Improve transport service documentation

[sgunes-wirepas]

Improve argparse parameters so that --help can be used as documentation and source of truth. After this, the table in docker/docker-compose/README.md can be removed and, a new one added in transport service readme based on the help text.

Briefly:

• Feature-level documentation (for example buffering / black hole prevention, filtering)
• Various improvements and fixes to argument help texts
• docker run wirepas/gateway_transport_service wm-gw --help should now work
• Deprecate WM_SERVICES_MQTT_ALLOW_UNTRUSTED; it wasn't used before and I think no need to connect it now

Example output

Using upb implementation for protobuf
usage: wm-gw [-h] [--version] [--settings $WM_GW_FILE_SETTINGS]
             [--mqtt_hostname $WM_SERVICES_MQTT_HOSTNAME]
             [--mqtt_username $WM_SERVICES_MQTT_USERNAME]
             [--mqtt_password $WM_SERVICES_MQTT_PASSWORD]
             [--mqtt_port $WM_SERVICES_MQTT_PORT]
             [--mqtt_ca_certs $WM_SERVICES_MQTT_CA_CERTS]
             [--mqtt_certfile $WM_SERVICES_MQTT_CLIENT_CRT]
             [--mqtt_keyfile $WM_SERVICES_MQTT_CLIENT_KEY]
             [--mqtt_cert_reqs $WM_SERVICES_MQTT_CERT_REQS]
             [--mqtt_tls_version $WM_SERVICES_MQTT_TLS_VERSION]
             [--mqtt_ciphers $WM_SERVICES_MQTT_CIPHERS]
             [--mqtt_persist_session [$WM_SERVICES_MQTT_PERSIST_SESSION]]
             [--mqtt_force_unsecure [$WM_SERVICES_MQTT_FORCE_UNSECURE]]
             [--mqtt_reconnect_delay $WM_SERVICES_MQTT_RECONNECT_DELAY]
             [--mqtt_max_inflight_messages $WM_SERVICES_MQTT_MAX_INFLIGHT_MESSAGES]
             [--mqtt_use_websocket [$WM_SERVICES_MQTT_USE_WEBSOCKET]]
             [--mqtt_rate_limit_pps $WM_SERVICES_MQTT_RATE_LIMIT_PPS]
             [--gateway_id $WM_GW_ID] [-fp [FULL_PYTHON]] [-gm $WM_GW_MODEL]
             [-gv $WM_GW_VERSION] [-gmss $WM_GW_MAX_SCRAT_SIZE]
             [-iepf $WM_GW_IGNORED_ENDPOINTS_FILTER]
             [-wepf $WM_GW_WHITENED_ENDPOINTS_FILTER]
             [--buffering_max_buffered_packets $WM_GW_BUFFERING_MAX_BUFFERED_PACKETS]
             [--buffering_max_delay_without_publish $WM_GW_BUFFERING_MAX_DELAY_WITHOUT_PUBLISH]
             [--buffering_action $WM_GW_BUFFERING_ACTION]
             [--buffering_minimal_sink_cost $WM_GW_BUFFERING_MINIMAL_SINK_COST]
             [--log_level $WM_DEBUG_LEVEL]
             [--debug_incr_data_event_id [$WM_SERVICES_DEBUG_INCR_EVENT_ID]]
             [-s HOST] [-p PORT] [-u USERNAME] [-pw PASSWORD] [-t TLSFILE]
             [-ua [UNSECURE_AUTHENTICATION]] [-i GWID]
             [--mqtt_allow_untrusted [$WM_SERVICES_MQTT_ALLOW_UNTRUSTED]]
             [--buffering_stop_stack $WM_GW_BUFFERING_STOP_STACK]

Wirepas Gateway Transport Service

Each parameter below can also be set with the environment variable
shown next to it (i.e. $WM_GW_ID). A parameter given on the command
line overrides the environment variable.

options:
  -h, --help            show this help message and exit

main:
  --version             show program's version number and exit

file_settings:
  --settings $WM_GW_FILE_SETTINGS
                        A yaml file with argument parameters (see help for
                        options). (default: None)

mqtt:
  --mqtt_hostname $WM_SERVICES_MQTT_HOSTNAME
                        MQTT broker hostname. (default: None)
  --mqtt_username $WM_SERVICES_MQTT_USERNAME
                        MQTT broker username. (default: None)
  --mqtt_password $WM_SERVICES_MQTT_PASSWORD
                        MQTT broker password. (default: None)
  --mqtt_port $WM_SERVICES_MQTT_PORT
                        MQTT broker port. (default: 8883)
  --mqtt_ca_certs $WM_SERVICES_MQTT_CA_CERTS
                        Path to the Certificate Authority certificate files
                        that are to be treated as trusted by this client.
                        (default: None)
  --mqtt_certfile $WM_SERVICES_MQTT_CLIENT_CRT
                        Path to the PEM encoded client certificate. (default:
                        None)
  --mqtt_keyfile $WM_SERVICES_MQTT_CLIENT_KEY
                        Path to the PEM encoded client private key. (default:
                        None)
  --mqtt_cert_reqs $WM_SERVICES_MQTT_CERT_REQS
                        Defines the certificate requirements that the client
                        imposes on the broker. (choices: [CERT_REQUIRED,
                        CERT_OPTIONAL, CERT_NONE]) (default: CERT_REQUIRED)
  --mqtt_tls_version $WM_SERVICES_MQTT_TLS_VERSION
                        Specifies the version of the SSL / TLS protocol to be
                        used. (choices: [PROTOCOL_TLS, PROTOCOL_TLS_CLIENT,
                        PROTOCOL_TLS_SERVER, PROTOCOL_TLSv1, PROTOCOL_TLSv1_1,
                        PROTOCOL_TLSv1_2]) (default: PROTOCOL_TLSv1_2)
  --mqtt_ciphers $WM_SERVICES_MQTT_CIPHERS
                        A string specifying which encryption ciphers are
                        allowable for this connection. (default: None)
  --mqtt_persist_session [$WM_SERVICES_MQTT_PERSIST_SESSION]
                        When True the broker will buffer session packets
                        between reconnection. (default: False)
  --mqtt_force_unsecure [$WM_SERVICES_MQTT_FORCE_UNSECURE]
                        When true, connect to the broker without TLS.
                        (default: False)
  --mqtt_reconnect_delay $WM_SERVICES_MQTT_RECONNECT_DELAY
                        Time in seconds to keep trying to reconnect when the
                        connection to the broker is lost. If it expires, the
                        service exits. 0 to retry forever. (default: 0)
  --mqtt_max_inflight_messages $WM_SERVICES_MQTT_MAX_INFLIGHT_MESSAGES
                        Max inflight messages for messages with qos > 0
                        (default: 20)
  --mqtt_use_websocket [$WM_SERVICES_MQTT_USE_WEBSOCKET]
                        When true the mqtt client will use websocket instead
                        of TCP for transport (default: False)
  --mqtt_rate_limit_pps $WM_SERVICES_MQTT_RATE_LIMIT_PPS
                        Max rate limit for the mqtt client to publish on mqtt
                        broker. It can be set to protect the broker from very
                        high usage when one or more gateways are offline for a
                        while and publish all their buffers when connection to
                        broker is restored. 0 to disable the limit. (default:
                        0)

gateway:
  --gateway_id $WM_GW_ID
                        Id of the gateway. It must be unique on same broker.
                        When empty, an id is generated based on the network
                        interface MAC address (uuid.getnode()). The id is used
                        in MQTT topics without escaping, so special MQTT
                        characters (+, #, /) should be avoided. (default:
                        None)
  -fp [FULL_PYTHON], --full_python [FULL_PYTHON]
                        Do not use C extension for optimization.
  -gm $WM_GW_MODEL, --gateway_model $WM_GW_MODEL
                        Model name of the gateway. (default: None)
  -gv $WM_GW_VERSION, --gateway_version $WM_GW_VERSION
                        Version of the gateway. (default: None)
  -gmss $WM_GW_MAX_SCRAT_SIZE, --gateway_max_scratchpad_size $WM_GW_MAX_SCRAT_SIZE
                        Maximum scratchpad size a gateway can accept. If
                        scratchpad is bigger it must be sent as chunks smaller
                        or equal to this value (default: None)

filtering:
  Filters to limit which packets received from the Wirepas
  network are published to the MQTT broker. Both filters apply
  to uplink traffic only and select packets based on their
  destination endpoint. Downlink traffic is never filtered.
  
  Both parameters accept a list of endpoints (i.e. [1,2,3]), a
  range of endpoints (i.e. [1-3]), or a combination of both
  (i.e. [1,2,10-15]). Valid endpoint values are 0-255. An
  endpoint cannot be in both lists at the same time.

  -iepf $WM_GW_IGNORED_ENDPOINTS_FILTER, --ignored_endpoints_filter $WM_GW_IGNORED_ENDPOINTS_FILTER
                        Destination endpoints list to ignore. Packets sent to
                        these endpoints are not published at all. (default:
                        None)
  -wepf $WM_GW_WHITENED_ENDPOINTS_FILTER, --whitened_endpoints_filter $WM_GW_WHITENED_ENDPOINTS_FILTER
                        Destination endpoints list to whiten (i.e. blank out
                        the payload). Packets sent to these endpoints are
                        published without the payload content, only the
                        payload size is kept. (default: None)

buffering:
  If the MQTT connection is lost, transport service might end up
  buffering uplink packets and never send them to the MQTT broker,
  becoming a "black hole".
  
  Transport service can be configured to detect this and avoid it in
  different ways, which can be selected by WM_GW_BUFFERING_ACTION
  parameter. By default, transport service will buffer outgoing MQTT
  messages without any limit and retry connecting to the broker.
  
  The black hole prevention can be enabled by setting
  WM_GW_BUFFERING_MAX_BUFFERED_PACKETS or
  WM_GW_BUFFERING_MAX_DELAY_WITHOUT_PUBLISH parameter.
  
  Different actions are described below:
  
  * 'raise_sink_cost'
    Sink costs of sinks connected to this gateway are raised to
    discourage nodes from connecting to sinks under this gateway.
  
  * 'stop_stack'
    Sinks connected to this gateway are stopped to ensure nodes are
    not connected to sinks under this gateway.
  
  * 'drop_packets'
    Enabled only by WM_GW_BUFFERING_MAX_BUFFERED_PACKETS;
    WM_GW_BUFFERING_MAX_DELAY_WITHOUT_PUBLISH cannot be used with
    this action. Internal publish queue size is limited to
    WM_GW_BUFFERING_MAX_BUFFERED_PACKETS and the oldest MQTT messages
    are dropped if necessary. Drops are reported in the log
    periodically.
  
  Once the MQTT connection is reestablished, the transport service
  waits until all buffered messages have been successfully published
  before lowering sink costs or starting sinks again. This is done to
  prevent modifying sink parameters in unstable connections with
  intermittent disconnects. Also see the WM_SERVICES_MQTT_RATE_LIMIT_PPS
  parameter.
  
  When lowering sink costs, the value of
  WM_GW_BUFFERING_MINIMAL_SINK_COST is used. It is also applied to
  sinks at startup, even when black hole prevention is disabled:
  unlike most sink configuration parameters, sink cost cannot be set
  over the MQTT interface, so a raised cost left behind by an earlier
  gateway configuration could not be lowered by the backend
  otherwise.

  --buffering_max_buffered_packets $WM_GW_BUFFERING_MAX_BUFFERED_PACKETS
                        Maximum number of messages to buffer before taking an
                        action. 0 will disable feature (default: 0)
  --buffering_max_delay_without_publish $WM_GW_BUFFERING_MAX_DELAY_WITHOUT_PUBLISH
                        Maximum time to wait in seconds without any successful
                        publish with packet queued before taking an action. 0
                        will disable feature (default: 0)
  --buffering_action $WM_GW_BUFFERING_ACTION
                        Action to take when the buffer limit is reached. When
                        empty, it is assumed to be 'raise_sink_cost'.
                        (choices: [raise_sink_cost, stop_stack, drop_packets])
                        (default: None)
  --buffering_minimal_sink_cost $WM_GW_BUFFERING_MINIMAL_SINK_COST
                        Minimal sink cost for a sink on this gateway. Can be
                        used to minimize traffic on a gateway, but it will
                        reduce maximum number of hops for this gateway
                        (default: 0)

debug:
  --log_level $WM_DEBUG_LEVEL
                        Log level of the transport service. 'debug' level
                        might generate too much logs and is not recommended to
                        be used in a production system. (choices: [debug,
                        info, warning, error, critical]) (default: info)
  --debug_incr_data_event_id [$WM_SERVICES_DEBUG_INCR_EVENT_ID]
                        When true the data received event id will be
                        incremental starting at 0 when service starts.
                        Otherwise it will be random 64 bits id. (default:
                        False)

deprecated:
  -s HOST, --host HOST  Deprecated argument (it will be dropped from version
                        2.x onwards) please use --mqtt_hostname instead.
  -p PORT, --port PORT  Deprecated argument (it will be dropped from version
                        2.x onwards) please use --mqtt_port instead.
  -u USERNAME, --username USERNAME
                        Deprecated argument (it will be dropped from version
                        2.x onwards) please use --mqtt_username instead.
  -pw PASSWORD, --password PASSWORD
                        Deprecated argument (it will be dropped from version
                        2.x onwards) please use --mqtt_password instead.
  -t TLSFILE, --tlsfile TLSFILE
                        Deprecated argument (it will be dropped from version
                        2.x onwards) please use --mqtt_certfile instead.
  -ua [UNSECURE_AUTHENTICATION], --unsecure_authentication [UNSECURE_AUTHENTICATION]
                        Deprecated argument (it will be dropped from version
                        2.x onwards) please use --mqtt_force_unsecure instead.
  -i GWID, --gwid GWID  Deprecated argument (it will be dropped from version
                        2.x onwards) please use --gateway_id instead.
  --mqtt_allow_untrusted [$WM_SERVICES_MQTT_ALLOW_UNTRUSTED]
                        Not in use. Deprecated argument (it will be dropped
                        from version 2.x onwards) (default: False)
  --buffering_stop_stack $WM_GW_BUFFERING_STOP_STACK
                        Deprecated argument (it will be dropped from version
                        2.x onwards) please use --buffering_action instead.
                        (default: None)

[hound]

Black would make changes.

[sgunes-wirepas]

Not sure about this one; it's not nice to do an import here but this is the easiest way to get help text to print without a dbus connection.