`fetch_data()` should prefer `normalized_payload` over `decoded_payload`

[dewet22]

I've implemented a data normalizer as per TTN's guidelines, in order to separate message decoding from data parsing. In my simple case, for a custom sensor with my own OTA data format, I've defined the following uplink payload formatter functions:

function decodeUplink(input) {
  var decoded = {
    data: {
      bytes: input.bytes
    },
    warnings: [],
    errors: []
  };

  if (input.fPort === 8) {
    decoded.data.temp = input.bytes[0] <<8 | input.bytes[1];
    decoded.data.humi = input.bytes[2] <<8 | input.bytes[3];
    decoded.data.co2 = input.bytes[4] <<8 | input.bytes[5];
  }

  return decoded;
}

function normalizeUplink(input) {
  return {
    data: {
      air: {
        temperature: input.data.temp / 10,
        relativeHumidity: input.data.humi / 10,
        co2: input.data.co2,
      },
    },
    // warnings: ["warning 1", "warning 2"], // optional
    // errors: ["error 1", "error 2"], // optional (if set, the normalization failed)
  };
}

This structure is defined by a common data dictionary, meaning sensors can now be represented more consistently. For instance, in my uplink_message stanza in the storage/uplink_message API call, I get access to both:

{
  "result": {
    // ...
    "uplink_message": {
      "f_port": 8,
      "f_cnt": 456,
      "frm_payload": "AOwBuAIu",
      "decoded_payload": {
        "bytes": [0, 236, 1, 184, 2, 46],
        "co2": 558,
        "humi": 440,
        "temp": 236
      },
      "normalized_payload": [
        { "air": { "co2": 558, "relativeHumidity": 44, "temperature": 23.6 } }
      ],
      // ...
}

I think the logic for fetch_data() should be to prefer normalized_payload data (if present) over decoded_payload. I wanted to run the idea by you first before I bother going off writing any code for it.

[angelnu]

@dewet22 This makes a lot of sense.

In the past (custom ha integration), I was configuring the equivalent of a normalized in the ga side using a custom ui. This functionality got lost when I contributed the integration officially as the custom config ui was kinda hack. I though about a custom config but this would also likely be difficult to get approved.

So supporting normalized outputs give the option to move the logic to the ttn side which I like.

Now, thinking on backward compatibility, I am not sure if we should only return normalized when available or a combination of both decoded and normalized. The second has less risk but there might be name conflicts.

When it is a normalized measurement we should include additional Metadata such as the unit and a "nirmalised" flag that allows the hs integration to select proper icons, units, sensor type, etc

[dewet22]

So supporting normalized outputs give the option to move the logic to the ttn side which I like.

This really is the bottom line for me – in the cases where people have invested extensively in LoRa it'll all be managed in the TTN platform, including normalisation. I could've duplicated my normalisation using template sensors but that is really defeating the purpose of it all. So the idea is that you can trust normalised data coming from TTN to be what it says, instead of needing any more "interpretation" on the hass side.

Now, thinking on backward compatibility, I am not sure if we should only return normalized when available or a combination of both decoded and normalized. The second has less risk but there might be name conflicts.

I don't think it's that problematic. There's essentially three cases, and it can be handled with a configuration option at the integration level like "Use normalised values", which defaults to false for existing installations, and for new installations defaults to whatever the first dataset returned shows: if they have normalised data, choose true otherwise default to false:

1. People who haven't implemented normalisation at the TTN side (new or existing). Nothing changes for them.

2. Existing users who have implemented normalisation at TTN. Nothing changes for them until they flip the configuration switch, after which their devices will be given new entities based on the normalised values, and the old entities will go stale over time and HA should offer the ability to delete them. A bit of a migration and cleanup but it is in their control.

3. New users who have normalisation (i.e., myself) - it enables normalisation immediately and all's well. This could also be signposted explicitly during the setup flow ("we see you don't have normalised values in your data, consider looking at these TTN docs to see how you can feed higher fidelity data into Home Assistant")

When it is a normalized measurement we should include additional Metadata such as the unit and a "nirmalised" flag that allows the hs integration to select proper icons, units, sensor type, etc

Yeah, this would absolutely be possible by translating that normalisation schema to HA entities.