Skip to content

fix(shadows): ack map-entry removals as null in reported and desired#131

Merged
MathiasKoch merged 1 commit into
masterfrom
fix/map-unset-ack
Jun 11, 2026
Merged

fix(shadows): ack map-entry removals as null in reported and desired#131
MathiasKoch merged 1 commit into
masterfrom
fix/map-unset-ack

Conversation

@MathiasKoch

Copy link
Copy Markdown
Member

Summary

Applying a map delta with Patch::Unset removed the entry locally but the acknowledgment could not express the removal: ReportedLinearMap / ReportedHashMap held plain K -> R entries, so into_partial_reported silently skipped unset keys (the ack sent {}), and desired_cleanup ignored them. After the ack, the cloud document kept the stale entry in reported and the "unset" tombstone in desired.

Empirically (replicating apply_delta_and_ack's two outputs before this change):

delta in:        {"ports":{"p1":"unset"}}
ack reported:    {"ports":{}}     <- merge no-op, stale entry survives
desired cleanup: None             <- tombstone survives

Consequences fixed

  1. Re-add never reaches the device: re-adding an entry identical to the stale reported value produces NO delta (desired == reported field-for-field). Concretely for wifi.known_networks: remove a network, later re-add the same network - the device never reconnects, with no error anywhere.
  2. Permanent pending delta: the "unset" tombstone in desired keeps the delta document non-empty forever; the device reprocesses it as a no-op on every boot/GET.

Fix

Reported map entries are now Patch<R>:

  • Patch::Set serializes transparently, so present entries keep the exact same wire format - no change for existing consumers' shadow documents.
  • Patch::Unset serializes as null, which AWS treats as key deletion. The unset arm of into_partial_reported now emits it, and desired_cleanup mirrors the removal, so the single apply_delta_and_ack update nulls the key in both sections:
ack reported:    {"ports":{"p1":null}}
desired cleanup: {"ports":{"p1":null}}
  • ReportedDiff always retains Unset entries (a removal must reach the cloud regardless of what was previously reported).

Both the heapless LinearMap and std HashMap implementations are updated symmetrically.

Compatibility

  • Wire format for present entries: unchanged (transparent Set).
  • SCHEMA_HASH / KV persistence: untouched - no flash migration for KV consumers.
  • API: ReportedLinearMap.0 / ReportedHashMap.0 now hold Patch<R> values; the From<Map<K, R>> impls wrap entries in Patch::Set, so construction sites keep compiling.

Tests

  • test_linear_map_unset_acks_null_in_reported_and_desired / test_hash_map_unset_acks_null_in_reported_and_desired - serialized ack payloads are {"k":null} on both sides.
  • test_reported_linear_map_diff_retains_unset - diff keeps removals, still prunes unchanged Set entries.
  • test_map_unset_ack_nulls_reported_and_desired (shadow level) - full parse_delta -> apply_delta -> ack pipeline on a shadow_root fixture.

Changelog

Fixed

  • Map-entry removals (Patch::Unset) are now acknowledged as null in both reported and desired, so AWS deletes the key instead of keeping a stale entry + tombstone (which silently swallowed any later re-add of an identical entry)

Applying a map delta with Patch::Unset removed the entry locally but the
acknowledgment could not express the removal: ReportedLinearMap /
ReportedHashMap held plain K -> R entries, so into_partial_reported
silently skipped unset keys (ack sent {}), and desired_cleanup ignored
them. The cloud document kept the stale entry in reported and the
"unset" tombstone in desired.

Consequences fixed:
- Re-adding an entry identical to the stale reported value produced NO
  delta (desired == reported field-for-field), so the device never saw
  the re-add. For wifi known_networks: remove network, re-add the same
  network -> device never reconnects.
- The desired tombstone kept a permanent pending delta, reprocessed as
  a no-op on every boot/GET.

Fix: reported map entries are now Patch<R>. Patch::Set serializes
transparently (present entries keep the exact same wire format) and
Patch::Unset serializes as null, which AWS treats as key deletion. The
unset arm of into_partial_reported emits Patch::Unset, and
desired_cleanup mirrors the removal so both sections are nulled in the
single apply_delta_and_ack update. ReportedDiff always retains Unset
entries (a removal must reach the cloud regardless of prior report).

Both heapless::LinearMap and std HashMap impls updated; regression
tests assert the serialized ack is {"k":null} on both sides.
@MathiasKoch MathiasKoch merged commit 5aea51c into master Jun 11, 2026
5 checks passed
@MathiasKoch MathiasKoch deleted the fix/map-unset-ack branch June 11, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants