Skip to content

fix: align UTF-8 decoding across payload utilities - #30

Merged
jinxiao merged 1 commit into
mainfrom
fix/utf8-decoding-26-28
Oct 4, 2026
Merged

jinxiao merged 1 commit into
mainfrom
fix/utf8-decoding-26-28

Conversation

@jinxiao

@jinxiao jinxiao commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Issue

Closes #26
Closes #27
Closes #28

Summary

Consecutive malformed UTF-8 bytes were collapsed into one replacement character in three utilities. Parser and Parameters also retained BOMs where the pinned TextDecoder removes them, so BOM-prefixed JSON could become a string or fail transformation. Parser's gzip JSON path split truncated sequences differently from the reference.

Reuse the existing Commons UTF-8 decoder, retaining each path's TextDecoder/Buffer BOM behavior and Parser fallback order. These three issues share one decoding problem, so they map to this focused PR. The separately tested Signer fix was merged as PR #29 before this work began.

Add 56 fixed outcomes generated from the actual TypeScript 2.35.0 packages and a reproducible generator. Record the module review scope, baseline evidence and local Linux limitations.

Testing

  • CGO_ENABLED=0 go test ./parser/... ./parameters/... ./jmespath/... -run 'Test(Base64|Transform|Powertools)UTF8Reference' -count=1: new tests reproduced all three modules' differences before the fix.
  • CGO_ENABLED=0 uv run --no-project python tools/modules.py check --only parser --only parameters --only jmespath: passed all packaged tests, vet, tidy verification and three independent consumers after the fix.
  • The pinned reference corpus contains 20 Parser, 16 Parameters and 20 JMESPath cases; Parameters also verifies Auto and untransformed byte input.
  • Documentation navigation and guide coverage checks passed.
  • Audit baseline: 25 native Windows modules passed; the exact baseline main commit's complete Linux CI passed all 31 repository modules, including KMS/frozen X-Ray, and both Lambda architectures. Required native PR checks will pass before merge. Local Docker's Linux engine was unavailable; no AWS cloud tests were run.

Risks and compatibility

No public API or dependency versions change. Outputs for malformed UTF-8 and BOM-prefixed inputs now follow the pinned reference, including correctly parsing BOM-prefixed plain JSON. Gzip retains its separate BOM behavior and existing fallback/error policy. Arbitrary malformed Unicode, service and performance gates retain their documented scope.

Release notes

  • parser | fix | Match reference UTF-8 replacement and BOM handling when decoding Base64 text and gzip JSON.
  • parameters | fix | Preserve reference UTF-8 replacement and remove leading BOMs in byte and binary parameter transforms.
  • jmespath | fix | Decode malformed UTF-8 subparts consistently in Powertools Base64 and gzip functions.

Checklist

  • The issue records the scope and acceptance criteria
  • Tests and documentation match the change, or I explained why they are not applicable
  • I kept CGO disabled and did not run cloud AWS tests without authorization
  • I excluded credentials, personal configuration, and generated build artifacts

@jinxiao
jinxiao merged commit f57274e into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant