Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,8 @@ jobs:
run: uv run --no-project python tools/test_runtime_ci.py
- name: Install pinned TypeScript reference dependencies
run: npm ci --prefix tools/reference
- name: Audit pinned reference dependencies against the official registry
run: npm audit --prefix tools/reference --registry=https://registry.npmjs.org --audit-level=low
- name: Build both architectures and run Lambda, streaming, and Valkey simulations
run: uv run --no-project python integration/local/run.py --skip-module-checks
- name: Verify Batch logs and spans from this run
Expand Down
2 changes: 1 addition & 1 deletion docs/THIRD_PARTY_DEPENDENCIES.json
Original file line number Diff line number Diff line change
Expand Up @@ -644,7 +644,7 @@
},
{
"package": "protobufjs",
"version": "7.5.4",
"version": "7.6.6",
"license": "BSD-3-Clause",
"source": "protobufjs/protobuf.js",
"scope": "Development reference generator only"
Expand Down
2 changes: 1 addition & 1 deletion integration/kafkamodes/testdata/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Kafka event mode reference corpus

Generate from tools/reference with `node generate-kafka-modes.mjs`. The generator executes the installed CommonJS export of Powertools TypeScript v2.35.0 with avro-js 1.12.1 and protobufjs 7.5.4. No upstream implementation is patched.
Generate from tools/reference with `node generate-kafka-modes.mjs`. The generator executes the installed CommonJS export of Powertools TypeScript v2.35.0 with avro-js 1.12.1 and protobufjs 7.6.6. No upstream implementation is patched. The protobufjs security update changes four short-prefix error strings in two events; all remaining event fields, decoded values, callback positions and error categories are unchanged.

The 172 constructed events cover MSK and self-managed sources, SOURCE and JSON delivery, Glue/Confluent schema metadata, missing registry metadata, every key/value pair of text/JSON/Avro/Protobuf, parser transformations, repeated lazy reads, byte headers, retained original values and metadata, unknown fields, nulls, empty strings and missing fields. A FileDescriptorSet supplies the same proto2 schema to the official Go Protobuf runtime. Successful native messages compare protojson with protobufjs JSON; Avro bytes retain explicit byte tags. Errors compare complete names and messages, including the supplied Protobuf schema description.

Expand Down
8 changes: 4 additions & 4 deletions integration/kafkamodes/testdata/typescript-v2.35.0.json
Original file line number Diff line number Diff line change
Expand Up @@ -11262,7 +11262,7 @@
"value": null,
"error": {
"name": "KafkaConsumerDeserializationError",
"message": "Failed to deserialize Protobuf message: RangeError: index out of range: 0 + 10 > 0, message: , messageType: {\"edition\":\"proto2\",\"fields\":{\"id\":{\"type\":\"string\",\"id\":1},\"count\":{\"type\":\"int32\",\"id\":2},\"raw\":{\"type\":\"bytes\",\"id\":3}}}"
"message": "Failed to deserialize Protobuf message: RangeError: index out of range: 0 + 1 > 0, message: , messageType: {\"edition\":\"proto2\",\"fields\":{\"id\":{\"type\":\"string\",\"id\":1},\"count\":{\"type\":\"int32\",\"id\":2},\"raw\":{\"type\":\"bytes\",\"id\":3}}}"
}
},
{
Expand Down Expand Up @@ -11293,7 +11293,7 @@
"value": null,
"error": {
"name": "KafkaConsumerDeserializationError",
"message": "Failed to deserialize Protobuf message: RangeError: index out of range: 0 + 10 > 0, message: , messageType: {\"edition\":\"proto2\",\"fields\":{\"id\":{\"type\":\"string\",\"id\":1},\"count\":{\"type\":\"int32\",\"id\":2},\"raw\":{\"type\":\"bytes\",\"id\":3}}}"
"message": "Failed to deserialize Protobuf message: RangeError: index out of range: 0 + 1 > 0, message: , messageType: {\"edition\":\"proto2\",\"fields\":{\"id\":{\"type\":\"string\",\"id\":1},\"count\":{\"type\":\"int32\",\"id\":2},\"raw\":{\"type\":\"bytes\",\"id\":3}}}"
}
}
],
Expand Down Expand Up @@ -22635,7 +22635,7 @@
"value": null,
"error": {
"name": "KafkaConsumerDeserializationError",
"message": "Failed to deserialize Protobuf message: RangeError: index out of range: 0 + 10 > 0, message: , messageType: {\"edition\":\"proto2\",\"fields\":{\"id\":{\"type\":\"string\",\"id\":1},\"count\":{\"type\":\"int32\",\"id\":2},\"raw\":{\"type\":\"bytes\",\"id\":3}}}"
"message": "Failed to deserialize Protobuf message: RangeError: index out of range: 0 + 1 > 0, message: , messageType: {\"edition\":\"proto2\",\"fields\":{\"id\":{\"type\":\"string\",\"id\":1},\"count\":{\"type\":\"int32\",\"id\":2},\"raw\":{\"type\":\"bytes\",\"id\":3}}}"
}
},
{
Expand Down Expand Up @@ -22666,7 +22666,7 @@
"value": null,
"error": {
"name": "KafkaConsumerDeserializationError",
"message": "Failed to deserialize Protobuf message: RangeError: index out of range: 0 + 10 > 0, message: , messageType: {\"edition\":\"proto2\",\"fields\":{\"id\":{\"type\":\"string\",\"id\":1},\"count\":{\"type\":\"int32\",\"id\":2},\"raw\":{\"type\":\"bytes\",\"id\":3}}}"
"message": "Failed to deserialize Protobuf message: RangeError: index out of range: 0 + 1 > 0, message: , messageType: {\"edition\":\"proto2\",\"fields\":{\"id\":{\"type\":\"string\",\"id\":1},\"count\":{\"type\":\"int32\",\"id\":2},\"raw\":{\"type\":\"bytes\",\"id\":3}}}"
}
}
],
Expand Down
23 changes: 12 additions & 11 deletions kafka/protobuf/protobuf.go
Original file line number Diff line number Diff line change
Expand Up @@ -182,20 +182,21 @@ func longSchemaID(id any) bool {
return false
}

// uint32Prefix matches protobufjs's five-byte low-word read and ten-byte skip,
// including its reported position on truncated input. It is not a uint64 parser.
// uint32Prefix matches patched protobufjs's five-byte low-word read and ten-byte
// skip, including its one-byte near-end diagnostic. It is not a uint64 parser.
func uint32Prefix(buffer []byte) (uint32, int, error) {
var value uint32
for i := 0; i < 5; i++ {
if i < len(buffer) {
mask := byte(127)
if i == 4 {
mask = 15
}
value |= uint32(buffer[i]&mask) << uint(7*i)
if buffer[i] < 128 {
return value, i + 1, nil
}
if i >= len(buffer) {
return 0, len(buffer), rangeError(len(buffer), 1, len(buffer))
}
mask := byte(127)
if i == 4 {
mask = 15
}
value |= uint32(buffer[i]&mask) << uint(7*i)
if buffer[i] < 128 {
return value, i + 1, nil
}
}
if len(buffer) < 10 {
Expand Down
22 changes: 22 additions & 0 deletions kafka/protobuf/protobuf_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,28 @@ func bytesValue(data []byte) []any {
return out
}

func TestPatchedPrefixReadBounds(t *testing.T) {
for length := 0; length < 10; length++ {
buffer := make([]byte, length)
for i := range buffer {
buffer[i] = 0x80
}
_, position, err := uint32Prefix(buffer)
readSize := 1
if length >= 5 {
readSize = 10
}
expected := fmt.Sprintf("index out of range: %d + %d > %d", length, readSize, length)
if position != length || err == nil || err.Error() != expected {
t.Fatalf("length %d: position=%d error=%v; want %s", length, position, err, expected)
}
}
value, position, err := uint32Prefix([]byte{0xac, 0x02})
if err != nil || value != 300 || position != 2 {
t.Fatalf("valid near-end varint changed: %d, %d, %v", value, position, err)
}
}

func TestTypeScriptReference(t *testing.T) {
data, err := os.ReadFile("testdata/typescript-v2.35.0.json")
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion kafka/protobuf/testdata/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Protobuf differential scenarios

Generate with `node generate-kafka-protobuf.mjs` in tools/reference. The fixtures execute the actual CommonJS Kafka v2.35.0 consumer with protobufjs 7.5.4. The ESM import issue is recorded in the parent Kafka fixture README.
Generate with `node generate-kafka-protobuf.mjs` in tools/reference. The fixtures execute the actual CommonJS Kafka v2.35.0 consumer with protobufjs 7.6.6. The security update from 7.5.4 records the patched tool version and its one-byte bounds diagnostic for truncated prefixes shorter than five bytes. Successful values, callback positions, error categories, the schema and nine native-message results remain unchanged. The ESM import issue is recorded in the parent Kafka fixture README.

The 220 prefix scenarios use an application-supplied message decoder, as supported by the public messageType contract. They compare original buffers, reader positions, explicit length arguments, both decoding attempts, results and complete errors. Preference state persists across scenarios. Cases cover absent/null metadata, missing/null/numeric/Unicode schema IDs, Glue uint32 consumption, Confluent byte skips, negative positions, truncation, permissive Base64, successful preference reversals and first-error retention when both attempts fail. The additional 97 scenarios cover object length coercion, array IDs and primitive metadata. These cases prove wrapper behavior, not protobufjs's internal wire decoder.

Expand Down
Loading
Loading