Skip to content

maintenance: preserve metric string values - #4287

Open
zqr10159 wants to merge 2 commits into
apache:masterfrom
zqr10159:maintenance/metric-value-boundary
Open

maintenance: preserve metric string values#4287
zqr10159 wants to merge 2 commits into
apache:masterfrom
zqr10159:maintenance/metric-value-boundary

Conversation

@zqr10159

@zqr10159 zqr10159 commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

  • remove the arbitrary 32,700-byte per-cell truncation from the shared metrics representation
  • let Arrow VarCharVector.setSafe grow its variable-width data buffer as designed
  • decode metric cells explicitly as UTF-8
  • add exact-fidelity contracts around the previous boundary and through Arrow IPC

Boundary analysis

The previous 32,768-byte comment described Arrow initial buffer capacity, not a logical cell limit. Apache Arrow 18.1 documents that BaseVariableWidthVector.setSafe(int, byte[]) handles values beyond existing capacity and that the data buffer is reallocated as needed: https://arrow.apache.org/docs/18.1/java/reference/org.apache.arrow.vector/org/apache/arrow/vector/BaseVariableWidthVector.html#setSafe(int,byte%5B%5D)

CollectRep.MetricsData is the common in-memory representation and is serialized as Arrow IPC for Kafka and Redis paths. No universal 32,700-byte database or protocol limit exists at this layer. Deployment-specific transport limits apply to the complete serialized message, not silently to one metric cell; any future transport-specific limit must be enforced and reported at that output boundary.

Regression evidence

Before removing the truncation, the new contracts failed five times: values immediately above 32,700 bytes and a 100,000-byte value lost data in memory, and the 100,000-byte value was also truncated after Arrow IPC serialization and deserialization.

The final contracts cover ASCII, a three-byte code point, and a four-byte emoji immediately before, at, and after the old boundary, plus a 100,000-byte ASCII value. They require exact content and UTF-8 byte-length preservation. The IPC contract requires exact preservation after serialization and deserialization.

Validation

  • ./mvnw -pl hertzbeat-common-core -Dtest=CollectRepTest,KafkaMetricsDataSerializerTest test -DskipITs -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false — 18 tests passed
  • ./mvnw -pl hertzbeat-common-core test -DskipITs — 595 tests passed
  • Checkstyle, no hard-coded CJK scan, and both Git diff whitespace checks passed

AI assistance: used for draft implementation and test iteration.
Human validation: the previous truncation failed exact-fidelity regressions; the focused Arrow/IPC contracts and complete common-core suite pass after removing it.
Risk notes: very large metric cells consume proportional memory and contribute to whole-message transport limits. Those limits are deployment-specific and must fail observably at the relevant transport boundary; this shared layer no longer corrupts values silently.

@zqr10159 zqr10159 changed the title maintenance: bound metric string values maintenance: preserve metric string values Jul 30, 2026
@zqr10159

Copy link
Copy Markdown
Member Author

Author remediation update:

The arbitrary 32,700-byte truncation has been removed. Arrow VarChar storage grows through setSafe, preserving full UTF-8 values instead of silently changing metric data. The regression suite covers ASCII, CJK, emoji, exact boundaries, over-boundary input, a 100 KB value, and Arrow IPC round-trip behavior.

Focused tests (18) and the complete common-core suite (595 tests) passed; all current GitHub checks are green. The data-integrity blocker is resolved; maintainer review is still required.

@zqr10159
zqr10159 marked this pull request as ready for review July 31, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant