maintenance: preserve metric string values - #4287
Open
zqr10159 wants to merge 2 commits into
Open
Conversation
Member
Author
|
Author remediation update: The arbitrary 32,700-byte truncation has been removed. Arrow VarChar storage grows through setSafe, preserving full UTF-8 values instead of silently changing metric data. The regression suite covers ASCII, CJK, emoji, exact boundaries, over-boundary input, a 100 KB value, and Arrow IPC round-trip behavior. Focused tests (18) and the complete common-core suite (595 tests) passed; all current GitHub checks are green. The data-integrity blocker is resolved; maintainer review is still required. |
zqr10159
marked this pull request as ready for review
July 31, 2026 02:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VarCharVector.setSafegrow its variable-width data buffer as designedBoundary analysis
The previous 32,768-byte comment described Arrow initial buffer capacity, not a logical cell limit. Apache Arrow 18.1 documents that
BaseVariableWidthVector.setSafe(int, byte[])handles values beyond existing capacity and that the data buffer is reallocated as needed: https://arrow.apache.org/docs/18.1/java/reference/org.apache.arrow.vector/org/apache/arrow/vector/BaseVariableWidthVector.html#setSafe(int,byte%5B%5D)CollectRep.MetricsDatais the common in-memory representation and is serialized as Arrow IPC for Kafka and Redis paths. No universal 32,700-byte database or protocol limit exists at this layer. Deployment-specific transport limits apply to the complete serialized message, not silently to one metric cell; any future transport-specific limit must be enforced and reported at that output boundary.Regression evidence
Before removing the truncation, the new contracts failed five times: values immediately above 32,700 bytes and a 100,000-byte value lost data in memory, and the 100,000-byte value was also truncated after Arrow IPC serialization and deserialization.
The final contracts cover ASCII, a three-byte code point, and a four-byte emoji immediately before, at, and after the old boundary, plus a 100,000-byte ASCII value. They require exact content and UTF-8 byte-length preservation. The IPC contract requires exact preservation after serialization and deserialization.
Validation
./mvnw -pl hertzbeat-common-core -Dtest=CollectRepTest,KafkaMetricsDataSerializerTest test -DskipITs -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false— 18 tests passed./mvnw -pl hertzbeat-common-core test -DskipITs— 595 tests passedAI assistance: used for draft implementation and test iteration.
Human validation: the previous truncation failed exact-fidelity regressions; the focused Arrow/IPC contracts and complete common-core suite pass after removing it.
Risk notes: very large metric cells consume proportional memory and contribute to whole-message transport limits. Those limits are deployment-specific and must fail observably at the relevant transport boundary; this shared layer no longer corrupts values silently.