Skip to content

maintenance: define VictoriaMetrics label collisions - #4286

Open
zqr10159 wants to merge 2 commits into
apache:masterfrom
zqr10159:maintenance/vm-label-precedence
Open

maintenance: define VictoriaMetrics label collisions#4286
zqr10159 wants to merge 2 commits into
apache:masterfrom
zqr10159:maintenance/vm-label-precedence

Conversation

@zqr10159

@zqr10159 zqr10159 commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

  • define an explicit collision policy shared by the single-node and cluster VictoriaMetrics writers
  • keep job and instance as ordinary Prometheus custom labels, preserving the exact label set and series identity used before this PR
  • reject a metrics batch when monitor custom labels use HertzBeat-managed keys: __name__, __monitor_id__, __metrics__, or __metric__
  • report conflicting key names without logging their values; do not silently discard or rename user labels
  • document the upgrade check and clarify that existing VictoriaMetrics series are not rewritten

Upgrade behavior

  • monitors using job, instance, or ordinary custom labels require no migration and continue writing the same label set
  • before upgrade, rename any monitor custom label using one of the four HertzBeat-managed keys
  • a remaining managed-key collision is observable and fail-closed: that metrics batch is not sent, and the application log lists only the conflicting key names

Validation

  • regression contracts first failed on the previous head: the writer changed existing job/instance values and silently accepted managed-key collisions
  • ./mvnw -pl hertzbeat-warehouse -Dtest=VictoriaMetricsDataStorageTest test -DskipITs -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
    • 6 tests passed; Checkstyle passed
  • ./mvnw -pl hertzbeat-warehouse -am test -DskipITs
    • complete reactor passed; hertzbeat-warehouse ran 61 tests with 0 failures and 0 errors
  • git diff --cached --check

AI assistance: used for draft implementation and test iteration.
Human validation: the serialized write payload retains existing custom job and instance values, managed-key collisions produce a value-free diagnostic and no HTTP write, and the complete warehouse reactor passed locally.
Risk notes: monitors that currently use a HertzBeat-managed custom-label key stop writing VictoriaMetrics batches until the key is renamed. This is intentional and documented so the upgrade cannot silently split or relabel an existing series.

@zqr10159 zqr10159 changed the title maintenance: preserve VictoriaMetrics label identity maintenance: define VictoriaMetrics label collisions Jul 30, 2026
@github-actions github-actions Bot added the doc Improvements or additions to documentation label Jul 30, 2026
@zqr10159

Copy link
Copy Markdown
Member Author

Author remediation update:

The collision policy is now explicit and shared by both VictoriaMetrics writers. Existing Prometheus-style job, instance, and ordinary custom labels are preserved exactly. HertzBeat-managed keys are rejected with a key-only diagnostic and no write; user labels are not silently discarded or renamed. The upgrade note identifies the four managed keys that must be renamed.

Payload-continuity and no-write diagnostic tests passed (6 focused tests), and the full warehouse reactor passed (61 tests). Backend, E2E, license, and label checks are green. DOC CI failed only because an unchanged historical Gitee link returned HTTP 405; that failed job has been rerun. Maintainer review remains required.

@zqr10159

Copy link
Copy Markdown
Member Author

CI follow-up: the DOC CI rerun passed, including Dead Link Check and the documentation build. Backend, Maven E2E, image E2E, license, and label checks are also green; all current checks have completed successfully.

@zqr10159
zqr10159 marked this pull request as ready for review July 31, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend doc Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant