Summary
OTEL_METRICS_KNOWN_FIELD_LIST in src/otel/metrics.rs is out of sync with the keys the metric flatteners actually emit. compute_series_hash treats every top-level key that is NOT in this list as a series label, so the mismatch corrupts physical-series identity for metrics in two opposite ways: some real per-sample fields are treated as labels (fragmenting a series into many), and some names that are never emitted are listed (which can merge distinct series if a user attribute happens to use one of those names).
Background
compute_series_hash (src/otel/metrics.rs) hashes metric_name plus every top-level key/value on the flattened data point that is not present in OTEL_METRICS_KNOWN_FIELD_LIST. The result is stored per sample as __series_hash and used by the query layer to group samples into series. So the list must exactly match the set of non-label (sample/metric-level) keys the flatteners emit.
Problem 1: emitted keys missing from the list (series fragmentation)
These are emitted as top-level keys but are absent from the list, so their per-sample values get hashed into the series identity. Samples of the same physical series then get different __series_hash values and are split into many spurious series.
min — emitted by flatten_histogram (around line 351, via insert_number_if_some)max — emitted by flatten_histogram (around line 352)data_point_quantile_values — emitted by flatten_summary (around line 471) as a nested array of {quantile, value} objects
Impact: histogram series fragment whenever min/max change across scrapes (routine, especially for DELTA temporality), and summary series fragment as quantile values change every sample.
Problem 2: listed names that are never emitted (dead entries, with collision risk)
These appear in the list but are never emitted as top-level keys, so they exclude any real attribute of the same name from the series hash:
value (list) — only ever emitted nested inside the data_point_quantile_values objects (around line 484), never as a top-level key. value is a very common attribute name, so a user label named value would be silently dropped from series identity and distinct series would collide.quantile (list) — same situation (nested only, around line 479); collision risk for an attribute named quantile.data_point_quantile_values_quantile and data_point_quantile_values_value (list) — never emitted anywhere. Harmless dead entries (names are specific enough to be unlikely user labels), but they should be removed for correctness.
Proposed fix
- Add
min, max, and data_point_quantile_values to OTEL_METRICS_KNOWN_FIELD_LIST.
- Remove
value, quantile, data_point_quantile_values_quantile, and data_point_quantile_values_value.
- Update the list length constant and the
OTEL_METRICS_KNOWN_FIELD_LIST.len() test assertion accordingly.
- Add tests asserting that differing
min/max (histogram) and data_point_quantile_values (summary) do not change compute_series_hash, and that a data point attribute named value or quantile does change it.
Related
Same class of issue as exemplar attributes leaking into the series hash, discussed in #1662 (see also #1720). Found while auditing the flattening code after that fix.
Summary
OTEL_METRICS_KNOWN_FIELD_LISTinsrc/otel/metrics.rsis out of sync with the keys the metric flatteners actually emit.compute_series_hashtreats every top-level key that is NOT in this list as a series label, so the mismatch corrupts physical-series identity for metrics in two opposite ways: some real per-sample fields are treated as labels (fragmenting a series into many), and some names that are never emitted are listed (which can merge distinct series if a user attribute happens to use one of those names).Background
compute_series_hash(src/otel/metrics.rs) hashesmetric_nameplus every top-level key/value on the flattened data point that is not present inOTEL_METRICS_KNOWN_FIELD_LIST. The result is stored per sample as__series_hashand used by the query layer to group samples into series. So the list must exactly match the set of non-label (sample/metric-level) keys the flatteners emit.Problem 1: emitted keys missing from the list (series fragmentation)
These are emitted as top-level keys but are absent from the list, so their per-sample values get hashed into the series identity. Samples of the same physical series then get different
__series_hashvalues and are split into many spurious series.min— emitted byflatten_histogram(around line 351, viainsert_number_if_some)max— emitted byflatten_histogram(around line 352)data_point_quantile_values— emitted byflatten_summary(around line 471) as a nested array of{quantile, value}objectsImpact: histogram series fragment whenever
min/maxchange across scrapes (routine, especially for DELTA temporality), and summary series fragment as quantile values change every sample.Problem 2: listed names that are never emitted (dead entries, with collision risk)
These appear in the list but are never emitted as top-level keys, so they exclude any real attribute of the same name from the series hash:
value(list) — only ever emitted nested inside thedata_point_quantile_valuesobjects (around line 484), never as a top-level key.valueis a very common attribute name, so a user label namedvaluewould be silently dropped from series identity and distinct series would collide.quantile(list) — same situation (nested only, around line 479); collision risk for an attribute namedquantile.data_point_quantile_values_quantileanddata_point_quantile_values_value(list) — never emitted anywhere. Harmless dead entries (names are specific enough to be unlikely user labels), but they should be removed for correctness.Proposed fix
min,max, anddata_point_quantile_valuestoOTEL_METRICS_KNOWN_FIELD_LIST.value,quantile,data_point_quantile_values_quantile, anddata_point_quantile_values_value.OTEL_METRICS_KNOWN_FIELD_LIST.len()test assertion accordingly.min/max(histogram) anddata_point_quantile_values(summary) do not changecompute_series_hash, and that a data point attribute namedvalueorquantiledoes change it.Related
Same class of issue as exemplar attributes leaking into the series hash, discussed in #1662 (see also #1720). Found while auditing the flattening code after that fix.