fix(querier): serve sharded windows whose shards are at mixed compaction levels

[1linkovdim]

Summary

Fixes a query-time false-negative that silently returned zero results for a fully-intact time window whose sharded blocks happened to sit at different compaction levels.

pruneIncompleteShardedBlocks (pkg/querier/replication.go) grouped blocks by {compaction level, minTime} and dropped every block in any group missing a shard. But the N shards of one window can legitimately be at different compaction levels: when late-arriving source blocks re-merge only the shards their data touches (notably with compaction_split_by: stacktracePartition, where a small increment hashes to a subset of shards), those shards advance a level while the rest lag. Keying completeness on level then split one complete window into multiple sub-groups — each missing shards — and pruned all of them. With no lower-level fallback present, the query returns nothing, with no error and no log.

Fix

Check shard completeness per sharding (shard count) and per time instant, ignoring compaction level:

• For each distinct window start t, require every shard to have a trusted block covering t (minTime ≤ t < maxTime). A window whose shards span e.g. L3/L4 is now seen as complete; pruneSupersededBlocks then collapses the per-shard level duplicates (an L3 and its L4 descendant) down to one block per shard.
• Coverage, not an exact minTime match. A shard merged to a wider span still counts for the later windows it overlaps, so sibling shards' blocks there are not orphaned and silently pruned. (Mixed-span case from review — see below.)
• Different shard counts are distinct shardings, checked independently. A 2_of_4 and a 2_of_8 block cover different hash ranges (e.g. after a split-and-merge-shards reconfiguration), so they can't be pooled. This also removes the old shard length mismatch error, which is now structurally impossible.

Subtleties, each of which would otherwise reintroduce a silent miscount

1. Only blocks at/above the deduplication level count. Intermediate L2 split blocks are never served — pruneSupersededBlocks always drops them in favour of the deduplicated L1 ancestor — so letting an L2 block satisfy a shard would make an incomplete set look whole and under-count once the L2 is dropped.
2. pruneIncompleteShardedBlocks runs before pruneSupersededBlocks. An incomplete sharded set is dropped while its lower-level ancestors still exist, so those ancestors survive as a query-time fallback. (A kept sharded block would otherwise supersede its shared, un-sharded ancestor — the very fallback the missing shards need.)
3. Multiple surviving shardings force query-time dedup. If more than one sharding (shard count) survives into the plan for overlapping data (e.g. a complete _of_2 and a complete _of_4), they partition the same series differently, so a series can appear in both. The sharded fast path assumes each series is served by exactly one block, which only holds for a single complete sharding — so we set Deduplication=true to reconcile the overlap instead of double-counting. (Normally a window has one sharding, and when a new sharding is derived from an old one pruneSupersededBlocks collapses to one, so the fast path is preserved.)

A genuinely incomplete window (a shard absent at every trusted level) is still pruned — and now emits a WARN log, so this can never again silently hide data.

Mixed-span note (from review)

An earlier revision keyed completeness on {minTime, shardCount} (exact start match). That under-counted a transient mixed-span window — one shard merged to a wide block while a sibling still had separate blocks in the tail — by orphaning the tail blocks. The per-instant coverage check above resolves it. It's a self-healing transient (a compaction cycle merges the lagging shard and converges), but coverage handles it correctly rather than serving a partial result.

Scope

This is the v1 read path only (pkg/querier). v2's read path (pkg/querybackend + metastore-driven queryplan) has no analogous completeness guard and is unaffected.

A related, separate limitation remains and is not addressed here: a genuinely low-cardinality window (some shards legitimately empty, so the compactor omits them per compact.go) is also treated as incomplete and falls back to L1. The querier can't distinguish "empty shard" from "missing shard" without compactor-side metadata; that's a follow-up.

Why this layer (vs. lockstep-promoting shards in the compactor)

Compaction level means "how many merge generations / has this been deduplicated," not "is this window complete." Shards are independent partitions with no reason to share a generation count. The bug was the read path assuming level ≈ completeness; the fix removes that assumption where it lives. Forcing lockstep levels on the write path (e.g. emitting empty/placeholder blocks) would reshape the write path to satisfy a read-path invariant that shouldn't exist, at ongoing storage/I-O cost, and would remain racy. The fingerprint split mode already yields lockstep levels implicitly if that property is desired.

Testing

go test ./pkg/querier/ passes. Cases in Test_replicasPerBlockID_blockPlan:

• keep sharded window with shards at mixed compaction levels — the incident. Verified it fails pre-fix (empty plan) and passes after. Asserts Deduplication=false (complete set served directly).
• keep mixed-span shards when a wide block covers a later window — the mixed-span review case: a wide block + sibling tail blocks; all kept, no orphaning.
• collapse L3+L4 duplicate of a shard within a mixed-level window — a shard with both its L3 and derived L4 present collapses to the L4.
• prune sharded window with a shard missing at all levels — genuinely-missing shard still pruned, falls back to the non-sharded block (Deduplication=true).
• serve complete old sharding when a shard-count change leaves a partial new one — distinct shardings counted separately; verified it fails against upstream (shard-length-mismatch empties the plan).
• deduplicate when two complete shardings overlap — two complete, unrelated shardings both kept with Deduplication=true (no double-count).
• single sharding after superseding keeps the fast path — a derived _of_4 supersedes the _of_2; one sharding remains, Deduplication=false (guard doesn't over-trigger).
• do not let an intermediate L2 block satisfy shard completeness — a mid-merge window must not serve a shard alone off an about-to-be-dropped L2 block.

Test_pruneIncompleteShardedBlocks_logging — a WARN is emitted when a genuinely incomplete window is pruned, and not for a healthy mixed-level window. The pre-existing ignore_incomplete_shards case still passes (fallback preserved by the prune-before-supersede ordering).

🤖 Generated with Claude Code

[1linkovdim]

Moving back to draft until I decide how to address the pointed out issue by the bugbot.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0a6b318. Configure here.}