fix: bound scans match --all history payloads

[GautamSharma99]

Summary

Fixes #136.

scans match --all now processes scan history through bounded workbench pages and bounded model inputs instead of materializing every historical finding repeatedly in one workbench response and sending an ever-growing prompt for each later scan.

The command still matches every eligible scan pair, but resource use is bounded at each step and completed pair results are persisted as the command progresses.

What changed

Bounded matching plan

list-unmatched-scan-pairs now returns scan IDs rather than embedding finding documents in the plan.

The workbench exposes at most 64 pair positions per response. Pair positions are derived directly from the stable triangular scan-pair ordering, so requesting a later page does not rebuild or retain all earlier pairs.

Saved comparisons remain excluded unless --force is used. Because each completed pair is saved immediately, a failed or interrupted --all run can resume without repeating successful work.

Bounded finding retrieval

Added a dedicated get-scan-matching-inputs workbench command. Each response contains at most:

• 32 findings;
• 512 KiB of compact matching-input JSON.

Finding pages are read with SQL LIMIT and OFFSET; the workbench no longer loads a scan’s complete finding set to return one page. Oversized details_json values are rejected through a database byte-length check before JSON expansion, and the complete encoded matching input is checked again before publication.

This keeps every workbench response comfortably below the existing 4 MiB subprocess-output ceiling.

Bounded model comparisons

For each scan pair, the CLI compares one bounded page from the earlier scan with one bounded page from the later scan. It iterates the Cartesian page set, so findings in different pages can still be matched without placing the complete scans in one model context.

Each model request is therefore limited to approximately 1 MiB of finding JSON plus the fixed comparison instructions, rather than growing with repository history.

Deterministic reconciliation

Confirmed matches returned by separate page comparisons are reconciled before persistence:

• occurrence IDs are joined through deterministic connected components;
• split and combined findings across page boundaries become one match group;
• occurrence IDs and groups are sorted before saving;
• the lexicographically first supporting reason is retained for a merged group;
• uncertain pairs are deduplicated deterministically;
• conflicting confirmed and uncertain results remain rejected.

The reconciled result preserves the existing per-pair comparison contract.

Bounded retained output

Raw batch results and the final serialized pair result are limited to 1 MiB. An oversized result stops with a controlled message before save-scan-comparison is invoked.

This bounds the CLI’s in-memory reconciliation state and prevents an unexpectedly large result from being passed to the workbench subprocess.

Documentation

The TypeScript README now documents the 64-pair workbench page, 32-finding and 512 KiB matching-input page, cross-page reconciliation, and 1 MiB result ceiling.

Tests

Updated the match-all fixtures to use the paged workbench contract and added regression coverage that verifies:

• matching plans continue across multiple pair pages;
• empty pair pages do not stop pagination;
• finding pages are submitted as separate bounded model calls;
• the Cartesian set of before/after finding pages is processed;
• confirmed matches spanning several batches are reconciled into one stable group;
• pair results are saved independently;
• existing match counts and summary output remain correct;
• empty scan comparisons are saved without starting Codex;
• confirmed/uncertain conflicts are not persisted;
• accumulated output over 1 MiB fails before persistence.

The triangular pair-offset calculation was also checked across multiple history sizes, and both new workbench commands were parsed through the bundled Python entry point.

Verification

• pnpm run types
• pnpm run format
• pnpm run build
• Python syntax compilation for modified workbench modules
• bundled workbench command parsing
• focused CLI workbench and semantic-comparison tests
• PATH="/opt/homebrew/bin:$PATH" pnpm run test

Full suite: 472 passed, 6 expected platform/integration skips, 0 failed.

[rajpratham1]

This pull request improves the scalability of scans match --all by replacing bulk loading with paginated retrieval of scan pairs and finding inputs. The implementation introduces bounded page sizes for scan pairs and findings, enforces a maximum serialized input size, validates pagination cursors, and reconciles matches generated across multiple batches before persisting the final comparison. Additional safeguards reject oversized automatic matching results with a clear error, and the workbench CLI and README are updated to document the new behavior and operational limits. Based on the visible changes, the implementation is well-aligned with the stated goal of bounding resource usage while maintaining matching correctness, and no blocking issues are apparent.