Repository navigation
M1: trustworthy benchmark inputs and an honest CI gate (#97) - #102
Merged
Merged
Conversation
The e2e job now starts apps/api/dist on loopback against its MongoDB service with a per-run database, agent and API key and no provider key, waits for /health with a bounded deadline, runs the 15-check proof-pack, proves the recorded PID alive before and after, and stops only that PID. The verdict JSON must contain all 15 checks passing; API log, PID and verdict upload on failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add scripts/tsconfig.json and a check-types script so turbo's check-types covers scripts/ source and tests, then fix every inherited error without suppression: - mocked helper returns vi.mocked(value), keeping mock APIs and the real callable and return types - stale fixtures completed against current types (ingest rejection counters, timestamp validation, case outcomes, candidate traces, run identity, e2eQa envelope, eval cases and search responses) - queue-settle doubles typed as MongoDBManagerHost, the type the method under test takes - run-benchmark passes the manager through the engine's own manager-to-host cast Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rom's 2026-10-18 grant: the e2e MongoDB service gets the VOYAGE_API_KEY secret so autoEmbed serves the vector and hybrid lanes. The job fails before starting the API when the secret is empty, and fails after readiness unless the API capability table reports hybrid and vector available, so a green run cannot come from the $text fallback. The API process itself still gets no provider key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…apter diagnostics (#94) Implements the approved 1197 shape table plus resolutions r3 for the LongMemEval adapter: any-element recognition with format-ambiguous, identity preflight before ordinary rejection (OPEN-26), typed value-free rejections in adapterDiagnostics, informational empty-turn skips, and BenchmarkAdapterRejectedError for all-rejected. Adapter totals map into rejectedTurns/rejectedConversations on ingest and import receipts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scenario runs count adapterDiagnostics into rejectedTurns and rejectedConversations, checkpoints persist adapterDiagnosticsTotals, and resume fails on contradictory counts. A missing totals field reads as zero only when the loaded dataset has no adapter diagnostics. Adds the pinned LongMemEval-S acceptance test (32 / 13 / 12 / 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uestion-no-sessions (#94) OPEN-5 per the 1206 Q4 ruling: a haystack emptied only by informational skips is rejected once; the skips stay informational. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#94) OPEN-8 per the 1206 Q4 ruling: a non-abstention question with missing or empty answer_session_ids keeps its gold and is retrieval-ineligible with ineligibleReason "no-evidence"; _abs questions keep the abstention label. Widens the engine union additively, and its type-only mirror in the benchmark runner's case-execution params so check-types passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d category 5 as abstention (#95) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…val scores (#95) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ns (#95) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A write that sends searchBarrier stamps a fresh searchWriteToken on the source and its chunk, checks every target index statically before writing, then probes each index for that exact version within a bounded wait. The outcome is returned as searchBarrier; unsupported targets reject with 400 BARRIER_UNSUPPORTED before any write, and an uncertain commit is annotated commitState unknown. Ordinary writes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A structured write that is quarantined never reaches structured_mem, so the barrier returns committed not-yet-searchable with reason quarantined, every target pending at 0 attempts, and makes no probe or paid call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Budget refusals throw SearchBudgetDeniedError, exhausted warmup retries throw index-not-ready, and an error at any hybrid tier stops the lane instead of escalating to a weaker tier. Classic $text runs only as the planned lexical tier when neither Search lane exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A non-canonical session key has no position in the 1..M walk, so its entry keeps only sampleIndex and follows the canonical session entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each failed probe logs one warn with the index name, the error class and a numeric code when present; never the message, query or token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…101) Typed tier failures, per-index text availability, and no BSON text indexes for structured_mem or procedures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…acks (#101) Every lane, tier and phase now reports a bounded {lane, stage, outcome, reason} row; failures and budget denials travel as typed errors. - KB (1f): an error stops the lane; zero hits keep the planned tiers; classic $text runs only when Atlas Search is absent. - Event lanes (1c): per-branch rows; denials are typed, never skipped. - Episodes and graph (1h): a $search error throws typed; no $regex retry. - searchV2 (1b): one failure policy, not-planned and zero-lane rows, hybrid aggregates its sub-lanes, backstops only after healthy zeros, the hybrid backstop's rows merge, outcomeStatus on the metadata. - Executor (E1/E3): rows keep their pass; partial passes stop follow-ups. - Manager (1g): onOutcomes sinks on search and searchKB, typed request failures, legacy re-run only after a complete empty, legacy row sink. - Public exports for the outcome types and typed errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bridge returns each call's outcome summary; one pure mapper serves /v1/search, /v1/search-kb and /v1/search-detailed: 200 complete/partial, 503 SEARCH_LANES_UNAVAILABLE, 503 SEARCH_BUDGET_EXHAUSTED, 500 SEARCH_LANES_FAILED. Typed strict-mode errors map through the same table, and wire rows are bounded to their enum fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#101) MemongoClientError parses outcomes; silent mode keeps its degradation marker and adds the API code and outcomes; searchDetailed metadata types laneOutcomes and outcomeStatus. MCP passes them through unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The classic $text lane exists only where the engine creates its index (no Atlas Search), and an unknown probe reports it unavailable. The table no longer claims a silent $text fallback. The rule is re-exported through the bridge rather than copied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three search routes document the outcome summary on 200 bodies, the detailed laneOutcomes/outcomeStatus metadata, and the 503/500 search failure codes; the contract table now lists 503 for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decision 18: a vector stage, a warmup retry and an inline event vector branch charge both halves or neither. The request ledger labels its counts as engine query work and breaks out retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…als (#101) Decision 18 in searchV2: the planned paths, and conversation evidence when exact evidence was asked for, reserve their cost-table estimate before optional work starts. An unschedulable required lane is a typed budget-reservation denial. A retrieval equivalent to one that already completed in the request is reused instead of re-spent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s into the barrier (#93) search reads the agent partition the structured and procedure writes use; searchDetailed asks exact evidence of the planted session turns. Both checks require the planted marker, and the four write checks require searchBarrier.status searchable. Predicates are importable and tested offline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#101) Pass 1, the exact-evidence follow-up and the corrective pass run on one request ledger: required lanes of every pass run or are reused, and the corrective pass's new optional backstop is denied with budget-reservation. The equivalence key drops the caller's raw limit, since the resolved limit already shapes every lane. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…93) The proof-pack and its key guard move out of tier A into a proof-pack job whose atlas-local service alone carries VOYAGE_API_KEY; tier A is keyless and requires the search-barrier suite. On failure the job uploads the service log and the proof database's search-index status, each bounded and scrubbed of URIs, keys and document text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dger (#101) search() and searchDetailed() run their opt-in legacy re-run on the same ledger as the v2 search it double-checks, and searchDetailed reports the whole request's snapshot. The direct KB route (K12) runs on a ledger with its lane reserved first; an unschedulable lane rejects typed with a budget-reservation row and never runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arch marker hit (#93) memongo-sol (1215) refused: a hit with the expected scopeRef under another scope passed the partition check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI run 39163428188 booted the API 1 s before its autoEmbed vector indexes became queryable, so capabilities snapshotted vectorSearch:false and the startup guard refused the run. Strict readiness waits up to 180 s and fails startup loudly instead of booting with lanes off; the health wait now has headroom over it. Also log the text-tier availability probe failure instead of swallowing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API log is printed to the job log and uploaded on failure. It now goes through the same scrub and bound as the service log; if scrubbing fails the raw log is removed and the job fails. Resolves the 1216 review finding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
romiluz13
marked this pull request as ready for review
October 18, 2026 21:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone 1 integration (#97): trustworthy benchmark inputs and an honest CI gate. Spec #90.
Draft. This is for the first full CI run. Merging is Rom's decision.
Tickets in this branch
check-types$textor$regexfallback after an errorBehavior changes (intentional)
$textis no longer used for them.searchWriteTokenon 8 search index definitions, andtemporalScopeonstructured_mem_text. Existing deployments rebuild those indexes once.idx_structured_textandidx_procedures_textare no longer created; existing copies can be dropped manually.laneOutcomes.VOYAGE_API_KEY. Its paid barrier probes are capped at 60 query embeddings per run, as Rom approved.Local evidence
bun run check-types(forced, no cache),bun run lint, and every package's tests. Results: engine 4144, api 752, client 129, bridge 134, mcp 254, lib 229, tools 120, scripts 1526 passed (4 skipped).CI evidence at head 246294d
writeEventandaddreportsearchBarrier=searchableon 4 targets each;writeStructuredandwriteProcedureon 2 each. So atlas-local acceptsexact: trueon autoEmbed vector indexes.Negative controls
Each is one patch on a throwaway draft PR off 246294d. Each fails only the proof-pack job, on its named line, while quality and tier-A e2e stay green.
API pid … exited before becoming readyverdict lacks passing required checks: relevanceReportFAIL search: search returned no resultsVOYAGE_API_KEY secret is empty(key guard)Review
$texton non-Search deployments) awaits Rom.🤖 Generated with Claude Code