fix(prompt): relocate discovered tool catalog to variable suffix - #13
Merged
Conversation
Cache hit rate remained ~11% after fix/cache-suffix-relocate (b9e53d1) despite the systemVariableSuffix relocation. 2026-05-26 dogfood diffed two adjacent main-turn system prompts on the same DM session: 04:39:04 turn=0 (16468B) vs 05:01:36 turn=2 (16576B): +108 bytes, diff at offset 11922 inside `## Tool Catalog`. The newer prompt has `- WebSearch: Search the web ...` and `- Dispatch: Delegate a focused task ...` lines that the older one lacks — ToolSearch promoted them between turns and the catalog grew. Same DM, two adjacent turn=0 query loops with no promotion in between (05:00:51 vs 05:01:22) were byte-identical, confirming fix/cache-suffix-relocate works on its own axis but the catalog section is a second prefix-break source. OpenAI Responses API auto-cache fingerprints by wire-byte prefix; any change after offset N invalidates the cache from N onward. Result: a turn that promotes a deferred tool truncates cache to the small block preceding the catalog growth (the 1,536-token fingerprint that recurs across usage.jsonl). Sum cacheRead/sum input on gpt-codex-mid main = 6.3% pre-fix, 11.8% post-fix-suffix-relocate, expected 60-80%+ once the catalog stabilizes. Fix: `## Tool Catalog` in the stable system section now renders only the always-loaded subset (alwaysLoad-tagged tools + ToolSearch, fixed membership for the lifetime of a query loop). Discovered (already ToolSearch-promoted) deferred tools go into a new `<system-reminder>` block in the variable suffix that `injectSystemReminderIntoLastUserMessage` places on the last user message — same path systemVariableSuffix uses since b9e53d1. Tools array passed to the provider is unchanged (full inline+discovered list ordered inline-first), so the tools API field stays cache-friendly via the existing prefix ordering. Backward compat: callers that don't pass `inlineCatalogTools` (tests, custom systemPrompt shapes) still render the full `tools` array in the catalog. The discovered-tools reminder is empty unless explicitly populated. Tests: prompt.test.ts adds a `cache anchoring under discovered tool promotion` describe block. Pre-fix verified: 2 of 4 new subtests fail on bare main (`keeps stable prefix byte-identical when discoveredTools grows` + `renders discovered tool descriptions in the variable suffix only`). Post-fix: 1753/1753 PASS. typecheck clean, build clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes prompt prefix-cache instability by making the stable system prompt’s ## Tool Catalog byte-stable across turns, even as ToolSearch promotes deferred tools during a session.
Changes:
- Split the per-turn tool catalog into (1) a stable “inline” subset for the stable
## Tool Catalogsection and (2) a discovered-tools list rendered only in the variable suffix. - Update split-render hook wiring so discovered (already-promoted) tools are appended via
systemVariableSuffix(injected into the last user message). - Add regression tests to ensure the stable prefix remains byte-identical under discovered tool promotion.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/tools/deferred-loading.ts | Extends TurnToolCatalog with inlineTools + discoveredCatalogTools and builds stable vs discovered subsets. |
| src/query.ts | Updates default TurnToolCatalog initialization to include new fields. |
| src/prompt.ts | Renders stable tool catalog from inlineCatalogTools and adds a variable-suffix reminder for discovered tools. |
| src/prompt.test.ts | Adds regression tests verifying stable prefix invariance and correct suffix routing. |
| src/agents/hooks/split-render.ts | Plumbs inlineCatalogTools/discoveredCatalogTools into split prompt rendering. |
| src/agents/hooks/forward-progress-to-channel.test.ts | Updates test HookContext stub for the extended TurnToolCatalog shape. |
| src/agents/hooks/deferred-tools.ts | Ensures TurnToolCatalog is self-consistent on the system-prompt-override path. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+15
to
+19
| // `discoveredCatalogTools` is the this-turn promoted-via-ToolSearch subset; | ||
| // its membership changes between turns, so the prompt renders it inside the | ||
| // variable suffix (injected into the last user message) instead of the | ||
| // stable system section. Both are still in `tools` so the provider's tools | ||
| // array carries every callable schema this turn. |
Comment on lines
+721
to
+725
| return `<system-reminder> | ||
| The following deferred tools were loaded via ToolSearch earlier in this session and are now callable. Full schemas live in the tools API description field — call them by name: | ||
| ${discoveredCatalogTools | ||
| .map(tool => (tool.whenToUse ? `- ${tool.name}: ${tool.whenToUse}` : `- ${tool.name}`)) | ||
| .join('\n')} |
Comment on lines
+712
to
+714
| // under discoveredTools changes. The block name lists the tool names + their | ||
| // whenToUse so the model has the same one-line hint it had pre-fix; full | ||
| // schemas still live in the provider's tools API field. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
2026-05-26 dogfood §cache hit rate root-cause. After
fix/cache-suffix-relocate(b9e53d1) moved TodoList + deferred-reminder out ofinstructions, the cache hit rate jumped from 6% to 11% — far below the expected 60-80%. Diffing two main-turn system prompts on the same DM session showed## Tool Catalogwas the second prefix-break source: ToolSearch promotion of a deferred tool appends a new line to the catalog inside the stable system section, breaking OpenAI's wire-byte prefix-cache fingerprint from the catalog onward. RecurrentcacheRead = 1536across usage.jsonl was the smoking gun.What
## Tool Catalogin the stable system section now renders only the always-loaded subset (alwaysLoad-tagged tools + ToolSearch). Membership is fixed for the lifetime of a query loop, so the section is byte-stable. Already-promoted deferred tools render in a new<system-reminder>block inside the variable suffix, whichinjectSystemReminderIntoLastUserMessagelands on the last user message — same pathsystemVariableSuffixhas used since b9e53d1.Provider
toolsarray is unchanged (inline + discovered, inline ordered first), so the tools-field prefix stays cache-friendly via ordering.Backward compat
Callers that don't pass
inlineCatalogTools(tests, custom systemPrompt shapes) still render the fulltoolsarray in the catalog. The discovered-tools reminder is empty when nodiscoveredCatalogToolsis supplied.Test plan
cache anchoring under discovered tool promotiondescribe block (4 subtests, 2 are core invariants)keeps stable prefix byte-identical when discoveredTools grows+renders discovered tool descriptions in the variable suffix onlyfail on bare main🤖 Generated with Claude Code