⚡ Copilot Token Optimization2026-10-26 — Build Test Suite #10652

Target Workflow: Build Test Suite

Source report: #10598
Estimated cost per run: ~$20.16 (report; AIC 18.5–22.8 per run)
Total tokens per run: ~4K reported (input+output only); real context is ~430–680K incl. cache reads (394K–640K cache-read + 35–60K cache-write per run)
Cache hit rate: ~90%+ (cache reads dominate; input uncached ~470 tokens)
LLM turns: 8–12 (3 runs)
Model: claude-sonnet-5.5

The workflow ranks #1 by cost. "Smoke Copilot" is excluded (open optimization issue).

Current Configuration

Evidence: runs 40166021945, 40146905683, 40146872159 (agent-session.jsonl).

Findings

• The agent re-derives a deterministic shell script every run: batched clones, installs, per-project test loops. Cost is driven by 8–12 turns, each re-reading a ~50K+ cached context, plus sonnet pricing.

• Retry/diagnostic turns are consistent. Hypotheses, not certainties:

  • dotnet: restore/build fails, then an inspection call, then a retry with DOTNET_CLI_HOME/XDG_RUNTIME_DIR (run 40166021945 calls 3→5→8; run 40146872159 calls 7→8).
  • Maven: ~/.m2 is not writable, then a retry with -Dmaven.repo.local=/tmp/gh-aw/agent/m2 (run 40166021945 calls 6→7→9; run 40146872159 calls 10→11).
  • Those fixes are the same every run, so they could be baked in up front and the diagnostic turns removed.

• The prompt's Java Maven settings recipe uses ~/.m2, which hits the problem above.

Recommendations

1. Move the deterministic build/test matrix into pre-agent steps: (highest impact)

Estimated savings: ~6–10 of 8–12 turns, ~350–550K cached tokens/run (~75–85%), and a large part of the cost.

Add a script (e.g. scripts/ci/build-test-matrix.sh) that clones, installs and tests all 8 ecosystems, with the known env fixes built in (DOTNET_CLI_HOME, DOTNET_SKIP_FIRST_TIME_EXPERIENCE, XDG_RUNTIME_DIR, -Dmaven.repo.local=$RUNNER_TEMP/m2, Maven proxy settings). It writes /tmp/gh-aw/build-test/results.json with per-project status, counts and a short error tail. Run it via steps: in the frontmatter. Note that pre-agent steps run outside the AWF sandbox, so verify they still exercise the firewall. If the point of this workflow is to test building through AWF, instead run the script as one single bash call from the agent. This still cuts turns from ~10 to ~3.

The agent prompt then shrinks to: read results.json, post a single comment with the table, add the build-test label if all pass and the trigger was a PR.

2. If the agent must stay in the sandbox: replace the per-task prose with one script call

Estimated savings: ~3–6 turns/run (~30–50%).

Replace the 8 task sections with "run bash scripts/ci/build-test-matrix.sh once, then summarize". This removes the dotnet and Maven retry turns, which are identified above as hypotheses of redundant work.

3. Drop the unused GitHub MCP server

Estimated savings: ~8–12K tokens of tool schema per turn (~10 turns, ~80–120K cached tokens/run, ~15%).

The github: tool was called 0 times in all 3 runs. Remove it from tools:. If a PR number is needed, use safe-outputs context. Otherwise, at minimum restrict toolsets: [pull_requests].

tools:
  bash:
    - "*"
# github: removed

4. Use a cheaper model

Estimated savings: ~50–70% of cost/run.

The work is mechanical summarization; claude-haiku-4.5 (used by the BYOK smoke) or gpt-5.4-mini should suffice once the script does the work.

engine:
  id: copilot
  model: claude-haiku-4.5

5. Prompt trim

Estimated savings: ~2K tokens/turn (~15–20K total). Once recommendation 1 or 2 lands, remove the per-ecosystem blocks and the 18-row template (keep a compact one).

6. Network group review

No token savings; only cleanup. All listed groups map to ecosystems that are exercised, so keep them as is. Verify jsr.io and dl.deno.land are needed.

Tool-call cap

Sample: 3 successful runs (40166021945, 40146905683, 40146872159), non-final tool calls = 9, 10, 12 (the add_comment call is the final one). p50 = 10, p95 = 12 (nearest rank).

No --max-num-tool-calls recommendation. 2 of 3 runs (67%) are within two calls of the observed maximum (≥20% threshold), and n=3 is barely at the minimum. More data is needed. Once Recommendation 1 or 2 lands, recompute the baseline.

Expected Impact

Implementation Checklist

• Create scripts/ci/build-test-matrix.sh with the dotnet/Maven env fixes built in
• Decide: pre-agent steps: (outside AWF) vs single in-sandbox bash call
• Remove the unused github: tool from .github/workflows/build-test.md
• Switch engine.model to a lower-cost model
• Shrink prompt to read results and comment
• Recompile: gh aw compile .github/workflows/build-test.md
• Post-process: npx tsx scripts/ci/postprocess-smoke-workflows.ts
• Verify CI passes on PR
• Compare token usage on new run vs baseline

Generated by Daily Copilot Token Optimization Advisor · copilot · auto · ~27 AIC · ⊞ 11.4K · ◷