Repository navigation
Research: measure and cut token spend per Bot turn (prompt overhead, history growth, tool output, fan-out) #796
Description
Why
Every FlyBot turn spends a large and unmeasured share of its input tokens on things that are not the task: the group notice, the member roster, the speaking order, the full conversation so far, and every tool result that has ever been pasted into it. Nobody has measured that split, so nobody can tell a real saving from a change that only feels cheaper. On this host the delivery group has 10 Bots; one user message in the group can start up to 10 model calls, each carrying its own copy of the same context.
This issue is a research task, not a refactor: measure first, then land only what the measurement justifies.
What to measure (per turn, on this host)
- Fixed prompt overhead. The group notice, the member list, the speaking order, the tree block, and the injected memory files are re-sent on every turn of every member. Measure their size in tokens and multiply by turns per day.
- Conversation growth. Tokens per turn as a function of turns already taken in that conversation, before and after a checkpoint/compaction.
- Tool-result weight. The share of history that is raw tool output (GitHub JSON, test logs, file reads). Report the largest contributors and whether anything ever trims them.
- Fan-out cost. Model calls per user message in a group round, and the tokens each member re-reads that the others also read.
- Retry and duplicate work. Turns that were stopped, restarted, or repeated because a Bot re-read the same state — count them, because they are pure cost.
Measurement protocol
- Take one ordinary working day and one group round as the sample. Record input and output tokens per turn from the provider side, not from an estimate in the prompt.
- Produce a table: turn id, Bot, phase, input tokens, output tokens, and a category split for the input (fixed overhead / history / tool results / task).
- State the method so a second person can reproduce the numbers on another day.
Candidate savings (each must be proven, not assumed)
- Sending the group notice only when it changed, or compressing it to the rules that actually change behaviour.
- Roster and speaking order trimmed to the names that matter for that turn.
- Tool results summarised at the point they enter history, with the full text kept in a file.
- Earlier turns condensed more aggressively, with the exact wording still reachable through search rather than carried inline.
- Fewer members called per round when the work is sequential anyway.
Acceptance
- A report committed to the repository with the measured split above, and the method behind it.
- At least one landed change that cites a measured before/after in tokens per turn, on the same workload.
- Nothing in the safety rules is traded away for the savings: the attribution rules, the verdict rules, and the per-role write limits stay, and the report says which of them cost what.
Out of scope
pricing negotiation, provider choice, and per-team budgets (that is #645); this issue is only about spending fewer tokens for the same work.
Activity
Birfy commentedon Oct 23, 2026
Split, and only the measurable half is ready.
#796stays the umbrella: which savings to take changes what every Bot is sent on every turn, so it needs the owner's word. The measurement does not, and it needs nothing from CI or the provider.Split out:
#799— Token report: read the existing usage ledger into per-turn numbers and a reproducible prompt split (enhancement | area:automation | agent-ready, hanging under this epic).Why it is not starting from zero:
core/src/host/usage.jsalready writes a ledger with one row per billed attempt and keepsinput,cacheRead,cacheWriteandoutputapart (tokensOf():44-49; schema:142-152; file$DSH_HOME/usage.sqlite).core/dev/llm-tap.mjsalready logs the provider'susageper call plus aprefixdiff, and can dump request bodies for replay. Only the UI reads the ledger (core/src/client/usage-page.js); there is no per-turn distribution, no growth curve, no split, and no report.One number from this host's ledger, read read-only from a copy and covering 2026-10-20T07:34Z → 2026-10-22T22:05Z (16,399 calls):
That reframes the work: the biggest single block is not the group notice, it is the history that every turn re-reads, and any report that prints one "tokens" column will hide the difference between a cheap cache read and a fresh input token.
#799therefore requires the columns to stay apart and every number to carry its provenance.Scope guard:
#799may not touch prompts,core/src/**, workflows, or start a third metrics framework —#791defines the RSI baseline metrics and#357is the Nightly smoothness dashboard.#645stays out of scope, as you wrote.feat(dev): read the usage ledger into a token report (#802)
Ask for list_children only when more than one child fits a part (Thin…
Keep a group's fixed rules in its standing section, not in every turn (…
Keep children's load out of a tree coordinator's system prompt (Think…
Ask for list_children only when more than one child fits a part (Thin…
Birfy commentedon Oct 23, 2026
Written-by: Dev
Token audit of a Bot turn (owner request), two fixes merged:
9fe5c26): a group's fixed rules (members / face roster,@Namecalls, notice, admin / lead notes, face routing rule, reply rule) now live in the member Session'sbot-roomsystem-prompt section, which is rendered every step and survives a checkpoint, instead of in every stored turn. Turn text: 4 Bots, 713-char notice, 80 turns went 161,750 → 46,790 chars (−71%). A face room went 25,236 → 8,491 chars (−66%). A notice change is said in one line.9f4aa52): a tree coordinator's system prompt no longer changes at every child turn start or end. Before, 6 changes per 3-child round each re-sent ≥ ~11.5k chars on DeepSeek; now there are 0. Load is read withlist_childrenwhen more than one child fits.Audited, no change: the brief is sent once per Session, transcript windows are unseen-only deltas, relay and
message_bottext is plain, the friend fence is minimal, and FlyBot tool schemas (~16.4k chars) are stable and cached. Schedules add only the[Scheduled task]prefix. Face[PASS]fan-out is already cut after the first answer.Left open (behavior or design decisions, not fixed here): cutting face
[PASS]calls further, which would change routing, and trimming the largest tool descriptions (message_bot,update_group,save_capability).Keep children's load out of a tree coordinator's system prompt (Think…
Birfy commentedon Oct 24, 2026
Assigned to @hsliuustc0106 (2026-10-24, owner's request).
Two things changed since this issue was written, and they narrow what the measurement has to cover:
9fe5c261moved a group's fixed rules out of every turn into the room's standing section, so the "group notice re-sent on every turn of every member" cost named in the original scope no longer exists as written. Measure the new shape instead: the standing section, the member roster, the speaking order, the tree block and the injected memory files.9f4aa524took a coordinator's child-load block out of the system prompt, so the tree block is smaller than when this was filed.Start from a baseline on the current
mainand report the split per turn before proposing any change. Numbers, not intuition: tokens per turn for (a) fixed prompt, (b) history growth before and after a checkpoint, (c) raw tool output.