Repository navigation
Step 0: extract default harness + usage five fields (#874 B) #903
Description
Parent
Implements Option B / Step 0 of #874.
Out of scope (this issue)
- Lite / Medium / Heavy tiers (Step 1)
- Magpie
- [RFC] Collaboration model between FlyBot and agent-CI-Infra/bot #871 human-confirm automation
- Edits under
core/dev/harness/ - Secrets / API keys in the issue, CI, or commits
Goal
- Extract a declarative harness layer; today's setup becomes a single
defaultharness (prompt / tools / model / step·retry / verifiers). - Keep behavior neutral vs main before the extract.
- Extend usage ledger so each recorded call carries five fields:
harness(=default), issue·PR, tool-call count, wall clock, token in/out (reuse existing token columns).
Three mirrors (must stay in step)
Per scripts/ds-bot-patches.mjs / #34 — Step 0 must leave these three equal after any core change:
| Role | Path |
|---|---|
| Upstream pin | ds-bot/ (tree pinned by ds-bot.pin.json) |
| Patch series | patches/ds-bot/ (*.patch + files/) |
| Committed result | core/ |
Check: node scripts/ds-bot-patches.mjs check (also covered by npm run verify).
Edit path: change core/, then generate / apply as documented — do not edit ds-bot/ by hand.
Tree prompt exception (explicit)
@flybot/tree keeps its own marked prompt block under plugins/tree (not one of the three mirrors). Step 0 must list and preserve that exception: tree prompt injection stays outside the default harness file if that is today's behavior; document the exception in the PR; eval P3-tree-split must still pass.
Acceptance (merge gate)
- Unit tests (
core/test/usage.test.jsor equivalent): missing any of the five new usage fields → red → green. - One real
$DSH_HOME/bot/usage.sqlite(calls) sample row with all five fields. Usage page screenshots do not count. - Behavior parity vs pre-extract baseline:
npm run verify,node scripts/ds-bot-patches.mjs check, andcore/dev/evalspecs unchanged. - Control set of 10 frozen items below: each run records commit/spec id, pass/fail, and a usage sample. Changing the set requires a separate PR.
Control set (frozen)
| # | Tier | Item | Anchor |
|---|---|---|---|
| 1 | simple | eval memory-recall |
spec id |
| 2 | simple | PR #880 | merge b6ab4b2f223adde424c6d80eae6af9070c19e4d4 |
| 3 | simple | PR #886 | merge 0c487992d744a2357a08f467d6cb1dc431343070 |
| 4 | medium | eval P3-tree-split |
spec id |
| 5 | medium | eval P4-invite-approve |
spec id |
| 6 | medium | PR #898 | merge 9f4aa524eb1c05e5f2127ec11c5dd05d172c2f2a |
| 7 | medium | PR #897 | merge 9fe5c261479057b472452f5ab19c92d485e0cbff |
| 8 | easy-INC | eval P6-timeline |
spec id |
| 9 | easy-INC | PR #869 | merge 690093bc7b8f22945efc0050ac4ca44b438f6f8f |
| 10 | easy-INC | PR #900 | merge abaebb2bab22cfd6b890b1b996476d1c66b5c1e2 |
VM runs only this list (no demo extras). Isolate with e.g. export DSH_HOME=/var/flybot-test.
Local / VM reproduce
npm run verify
node scripts/ds-bot-patches.mjs check
node core/dev/eval/run.mjs --list
node core/dev/eval/run.mjs --spec <id> --text <artifact> --json out.json
node --test core/test/usage.test.js core/test/usage-data.test.js
# VM install + one real turn
npx -y @deepseek-ai/dsh@0.2.0-rc.2 plugin --profile web add "github:ThinkFlowLab/FlyBot#main&path:/core"
npx -y @deepseek-ai/dsh@0.2.0-rc.2 plugin --profile web add "github:ThinkFlowLab/FlyBot#main&path:/plugins/tree"
npx -y @deepseek-ai/dsh@0.2.0-rc.2 web --port 3080
# after a turn:
sqlite3 "${DSH_HOME:-$HOME/.dsh}/bot/usage.sqlite" ".schema calls" "SELECT * FROM calls ORDER BY time DESC LIMIT 5;"Ledger path: ${DSH_HOME:-$HOME/.dsh}/bot/usage.sqlite (or bot.config.home/usage.sqlite).
Notes
- Step 1 (tiers + unit success cost / ROI) is out of this issue.
- Room ACK: scheme B + this acceptance + control set 10 (2026-10-24).
Activity
feat(#903): Step 0 default harness + usage five fields
Birfy commentedon Oct 24, 2026
Triaged 2026-10-24: this is a step with a clear scope, but it rests on a decision that is not recorded anywhere — so it waits, and here is exactly what is missing.
Missing:
#874is labelledneeds-humanprecisely because picking an option is the owner's call, and its only comment says so. This issue says "Implements Option B / Step 0 of#874", but no comment on#874records that B was chosen. Nothing here should start before that one line exists — a Bot refactoring the harness because an issue title asserted a choice is how a wrong option gets implemented thoroughly.To unblock (one sentence from the owner): "Option B is chosen" on
#874(or a short note on this issue saying Step 0 is approved on its own). The moment that exists, this isagent-readyas written: the scope, the three-mirror rule (node scripts/ds-bot-patches.mjs check), the@flybot/treeprompt exception and the out-of-scope list are all already specific enough for a Dev to start.Why not
agent-readymeanwhile:needs-human, overneeds-info— the gap is a decision, not information someone can go and fetch.Not merged into
#874: this is one work item with its own acceptance;#874stays the umbrella.fix(#903): real usage fields from turn events + sqlite sample
docs(core-host): document default harness + usage ledger v3 (#903)
fix(#903): drop unused mkdirSync in usage-fields test
Step 0: default harness + usage five fields (#903) (#908)
Fix mock admin @ALL and in-place compaction in smoke (#917) (#925)