Repository navigation
[RFC] Unify FlyBot's dev harness into a single, versioned layer #874
Description
Problem
FlyBot has no single harness. Its knobs are spread across five places:
- prompt in
prompt.jsandAGENTS.md - tools in
tools.js - step and retry limits in
turns.js - model in
models.js - verification in CI and merge-gate
Every one of these is fixed per bot, so we can't tier work by difficulty, compare variants, or let an RSI loop evolve the harness safely.
Options
A. Status quo plus per-knob tweaks
- Pros: no refactor and no regression risk. Each fix is small.
- Cons: no single thing to version, compare, or roll back. Tiering would need special cases across five files, and any RSI loop would have to edit scattered code.
B. Unified declarative harness layer (proposed)
A harness is one config that declares prompt, tools, model, step/retry budget, and verifiers. The runtime loads the config chosen for each task.
- Pros:
- Lite, Medium, and Heavy become three configs.
- HarnessPro can evolve those files, with every change committed to git, so it's auditable and easy to roll back.
- It's easy to A/B test, and cost can be tracked per harness.
- Cons:
- Upfront refactor of the host code paths, with a risk of behavior drift. Mitigation: step 0 must be behavior-neutral and verified by
core/dev/eval. - Config can't express everything, and some logic will still live in code.
- Upfront refactor of the host code paths, with a risk of behavior drift. Mitigation: step 0 must be behavior-neutral and verified by
C. Adopt an external harness framework
For example, run coding tasks through an existing coding-agent runtime.
- Pros: mature tooling, and less to maintain ourselves.
- Cons:
- Its abstractions don't match ours: Bot Social, multi-bot rooms, and the
Written-by:and merge-gate contracts would all have to be bridged. - It adds an outside dependency.
- We'd lose control over prompt caching and cost.
- Its abstractions don't match ours: Bot Social, multi-bot rooms, and the
D. Fully online self-modifying harness (Harness²-style)
- Pros: adapts to new kinds of issues without an offline cycle.
- Cons:
- Hard to predict and audit.
- Each task needs extra trial runs, so it costs more.
- It's a poor fit for a merge gate.
- Better used only as an experience library later on.
Recommendation
Option B, rolled out in phases:
- Step 0: extract the harness layer with no behavior change. The current setup becomes a single
defaultharness. - Step 1: add Lite, Medium, and Heavy tiers, with routing labels set at triage and HarnessPro-style offline evolution checked against real CI,
core/dev/eval, andrsi-baseline. Add the issue/PR number, tool-call count, and wall time to the usage records. - Step 2: add a Harness²-style experience library, kept separate per repo or tech stack.
Risks
- Gaming weak tests. Add hidden tests or human spot checks.
- Refactor regressions in step 0.
- Noisy transfer of lessons between repos.
- Merging still requires human confirmation ([RFC] Collaboration model between FlyBot and agent-CI-Infra/bot #871).
Activity
Birfy commentedon Oct 23, 2026
Triaged as a design decision, not a task: labelled
enhancement|area:automation|needs-human. Notagent-ready— the first thing it asks for is a choice no agent should make for the owner.What the repo actually shows today
The five knobs are not five independent homes. At
1d81580e:core/src/host/prompt.js,tools.js,turns.js,models.js.ds-bot/src/host/…, plus the patch set underpatches/ds-bot/files/— every change there must be byte-identical in all of them (ds-bot-patches check), which the feat(rooms): mint and withdraw a local room invite link (#852) #861 and feat(rooms): answer a local room's invite link (#852) #864 PRs both had to satisfy.plugins/tree/src/prompt.js.core/dev/eval/exists (run.mjs,judge.mjs,rubric.mjs,specs.mjs,guard.mjs), so the step-0 gate the proposal names is real.core/dev/harness/is not a harness in this sense: it isds-bot-test-harness, a private package pinning@deepseek-ai/dsh 0.2.0-rc.2,jsdomandoxlintfor the unit tests. The name will mislead whoever picks this up.So the cost of Option B is not "one config file": step 0 has to move the same logic through
core/, three mirror copies, and a plugin prompt, withds-bot-patchesandarch-docsas gates. That is the number to weigh, and the proposal does not mention it.Three options
A. Status quo, per-knob tweaks. No refactor, no shared artifact; every future tier comparison stays a special case across the mirrors. Cheapest now, most expensive per change.
B. Unified declarative harness, as proposed. One config declares prompt, tools, model, step/retry budget and verifiers; the runtime loads the config for a task; Lite/Medium/Heavy are three configs. Step 0 must be behavior-neutral across the mirrors above, or the first regression lands in
ds-bot/where a plugin patch also lives. Step 1 needs a routing label set at triage, which does not exist — and adding one means deciding who owns it and who may write it (see #871), plus it interacts withrisk:T*, which the controller owns and no bot writes by hand.C. Declarative manifest only — no runtime move. Keep the code where it is, add one versioned
harness.jsonper tier that names the existing knobs (prompt file, tool set, model, budgets, verifier list) and a loader that reads it for reporting and for A/B, while the host keeps executing today's code paths. Gives versioning, comparison and rollback with no step-0 behavior risk; tiers become comparable before anything is refactored. It cannot express new behavior, and someone still has to keep the manifest honest against the code — which is exactly whatcore/dev/evalcan check.(C is not in the proposal; it is the option that buys the two things the issue actually asks for — comparison and rollback — without the refactor B defers its risk into.)
D and the self-modifying variant are out of scope until one of the above exists: both assume a gate that does not exist yet, and a merge gate that cannot audit what it merges is worse than a slower one.
Relation to #871
holdis human-only;risk:T*is the controller's).core/— a step-0 refactor and an unmerged mirror change in flight are the same conflict we keep paying for.What I need from the owner: A, B or C, and for B/C whether tier routing may be a label at all. With that, the first implementable issue is small either way (C: manifest + loader + one eval that reads it; B: the step-0 extraction, gated on the mirrors).
Step 0: default harness + usage five fields (#903) (#908)
hsliuustc0106 commentedon Oct 24, 2026
we decide to choose decision B
hsliuustc0106 commentedon Oct 24, 2026
Option B Step 0–2 done on main: #908 (default harness), #933 (Lite/Medium/Heavy), #941 (repo-scoped experience library). No online self-mod / hot-path tier changes. Further harness work = new issues. Closing this RFC.
Add Lite, Medium, and Heavy harness tiers (ThinkFlowLab#933)
State the reference-first hard rule for architecture and review-pr (T…
Add a repo-scoped experience library for harness step 2 (ThinkFlowLab…
Read the body on a queue entry, and write enqueue-guard on merge_group (