Skip to content

feat(cli): harness init scaffold, BYOH upstreaming flow, and descriptor PR template#147

Merged
slowdini merged 1 commit into
devfrom
feat/byoh-harness-init
Jul 16, 2026
Merged

feat(cli): harness init scaffold, BYOH upstreaming flow, and descriptor PR template#147
slowdini merged 1 commit into
devfrom
feat/byoh-harness-init

Conversation

@slowdini

Copy link
Copy Markdown
Owner

Closes #138 (part of the descriptor epic #122, step 7 — make the BYOH path agent-followable).

What changed

  • New subcommand eval-magic harness init <name> — scaffolds .eval-magic/harnesses/<name>.toml, a fully commented descriptor template (only label live; every optional table a commented-out, inline-explained example with VERIFY-don't-guess prompts) plus <name>-notes.md, the verification-record skeleton. The scaffold is lint-clean as written and registers the name as a baseline harness immediately. --stdout prints the template for redirection, --force overwrites, non-kebab names are rejected, and scaffolding a registered name (e.g. claude-code) prints an overlay note instead of erroring.
  • Embedded template assets harnesses/template.toml / harnesses/template-notes.md, packaged with the crate (pinned by the package test). Unit drift-guards enforce that the template mentions every schema field and named-capability value, and that the fully-uncommented examples form one coherent, invariant-clean descriptor.
  • docs/byoh.md: new "Upstreaming your descriptor" section (exactly which capabilities are data vs code, the four-file data-only PR shape, how to open it with the template) and a "Verify, don't guess" section; the five-minute version and workflow now lead with harness init.
  • New PR template .github/PULL_REQUEST_TEMPLATE/harness-descriptor.md for data-only descriptor contributions: scope check (existing named capabilities only, no [guard]), expected diff shape, verification evidence, and the don't-guess attestation. Ordinary PRs stay template-less.
  • Docs touches: progressive-enhancements.md ("Adding a new harness" + Guardrails reference the scaffold and PR template), README Harnesses/Documentation/Bundled-assets sections, and the top-level --help examples.

Before / after

Before, the five-minute BYOH flow started from a bare heredoc:

mkdir -p .eval-magic/harnesses
cat > .eval-magic/harnesses/cool-custom-harness.toml <<'TOML'
label = "cool-custom-harness"
TOML

After:

$ eval-magic harness init cool-cli
Scaffolded harness descriptor: .eval-magic/harnesses/cool-cli.toml
Scaffolded notes skeleton:     .eval-magic/harnesses/cool-cli-notes.md

✓ TOML syntax + schema
✓ user-layer restrictions ([guard] stays built-in-only)
✓ cross-field invariants
Linted .eval-magic/harnesses/cool-cli.toml: all checks passed.

Next:
  1. Fill in verified values — follow the template's comments; never guess a flag
     (record each value's source in .eval-magic/harnesses/cool-cli-notes.md).
  2. Re-lint after every edit: eval-magic harness lint .eval-magic/harnesses/cool-cli.toml
  3. Smoke eval: eval-magic run --harness cool-cli  (dispatch, then ingest + finalize)
  4. Upstreaming the proven descriptor: docs/byoh.md "Upstreaming your descriptor".

Verification

  • cargo test — 430 lib + 94 cli + 69 run tests pass (10 new integration tests in tests/cli/harness_init.rs, 4 new unit drift-guards; both guards mutation-tested)
  • cargo fmt --check and cargo clippy --all-targets -- -D warnings clean
  • Manual: harness init cool-cli → lint → harness list (shows cool-cli project baseline), overlay note on claude-code, --stdout mode, harness init --help, top-level --help examples block

Notes for review

  • CLI surface addition (harness init) with thorough --help; no schema changes; docs updated across byoh.md / progressive-enhancements.md / README.
  • --probe is still only "planned" everywhere — the template and docs never print it as runnable.

🤖 Generated with Claude Code

…or PR template

Closes #138 (descriptor epic #122, step 7): make the bring-your-own-harness
path agent-followable end to end.

- `eval-magic harness init <name>` scaffolds a commented descriptor template
  (every field explained inline, VERIFY-don't-guess prompts, lint-clean as
  written) plus a notes-file skeleton for the per-field verification record,
  into the project-local layer; `--stdout` prints the template, `--force`
  overwrites, already-registered names get an overlay note instead of an error.
- Templates ship as embedded assets (harnesses/template.toml,
  harnesses/template-notes.md); drift guards pin that the template mentions
  every schema field and capability name and that the fully-uncommented
  examples form one coherent descriptor.
- docs/byoh.md gains "Upstreaming your descriptor" (data-vs-code split, the
  four-file PR shape) and "Verify, don't guess"; progressive-enhancements.md,
  README, and the --help examples now route through `harness init`.
- New .github/PULL_REQUEST_TEMPLATE/harness-descriptor.md scopes data-only
  descriptor PRs: diff shape, verification evidence, don't-guess attestation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@slowdini
slowdini merged commit 046c77c into dev Jul 16, 2026
6 checks passed
@slowdini
slowdini deleted the feat/byoh-harness-init branch July 16, 2026 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(byoh): bring-your-own-harness guide, descriptor template, harness init scaffold

1 participant