Feat/byoh layered descriptors#145
Merged
Merged
Conversation
Carve descriptor provenance into a new descriptor::layers module (Layer, DescriptorSource, embedded_sources) and rebuild the registry on top of it: entries now record their contributing sources, build_registry takes owned DescriptorSource values and returns a Result (duplicate labels error instead of panicking), and the static becomes a OnceLock whose lazy fallback loads the embedded built-ins only — the seam #136's user-supplied layers feed into next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--harness now parses as a plain string and resolves against the descriptor registry in run_context_with_bootstrap, so runtime-loaded descriptor layers (#136) can name harnesses clap never saw. The PossibleValuesParser goes away with the compile-time-complete registry assumption: unknown names now surface the registry's own actionable error (unknown harness 'x'; known harnesses: ...) with exit 1 instead of clap's invalid-value rejection with exit 2, and run --help names the built-ins in the flag's doc text instead of [possible values:]. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A descriptor with just a label now loads: skills_dir leaves the schema's required set and becomes Option<String>, and HarnessAdapter::skills_dir returns Option<PathBuf>. Two new load-time invariants keep the combination coherent — [staging] configured without skills_dir and [guard] without skills_dir are both rejected, since staging copies into the skills dir and the guard keeps its marker/manifest there. Guard teardown and the armed-marker sweep skip skills_dir-less harnesses; staging call sites keep a PathBuf via expect, justified because the run preflight (upcoming) forces --no-stage for such harnesses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Descriptor sources now stack in precedence order — embedded built-ins, user-global ($EVAL_MAGIC_CONFIG_DIR, $XDG_CONFIG_HOME/eval-magic, or ~/.config/eval-magic, under harnesses/*.toml), project-local (.eval-magic/harnesses/*.toml), and a one-off --harness-file — with field-level merge: a later file overrides individual fields of an earlier descriptor with the same label, or defines a new harness outright. Every file passes the schema gate individually (only label is required, so partial overrides validate too) and merged results re-run the full invariant suite with a provenance chain naming every contributing file. Policy: broken discovered files are skipped with a warning pointing at harness lint — they never brick the CLI; a broken --harness-file is a hard error. User-layer files may not declare [guard] or flip run.supports_guard (fail-open safety until the guard engine opens up); unguarded runs fall back to the detect-stray-writes audit. When --harness is omitted, a --harness-file descriptor's label becomes the invocation's default harness. Test helpers set EVAL_MAGIC_CONFIG_DIR="" so a developer's user-global descriptors never leak into the suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
harness list surveys the registry — label (with the session default marked), contributing layers (built-in + project when merged), and the enhancements each resolved descriptor declares. harness show prints one harness's resolved post-merge descriptor as authorable TOML behind a provenance comment header, ready to copy into a layer file; descriptor structs gain Serialize (defaults omitted) and the toml dependency turns on display for it. harness lint validates a descriptor file — TOML + schema, the user-layer guard restriction, and cross-field invariants merged onto the registered harness with the same label, so partial overrides are checked against their real merge target — or, given a name, strictly re-lints every discovered layer file, surfacing exactly what registry init skipped with a warning. ✓/✗ report lines follow the validate idiom; a reserved --probe live-dispatch check is documented as deferred. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The harness preflight now follows the #126 model: each enhancement the selected harness doesn't declare produces a warning naming the fallback that carries the run, and the run proceeds degraded. --guard on a guard-less harness (e.g. opencode) is forced off with a pointer at the detect-stray-writes audit instead of erroring; a harness without a skills_dir falls back to --no-stage with each SKILL.md inlined; a missing transcript parser warns that transcript_check grades as unverifiable with llm_judge carrying the grading; requested models without a descriptor model flag are recorded as provenance only. Only genuinely contradictory flag combinations (--bootstrap/--stage-name where the descriptor declares them incompatible with --no-stage) remain hard errors. ingest prints a matching note on transcript-less harnesses, which previously skipped transcript work silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A descriptor with just a label and an exec_template now earns real dispatch guidance: cli_next_steps renders a generic iterate-dispatch recipe around the rendered exec command, and dispatch-manifest.md gains a generic '## Dispatch recipe' section with the final-message capture contract. A baseline descriptor (no exec template) gets the harness-agnostic handoff naming outputs/final-message.md and the ingest line. Built-in harnesses all declare their own templates, so the golden artifacts are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three integration tests pin the bring-your-own-harness contract: a project-local descriptor with just a label and exec template carries a complete run — fallback warnings, generic recipes in RUNBOOK.md and dispatch-manifest.md, simulated dispatches, ingest with the stray-writes audit, and llm_judge grading; a --harness-file descriptor becomes the invocation's default harness; and a descriptor referencing the codex-items parser by name gets full transcript ingest (parsed tool invocations in run.json) with zero code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New docs/byoh.md — the user-facing bring-your-own-harness guide: the minimum viable descriptor, the exec-template contract (eval_root cwd, final-message recovery), layer order and field-level merge semantics, the named-capability reference, the guard restriction, the warning/fallback table, and the lint/show/list workflow. progressive-enhancements.md catches up with the layered registry (label is the only required field, warn-never-reject preflight, layers.rs and registry init in the code map) and its Adding-a-new-harness checklist now starts from a user descriptor instead of the retired enum variant step. README gains the BYOH paragraph, a docs-table row, and the guard-fallback wording; --help's examples cover harness lint/list/show and --harness-file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The hidden guard/guard-codex hooks fire on every PreToolUse in a dispatched session and only ever need the embedded descriptors, so cli::run() no longer runs init_registry for them — the lazy embedded-only registry fallback serves the hook path, and a broken user descriptor can't add a skip-warning (or discovery latency) per tool call. Also from review: refresh the three doc comments that still described the retired --harness value parser, caveat harness show's copy-paste claim for guarded built-ins (drop [guard] before reusing as a user layer), and reword lint's name-target failure summary to say the ✗ lines may belong to other discovered files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #136