Follow-up to #136, which shipped eval-magic harness lint <name|file> with schema, user-layer, and cross-field invariant checks. The subcommand's --help and docs/byoh.md reserve a --probe flag for the piece lint can't prove statically: that the descriptor's exec template actually drives the harness.
Idea
harness lint <name|file> --probe renders dispatch.exec_template with a trivial prompt in a throwaway temp dir (filling <eval-root>, <dispatch_prompt_path>, <outputs_dir>, empty {model_arg}/{guard_args}), executes it, and verifies the final-message contract: outputs/final-message.md exists and is non-empty afterwards. This closes the last gap in the BYOH loop — an agent authoring a descriptor from knowledge of its own CLI can prove the recipe end-to-end before dispatching a real run.
Open questions
- Probe cost/safety: it invokes a real agent CLI (network, tokens). Definitely opt-in; probably needs a timeout and a clear "about to execute" banner.
- What counts as pass: final-message recovery only, or also events-file presence when
[transcript] is declared (proving events_filename lines up with the template's capture redirect)?
- Whether the probe should also validate
parallel_command_template/judge_command_template shapes by rendering (not executing) them.
- CI story: the built-ins' probes require the harness binaries on PATH — likely a manually-run check, not a test-suite member.
Refs #136.
Follow-up to #136, which shipped
eval-magic harness lint <name|file>with schema, user-layer, and cross-field invariant checks. The subcommand's--helpanddocs/byoh.mdreserve a--probeflag for the piece lint can't prove statically: that the descriptor's exec template actually drives the harness.Idea
harness lint <name|file> --proberendersdispatch.exec_templatewith a trivial prompt in a throwaway temp dir (filling<eval-root>,<dispatch_prompt_path>,<outputs_dir>, empty{model_arg}/{guard_args}), executes it, and verifies the final-message contract:outputs/final-message.mdexists and is non-empty afterwards. This closes the last gap in the BYOH loop — an agent authoring a descriptor from knowledge of its own CLI can prove the recipe end-to-end before dispatching a real run.Open questions
[transcript]is declared (provingevents_filenamelines up with the template's capture redirect)?parallel_command_template/judge_command_templateshapes by rendering (not executing) them.Refs #136.