Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
8333cdf
Merge pull request #112 from slowdini/main
slowdini Jun 25, 2026
3daa9f9
refactor(instant-magic): reduce plan mode to single harness-agnostic
Jun 29, 2026
5f14ec4
Merge pull request #123 from slowdini/refactor/generic-plan-mode
slowdini Jun 29, 2026
95dec3a
refactor(run-mode): delete the in-session-only switch-condition/reset…
slowdini Jun 29, 2026
37449a8
refactor(run-mode): remove the in-session dispatch mechanism
slowdini Jun 29, 2026
fc6fd4f
Merge pull request #124 from slowdini/refactor/remove-cli-dispatch
slowdini Jun 30, 2026
34f325d
feat(instant-magic): retire run mode, collapse docs
Jul 2, 2026
33a7bae
Merge pull request #125 from slowdini/refactor/collapse-run-mode-vocab
slowdini Jul 2, 2026
3b1f98e
refactor(adapters): move harness-specific code into per-harness modules
slowdini Jul 6, 2026
cc009de
refactor(adapters): tier the HarnessAdapter trait into baseline + enh…
slowdini Jul 6, 2026
f38e1cf
refactor(adapters): drop the orphaned per-harness plan-mode renderers
slowdini Jul 6, 2026
0168eba
docs(harness): replace the parity doc with progressive-enhancement docs
slowdini Jul 6, 2026
5942b40
Merge pull request #127 from slowdini/refactor/adapters-per-harness-m…
slowdini Jul 6, 2026
c2be6ba
Merge branch 'dev' into refactor/adapters-tiered-trait
slowdini Jul 6, 2026
19df238
Merge pull request #128 from slowdini/refactor/adapters-tiered-trait
slowdini Jul 6, 2026
f52be3f
Merge branch 'dev' into docs/harness-progressive-enhancements
slowdini Jul 6, 2026
4910d15
Merge pull request #129 from slowdini/docs/harness-progressive-enhanc…
slowdini Jul 6, 2026
cb09d61
fix(sandbox): cover every harness config dir in guard tamper rule and…
slowdini Jul 7, 2026
98a493a
Merge pull request #131 from slowdini/fix/harness-config-dir-isolation
slowdini Jul 7, 2026
932dbeb
refactor(adapters): de-static the HarnessAdapter trait surface
slowdini Jul 8, 2026
72d50d0
Merge pull request #140 from slowdini/refactor/de-static-adapter-trait
slowdini Jul 9, 2026
4836b0a
refactor(adapters): extract shared JSONL reader
slowdini Jul 10, 2026
f225fa8
refactor(adapters): fold plugin-shadow preflight behind the trait
slowdini Jul 10, 2026
f1923f5
refactor(adapters): adapter-supplied tool vocabulary
slowdini Jul 10, 2026
05a41a7
refactor(adapters): render judge recipes from one structural template
slowdini Jul 10, 2026
b2c1a21
refactor(run): derive Default for the remaining harness-naming impls
slowdini Jul 10, 2026
c041239
Merge pull request #141 from slowdini/refactor/absorb-out-of-trait-co…
slowdini Jul 10, 2026
bf7048c
test(golden): pin generated harness artifacts as committed fixtures
slowdini Jul 10, 2026
6259f9b
feat(descriptors): harness descriptor schema, TOML loading, and load-…
slowdini Jul 10, 2026
151bf43
feat(descriptors): generic DescriptorAdapter over the embedded descri…
slowdini Jul 10, 2026
45ebdf3
refactor(adapters): route adapter_for through the descriptor registry
slowdini Jul 10, 2026
c7a7e29
docs(descriptors): enhancements are descriptor fields or named capabi…
slowdini Jul 10, 2026
b1dd043
refactor(descriptors): carve invariant validation into descriptor/val…
slowdini Jul 10, 2026
d61607a
Merge pull request #142 from slowdini/feat/harness-descriptors
slowdini Jul 10, 2026
4a4c6c7
fix(core): preserve explicit-null staged_skill_slug through deseriali…
slowdini Jul 14, 2026
ffd4dd4
test(compat): pin conditions.json round-trips and --harness CLI errors
slowdini Jul 14, 2026
9f20c72
refactor(adapters): key the descriptor registry by label
slowdini Jul 14, 2026
8472735
refactor(core)!: resolve --harness against the descriptor registry
slowdini Jul 14, 2026
a10778f
refactor(adapters): carve the registry into its own module
slowdini Jul 14, 2026
18e2674
Merge pull request #143 from slowdini/feat/string-keyed-harness
slowdini Jul 14, 2026
2369f3d
refactor(adapters): stage the registry for layered descriptor sources
slowdini Jul 14, 2026
211d060
refactor(cli)!: resolve --harness after parsing, not in clap
slowdini Jul 14, 2026
63bd8a8
feat(descriptors): make skills_dir optional for baseline harnesses
slowdini Jul 14, 2026
c23f739
feat(byoh): layered user-supplied harness descriptors
slowdini Jul 14, 2026
3453d8b
feat(cli): harness list, show, and lint subcommands
slowdini Jul 14, 2026
0eda51e
feat(run): preflight warns about undeclared enhancements, never rejects
slowdini Jul 14, 2026
855f6d2
feat(dispatch): generic recipes for descriptor-only harnesses
slowdini Jul 14, 2026
b2c2da2
test(byoh): end-to-end acceptance coverage for descriptor-only harnesses
slowdini Jul 14, 2026
f12d6ac
docs(byoh): authoring guide, enhancement-contract updates, README + help
slowdini Jul 14, 2026
ff31449
fix(guard): keep the hook entry points off layered descriptor discovery
slowdini Jul 14, 2026
432f975
Merge pull request #145 from slowdini/feat/byoh-layered-descriptors
slowdini Jul 15, 2026
2151364
test(guard): byte-pin deny verdicts and fresh hook files
slowdini Jul 15, 2026
accdaba
feat(guard): generic guard engine over embedded descriptor data
slowdini Jul 15, 2026
ac35f27
feat(cli): generic guard-hook subcommand for descriptor-driven guards
slowdini Jul 15, 2026
2222e5f
feat(run): hard preflight rejection of --guard on user-only harnesses
slowdini Jul 15, 2026
2973efe
docs(guard): descriptor-data guard story across docs and messages
slowdini Jul 15, 2026
779a4aa
Merge pull request #146 from slowdini/feat/guard-generic-engine
slowdini Jul 16, 2026
76bdb75
feat(cli): harness init scaffold, BYOH upstreaming flow, and descript…
slowdini Jul 16, 2026
046c77c
Merge pull request #147 from slowdini/feat/byoh-harness-init
slowdini Jul 16, 2026
25e7a55
feat(transcript): declarative extractor tier for flat event streams
slowdini Jul 16, 2026
12fb7b9
Merge pull request #148 from slowdini/feat/transcript-extract-tier
slowdini Jul 16, 2026
ec0ed42
feat(run): auto-arm the write guard; scope preflight warnings
slowdini Jul 17, 2026
892d0ad
Merge pull request #149 from slowdini/feat/guard-auto-arm
slowdini Jul 17, 2026
cb20cdf
chore: bump version to 0.5.0
github-actions[bot] Jul 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Golden fixtures are byte-exact — never EOL-normalize them.
tests/golden/** -text
65 changes: 65 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE/harness-descriptor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
<!--
Harness-descriptor contribution PR — data-only.
Use for: adding a new built-in harness descriptor (or extending one) that only
reuses EXISTING named capabilities. New parsers, slug capabilities, shadow
preflights, and guard support are CODE contributions — one capability per PR,
separately from this one (see docs/progressive-enhancements.md "Guardrails").
Guide: docs/byoh.md "Upstreaming your descriptor".
Title: feat(harness): add <label> descriptor
-->

## Harness

- **Label:** `<label>`
- **Harness CLI + version verified against:** <!-- e.g. `cool-cli 1.4.2` -->
- **What it is / docs link:**

## Scope check (data-only)

- [ ] This PR adds/changes descriptor **data only** — no parser, slug, shadow,
or guard code (those are separate one-capability-per-PR contributions)
- [ ] `[transcript]`/`[shadow]`/`[staging]` (if declared) reuse an existing named
capability (`claude-stream-json` / `codex-items` / `opencode` / `claude-plugins`)
or ingest through a `[transcript.extract]` block (declarative data)
- [ ] No `[guard]` table and no `run.supports_guard = true` in this PR

## The diff (expected shape)

- [ ] `harnesses/<label>.toml` — the descriptor
- [ ] `src/adapters/descriptor.rs` — the `EMBEDDED_DESCRIPTORS` entry (mechanical
registration only)
- [ ] `docs/<label>-notes.md` — verified implementation notes (promote your
project's `.eval-magic/harnesses/<label>-notes.md` — `harness init`
scaffolds it)
- [ ] `README.md` — new support-table row (only wired enhancements checked;
❌ elsewhere)

## Verification evidence

- [ ] Lint is clean — paste the output:

```
$ eval-magic harness lint harnesses/<label>.toml
```

- [ ] A smoke eval ran end-to-end on this harness
(`run --harness <label>` → dispatch → `ingest` → `finalize`); describe or
link the result:
- [ ] Every declared enhancement was exercised by that run (e.g. staged skill
discovered natively, events file parsed by the named parser, model flag
appeared in the generated recipes)

## Don't-guess attestation

Every flag, filename, and event shape in the descriptor is traced to the
harness's own documentation or to output I observed — nothing is inferred by
analogy with another harness. `docs/<label>-notes.md` records, per value: the
value, its source (docs URL / `--help` output / observed events file), and the
harness version.

- [ ] The notes file's Verification record covers every non-comment field in
the descriptor

## Related

Issues/PRs:
8 changes: 5 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@

`eval-magic` is a Rust CLI crate. The binary entry point is `src/main.rs`; reusable logic lives in
`src/lib.rs` and submodules such as `cli/`, `pipeline/`, `sandbox/`, `validation/`, and
`workspace/`. JSON schemas are tracked in `schema/`, harness profiles in `profiles/`, and design or
parity notes in `docs/`. Integration tests are split by surface area under `tests/cli/` and
`tests/run/`; unit tests usually live beside the module they exercise.
`workspace/`. JSON schemas are tracked in `schema/`, harness descriptors in `harnesses/`, shared
prompt profiles in `profiles/`, and development docs (the harness enhancement contract,
per-harness notes) in `docs/`. Integration
tests are split by surface area under `tests/cli/` and `tests/run/`; unit tests usually live
beside the module they exercise.

## Build, Test, and Development Commands

Expand Down
62 changes: 61 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "eval-magic"
version = "0.4.1"
version = "0.5.0"
edition = "2024"
description = "One-stop CLI for running skill evals — measure whether an agent skill actually shifts behavior."
license = "MIT"
Expand Down Expand Up @@ -44,6 +44,10 @@ regex = { version = "1.12.3", default-features = false, features = ["std", "perf
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.150", features = ["preserve_order"] }
thiserror = "2.0.18"
# Harness descriptor files (harnesses/*.toml). `display` serializes the
# resolved (layer-merged) descriptor back to authorable TOML for
# `harness show`.
toml = { version = "0.9", default-features = false, features = ["parse", "serde", "display"] }
walkdir = "2.5.0"

[dev-dependencies]
Expand Down
Loading
Loading