ifrit98 · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,33 @@
+# Explainer — instructions for coding agents
+
+This file is for agents other than Claude Code. Claude Code reads `CLAUDE.md`, which loads the same principles through an import that other agents do not follow.
+
+In this folder, act as an explanation compiler. Before you write any explanation, also a short answer, read `plugin/skills/explain/references/principles.md` and follow it.
+
+For a non-trivial explanation, follow the `explain` skill (`plugin/skills/explain/SKILL.md`). Use `plugin/skills/video/SKILL.md` to go straight to an animated explainer, and `plugin/skills/verify/SKILL.md` to check an existing explainer. Their reference files are in `plugin/skills/explain/references/`.
+
+## This repo
+
+The repo is three things: the toolkit (`explainer_kit/`), the Claude Code plugin (`plugin/`, listed by `.claude-plugin/marketplace.json`), and the examples (`explainers/`). Human-facing docs live in `docs/`. When you change the toolkit or the workflow, update the matching doc page and `ROADMAP.md`.
+
+Run the CLI through uv: `uv run explainer <command>`.
+
+```bash
+uv run explainer new <slug> --stage 1 2 3 4   # scaffold model.md, model.yaml, narrative.md, and the chosen renderings
+uv run explainer check                        # every example against its model.yaml
+uv run explainer coldread <slug> --run        # first-viewing read by a fresh agent
+uv run explainer findings --stats             # how many review findings authors adopt, per tool
+uv run explainer eval chat                    # chat answers under each system prompt, graded blind
+uv run explainer eval artifact --split dev    # held-out subjects built under each condition, read and graded blind
+uv run explainer render <slug> --draft        # fast silent layout pass
+uv run pytest -q                              # toolkit tests
+```
+
+The review and eval commands (`probe`, `coldread`, and `quiz` with `--run`, and `eval`) call the Claude CLI (`claude -p`). They need it installed and signed in, whichever agent runs them.
+
+Reference implementations by stage are indexed in `explainers/README.md`.
+
+## Rules
+
+- This repo is public. Never copy material from private projects that use the toolkit (course content, learner records) into it.
+- The examples in `explainers/` are frozen as reference implementations and regression fixtures: `explainer check` must pass on them.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,15 @@
 # Changelog
 
+## 0.8.0 — 2026-10-19
+
+Measure the toolkit, not the examples.
+
+- **`explainer eval artifact`**: 30 held-out subjects (10 dev, 20 test) with fixed quizzes. A fresh agent builds each explainer under each condition (no guidance, the principles, the explain skill at the quick or published tier); a second answers the quiz from the renderings alone, quoting the passage behind each answer (unquotable answers score 0); a third grades blind. [docs/evals.md](docs/evals.md#the-artifact-eval).
+- The examples are frozen as reference implementations and regression fixtures (ROADMAP §15).
+- [What the artifact eval found](docs/evals.md#what-the-artifact-eval-found), on the 10 dev subjects: the quiz is at its ceiling; the principles carry the measurable value (40% shorter, best mean effort rank, lowest cost); the quick tier makes the most errors; the published tier costs 13 times as much with no consistent measured gain. A person's ranking has to confirm the grader before it steers a change.
+- `AGENTS.md` for agents other than Claude Code: no `@` imports, no `.claude/` paths.
+- Docs: the author's eight understanding questions and the blind test's seven are described as two sets; "at least three dev subjects" everywhere.
+
 ## 0.7.0 — 2026-10-04
 
 Reviews that run themselves, and a record of which findings matter.

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -18,6 +18,7 @@ uv run explainer check                        # every example against its model.
 uv run explainer coldread <slug> --run        # first-viewing read by a fresh agent (claude -p), saved with numbered findings
 uv run explainer findings --stats             # how many review findings authors adopt, per tool
 uv run explainer eval chat                    # chat answers under each system prompt, graded blind (needs the claude CLI)
+uv run explainer eval artifact --split dev     # held-out subjects built under each condition, read and graded blind
 uv run explainer render <slug> --draft        # fast silent layout pass
 uv run explainer render <slug> --review       # 1080p60, voice, captions, chapters, review sheet
 uv run pytest -q                              # toolkit tests

diff --git a/ROADMAP.md b/ROADMAP.md
@@ -20,6 +20,7 @@ Status: `proposed` · `next` · `in progress` · `done`
 | 12 | [The narrative pass](#12-the-narrative-pass) | done (v0.5.0), with open items |
 | 13 | [Rebalance: omission and excess both cost](#13-rebalance-omission-and-excess-both-cost) | done (v0.6.0), with open items |
 | 14 | [Reviews that run themselves](#14-reviews-that-run-themselves) | done (v0.7.0), with open items |
+| 15 | [Measure the toolkit, not the examples](#15-measure-the-toolkit-not-the-examples) | in progress (eval shipped in v0.8.0) |
 
 ---
 
@@ -210,3 +211,34 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o
 - Measure adoption again under the tuned prompts; if edge findings stay under 10%, drop the per-unit edge lists and keep only blocking findings and cuts.
 - The cold read on a video re-reads its review sheet from scratch each round; a round could read only the changed captions.
 - Run review rounds on more than one model, as with the chat eval.
+
+## 15. Measure the toolkit, not the examples
+
+**Problem.** From v0.4 to v0.7, each round reviewed the same ten examples, fixed what the reviewers found, and often changed a principle or a review prompt in response. That fits the toolkit to those examples. The review prompts were tuned on adoption figures from the same examples they were then judged on; ste-80 took six cold-read rounds, which shows that a reviewer always finds something, not that the pipeline improved. Several rules in §10 and §11 trace back to one example's failure, and nothing showed that they help on a new subject. The only measure on new material was the chat eval: six questions, which the chat rule was also tuned against.
+
+**Built.**
+
+- The examples are frozen as reference implementations and regression fixtures: `explainer check` must pass; no more review rounds.
+- `evals/artifact/questions.yaml`: 30 held-out subjects (10 dev, 20 test), none an example or a chat-eval question, each with a fixed quiz of four questions; the last is a prediction about a case the explanation need not mention.
+- `explainer eval artifact`: per subject and condition, a fresh agent builds the explainer (none, the principles, or the explain skill at the quick or published tier), a second reads only the renderings and answers the quiz with a quote for each answer (an answer whose quote is not in the renderings scores 0), and a third grades blind. [docs/evals.md](docs/evals.md#the-artifact-eval).
+
+**Rules.**
+
+- Change the principles or a skill only for a failure seen in at least three dev subjects; confirm the change on test before a release; never change anything for one test subject.
+- Remove a rule when removing it costs nothing on dev.
+- Calibrate the grader against a person: the author rates a sample, and the grader's ranking must agree before its numbers steer a change.
+
+**Evidence** (dev baseline, [docs/evals.md](docs/evals.md#what-the-artifact-eval-found)).
+
+- The quiz is at its ceiling (94–100% in every condition, with a strong reader and with Haiku): a model reader cannot measure the difference. The same holds for the published tier's blind test.
+- The principles alone cut length 40% and rank first on effort in every grading pass, at the lowest cost. The quick tier is shortest but ranks below them and makes the most errors; the published tier costs 13× as much for no measured gain.
+- The first run was invalid: builders inside the repo loaded its CLAUDE.md in every condition. Builds now run outside every project.
+
+**Next.**
+
+- A person ranks a sample (principles, quick, published on three subjects); the grader's ranking is used only if it agrees.
+- If it agrees: decide what the skill keeps. Candidates: the model file and the reviews only for the published tier, or the reviews dropped where they do not change the result.
+- A quiz that discriminates: harder subjects, or a reader limited in another way (time, words read), since a weaker model was not enough.
+- `plugin/skills/explain/references/html.md` says "the seven understanding questions from principles.md §7", but §7 has eight. Fix it with the principles rewrite, because skill text is measured.
+- The blind test (`explainer_kit/review.py`) does not ask "Could anything be cut with no loss?" Decide whether it should when the objective becomes least *wasted* effort.
+
diff --git a/docs/cli.md b/docs/cli.md
@@ -22,6 +22,7 @@ With the plugin installed, run `explainer <command>` from any project. In the Ex
 | `explainer say "<text>" [--voice V] [--speed S]` | Synthesize one line and play it. Use it to choose a voice or test a lexicon spelling. |
 | `explainer frames <slug> [--draft] [--every N]` | Rebuild the contact sheet from an existing render. |
 | `explainer eval chat [--conditions name=source ...]` | Answer the questions in `evals/chat/questions.yaml` under several system prompts with fresh `claude -p` calls, grade them blind, and write a report. See [Evals](evals.md). |
+| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:<rev>`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). |
 
 ## `check`
 

diff --git a/docs/concepts.md b/docs/concepts.md
@@ -145,21 +145,26 @@ For learners, a result is worth more after a prediction. Pages put a predict gat
 
 ## The understanding test
 
-Before an explanation is done, it is checked against seven questions:
+There are two sets of questions, and they are not the same.
+
+**The author's checklist** ([principles.md](../plugin/skills/explain/references/principles.md) §7) has eight questions. The author asks them while writing:
 
 1. Can the reader identify the main objects?
-2. Can the reader explain how they relate?
+2. Can the reader explain how the objects relate?
 3. Can the reader see the central causal chain?
 4. Can the reader predict what changes when an important variable changes?
 5. Can the reader separate assumptions from observations?
-6. Can the reader rebuild the explanation without the exact wording?
-7. Does the representation impose mental simulation that another medium would remove?
+6. Can the reader rebuild the high-level explanation without the exact wording?
+7. Does the representation impose mental simulation that a different medium would remove?
+8. Could anything be cut with no loss?
 
 A "no" means revise, or escalate one stage.
 
+**The blind test** asks a fresh reader seven questions, defined in [`explainer_kit/review.py`](../explainer_kit/review.py) (`UNDERSTANDING`). They overlap with the checklist, but they are a different set. The reader names the main objects, states how they relate and the central causal chain, predicts the effect of the most important variable, separates an assumption from an observation, and rebuilds the explanation in at most five sentences. The seventh asks which part was hardest to follow and what would have made it easier. The blind test does not ask the cut question.
+
 ### Blind understanding test
 
-An author grading their own explanation is a weak test. `explainer quiz <slug> --rendering <r>` prints a prompt for a fresh agent that sees only one rendering. It answers the seven questions plus the model's quiz items, and its answers are scored against `explainer quiz <slug> --rubric`. A test must also be able to fail: in the first run, a deliberately broken copy of the softmax prose (it claimed a high temperature can change which token is first) scored 0 on the quiz item about that claim, while the real prose passed. Results live in `explainers/<slug>/review/understanding.md`.
+An author grading their own explanation is a weak test. `explainer quiz <slug> --rendering <r>` prints a prompt for a fresh agent that sees only one rendering. It answers the blind test's seven questions plus the model's quiz items, and its answers are scored against `explainer quiz <slug> --rubric`. A test must also be able to fail: in the first run, a deliberately broken copy of the softmax prose (it claimed a high temperature can change which token is first) scored 0 on the quiz item about that claim, while the real prose passed. Results live in `explainers/<slug>/review/understanding.md`.
 
 ## Where the rules live
 

diff --git a/docs/evals.md b/docs/evals.md
@@ -1,6 +1,50 @@
 # Evals
 
-The checks and reviews in Explainer test artifacts: one explainer at a time. The most common use is different: a question in chat, answered with the principles and no files. `explainer eval chat` measures that use.
+The checks and reviews in Explainer test one explainer at a time. They make that explainer better; they do not show that the toolkit helps on the next subject. Tuning the principles and the review prompts against the same ten examples fits them to those examples. Two evals measure the toolkit itself, on subjects it was never tuned on:
+
+- `explainer eval chat`: a question in chat, answered with the principles and no files (the most common use).
+- `explainer eval artifact`: an explainer built from scratch by a fresh agent, under each condition, then read and graded blind.
+
+## The artifact eval
+
+`evals/artifact/questions.yaml` holds 30 subjects that are neither examples nor chat-eval questions, from GPS to rent caps: 10 `dev`, 20 `test`. Each has the question, the misconception to avoid, and a fixed quiz of four questions with expected answers. The last is a prediction: a case the explanation need not mention, which the reader answers from the mechanism it gives. The quiz does not depend on what the builder chose to cover.
+
+```bash
+uv run explainer eval artifact --split dev                          # none, principles, quick, published
+uv run explainer eval artifact --only float-sum --conditions none quick
+uv run explainer eval artifact --conditions none old=git:v0.6.0 principles
+```
+
+For each subject and condition, three fresh `claude -p` agents run, with no settings files, no MCP servers, and no stdin:
+
+1. **Build.** The builder gets the question and the reader, and writes `explanation.md`, `diagram.md`, or `index.html` (no video) in an empty project folder. Conditions differ only in guidance: `none`; `principles` (principles.md appended to the system prompt; also `git:<rev>` or a file); `quick` and `published` (the plugin loaded and the explain skill at that tier, running this checkout's CLI).
+2. **Read.** The reader gets copies of the renderings only (no `model.md`) and answers the quiz. Each answer quotes the passage it rests on. An answer whose quote is not in the renderings scores 0, so a reader that answers from its own knowledge earns nothing. The reader also lists where it would stop or reread, and what it did not need.
+3. **Grade.** The grader sees each subject's explanations and answers under shuffled labels, scores each answer 0–2 against the expected one, and checks the text for the misconception and for factual errors.
+
+Every stage saves its output, and a rerun into the same folder skips what exists. The report (`evals/artifact/results/<date>-<split>/report.md`) gives, per condition: the grounded quiz score, the raw score, ungrounded answers, misconceptions taught and corrected, errors, stops and excess per explanation, words, the forms chosen, and the builder's cost and time.
+
+**How to use it.** Change the principles or a skill only for a failure that appears in at least three `dev` subjects. Then run `test` to confirm the change before a release. Do not change anything in response to one test subject, and remove a rule when removing it costs nothing on `dev`.
+
+## What the artifact eval found
+
+Dev baseline, 2026-10-04, v0.7.0 principles and skills; builder, reader, and grader on the `claude` CLI's default model. [Report](../evals/artifact/results/2026-10-04-dev/report.md); the same builds read by Haiku: [report](../evals/artifact/results/2026-10-04-dev-haiku-reader/report.md).
+
+| condition | effort rank (1–4) | quiz, strong reader | quiz, Haiku reader | words | errors | cost | minutes |
+|---|---|---|---|---|---|---|---|
+| none | 3.1–3.3 | 100% | 95% | 1,689 | 3–4 | $0.28 | 1.5 |
+| principles | 1.7–1.9 | 99% | 98% | 1,006 | 1–4 | $0.21 | 1.0 |
+| quick (skill) | 2.5–2.9 | 95% | 94% | 606 | 4–9 | $0.46 | 1.6 |
+| published (skill) | 2.2–2.5 | 95% | 95% | 1,016 | 1–2 | $2.81 | 11.8 |
+
+Effort rank: the grader's ranking of each subject's four explanations by the reader's effort, averaged over 10 subjects; the range covers three grading passes. Errors: factual errors the grader found, in all 10 explanations; the range covers the same three grading passes. Cost: the builder plus the reviews it ran.
+
+- **The quiz is at its ceiling.** Every condition scores 94–100%, with a strong reader and with Haiku. A model reader answers from any competent explanation, so the quiz cannot rank the conditions. The published tier's own blind test is the same kind of instrument, so it is weak for the same reason.
+- **The principles carry the measurable value.** Appended to the system prompt, they cut length by 40% (1,689 → 1,006 words), came first in the effort ranking in all three passes, and cost the least: a shorter explanation is cheaper to write.
+- **The skill adds cost without a consistent measured gain over the principles.** The quick tier is the shortest (606 words) but ranks below the principles and has the most errors, several from compression ("three spheres meet at a point"; they meet at two). The published tier costs 13 times as much as the principles and takes 12 times as long, for a lower rank and the same length; its errors (1–2) overlap the principles' (1–4), fewer in one pass and more in another.
+- **Almost everything is prose.** With no guidance, 5 of 10 builders made a page; with the principles, none did; the skill made 3 diagrams in 20 builds. These subjects may simply be prose-shaped; the eval does not test video.
+- **The ranking is a model's taste, and it is noisy.** Changing only what the grader saw (Markdown as written instead of stripped of inline code) moved the quick tier from 2.5 to 2.9. The principles-first and none-last results held in every pass; the order of quick and published did not. A person's ranking of a sample has to confirm it before it steers a change.
+
+Two faults in the first run, both fixed: builders ran inside this repo, so Claude Code loaded its CLAUDE.md (and the principles) in every condition, including `none`; and the grader read Markdown with inline code removed, so formulas looked like blanks. Builds now run in a temp folder outside every project, with a guard.
 
 ## The chat eval
 
@@ -19,7 +63,7 @@ Each answer comes from a fresh `claude -p` call in an empty folder, with no tool
 
 The report goes to `evals/chat/results/<date>/report.md`, with every answer and `grades.json`.
 
-## What it found
+## What the chat eval found
 
 Two runs on 2026-10-04 (answers and grades by the `claude` CLI's default model). Run a: the principles before the chat rule. Run b: after it. Each run is graded on its own, so compare conditions within a run.
 
@@ -39,7 +83,7 @@ The current principles now give the shortest answers of the three (292 words, ag
 
 Reports: [run a](../evals/chat/results/2026-10-04-a/report.md), [run b](../evals/chat/results/2026-10-04-b/report.md).
 
-## Limits
+## Limits of the chat eval
 
 - Six questions, one grader, one model. Treat a difference of a few tenths of a point as noise.
 - The grader is a model with its own taste. It sees the reader and the must points, not the principles.