diff --git a/AGENTS.md b/AGENTS.md index 63d9db5..d610247 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,6 +19,7 @@ uv run explainer coldread --run # first-viewing read by a fresh ag uv run explainer findings --stats # how many review findings authors adopt, per tool uv run explainer eval chat # chat answers under each system prompt, graded blind uv run explainer eval artifact --split dev # held-out subjects built under each condition, read and graded blind +uv run explainer eval rederive --builds # a fresh reader works out a result each explanation never states, graded blind uv run explainer render --draft # fast silent layout pass uv run pytest -q # toolkit tests ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index bad7aa0..37d8c92 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## Unreleased + +Rederivation: can a reader work out what an explanation never states? + +- **A rederivation target for every subject** in `evals/artifact/questions.yaml`: a question the explanation was never asked to answer, the expected steps (building blocks a good explanation gives, and derived steps), and the result. A fresh agent checked each one (`explainer eval targets`, `evals/artifact/rederive-checks.json`), and all 30 were frozen before the first run. +- **The rederivation stage** (`explainer eval rederive --builds `; `eval artifact` runs it too). A fresh reader works the target out from the renderings, naming the part of the explanation each step rests on; a grader marks each expected step supported, unsupported, missing, or stated, blind, in three passes. Stated steps are left out of the score. [docs/evals.md](docs/evals.md#rederivation). +- [What it found](docs/evals.md#what-the-dev-baseline-found) on the v0.8 dev builds: it is also near its ceiling (90–100% of steps supported). Under the rule fixed before the run, one pair separates, none > published, by about two steps of 38, from one subject where the published explanation lacks part of a building block (when the Sun's and Moon's tides add), a gap the quiz also found. Reported with it: one target that overclaims (float-sum), explanations that state the derived steps in 3 of 10 subjects, and a partial building block graded inconsistently. +- `explainer eval human` and `evals/artifact/human-check.md`: a person works out a target, graded with the same rubric. + ## 0.8.0 — 2026-10-19 Measure the toolkit, not the examples. diff --git a/CLAUDE.md b/CLAUDE.md index b20121c..bafaa18 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,7 @@ uv run explainer coldread --run # first-viewing read by a fresh ag uv run explainer findings --stats # how many review findings authors adopt, per tool uv run explainer eval chat # chat answers under each system prompt, graded blind (needs the claude CLI) uv run explainer eval artifact --split dev # held-out subjects built under each condition, read and graded blind +uv run explainer eval rederive --builds # a fresh reader works out a result each explanation never states, graded blind uv run explainer render --draft # fast silent layout pass uv run explainer render --review # 1080p60, voice, captions, chapters, review sheet uv run pytest -q # toolkit tests diff --git a/ROADMAP.md b/ROADMAP.md index df5b402..9fe58ca 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -221,6 +221,7 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o - The examples are frozen as reference implementations and regression fixtures: `explainer check` must pass; no more review rounds. - `evals/artifact/questions.yaml`: 30 held-out subjects (10 dev, 20 test), none an example or a chat-eval question, each with a fixed quiz of four questions; the last is a prediction about a case the explanation need not mention. - `explainer eval artifact`: per subject and condition, a fresh agent builds the explainer (none, the principles, or the explain skill at the quick or published tier), a second reads only the renderings and answers the quiz with a quote for each answer (an answer whose quote is not in the renderings scores 0), and a third grades blind. [docs/evals.md](docs/evals.md#the-artifact-eval). +- Rederivation (PRD-002): each subject also has a frozen, model-checked target, a result its explanation was never asked to state. A fresh reader works it out from the renderings, naming the part each step rests on; a grader marks each expected step blind, in three passes. `eval artifact` runs it; `eval rederive --builds ` runs it on saved builds. [docs/evals.md](docs/evals.md#rederivation). **Rules.** @@ -233,12 +234,14 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o - The quiz is at its ceiling (94–100% in every condition, with a strong reader and with Haiku): a model reader cannot measure the difference. The same holds for the published tier's blind test. - The principles alone cut length 40% and rank first on effort in every grading pass, at the lowest cost. The quick tier is shortest but ranks below them and makes the most errors; the published tier costs 13× as much for no measured gain. - The first run was invalid: builders inside the repo loaded its CLAUDE.md in every condition. Builds now run outside every project. +- Rederivation is also near its ceiling (90–100% of steps supported). One pair separates under the rule fixed before the run, none > published, by about two of 38 steps, from one subject (two-tides) where the published explanation lacks part of a building block (when the Sun's and Moon's tides add), a gap the quiz also found. Two target defects and three subjects whose explanations state the derived steps are reported with it ([docs/evals.md](docs/evals.md#what-the-dev-baseline-found)). **Next.** - A person ranks a sample (principles, quick, published on three subjects); the grader's ranking is used only if it agrees. - If it agrees: decide what the skill keeps. Candidates: the model file and the reviews only for the published tier, or the reviews dropped where they do not change the result. - A quiz that discriminates: harder subjects, or a reader limited in another way (time, words read), since a weaker model was not enough. +- Open: whether PRD-003 adopts rederivation as its measure. With a strong model reader it mostly re-detects missing building blocks. Options: a weaker or constrained reader, harder targets with more derived steps (and a checker that tests "not stated" against real explanations), or human readers (`evals/artifact/human-check.md`). Recommendation: get two or three human checks before adopting it. - `plugin/skills/explain/references/html.md` says "the seven understanding questions from principles.md §7", but §7 has eight. Fix it with the principles rewrite, because skill text is measured. - The blind test (`explainer_kit/review.py`) does not ask "Could anything be cut with no loss?" Decide whether it should when the objective becomes least *wasted* effort. diff --git a/docs/cli.md b/docs/cli.md index 9ad7f7e..ccb9884 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -22,7 +22,10 @@ With the plugin installed, run `explainer ` from any project. In the Ex | `explainer say "" [--voice V] [--speed S]` | Synthesize one line and play it. Use it to choose a voice or test a lexicon spelling. | | `explainer frames [--draft] [--every N]` | Rebuild the contact sheet from an existing render. | | `explainer eval chat [--conditions name=source ...]` | Answer the questions in `evals/chat/questions.yaml` under several system prompts with fresh `claude -p` calls, grade them blind, and write a report. See [Evals](evals.md). | -| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). | +| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report; then run the rederivation stage on the same builds. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). | +| `explainer eval rederive --builds [--only id ...] [--out folder]` | The rederivation stage alone, on the builds of an earlier artifact run (only read, never changed): in three passes, a fresh reader works out each subject's target question from the renderings and names the part each step rests on, and a grader marks each expected step blind. Writes `rederive.md` and `rederive-scores.json`. See [Evals](evals.md#rederivation). | +| `explainer eval targets [--only id ...]` | Check each rederivation target in `evals/artifact/questions.yaml` with a fresh agent, and record the verdicts in `evals/artifact/rederive-checks.json`. Only targets changed since their last check are checked. | +| `explainer eval human --builds --chain ` | Grade a person's working on a rederivation target with the same rubric as the model readers. See [`evals/artifact/human-check.md`](../evals/artifact/human-check.md). | ## `check` diff --git a/docs/evals.md b/docs/evals.md index 1836f83..193fb11 100644 --- a/docs/evals.md +++ b/docs/evals.md @@ -46,6 +46,55 @@ Effort rank: the grader's ranking of each subject's four explanations by the rea Two faults in the first run, both fixed: builders ran inside this repo, so Claude Code loaded its CLAUDE.md (and the principles) in every condition, including `none`; and the grader read Markdown with inline code removed, so formulas looked like blanks. Builds now run in a temp folder outside every project, with a guard. +## Rederivation + +The quiz checks what an explanation says. A reader who has grokked a subject can do more: from its building blocks, work out a result nobody told them. An explanation of entropy as "the average length of the best code for a source" never mentions KL divergence, yet a reader who holds that idea can derive it: messages from p sent with a code built for q cost −log q each instead of −log p, so the average extra cost is Σ p log(p/q), never negative, because no code beats the one built for the true source. The rederivation stage tests that, on the same held-out subjects. + +**Targets.** Each subject in `evals/artifact/questions.yaml` has a `rederive` target: a question the explanation was never asked to answer, 3–6 expected steps, and the result. Each step is a building block (an idea a good explanation gives) or a derived step (worked out from the steps before it). A fresh agent checked every target (`explainer eval targets`, verdicts in `evals/artifact/rederive-checks.json`) against five tests: every step and the result are true; every building block comes from the subject's question, the quiz's expected answers, or background this reader already has; every derived step follows from earlier steps plus that background and brings in no new fact about the subject; each step is one any valid derivation needs; and a good answer to the subject's question would not state the result. The second and third tests matter most: a target that needs an idea a focused explanation may leave out would reward length, not understanding. All 30 passed (two after a revision) and were frozen in one commit before the first run; a test fails if a results folder's targets differ from the file. + +**Reader and grader.** The reader gets the renderings and the target's question, never the steps or the result. It works the question out step by step, and for each step names the part of the explanation it rests on (a heading, paragraph, figure, or a control on a page) and says what that part gives it. Reader and grader get the same rule: background this reader already has may be used without a citation; every fact about the subject must come from a part the reader names. The grader sees each subject's explanations and the readers' working under shuffled labels, and marks each expected step, credited in any order, merged, split, or by an equivalent route: + +- **supported:** made, and the cited parts carry the ideas it rests on (or it uses only background on top of earlier steps); +- **unsupported:** made from the reader's own knowledge, or from a part that does not carry the idea; +- **missing;** +- **stated:** a derived step the explanation states outright, so the reader looked it up. A stated step, or a stated result, is left out of the score and counted, so a long explanation is neither rewarded nor penalized for spelling out what the reader was meant to derive. + +The headline is the **supported** share of expected steps; **raw** is the share made at all; **result right** the share of results that match. A run makes three **passes**: each is a fresh reader run over every build, graded once under its own label shuffle, so the spread covers both the reader's and the grader's variation. **The separation rule**, fixed before the first run: condition A beats condition B when A's supported score is higher than B's in all three passes. + +```bash +uv run explainer eval rederive --builds evals/artifact/results/2026-10-04-dev # the stage alone, on saved builds +uv run explainer eval artifact --split dev # builds, quiz, then this stage +``` + +### What the dev baseline found + +Dev baseline, 2026-10-20: the 40 v0.8 builds (10 subjects × 4 conditions), read and graded three times by the `claude` CLI's default model; no new builds. [Report](../evals/artifact/results/2026-10-20-dev-rederive/rederive.md). + +| condition | supported (3 passes) | result right | stated steps | words | quiz (grounded) | +|---|---|---|---|---|---| +| none | 100%, 100%, 100% | 100%, 100%, 100% | 2, 2, 3 | 1,689 | 100% | +| principles | 95%, 100%, 97% | 89%, 89%, 100% | 4, 4, 5 | 1,006 | 99% | +| quick (skill) | 92%, 100%, 90% | 100%, 100%, 89% | 2, 2, 2 | 606 | 95% | +| published (skill) | 95%, 95%, 95% | 89%, 78%, 89% | 3, 3, 3 | 1,016 | 95% | + +**Under the separation rule, one pair separates: none > published** (100% against 95% in every pass). No other pair does. + +**What that result is made of.** About 38 expected steps count per condition, so 95% is two steps short. The gap comes from one subject, two-tides, and from half of one building block. Block 2 bundles two ideas: the Sun's tide is about half the Moon's, and at new and full Moon the two line up and add. The published explanation gives the ratio but never says when the tides add (no spring or neap tides), so the grader marked the block unsupported in all three passes. That is the gap the quiz already found (published scored 6 of 8 on two-tides, 0 on the spring-tide question). The quick explanation also gives the ratio, and only mentions in a "not covered" line that the Sun's bulges add to or partly cancel the Moon's; the grader marked its block unsupported, supported, and missing in the three passes, so it does not separate. + +**How much a single subject moves between passes.** The same quick two-tides build scored 3/4, 4/4, and 1/4 supported steps in the three passes. A difference of a step or two in one subject is within what one fresh reader and one grading change on their own. + +**Known defects, found after the run** (the targets are frozen, so they stay and are reported): + +- **The float-sum target overclaims.** Its last derived step argues that adding 0.1 ten times cannot land exactly on 1.0 because each addition rounds. Rounding errors can cancel (`0.1` added five times does give exactly `0.5`), so the honest answer from the building blocks is "not guaranteed", which is what the principles and published readers said. They were marked wrong in passes 1 and 2; in pass 3 the published reader gave the same hedged answer and was marked right, so the grader was not consistent either. This lowers principles' and published's "result right" (published's also falls on two-tides in passes 2 and 3) and cost each one supported step in pass 1. The separation does not depend on it: two-tides alone carries none > published. +- **Real explanations stated derived steps outright in 3 of 10 subjects,** more often than the target checker predicted: the gps-relativity clock argument (three conditions), the greenhouse no-lapse-rate case (both derived steps, in principles, quick, and published), and the two-tides right-angle step (none and principles). Those steps are left out of the score, so for greenhouse three conditions were scored on building blocks alone. +- **Compound building blocks are graded inconsistently when half present.** The quorum block "any two majorities share a server, so the next group sees every committed write" is half present in the principles and quick explanations (they give the overlap, for conflicting writes, not for reads); the grader marked it unsupported in two of three passes. The two-tides block above is the same kind, and it carries the only separation. + +**Does the order follow length?** Roughly. On the mean supported share the order is none (100%) > principles (97%) > published (95%) > quick (94%); by words it is none (1,689) > published (1,016) ≈ principles (1,006) > quick (606). The one separation goes in length's direction. The word count also leaves out what the grader reads in pages' scripts, and only none made pages (5 of 10). + +**What we take from it (interpretation, not the rule).** With a strong model reader, rederivation mostly re-detects missing building blocks, which the quiz already catches; it does not yet show a difference in understanding between conditions. The one separation is small, comes from half of one building block in one subject, and favors the longest explanation. Whether PRD-003 adopts this stage, and what to change first (a weaker or constrained reader, harder targets with more derived steps, or human readers), is open: see ROADMAP §15. + +**Human check:** none yet. A brief for testers is in [`evals/artifact/human-check.md`](../evals/artifact/human-check.md); results will be recorded here. + ## The chat eval `evals/chat/questions.yaml` holds six questions about phenomena (why ice floats, why TCP starts slowly, how a wing makes lift, why a hash table is fast, why we see one side of the Moon, why salt melts ice). For each one it lists: diff --git a/docs/prds/PRD-002-rederivation-eval.md b/docs/prds/PRD-002-rederivation-eval.md new file mode 100644 index 0000000..f55894b --- /dev/null +++ b/docs/prds/PRD-002-rederivation-eval.md @@ -0,0 +1,183 @@ +# PRD-002: A rederivation eval + +Status: approved, 2026-10-20 · Part 2 of 3 (PRD-001: ship v0.8; PRD-003: Olah-compass principles) + +## Why + +The v0.8 quiz is at its ceiling. Every condition scores 94–100%, because a model reader answers from any competent explanation (`docs/evals.md`). So the quiz cannot tell PRD-003 whether a change to the principles helps. + +An explanation here aims for the reader to grok the subject: once they hold its building blocks, they can work out results nobody told them. Rederivation tests exactly that. + +An example from outside the 30 subjects: an explanation of entropy as "the average length of the best code for a source" never mentions KL divergence, but a reader who holds that idea can derive it. + +- Messages from p sent with a code built for q cost −log q each instead of −log p. +- So the average extra cost is Σ p log(p/q). +- Its properties follow without being taught: + - it is never negative, because no code beats the one built for the real source; + - it is zero only when p = q; + - it is not symmetric; + - it is infinite when q gives zero probability to something p produces. + +Each quiz already ends with a one-step "predict" item. This PRD adds the several-step version and grades the chain of reasoning, not only the answer. + +The reader is a model that already knows most results, so a right answer alone proves nothing. The fix: every step must rest on a part of the explanation that the reader names, and the grader checks that the part carries the idea. + +## Goals + +### 1. A rederivation target for every subject, frozen before any run + +Each of the 30 subjects in `evals/artifact/questions.yaml` gets a target with three parts: + +- **the question** the reader works out; +- **the expected steps:** 3–6 of them, one sentence each. Each step is an idea that any valid derivation needs, not the bookkeeping of one route. Each is marked as one of: + - a **building block**, which a good explanation gives; + - a **derived step**, which the reader works out from earlier steps. +- **the result:** the expected conclusion, specific enough to be right or wrong. + +Each target passes three tests: + +1. **Allowed building blocks only.** Every step uses one of: + - an idea from the subject's question; + - an idea from the quiz's expected answers; + - background at the reader's level. That level is set by the `reader` line in `questions.yaml`: school mathematics and science, and basic programming. + + A target that needs an idea a focused explanation can leave out rewards length, not understanding. The conditions average from 606 to 1,689 words, so that bias would be large. PRD-003 would then tune the principles toward length, the opposite of their aim. +2. **Several steps:** at least three. +3. **Not stated:** a good answer to the subject's question would not state the result. + +Claude drafts the targets. A fresh `claude -p` checker sees the subject's question, its quiz with the expected answers, and the target, then applies the three tests and returns pass or fail with a reason. + +- A failed target is rewritten and checked again, at most twice. +- A subject whose target still fails gets no target, and the report lists it. +- The checker's verdicts are saved with the targets. + +All 30 targets, test subjects included, are committed in one commit before the first rederivation run. Writing them now means the PRD-003 changes cannot shape them. The owner may skim them; nothing waits on that. + +### 2. A rederive stage in the artifact eval + +**Reader.** The reader gets the renderings only, as today, and the target's question. It does not see the expected steps or the result. It works the question out step by step. For each step it gives: + +- the idea the step uses; +- the part of the explanation that idea comes from: a heading, a paragraph, a figure or diagram, or a control on a page; +- that idea in its own words. + +If the explanation lacks an idea the reader needs, it says so and stops there. A gap is evidence. + +**Outside-knowledge rule.** The same text goes in the reader prompt and the grader prompt: + +- background at the reader's level may be used without citing anything; +- any fact about the subject must come from a part of the explanation that the reader names. + +**Grader.** The grader works blind, under shuffled labels, as today. For each explanation it sees the text, the reader's chain, and the target's expected steps and result. The expected steps are a checklist of ideas, not a script. A step counts when the reader's chain establishes the same idea in any order, whether merged with another step, split in two, or reached by an equivalent route. The grader marks each expected step as one of: + +- **supported:** the reader made the step, and either the cited parts carry the ideas it rests on, or it uses only reader-level background on top of earlier steps; +- **unsupported:** the reader made the step from its own knowledge, or from a part that does not carry the idea; +- **missing;** +- **stated:** a derived step that the explanation states outright, so the reader looked it up instead of working it out. + +The grader also marks whether the result is right, and whether the explanation states the result outright. + +**Scores.** Each condition gets three scores: + +- **supported:** the share of expected steps that are supported; +- **raw:** the share made at all; +- **result right.** + +These mirror the quiz's grounded and raw scores. + +A stated derived step, or a stated result, is left out of that explanation's score: out of both the count and the total. The report counts these per condition. That way a long explanation is neither rewarded nor penalized for spelling out what the reader was meant to derive. + +**Passes.** Every run of the stage makes three passes. In each pass, a fresh reader works through every build, and the grader grades it once under its own label shuffle. The spread across passes then covers both the reader's variation and the grader's. + +The stage runs in every `eval artifact` run from now on, because PRD-003 needs it on new builds. It can also run alone on the saved builds of an earlier run. + +### 3. A dev baseline on the saved v0.8 builds + +- **No new builds.** The results go in a new folder that reuses the 40 builds saved in `evals/artifact/results/2026-10-04-dev/` (10 subjects × 4 conditions). +- **Models.** The reader and the grader use the `claude` CLI's default model, as the v0.8 baseline did. +- **Three passes,** as defined in Goal 2, each reported. The v0.8 ranges came from three different gradings, one of them the faulty first run, so they are no model to copy. +- **The report** gives, per condition, the three scores (with their range across passes), next to words and the v0.8 quiz score, plus a table per subject. + +### 4. The result is written up, whatever it is + +`docs/evals.md` gets a "Rederivation" section. It says what the eval measures and gives the three target tests. Then it reports what the dev baseline found, judged by the rule below. + +**The separation rule, fixed before the first run.** Condition A beats condition B when A's supported score is higher than B's in all three passes. + +The section says: + +- which pairs of conditions separate; +- whether the order follows word count, which would mean the score measures length; +- if no pair separates, that the eval does not yet rank the conditions, and that PRD-003 should not adopt it as its measure without changes. + +The "Next" list in ROADMAP §15 and the "Unreleased" section of `CHANGELOG.md` record the stage and its result. + +### 5. A human check that does not block + +`evals/artifact/human-check.md` is a one-page brief for a person who reads one explanation and works out its target on paper, before seeing the expected steps. Their chain is graded with the same rubric. `docs/evals.md` then records whether the person's score and the model reader's score on that explanation agree. + +PRD-002 is done without any human results; they are added when they come in. + +## Non-goals + +- **No change to the principles, the skills, or other measured text.** That is PRD-003. +- **No new builds and no run on the test split.** A test run needs new builds for all 20 test subjects. It runs when PRD-003 confirms a change. +- **No changed-world variants and no reader run without an explanation.** They were cut to keep this simple. +- **No change to the quiz, its scoring, or the v0.8 reports.** + +## Anti-goals + +- Changing a target, its steps, or the separation rule after seeing any rederivation score. +- A target that rewards length because it needs an idea outside the allowed building blocks. +- Claiming a separation the rule does not support, or presenting a 10-subject dev run as more than that. + +## Constraints + +- **The repo is public (MIT).** Targets and docs contain no personal or private-project context. +- **The shipped v0.8 record stays as it is.** `evals/artifact/results/2026-10-04-dev/` and `2026-10-04-dev-haiku-reader/` stay byte-identical: `git diff --exit-code` on them is clean. +- **The builder never sees a target.** Build prompts are unchanged. +- **The reader never sees** the expected steps or the result. +- **Grading stays blind** under shuffled labels. +- **Every stage saves its output,** and a rerun skips what exists, as today. +- **Tests run offline,** without the `claude` CLI, like the existing artifact eval tests. + +## Decided + +All decided by the owner on 2026-10-19. + +- **Rederivation is the test of grokking.** Once the reader has the building blocks, they can rederive the results and the rest of the theory from first principles. +- **A citation names a part of the explanation and restates its idea.** Word-for-word quotes are too strict, and an idea carried by a figure cannot be quoted. +- **The owner's time is not required.** Claude drafts the targets, a model checks them, and the owner may skim. +- **Simpler beats complete.** No changed-world variants and no control run without an explanation. +- **Human calibration happens when a tester is free.** It blocks nothing. +- **Targets are frozen before the first run.** + +## Calls made by Claude (check these) + +Approving this PRD approves these calls. Nothing else waits on the owner. + +- **The allowed-building-blocks test:** the question, the quiz's expected answers, and reader-level background. This keeps the score from rewarding length. +- **Targets for all 30 subjects now;** only dev runs in this PRD. +- **The separation rule:** higher in all three passes. +- **A pass is a fresh reader run, graded once under its own shuffle.** This triples the reader calls in every run. Reads are cheap next to builds, and only this design tests whether a difference survives the reader's own variation. +- **Credit for other routes.** Expected steps are a checklist of ideas, credited in any order or form. Each step must be one that any valid derivation needs. +- **Stated derived steps and stated results are left out of the score and counted.** They are neither rewarded nor penalized. +- **Scores:** supported, raw and result-right, mirroring the quiz. +- **The rederive stage joins every artifact eval run** from now on. +- **No release is decided here.** The owner decides at the end whether PRD-002 ships as v0.9.0 or waits for PRD-003. + +## Left to the builder + +- Field names in `questions.yaml`, and the prompt wording beyond the outside-knowledge rule. +- The CLI form for running the stage on an earlier run's builds. +- How the new folder reuses the builds, by copy or by reference, provided the v0.8 folders stay unchanged. +- Whether the grader sees a subject's four explanations in one call, as today, or one at a time, provided the labels stay shuffled. + +## Done when + +- Each of the 30 subjects has a target whose checker verdict is pass, or is listed as having none. The targets were committed before any rederivation result. +- The dev baseline is in a new results folder with three passes. Its report shows the scores next to word count and the quiz. +- `docs/evals.md` states the result under the separation rule, including "no separation" if that is the outcome. +- The v0.8 results folders are unchanged. +- `evals/artifact/human-check.md` exists. +- Tests pass offline, `explainer check` passes, and CI is green. diff --git a/docs/superpowers/plans/2026-10-20-rederivation-eval.md b/docs/superpowers/plans/2026-10-20-rederivation-eval.md new file mode 100644 index 0000000..8b983b2 --- /dev/null +++ b/docs/superpowers/plans/2026-10-20-rederivation-eval.md @@ -0,0 +1,440 @@ +# Rederivation Eval Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a rederivation stage to the artifact eval, give all 30 subjects a frozen, model-checked target, and report a three-pass dev baseline on the saved v0.8 builds, whatever it shows. + +**Architecture:** +- One new module, `explainer_kit/rederive.py`, holds the targets (validation, hashing, checker), the reader, the grader, scoring, the run, the report, and the grading of a human chain. +- `artifact_eval.py` changes only where the reader's file copies are made: that code is extracted into `copy_for_reader` so both stages share it. +- The CLI gains three `eval` suites (`targets`, `rederive`, `human`), and `eval artifact` runs the rederive stage after it grades. +- The stage reads builds from a `--builds` folder and writes only to `--out`. That is how the baseline reuses the v0.8 builds without touching them. +- Work happens on branch `prd-002-rederivation` from `main`. + +**Tech Stack:** Python 3 / uv, pytest, PyYAML, the `claude` CLI through `explainer_kit.agent.ask`. + +**Spec:** `docs/prds/PRD-002-rederivation-eval.md` + +## Global Constraints + +- **Measured text does not change:** `git diff --stat main -- plugin/skills explainers` is empty at every commit. +- **The builder never sees a target.** In `artifact_eval.py`, `TASK`, `SKILL_TIER` and `build()` are unchanged: `git diff main -- explainer_kit/artifact_eval.py` touches only `read()` and the new `copy_for_reader`. +- **The v0.8 record stays byte-identical:** `git status --porcelain -- evals/artifact/results/2026-10-04-dev evals/artifact/results/2026-10-04-dev-haiku-reader` is empty at every commit. +- **Targets are frozen at the Task 2 commit.** Afterwards, `rederive` blocks in `evals/artifact/questions.yaml` never change. A test (Task 5) fails if a results folder's snapshot differs from the file. +- **The repo is public.** No committed file holds an absolute path, a username, an email address, a key, or private-project context. +- **The reader prompt and the grader prompt contain the same constant, `OUTSIDE_RULE`:** + > Background that this reader already has (see the reader above) may be used without a citation. Every fact about the subject must come from a part of the explanation that the reader names. +- **The separation rule, verbatim:** "Condition A beats condition B when A's supported score is higher than B's in all three passes." +- **A pass** is a fresh reader run over every build, then one grading per subject under its own label shuffle (`random.Random(k)` for pass `k`). +- **Tests run offline.** They stub `agent.ask` and `agent.subprocess.run` and never call the `claude` CLI. +- **The baseline uses the `claude` CLI's default model,** so no `--model` and no `--reader-model`. + +## Review Focus + +1. **A build with no renderings** (a failed build). Expect the reader to be skipped and every step marked missing. The run must not crash, and the report shows it. The test is in Task 3, `test_read_chain_skips_a_build_with_no_renderings`. +2. **A rerun after the targets changed.** Expect the run to refuse, naming the changed subjects, so no folder mixes two target versions. The test is in Task 5, `test_rerun_refuses_changed_targets`. +3. **Absolute paths leaking into saved output** (public repo). Expect the source folder saved relative to `--out`, and no prompt or temp path saved. The test is in Task 5, `test_outputs_hold_no_absolute_paths`. +4. **Grader replies that don't fit:** too few or too many marks, an unknown word, "stated" on a building block, or a missing label. Expect normalisation, not a crash or a skewed score. The test is in Task 4, `test_marks_are_normalised`. +5. **A page whose idea lives in a control.** Expect the grader to see the page's script as well as its visible text, so a reader citing a slider can be checked. The test is in Task 4, `test_grader_text_includes_page_scripts`. + +--- + +### Task 0: Branch + +- [ ] **Step 1:** Create the branch. In the PRD, change `Status: draft, 2026-10-19` to `Status: approved, 2026-10-20`; committing it before any run is what pre-registers the separation rule. Commit the spec and the plan. + +```bash +git switch -c prd-002-rederivation +git add docs/prds/PRD-002-rederivation-eval.md docs/superpowers/plans/2026-10-20-rederivation-eval.md +git commit -m "PRD-002: a rederivation eval, and its plan" +``` + +### Task 1: Targets — format, validation, and the checker + +**Files:** +- Create: `explainer_kit/rederive.py` +- Modify: `explainer_kit/cli.py`. Add `targets` to the `eval` suite choices; `cmd_eval` dispatches it. +- Create: `tests/test_rederive.py` + +**Interfaces:** +- Produces: + - `checks_file() -> Path`, which returns `project_root()/"evals"/"artifact"/"rederive-checks.json"`. + - `target_problems(t: dict) -> list[str]`. + - `target_hash(t: dict) -> str`: the first 12 hex digits of the sha256 of `json.dumps(t, sort_keys=True, ensure_ascii=False)`. + - `check_prompt(subject: dict, reader: str) -> str`. + - `check_targets(only: list[str] | None = None, model: str | None = None) -> Path`. +- The target format in `questions.yaml`, under each subject: + + ```yaml + rederive: + q: "…" + steps: + - {kind: block, text: "…"} + - {kind: derived, text: "…"} + result: "…" + ``` + +- The checks file maps `{subject_id: {"verdict": "pass"|"fail", "tests": {...}, "reason": str, "hash": str, "attempt": int}}`. + +- [ ] **Step 1: Write the failing tests** in `tests/test_rederive.py`. + - `test_target_problems`: + - a valid target (3 steps, at least one `derived`, non-empty `q` and `result`) gives `[]`; + - each of these gives a non-empty list: 2 steps; 7 steps; no `derived` step; kind `"lemma"`; empty `result`; missing `q`. + - `test_target_hash_is_stable_and_sensitive`: the same dict in a different key order hashes the same, and changing one step's text changes the hash. + - `test_check_prompt_shows_the_building_blocks`. For the first dev subject, given a target, the prompt contains: + - every quiz `expect`, the reader line, every step text with its kind, and the result; + - the phrases `"any valid derivation needs"`, `"would not state"`, and `"no new fact about the subject"`. + - `test_check_targets_records_and_resumes`. Monkeypatch `rederive.checks_file` to a tmp file and `ae.questions_file` to a tmp YAML with two subjects that have targets, and a stub `agent.ask` that returns: + + ```json + {"tests": {"correct": true, "blocks": true, "derived_follow": true, "steps": true, "not_stated": true}, "reason": "ok"} + ``` + + for one subject, and `"derived_follow": false` for the other. Assert: + - verdicts `pass` and `fail` (the verdict is computed in code as all five tests true, never read from the reply); + - each `hash` equals `target_hash`, and `attempt == 1`; + - a second call makes **no** `agent.ask` call; + - after editing the failing target and calling again, that subject has `attempt == 2` and only it was asked. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: FAIL. The module is not found. +- [ ] **Step 3: Implement the four functions in `explainer_kit/rederive.py`.** + - Module docstring style matches `artifact_eval.py`. + - `CHECKER_SYSTEM = "You are a careful scientist and teacher. You check test questions. Reply with JSON only."` + - The checker prompt asks for five booleans: + - `correct`: the steps and the result are true; + - `blocks`: every `block` step is an idea from the subject's question, from the quiz's expected answers, or background this reader already has; + - `derived_follow`: every `derived` step follows from the steps before it plus background this reader already has, and introduces no new fact about the subject (otherwise only explanations that happen to contain that fact could support it, and length wins); + - `steps`: there are at least three steps, and each is an idea any valid derivation needs, not one route's bookkeeping; + - `not_stated`: a good answer to the subject's question would not state the result. + + It also asks for a one-sentence `reason` that names any failing step. + - `check_targets` walks `ae.load("all", only)[1]`. It skips subjects without `rederive` and subjects whose saved hash matches. It sets `attempt` to the saved attempt plus 1 when the hash changed. It writes the file sorted by id, with a trailing newline. + - `cmd_eval` with `suite == "targets"` calls `check_targets(args.only, args.model)` and prints a count of pass and fail. +- [ ] **Step 4: Run the tests to verify they pass.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: 4 passed. +- [ ] **Step 5: Commit.** + `git add explainer_kit/rederive.py explainer_kit/cli.py tests/test_rederive.py && git commit -m "rederive: target format, validation, and the checker"` + +### Task 2: Draft, check and freeze all 30 targets + +This task is content work plus live checker calls (`claude -p`), about 30–60 calls. Nothing from Tasks 3–5 may exist in the working tree yet. The targets are written before the reader or grader prompts exist, and stay unchanged afterwards. + +**Files:** +- Modify: `evals/artifact/questions.yaml`. Add a `rederive` block to every subject, and add a header paragraph explaining the block and that the builder and reader never see the steps or the result. +- Create: `evals/artifact/rederive-checks.json` (generated). +- Modify: `tests/test_rederive.py` (one test). + +**How to draft a target.** The worked example below is from outside the 30 subjects and is not added to the file: + +```yaml +rederive: + q: "A code was built for the letter frequencies q of one language and is used for text from another, with frequencies p. How many extra bits per letter does it cost on average, and can that cost ever be negative?" + steps: + - {kind: block, text: "In the best code, a symbol of probability p gets a codeword of about -log2 p bits."} + - {kind: block, text: "Entropy is the average codeword length of the best code: the sum of -p log2 p."} + - {kind: derived, text: "With q's code, each letter costs -log2 q instead of -log2 p, so the average extra cost is the sum of p log2(p/q)."} + - {kind: derived, text: "No code beats the one built for the true source, so the extra cost is never negative, and it is zero only when p = q."} + result: "The sum of p log2(p/q) bits per letter (the KL divergence); never negative, zero only when p = q." +``` + +- A `block` step restates an idea from the subject's `q`, from a quiz `expect`, or from background at the reader's level. If you cannot point to one, the step is not a block, and the target likely rewards length: redraft it. +- Derived steps follow from the steps before them plus reader-level background, and bring in no new fact about the subject. The result follows from those steps. +- The question names a concrete case, quantity, sign, or limit that is not one of the quiz questions. + +- [ ] **Step 1: Write the failing test** `test_every_subject_has_a_checked_target`. For each subject in `ae.load("all")`, with `checks = json.loads(rederive.checks_file().read_text())`: + - if the subject has a `rederive` block: `target_problems(...) == []`, `checks[id]["verdict"] == "pass"`, and `checks[id]["hash"] == target_hash(...)`; + - otherwise: `checks[id]["verdict"] == "fail"` and `checks[id]["attempt"] == 3`. This is the "listed as having none" case. +- [ ] **Step 2: Run it to verify it fails.** + Run: `uv run pytest tests/test_rederive.py::test_every_subject_has_a_checked_target -q` + Expected: FAIL, because the checks file is missing. +- [ ] **Step 3: Draft the 30 `rederive` blocks** in `questions.yaml`, using the guidance above, and run the checker. + Run: `uv run explainer eval targets` + Expected: one verdict per subject in `evals/artifact/rederive-checks.json`. +- [ ] **Step 4: Rewrite failures, at most twice.** + - For each `fail`, rewrite the target using the checker's `reason`, then rerun the checker. The attempt count reaches 2, then 3. + - If a subject still fails at attempt 3, delete its `rederive` block and keep its `fail` entry. + - Ledger each deletion as a ruling. +- [ ] **Step 5: Run the test to verify it passes.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: all pass. +- [ ] **Step 6: Commit. This is the freeze commit.** + - `git add evals/artifact/questions.yaml evals/artifact/rederive-checks.json tests/test_rederive.py && git commit -m "Freeze the rederivation targets: 30 subjects, model-checked"` + - Ledger the commit hash as `FREEZE`. + +### Task 3: The reader + +**Files:** +- Modify: `explainer_kit/artifact_eval.py`. Extract `copy_for_reader(files: list[Path], shown: Path) -> list[Path]` from `read()`. It empties `shown`, then copies each file into it and returns the copies. `read()` calls it, and its behaviour is unchanged. +- Modify: `explainer_kit/rederive.py` +- Modify: `tests/test_rederive.py` + +**Interfaces:** +- Consumes: `ae.renderings(ws, sid)` and `ae.copy_for_reader`. +- Produces: + - `OUTSIDE_RULE: str`, the exact text in Global Constraints; + - `reader_prompt(files: list[Path], reader: str, question: str) -> str`; + - `chain_reply(text: str) -> dict`, which returns `{"steps": [{"step": str, "part": str, "idea": str}], "result": str, "gap": str}`; + - `chain_text(chain: dict) -> str`; + - `read_chain(subject: dict, cond: str, reader: str, builds: Path, out: Path, k: int, model: str | None) -> dict`, which saves to `out/"rederive"/f"pass-{k}"/"read"//f"{cond}.json"`. + +- [ ] **Step 1: Write the failing tests.** + - `test_reader_prompt_is_blind_to_steps_and_result`: + - it contains the target `q`, `OUTSIDE_RULE`, every file path, `"gap"` and `"part"`; + - it contains no step text and no result text from a real dev target. + - `test_chain_reply_tolerates_shapes`: + - valid JSON round-trips; + - `{"steps": "x"}` gives `steps == []`; + - `"no json"` gives `{"steps": [], "result": "", "gap": ""}`; + - step items that are not dicts are dropped. + - `test_chain_text_numbers_steps`: two steps give lines starting `1.` and `2.`, each with its part and idea, then a `Result:` line. + - `test_read_chain_uses_copies_and_leaves_builds_untouched`. Use a tmp builds folder with `work//a/explanation.md` and a stub `agent.ask` that captures `read=` and returns a chain. Assert: + - every `read` folder lies outside `builds`; + - the saved file exists at the path above; + - a second call makes no `agent.ask` call; + - the sha256 of every file under `builds` is unchanged. + - `test_read_chain_skips_a_build_with_no_renderings`: there is no `agent.ask` call, and the result has `"missing": True` and `steps == []`. + - The existing `tests/test_artifact_eval.py` must still pass. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: the new tests FAIL with `AttributeError`. +- [ ] **Step 3: Implement.** + - The reader prompt states, in order: + - the reader line; + - "Read ONLY these files" and the file list; + - the `.html` rule from `ae.reader_prompt`; + - the question; + - "work it out step by step from the explanation"; + - per step: what it establishes (`step`), the part it rests on (`part`: a heading, paragraph, figure or diagram, a page control, an earlier step number, or `background`), and that part's idea in its own words (`idea`); + - `OUTSIDE_RULE`; + - "if the explanation lacks an idea you need, say which in `gap` and stop there"; + - the JSON shape. + - Reuse `ae.READER_SYSTEM`. + - Make copies in `tempfile.mkdtemp()` and remove them afterwards. + - Retry the call once if `steps` comes back empty and `gap` is empty, as `ae.read` does. +- [ ] **Step 4: Run the tests to verify they pass.** + Run: `uv run pytest tests/test_rederive.py tests/test_artifact_eval.py -q` + Expected: all pass. +- [ ] **Step 5: Commit.** + `git add -u && git commit -m "rederive: the reader works the target out and cites the parts it rests on"` + +### Task 4: The grader and the scores + +**Files:** modify `explainer_kit/rederive.py` and `tests/test_rederive.py`. + +**Interfaces:** +- Consumes: `chain_text`, `OUTSIDE_RULE`, `ae.shown_text`, `ae.renderings`, and `ae.GRADER_SYSTEM`. +- Produces: + - `MARKS = ("supported", "unsupported", "missing", "stated")`; + - `grader_text(path: Path) -> str`; + - `grade_prompt(subject: dict, reader: str, entries: dict[str, dict]) -> str`, where `entries` maps a label to `{"text": str, "chain": str}`; + - `marks(raw: object, kinds: list[str]) -> list[str]`; + - `tally(marks: list[str], kinds: list[str], right: bool, stated: bool) -> dict[str, int]`, with keys `supported, made, counted, stated, right, results, results_stated`; + - `pass_scores(tallies: dict[str, list[dict]]) -> dict[str, dict]`. It maps a condition to `{"supported": float, "raw": float, "result": float | None}`, pooled over subjects; + - `separations(passes: list[dict[str, dict]]) -> list[tuple[str, str]]`; + - `grade_subject(subject: dict, key: dict[str, str], reader: str, builds: Path, out: Path, k: int, model: str | None) -> dict`. It saves `out/"rederive"/f"pass-{k}"/"grades"/f"{id}.json"` as `{"key": key, "grades": {cond: {"marks": [...], "result_right": bool, "result_stated": bool}}}`. + +- [ ] **Step 1: Write the failing tests.** + - `test_grader_text_includes_page_scripts`. `

Drag the slider

` gives text containing both `"Drag the slider"` and `"const loss"`. A `.md` file gives the same as `ae.shown_text`. + - `test_grade_prompt_is_blind_and_carries_the_rule`. The prompt contains: + - `OUTSIDE_RULE`; + - each expected step with its kind; + - the result; + - `"=== Explanation A ==="` and the chain text; + - the words `"any order"` and `"stated"`. + + It does not contain `"principles"` or `"skill"`, in any case. + - `test_marks_are_normalised`: + - `marks(["supported", "stated", "bogus"], ["block", "block", "derived", "derived"]) == ["supported", "supported", "missing", "missing"]`; + - `marks("x", ["block"]) == ["missing"]`; + - extra marks are dropped. + - `test_tally`: + - `tally(["supported", "unsupported", "stated", "missing"], ["block", "derived", "derived", "derived"], True, False)` gives `{"supported": 1, "made": 2, "counted": 3, "stated": 1, "right": 1, "results": 1, "results_stated": 0}`; + - with `stated=True`, it gives `right == 0`, `results == 0` and `results_stated == 1`. + - `test_pass_scores_pool_subjects`. For condition `a`, tallies `{"supported": 1, "made": 2, "counted": 2, …, "right": 1, "results": 1}` and `{"supported": 2, "made": 2, "counted": 3, …, "right": 0, "results": 1}` give supported `0.6`, raw `0.8` and result `0.5`. A condition with `results == 0` everywhere gives `result is None`. + - `test_separations_need_every_pass`. + + ```python + separations([ + {"a": {"supported": .7}, "b": {"supported": .5}, "c": {"supported": .6}}, + {"a": {"supported": .7}, "b": {"supported": .5}, "c": {"supported": .4}}, + {"a": {"supported": .6}, "b": {"supported": .6}, "c": {"supported": .3}}, + ]) == [("a", "c")] + ``` + - `test_grade_subject_maps_labels_back_and_fills_missing`. Use a stub grader reply that grades label `A` only: + - the saved grades are keyed by condition; + - the condition whose label is missing gets all `missing` marks and `result_right: False`; + - a rerun makes no call. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: the new tests FAIL. +- [ ] **Step 3: Implement.** + - `grader_text`: for `.html`, `ae.shown_text(path)` plus the text of each `` under a `--- page script ---` line; otherwise `ae.shown_text(path)`. Each explanation is truncated at `ae.TEXT_LIMIT`. + - The grader prompt states: + - the subject `q`, the reader, and the target `q`; + - the expected steps, numbered, with kind, and the expected result; + - `OUTSIDE_RULE`; + - "the expected steps are a checklist of ideas: credit a step when the chain establishes the same idea in any order, merged, split, or by an equivalent route"; + - the four marks as the PRD defines them, adding that `stated` applies to derived steps only and that building blocks are meant to be in the explanation; + - `result_right`, and `result_stated` (the explanation states the result outright); + - the reply shape `{"grades": {"