diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..63d9db5 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,33 @@ +# Explainer — instructions for coding agents + +This file is for agents other than Claude Code. Claude Code reads `CLAUDE.md`, which loads the same principles through an import that other agents do not follow. + +In this folder, act as an explanation compiler. Before you write any explanation, also a short answer, read `plugin/skills/explain/references/principles.md` and follow it. + +For a non-trivial explanation, follow the `explain` skill (`plugin/skills/explain/SKILL.md`). Use `plugin/skills/video/SKILL.md` to go straight to an animated explainer, and `plugin/skills/verify/SKILL.md` to check an existing explainer. Their reference files are in `plugin/skills/explain/references/`. + +## This repo + +The repo is three things: the toolkit (`explainer_kit/`), the Claude Code plugin (`plugin/`, listed by `.claude-plugin/marketplace.json`), and the examples (`explainers/`). Human-facing docs live in `docs/`. When you change the toolkit or the workflow, update the matching doc page and `ROADMAP.md`. + +Run the CLI through uv: `uv run explainer `. + +```bash +uv run explainer new --stage 1 2 3 4 # scaffold model.md, model.yaml, narrative.md, and the chosen renderings +uv run explainer check # every example against its model.yaml +uv run explainer coldread --run # first-viewing read by a fresh agent +uv run explainer findings --stats # how many review findings authors adopt, per tool +uv run explainer eval chat # chat answers under each system prompt, graded blind +uv run explainer eval artifact --split dev # held-out subjects built under each condition, read and graded blind +uv run explainer render --draft # fast silent layout pass +uv run pytest -q # toolkit tests +``` + +The review and eval commands (`probe`, `coldread`, and `quiz` with `--run`, and `eval`) call the Claude CLI (`claude -p`). They need it installed and signed in, whichever agent runs them. + +Reference implementations by stage are indexed in `explainers/README.md`. + +## Rules + +- This repo is public. Never copy material from private projects that use the toolkit (course content, learner records) into it. +- The examples in `explainers/` are frozen as reference implementations and regression fixtures: `explainer check` must pass on them. diff --git a/CHANGELOG.md b/CHANGELOG.md index 004cfa1..bad7aa0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## 0.8.0 — 2026-10-19 + +Measure the toolkit, not the examples. + +- **`explainer eval artifact`**: 30 held-out subjects (10 dev, 20 test) with fixed quizzes. A fresh agent builds each explainer under each condition (no guidance, the principles, the explain skill at the quick or published tier); a second answers the quiz from the renderings alone, quoting the passage behind each answer (unquotable answers score 0); a third grades blind. [docs/evals.md](docs/evals.md#the-artifact-eval). +- The examples are frozen as reference implementations and regression fixtures (ROADMAP §15). +- [What the artifact eval found](docs/evals.md#what-the-artifact-eval-found), on the 10 dev subjects: the quiz is at its ceiling; the principles carry the measurable value (40% shorter, best mean effort rank, lowest cost); the quick tier makes the most errors; the published tier costs 13 times as much with no consistent measured gain. A person's ranking has to confirm the grader before it steers a change. +- `AGENTS.md` for agents other than Claude Code: no `@` imports, no `.claude/` paths. +- Docs: the author's eight understanding questions and the blind test's seven are described as two sets; "at least three dev subjects" everywhere. + ## 0.7.0 — 2026-10-04 Reviews that run themselves, and a record of which findings matter. diff --git a/CLAUDE.md b/CLAUDE.md index 2919020..b20121c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,6 +18,7 @@ uv run explainer check # every example against its model. uv run explainer coldread --run # first-viewing read by a fresh agent (claude -p), saved with numbered findings uv run explainer findings --stats # how many review findings authors adopt, per tool uv run explainer eval chat # chat answers under each system prompt, graded blind (needs the claude CLI) +uv run explainer eval artifact --split dev # held-out subjects built under each condition, read and graded blind uv run explainer render --draft # fast silent layout pass uv run explainer render --review # 1080p60, voice, captions, chapters, review sheet uv run pytest -q # toolkit tests diff --git a/ROADMAP.md b/ROADMAP.md index caa82db..df5b402 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -20,6 +20,7 @@ Status: `proposed` · `next` · `in progress` · `done` | 12 | [The narrative pass](#12-the-narrative-pass) | done (v0.5.0), with open items | | 13 | [Rebalance: omission and excess both cost](#13-rebalance-omission-and-excess-both-cost) | done (v0.6.0), with open items | | 14 | [Reviews that run themselves](#14-reviews-that-run-themselves) | done (v0.7.0), with open items | +| 15 | [Measure the toolkit, not the examples](#15-measure-the-toolkit-not-the-examples) | in progress (eval shipped in v0.8.0) | --- @@ -210,3 +211,34 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o - Measure adoption again under the tuned prompts; if edge findings stay under 10%, drop the per-unit edge lists and keep only blocking findings and cuts. - The cold read on a video re-reads its review sheet from scratch each round; a round could read only the changed captions. - Run review rounds on more than one model, as with the chat eval. + +## 15. Measure the toolkit, not the examples + +**Problem.** From v0.4 to v0.7, each round reviewed the same ten examples, fixed what the reviewers found, and often changed a principle or a review prompt in response. That fits the toolkit to those examples. The review prompts were tuned on adoption figures from the same examples they were then judged on; ste-80 took six cold-read rounds, which shows that a reviewer always finds something, not that the pipeline improved. Several rules in §10 and §11 trace back to one example's failure, and nothing showed that they help on a new subject. The only measure on new material was the chat eval: six questions, which the chat rule was also tuned against. + +**Built.** + +- The examples are frozen as reference implementations and regression fixtures: `explainer check` must pass; no more review rounds. +- `evals/artifact/questions.yaml`: 30 held-out subjects (10 dev, 20 test), none an example or a chat-eval question, each with a fixed quiz of four questions; the last is a prediction about a case the explanation need not mention. +- `explainer eval artifact`: per subject and condition, a fresh agent builds the explainer (none, the principles, or the explain skill at the quick or published tier), a second reads only the renderings and answers the quiz with a quote for each answer (an answer whose quote is not in the renderings scores 0), and a third grades blind. [docs/evals.md](docs/evals.md#the-artifact-eval). + +**Rules.** + +- Change the principles or a skill only for a failure seen in at least three dev subjects; confirm the change on test before a release; never change anything for one test subject. +- Remove a rule when removing it costs nothing on dev. +- Calibrate the grader against a person: the author rates a sample, and the grader's ranking must agree before its numbers steer a change. + +**Evidence** (dev baseline, [docs/evals.md](docs/evals.md#what-the-artifact-eval-found)). + +- The quiz is at its ceiling (94–100% in every condition, with a strong reader and with Haiku): a model reader cannot measure the difference. The same holds for the published tier's blind test. +- The principles alone cut length 40% and rank first on effort in every grading pass, at the lowest cost. The quick tier is shortest but ranks below them and makes the most errors; the published tier costs 13× as much for no measured gain. +- The first run was invalid: builders inside the repo loaded its CLAUDE.md in every condition. Builds now run outside every project. + +**Next.** + +- A person ranks a sample (principles, quick, published on three subjects); the grader's ranking is used only if it agrees. +- If it agrees: decide what the skill keeps. Candidates: the model file and the reviews only for the published tier, or the reviews dropped where they do not change the result. +- A quiz that discriminates: harder subjects, or a reader limited in another way (time, words read), since a weaker model was not enough. +- `plugin/skills/explain/references/html.md` says "the seven understanding questions from principles.md §7", but §7 has eight. Fix it with the principles rewrite, because skill text is measured. +- The blind test (`explainer_kit/review.py`) does not ask "Could anything be cut with no loss?" Decide whether it should when the objective becomes least *wasted* effort. + diff --git a/docs/cli.md b/docs/cli.md index 3451e12..9ad7f7e 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -22,6 +22,7 @@ With the plugin installed, run `explainer ` from any project. In the Ex | `explainer say "" [--voice V] [--speed S]` | Synthesize one line and play it. Use it to choose a voice or test a lexicon spelling. | | `explainer frames [--draft] [--every N]` | Rebuild the contact sheet from an existing render. | | `explainer eval chat [--conditions name=source ...]` | Answer the questions in `evals/chat/questions.yaml` under several system prompts with fresh `claude -p` calls, grade them blind, and write a report. See [Evals](evals.md). | +| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). | ## `check` diff --git a/docs/concepts.md b/docs/concepts.md index 9e96e02..0f237ae 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -145,21 +145,26 @@ For learners, a result is worth more after a prediction. Pages put a predict gat ## The understanding test -Before an explanation is done, it is checked against seven questions: +There are two sets of questions, and they are not the same. + +**The author's checklist** ([principles.md](../plugin/skills/explain/references/principles.md) §7) has eight questions. The author asks them while writing: 1. Can the reader identify the main objects? -2. Can the reader explain how they relate? +2. Can the reader explain how the objects relate? 3. Can the reader see the central causal chain? 4. Can the reader predict what changes when an important variable changes? 5. Can the reader separate assumptions from observations? -6. Can the reader rebuild the explanation without the exact wording? -7. Does the representation impose mental simulation that another medium would remove? +6. Can the reader rebuild the high-level explanation without the exact wording? +7. Does the representation impose mental simulation that a different medium would remove? +8. Could anything be cut with no loss? A "no" means revise, or escalate one stage. +**The blind test** asks a fresh reader seven questions, defined in [`explainer_kit/review.py`](../explainer_kit/review.py) (`UNDERSTANDING`). They overlap with the checklist, but they are a different set. The reader names the main objects, states how they relate and the central causal chain, predicts the effect of the most important variable, separates an assumption from an observation, and rebuilds the explanation in at most five sentences. The seventh asks which part was hardest to follow and what would have made it easier. The blind test does not ask the cut question. + ### Blind understanding test -An author grading their own explanation is a weak test. `explainer quiz --rendering ` prints a prompt for a fresh agent that sees only one rendering. It answers the seven questions plus the model's quiz items, and its answers are scored against `explainer quiz --rubric`. A test must also be able to fail: in the first run, a deliberately broken copy of the softmax prose (it claimed a high temperature can change which token is first) scored 0 on the quiz item about that claim, while the real prose passed. Results live in `explainers//review/understanding.md`. +An author grading their own explanation is a weak test. `explainer quiz --rendering ` prints a prompt for a fresh agent that sees only one rendering. It answers the blind test's seven questions plus the model's quiz items, and its answers are scored against `explainer quiz --rubric`. A test must also be able to fail: in the first run, a deliberately broken copy of the softmax prose (it claimed a high temperature can change which token is first) scored 0 on the quiz item about that claim, while the real prose passed. Results live in `explainers//review/understanding.md`. ## Where the rules live diff --git a/docs/evals.md b/docs/evals.md index 815137b..1836f83 100644 --- a/docs/evals.md +++ b/docs/evals.md @@ -1,6 +1,50 @@ # Evals -The checks and reviews in Explainer test artifacts: one explainer at a time. The most common use is different: a question in chat, answered with the principles and no files. `explainer eval chat` measures that use. +The checks and reviews in Explainer test one explainer at a time. They make that explainer better; they do not show that the toolkit helps on the next subject. Tuning the principles and the review prompts against the same ten examples fits them to those examples. Two evals measure the toolkit itself, on subjects it was never tuned on: + +- `explainer eval chat`: a question in chat, answered with the principles and no files (the most common use). +- `explainer eval artifact`: an explainer built from scratch by a fresh agent, under each condition, then read and graded blind. + +## The artifact eval + +`evals/artifact/questions.yaml` holds 30 subjects that are neither examples nor chat-eval questions, from GPS to rent caps: 10 `dev`, 20 `test`. Each has the question, the misconception to avoid, and a fixed quiz of four questions with expected answers. The last is a prediction: a case the explanation need not mention, which the reader answers from the mechanism it gives. The quiz does not depend on what the builder chose to cover. + +```bash +uv run explainer eval artifact --split dev # none, principles, quick, published +uv run explainer eval artifact --only float-sum --conditions none quick +uv run explainer eval artifact --conditions none old=git:v0.6.0 principles +``` + +For each subject and condition, three fresh `claude -p` agents run, with no settings files, no MCP servers, and no stdin: + +1. **Build.** The builder gets the question and the reader, and writes `explanation.md`, `diagram.md`, or `index.html` (no video) in an empty project folder. Conditions differ only in guidance: `none`; `principles` (principles.md appended to the system prompt; also `git:` or a file); `quick` and `published` (the plugin loaded and the explain skill at that tier, running this checkout's CLI). +2. **Read.** The reader gets copies of the renderings only (no `model.md`) and answers the quiz. Each answer quotes the passage it rests on. An answer whose quote is not in the renderings scores 0, so a reader that answers from its own knowledge earns nothing. The reader also lists where it would stop or reread, and what it did not need. +3. **Grade.** The grader sees each subject's explanations and answers under shuffled labels, scores each answer 0–2 against the expected one, and checks the text for the misconception and for factual errors. + +Every stage saves its output, and a rerun into the same folder skips what exists. The report (`evals/artifact/results/-/report.md`) gives, per condition: the grounded quiz score, the raw score, ungrounded answers, misconceptions taught and corrected, errors, stops and excess per explanation, words, the forms chosen, and the builder's cost and time. + +**How to use it.** Change the principles or a skill only for a failure that appears in at least three `dev` subjects. Then run `test` to confirm the change before a release. Do not change anything in response to one test subject, and remove a rule when removing it costs nothing on `dev`. + +## What the artifact eval found + +Dev baseline, 2026-10-04, v0.7.0 principles and skills; builder, reader, and grader on the `claude` CLI's default model. [Report](../evals/artifact/results/2026-10-04-dev/report.md); the same builds read by Haiku: [report](../evals/artifact/results/2026-10-04-dev-haiku-reader/report.md). + +| condition | effort rank (1–4) | quiz, strong reader | quiz, Haiku reader | words | errors | cost | minutes | +|---|---|---|---|---|---|---|---| +| none | 3.1–3.3 | 100% | 95% | 1,689 | 3–4 | $0.28 | 1.5 | +| principles | 1.7–1.9 | 99% | 98% | 1,006 | 1–4 | $0.21 | 1.0 | +| quick (skill) | 2.5–2.9 | 95% | 94% | 606 | 4–9 | $0.46 | 1.6 | +| published (skill) | 2.2–2.5 | 95% | 95% | 1,016 | 1–2 | $2.81 | 11.8 | + +Effort rank: the grader's ranking of each subject's four explanations by the reader's effort, averaged over 10 subjects; the range covers three grading passes. Errors: factual errors the grader found, in all 10 explanations; the range covers the same three grading passes. Cost: the builder plus the reviews it ran. + +- **The quiz is at its ceiling.** Every condition scores 94–100%, with a strong reader and with Haiku. A model reader answers from any competent explanation, so the quiz cannot rank the conditions. The published tier's own blind test is the same kind of instrument, so it is weak for the same reason. +- **The principles carry the measurable value.** Appended to the system prompt, they cut length by 40% (1,689 → 1,006 words), came first in the effort ranking in all three passes, and cost the least: a shorter explanation is cheaper to write. +- **The skill adds cost without a consistent measured gain over the principles.** The quick tier is the shortest (606 words) but ranks below the principles and has the most errors, several from compression ("three spheres meet at a point"; they meet at two). The published tier costs 13 times as much as the principles and takes 12 times as long, for a lower rank and the same length; its errors (1–2) overlap the principles' (1–4), fewer in one pass and more in another. +- **Almost everything is prose.** With no guidance, 5 of 10 builders made a page; with the principles, none did; the skill made 3 diagrams in 20 builds. These subjects may simply be prose-shaped; the eval does not test video. +- **The ranking is a model's taste, and it is noisy.** Changing only what the grader saw (Markdown as written instead of stripped of inline code) moved the quick tier from 2.5 to 2.9. The principles-first and none-last results held in every pass; the order of quick and published did not. A person's ranking of a sample has to confirm it before it steers a change. + +Two faults in the first run, both fixed: builders ran inside this repo, so Claude Code loaded its CLAUDE.md (and the principles) in every condition, including `none`; and the grader read Markdown with inline code removed, so formulas looked like blanks. Builds now run in a temp folder outside every project, with a guard. ## The chat eval @@ -19,7 +63,7 @@ Each answer comes from a fresh `claude -p` call in an empty folder, with no tool The report goes to `evals/chat/results//report.md`, with every answer and `grades.json`. -## What it found +## What the chat eval found Two runs on 2026-10-04 (answers and grades by the `claude` CLI's default model). Run a: the principles before the chat rule. Run b: after it. Each run is graded on its own, so compare conditions within a run. @@ -39,7 +83,7 @@ The current principles now give the shortest answers of the three (292 words, ag Reports: [run a](../evals/chat/results/2026-10-04-a/report.md), [run b](../evals/chat/results/2026-10-04-b/report.md). -## Limits +## Limits of the chat eval - Six questions, one grader, one model. Treat a difference of a few tenths of a point as noise. - The grader is a model with its own taste. It sees the reader and the must points, not the principles. diff --git a/docs/prds/PRD-001-ship-v0.8.md b/docs/prds/PRD-001-ship-v0.8.md new file mode 100644 index 0000000..9c3dff3 --- /dev/null +++ b/docs/prds/PRD-001-ship-v0.8.md @@ -0,0 +1,115 @@ +# PRD-001: Ship v0.8, the toolkit eval and what it found + +Status: draft, 2026-10-19 · Part 1 of 3 (PRD-002: a rederivation eval; PRD-003: Olah-compass principles) + +## Why + +ROADMAP §15 ("Measure the toolkit, not the examples") is built but not shipped. The artifact eval runs, the dev baseline exists, and it found something that matters: + +- the reader quiz is at its ceiling; +- plain principles beat the skill tiers on effort; +- the published tier costs up to 13× as much for no measured gain. + +None of it is committed, so nobody else can see it, and the public repo still implies the v0.7 workflow is the measured best. + +Shipping it now does three things: + +- **Puts an honest negative result on the record** before the next round of work changes the principles. PRD-002 and PRD-003 build on this baseline, so it has to be fixed in history first. +- **Shows readers what was measured, including what didn't work.** +- **Clears the working tree,** so PRD-002 starts from a clean, tagged state. + +## Goals + +### 1. Release v0.8.0 containing the artifact eval and its results + +- **New files:** `explainer_kit/artifact_eval.py`, `tests/test_artifact_eval.py`, and `evals/artifact/`. The last holds the questions and both result sets, `2026-10-04-dev` and `2026-10-04-dev-haiku-reader`. +- **Modified files:** + - code: `agent.py`, `cli.py`, `runs.py`, and their tests; + - docs and plans: `docs/cli.md`, `docs/evals.md`, `CHANGELOG.md`, `ROADMAP.md`, `CLAUDE.md`, and `docs/prds/`. +- **Changelog:** the "Unreleased" section in `CHANGELOG.md` becomes `0.8.0 — `. +- **Tag and release:** tag `v0.8.0`, and publish a GitHub release in the same style as the earlier ones (title, one-line summary, bullets). +- **Version bumps** follow the v0.7.0 commit: `pyproject.toml`, `plugin/.claude-plugin/plugin.json`, and the version pin in `plugin/bin/explainer`. + +### 2. The release notes state the negative result plainly + +Use the repo's own words from `docs/evals.md` and ROADMAP §15. The notes must say: + +- the quiz is at its ceiling; +- the principles alone rank first on effort, at the lowest cost; +- the quick tier makes the most errors; +- the published tier costs 13× as much for no measured gain. + +They also say the first run was invalid and why. Builders loaded the repo's CLAUDE.md in every condition, and the grader read Markdown with inline code stripped (`docs/evals.md`). + +### 3. Fix the drift found on 2026-10-19, without touching measured text + +`plugin/skills/` and `plugin/skills/explain/references/principles.md` are the text the eval measures: the `principles` condition appends `principles.md` word for word. They stay unchanged in this release, so the v0.8.0 tag holds exactly the baseline PRD-002 and PRD-003 build on. + +- **AGENTS.md (currently untracked).** It is a copy of CLAUDE.md with "Claude" replaced by "Codex". That broke real paths (`.Codex/skills/`, `.Codex-plugin/marketplace.json`) and a command (`Codex -p`). Replace it with a short file that is accurate for any coding agent: + - read `plugin/skills/explain/references/principles.md` before writing an explanation; + - the skills live in `plugin/skills/`; + - the review and eval commands call the Claude CLI (`claude -p`). + + It must not depend on Claude Code's `@` imports, which other agents do not follow. +- **Question counts in the docs.** There are two different sets of understanding questions: + - the author's test in `principles.md` §7 has eight; the eighth is "Could anything be cut with no loss?"; + - the blind test in `explainer_kit/review.py:24-32` asks seven, and its seventh ("Which part was hardest to follow…") is not one of the author's eight. + + `docs/concepts.md:148` says "seven" and lists the author's first seven. Rewrite that passage so each set is described correctly, with its count and where it lives. + + Two related mismatches sit in measured skill text and stay as they are: "seven" in `plugin/skills/explain/references/html.md:89`, and the blind test not asking question 8. Record both in ROADMAP §15 for PRD-003. +- **The change rule.** ROADMAP §15 says a principle changes only for a failure seen in "at least three dev subjects"; `docs/evals.md:26` says "several". Use "at least three" in both. +- **ROADMAP index.** Add a §15 row to the index table. It currently ends at §14. + +### 4. Tests, checks and CI pass + +All tests pass, `explainer check` passes for every example, and CI is green on the release commit. + +## Non-goals + +- **No change to the principles, the skills, or the objective.** The README still says "least effort" after this release. The move to "least *wasted* effort" belongs to PRD-003, so the README, `principles.md`, and the graders change together. +- **No new eval subjects, no new eval runs, and no changes to scoring.** The results ship as they are. +- **No human calibration yet** (ROADMAP §15 "Next"). PRD-002 covers it. + +## Anti-goals + +- Softening the negative result, or presenting the dev baseline as more than a 10-subject dev run. The release must not read as a success story about the skill tiers. +- Shipping anything that changes what an existing example renders. + +## Constraints + +- **The repo is public (MIT).** + - `evals/artifact/` was scanned on 2026-10-19: no absolute paths, usernames, emails, API keys, or session IDs. Re-check it if anything is regenerated before the commit. + - Documents in `docs/prds/` hold no personal or private-project context. +- **Keep the release style** of v0.3.0–v0.7.0. + +## Decided + +- **The order:** this PRD ships first, then PRD-002 (eval), then PRD-003 (principles). (Owner, 2026-10-19.) +- **The negative result is published,** not held back. (Owner, 2026-10-19.) +- **The PRDs live in `docs/prds/` in this public repo.** (Owner, 2026-10-19.) + +## Calls made by Claude (check these) + +- **Measured skill text stays frozen in v0.8.** The doc fixes touch `docs/` and ROADMAP only. The two mismatches in skill text wait for PRD-003, which rewrites those files anyway. +- **AGENTS.md becomes a short, separate file, not a symlink.** A symlink to CLAUDE.md would pass along Claude Code `@` imports that other agents ignore, so they would still miss the principles. +- **The "least wasted effort" wording waits for PRD-003.** Changing only the README now would make it disagree with `principles.md` and the graders. + +## Left to the builder + +- The release title and summary line. +- Commit granularity before the tag: one commit, or one per goal. +- Whether to add a short "What the artifact eval found" paragraph to the README, linking to `docs/evals.md`. + +## Done when + +- `v0.8.0` is tagged and released on GitHub, and the release notes contain the four findings in Goal 2. +- `git status` is clean. +- `AGENTS.md` names no path or command that does not exist, and uses no `@` imports. +- `docs/concepts.md` describes both sets of questions correctly. ROADMAP §15 records the two skill-text mismatches for PRD-003. +- Docs and ROADMAP agree on "at least three". +- Tests, `explainer check`, and CI pass. + +## Follow-up outside the code + +- Pin Explainer on the GitHub profile by hand. GitHub has no API for profile pins. diff --git a/docs/superpowers/plans/2026-10-19-ship-v0.8.md b/docs/superpowers/plans/2026-10-19-ship-v0.8.md new file mode 100644 index 0000000..7d0ec6e --- /dev/null +++ b/docs/superpowers/plans/2026-10-19-ship-v0.8.md @@ -0,0 +1,246 @@ +# Ship v0.8.0 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Release v0.8.0 with the artifact eval, its dev results and an honest account of what it found, and fix the doc drift without touching measured skill text. + +**Architecture:** +- Mostly docs and release mechanics. The only code change is one repo-integrity test that guards AGENTS.md. +- Work happens on branch `release/v0.8.0`. It lands on `main` as one squash commit, which matches the repo's one-commit-per-version history (`v0.7.0: …`). The tag and the GitHub release are cut from that commit. + +**Tech Stack:** Python 3 / uv, pytest, the `explainer` CLI, git, GitHub CLI (`gh`). + +**Spec:** `docs/prds/PRD-001-ship-v0.8.md` + +## Global Constraints + +- Version `0.8.0` everywhere: + - `pyproject.toml` (`version = "0.8.0"`) + - `plugin/.claude-plugin/plugin.json` (`"version": "0.8.0"`) + - the pin in `plugin/bin/explainer` (`@v0.8.0`) +- **Measured text must not change.** `git diff --stat main -- plugin/skills explainers` must be empty at every commit. +- **The repo is public.** Nothing in `docs/prds/`, `docs/superpowers/` or `evals/artifact/` may carry personal or private-project context, absolute home paths, emails or keys. +- **The release notes state the negative result.** They must not read as a success story for the skill tiers. +- **Release style:** + - annotated tag `v0.8.0` with message `v0.8.0: measure the toolkit, not the examples`; + - GitHub release titled `v0.8.0 — Measure the toolkit, not the examples`. + +## Review Focus + +1. **Private data in the committed eval output.** `evals/artifact/results/*/work/` holds raw agent transcripts and review runs. Someone browsing the public repo would expect no home paths, emails or keys. A scan in Task 3, Step 1 pins this. +2. **A plugin pin that points at a tag that doesn't exist yet.** `plugin/bin/explainer` installs from `@v0.8.0`, so a user installing between the merge and the tag push gets a failure. + - Task 4 pushes the tag immediately after the merge. + - It then verifies the tag with `git ls-remote`. +3. **New Markdown files with broken relative links:** AGENTS.md, the PRDs, this plan and the eval reports. `test_relative_links_resolve` covers them all. Task 3, Step 4 runs the full suite with the new files in place. +4. **Measured text changed by accident**, for instance a sweeping find-and-replace of "seven". The `git diff --stat main -- plugin/skills explainers` check runs in Tasks 2 and 3. +5. **Mermaid rendering is checked only in CI.** Locally, `explainer check` runs without `--diagrams` because the Mermaid CLI comes through `npx`. Task 4 waits for a green CI, which runs `explainer check --diagrams`. + +--- + +### Task 1: AGENTS.md that works for any coding agent + +**Files:** +- Modify: `tests/test_repo.py` (append one test) +- Create/replace: `AGENTS.md` (the current file is untracked and broken) + +**Interfaces:** none (repo-integrity test only). + +- [ ] **Step 0: Branch** + +Run: `git switch -c release/v0.8.0` +Expected: `Switched to a new branch 'release/v0.8.0'` + +- [ ] **Step 1: Write the failing test** in `tests/test_repo.py` + +```python +def test_agents_md_is_self_contained(): + """AGENTS.md is read by agents other than Claude Code: no @ imports, no paths that do not exist.""" + text = (ROOT / "AGENTS.md").read_text() + assert not re.search(r"^@", text, re.M), "AGENTS.md uses Claude Code @ imports, which other agents ignore" + paths = [p for p in re.findall(r"`([\w.\-/]+/[\w.\-/]*)`", text) if not p.startswith("http")] + missing = [p for p in paths if not (ROOT / p.rstrip("/")).exists()] + assert not missing, f"AGENTS.md names paths that do not exist: {missing}" + assert "plugin/skills/explain/references/principles.md" in text + assert "claude -p" in text +``` + +- [ ] **Step 2: Run it and confirm it fails** + +Run: `uv run pytest -q tests/test_repo.py::test_agents_md_is_self_contained` +Expected: FAIL, naming `.Codex/skills/` (and `.Codex-plugin/marketplace.json`) as missing. + +- [ ] **Step 3: Rewrite `AGENTS.md`** + +Write it from `CLAUDE.md`, but drop the `@` import and every `.claude/` path. It must say: +- **Before writing an explanation:** read `plugin/skills/explain/references/principles.md`. +- **The skills** are at `plugin/skills/explain/SKILL.md`, `plugin/skills/video/SKILL.md` and `plugin/skills/verify/SKILL.md`. +- **What the repo is:** the toolkit (`explainer_kit/`), the plugin (`plugin/`) and the examples (`explainers/`). Docs live in `docs/`. Update the matching doc page and `ROADMAP.md` with every change. +- **The command list** from CLAUDE.md, noting that `coldread`, `blindtest`, `probe` with `--run`, and `eval` call the Claude CLI (`claude -p`). +- **CLAUDE.md's public-repo rule,** word for word. + +Keep it to about 40 lines. + +- [ ] **Step 4: Run the test and confirm it passes** + +Run: `uv run pytest -q tests/test_repo.py` +Expected: all pass. + +- [ ] **Step 5: Commit** + +```bash +git add AGENTS.md tests/test_repo.py +git commit -m "AGENTS.md for any coding agent: no @ imports, no paths that do not exist" +``` + +### Task 2: Doc drift: question counts, the change rule, ROADMAP §15 + +**Files:** +- Modify: `docs/concepts.md:146-156` (section "The understanding test") +- Modify: `docs/evals.md:26` +- Modify: `ROADMAP.md`: the index table (after the row for 14) and the "Next" list of §15 + +**Interfaces:** none. + +Note: `docs/evals.md` and `ROADMAP.md` already hold uncommitted v0.8 content (the artifact-eval section and §15). This task's commit includes it. That is intended. + +- [ ] **Step 1: Rewrite "The understanding test" in `docs/concepts.md`** so it describes two separate sets: + - **The author's checklist:** eight questions, copied verbatim from `plugin/skills/explain/references/principles.md` lines 79–86, item 8 included. It is used while writing. + - **The blind test:** seven questions that a fresh reader answers, defined in `explainer_kit/review.py` (`UNDERSTANDING`). Say that they overlap with the checklist but are not the same set: the blind test asks which part was hardest to follow, and it does not ask the cut question. Link `../explainer_kit/review.py`. + +- [ ] **Step 2: In `docs/evals.md:26`, replace** "a failure that appears in several `dev` subjects" **with** "a failure that appears in at least three `dev` subjects". + +- [ ] **Step 3: Add the index row to `ROADMAP.md`**, after row 14: + +```markdown +| 15 | [Measure the toolkit, not the examples](#15-measure-the-toolkit-not-the-examples) | in progress (eval shipped in v0.8.0) | +``` + +- [ ] **Step 4: Add two bullets to §15's "Next" list in `ROADMAP.md`:** + - `plugin/skills/explain/references/html.md` says "the seven understanding questions from principles.md §7", but §7 has eight. Fix it with the principles rewrite, because skill text is measured. + - The blind test (`explainer_kit/review.py`) does not ask "Could anything be cut with no loss?" Decide whether it should when the objective becomes least *wasted* effort. + +- [ ] **Step 5: Verify** + +Run: `grep -n "several \`dev\`" docs/evals.md; git diff --stat main -- plugin/skills explainers; uv run pytest -q tests/test_repo.py` +Expected: no grep hit, an empty diff, and all tests passing. + +- [ ] **Step 6: Commit** + +```bash +git add docs/concepts.md docs/evals.md ROADMAP.md +git commit -m "Docs: two sets of understanding questions, 'at least three' dev subjects, §15 in the roadmap index" +``` + +### Task 3: The release commit: eval, results, version 0.8.0, changelog + +**Files:** +- Add (now untracked): + - `explainer_kit/artifact_eval.py` + - `tests/test_artifact_eval.py` + - `evals/artifact/` + - `docs/prds/` + - `docs/superpowers/plans/` +- Add (now modified): + - `explainer_kit/agent.py`, `explainer_kit/cli.py`, `explainer_kit/runs.py` + - `tests/test_runs.py` + - `docs/cli.md`, `CLAUDE.md` + - `CHANGELOG.md`, `ROADMAP.md`, `docs/evals.md` (the last two already partly touched in Task 2) +- Modify: + - `pyproject.toml:3` + - `plugin/.claude-plugin/plugin.json:4` + - `plugin/bin/explainer:13` + - `CHANGELOG.md:3` + - `uv.lock`, if `uv lock` changes it + +**Interfaces:** Consumes Tasks 1–2. Produces the branch state that Task 4 merges. + +- [ ] **Step 1: Scan for private data** + +Run: `grep -rIlE '/Users/|/home/[a-z]|@gmail\.com|sk-[A-Za-z0-9]{20}|ANTHROPIC_API_KEY=|OPENAI_API_KEY=' evals/artifact docs/prds docs/superpowers AGENTS.md` +Expected: no output. If anything matches, stop and report it. Don't edit the eval output. + +- [ ] **Step 2: Bump the version.** Set `0.8.0` in the three places listed under Global Constraints, then run `uv lock`. + +- [ ] **Step 3: Write the changelog section.** In `CHANGELOG.md`, change `## Unreleased` to `## 0.8.0 — 2026-10-19`. Use the release date if it isn't today. Keep the existing summary line and bullets, and add: + - `AGENTS.md` for agents other than Claude Code: no `@` imports, no `.claude/` paths. + - Docs: the author's eight understanding questions and the blind test's seven are now described as two separate sets; "at least three dev subjects" is used everywhere. + - A link to the findings, written relative to `CHANGELOG.md`: target `docs/evals.md#what-the-artifact-eval-found`, text "What the artifact eval found". + +- [ ] **Step 4: Run the full suite and the model check** + +Run: `uv run pytest -q && uv run explainer check && git diff --stat main -- plugin/skills explainers` +Expected: +- all tests pass, including `test_plugin_manifest_and_skills` for the version pins and the link test over the new Markdown; +- `explainer check` exits 0; +- the diff is empty. + +- [ ] **Step 5: Commit** + +```bash +git add explainer_kit/artifact_eval.py tests/test_artifact_eval.py evals/artifact docs/prds docs/superpowers \ + explainer_kit/agent.py explainer_kit/cli.py explainer_kit/runs.py tests/test_runs.py \ + docs/cli.md CLAUDE.md CHANGELOG.md ROADMAP.md docs/evals.md pyproject.toml uv.lock \ + plugin/.claude-plugin/plugin.json plugin/bin/explainer +git commit -m "v0.8.0: the artifact eval, its dev results, and what it found" +git status --short # expect: empty +``` + +### Task 4: Land, tag, release + +**Files:** none in the repo. Release notes go in a temporary file outside the repo. + +**Interfaces:** Consumes the branch from Task 3. + +- [ ] **Step 1: Push and open the PR** + +Run: `git push -u origin release/v0.8.0 && gh pr create --base main --title "v0.8.0: measure the toolkit, not the examples" --body-file `. Use the same notes as Step 4. +Expected: a PR URL. + +- [ ] **Step 2: Wait for CI** + +Run: `gh pr checks --watch` +Expected: the `test` job passes, including `explainer check --diagrams` and the draft render. + +- [ ] **Step 3: Squash-merge, then tag at once** + +Write ``, a curated squash body in the v0.7.0 style: the CHANGELOG 0.8.0 bullets, then the attribution trailers once. Without it, GitHub joins the three branch commit messages and their trailers. + +Run: +```bash +gh pr merge --squash --subject "v0.8.0: measure the toolkit, not the examples" --body-file --delete-branch +git switch main && git pull --ff-only +git diff --stat HEAD # expect: empty, so the tag sits on the reviewed tree +git tag -a v0.8.0 -m "v0.8.0: measure the toolkit, not the examples" && git push origin v0.8.0 +git ls-remote --tags origin v0.8.0 +``` +Expected: an empty diff, then `git ls-remote` prints the tag. If either fails, stop before Step 4. + +- [ ] **Step 4: Publish the GitHub release** + +Run: `gh release create v0.8.0 --title "v0.8.0 — Measure the toolkit, not the examples" --notes-file ` + +The notes need three parts. + +**1. A one-line summary**, then the changelog bullets. + +**2. "What it found"**, with these facts and figures copied from `docs/evals.md`: +- **The quiz is at its ceiling.** Every condition scores 94–100% with a strong reader and with Haiku, so it cannot rank the conditions. +- **The principles carry the measurable value.** They cut length 40% (1,689 → 1,006 words), came first in effort rank in all three grading passes, and cost the least ($0.21 per build). +- **The quick tier** is the shortest at 606 words, ranks below the principles, and makes the most factual errors (6–9). +- **The published tier** costs 13 times as much as the principles ($2.81) and takes 11.8 minutes, for no measured gain. +- **This is a 10-subject dev run.** The effort ranking is a model's and is noisy; a person's ranking has to confirm it. + +**3. The first run was invalid**, for two reasons: builders loaded the repo's CLAUDE.md in every condition, and the grader read Markdown with inline code stripped. Both are fixed. + +Link `docs/evals.md#what-the-artifact-eval-found`. End with the Upgrade line, as v0.6.0 and v0.7.0 do: + +> **Upgrade:** plugin users get v0.8.0 on the next `/plugin` update; the `explainer` wrapper now pins `@v0.8.0`. `explainer eval artifact` needs the `claude` CLI. + +Use the corrected figures: quick's errors are 4–9, and the published tier has no *consistent* measured gain. Its errors (1–2) overlap the principles' (1–4). + +Expected: `gh release view v0.8.0` shows the title and notes. + +- [ ] **Step 5: Confirm the release** + +Run: `gh run list --branch main --limit 1 && gh release list --limit 3 && git status --short` +Expected: the latest `main` run passes, `v0.8.0` is listed first, and the working tree is clean. diff --git a/evals/artifact/questions.yaml b/evals/artifact/questions.yaml new file mode 100644 index 0000000..460b1b2 --- /dev/null +++ b/evals/artifact/questions.yaml @@ -0,0 +1,465 @@ +# Artifact eval: subjects the toolkit has never been tuned on. None is an example in explainers/ or a +# question in the chat eval. Each subject has a question (given to the agent that builds the explainer) and +# a quiz (given only to the reader, who answers from the finished explainer). The quiz is fixed here, so +# it does not depend on what the builder chose to cover. The last item of each quiz is a prediction +# (kind: predict): a case the explanation need not mention, answered from the mechanism it gives. +# +# Split: tune against `dev` only. Run `test` to confirm a change before a release, and do not change the +# principles or skills in response to a single test subject (docs/evals.md). +# +# Run: uv run explainer eval artifact --split dev (needs the `claude` CLI; see docs/evals.md) + +reader: "A technical reader: comfortable with school mathematics and science and with basic programming, and new to this subject." + +subjects: + # ------------------------------------------------------------------ dev + - id: gps-relativity + split: dev + q: "How does GPS find your position, and why does it need relativity?" + misconception: "The receiver sends a signal to the satellites, and they work out where it is." + quiz: + - q: "What does the receiver measure, and how does it turn that into a distance?" + expect: "The time each satellite's signal took to arrive (the signal carries the time it was sent); distance = speed of light × travel time." + - q: "Why does a receiver need four satellites for a position, not three?" + expect: "Its own clock is cheap and its time error is a fourth unknown, with the three coordinates; four distances give four equations." + - q: "Why would GPS fail within a day without relativity corrections, and by about how much?" + expect: "Satellite clocks run fast relative to ground clocks by about 38 µs a day (weaker gravity: +45 µs; speed: −7 µs); at the speed of light that is about 10 km of position error per day." + - q: "A receiver's clock is 1 µs fast, and it uses three satellites as if its clock were right. About how large is the error in the distances it computes?" + expect: "About 300 m per satellite: the speed of light × 1 µs; a fourth satellite would let it solve for the clock error." + kind: predict + + - id: vaccine-memory + split: dev + q: "How does a vaccine prepare the immune system for a disease it has never met?" + misconception: "A vaccine works by giving you a mild case of the disease, so you need to get sick for it to work." + quiz: + - q: "What must a vaccine contain or produce, and why does that not make you ill?" + expect: "An antigen (a piece of the pathogen the immune system recognizes, or instructions to make it), without a pathogen that can replicate and cause the disease." + - q: "Why is the response to a real infection after vaccination faster than the first response?" + expect: "Memory B and T cells specific to that antigen remain after vaccination, so the response starts from many matching cells within days instead of taking one to two weeks to select and multiply them." + - q: "Why does a vaccine against one virus not protect against an unrelated virus?" + expect: "Recognition is specific: antibodies and receptors match the shape of one antigen; a different virus has different antigens." + - q: "A virus changes the shape of the part that antibodies recognize. What happens to the protection from an old vaccine, and why?" + expect: "It weakens: the memory cells and antibodies match the old shape and bind the new one poorly; a vaccine with the new antigen restores it." + kind: predict + + - id: key-exchange + split: dev + q: "How can two people who have never met agree on a secret key over a channel that anyone can read?" + misconception: "They send the key over the channel in encrypted form." + quiz: + - q: "What operation makes this possible, and what makes it safe?" + expect: "Modular exponentiation (g^a mod p) is easy to compute, but recovering a from g^a mod p (the discrete logarithm) is infeasible for large p." + - q: "Why do both people end up with the same key, though the key itself is never sent?" + expect: "Each sends g^(own secret) mod p; each raises what it received to its own secret: (g^b)^a = (g^a)^b = g^(ab) mod p." + - q: "What attack does the exchange alone not stop, and what stops it?" + expect: "A man-in-the-middle who runs a separate exchange with each side; authentication (signatures, certificates) stops it." + - q: "With p = 23 and g = 5, Alice's secret is 6 and Bob's is 15. What do they send, and what key do they share?" + expect: "Alice sends 5^6 mod 23 = 8, Bob sends 5^15 mod 23 = 19; the key is 19^6 mod 23 = 8^15 mod 23 = 2." + kind: predict + + - id: two-tides + split: dev + q: "Why are there two high tides a day, not one?" + misconception: "The far-side bulge comes from the centrifugal force of Earth spinning on its axis." + quiz: + - q: "Why is there a bulge on the side of Earth away from the Moon?" + expect: "The Moon's pull weakens with distance: it pulls the near side more than Earth's center and the center more than the far side, so the oceans are stretched along the Earth–Moon line, with a bulge at each end." + - q: "Why do high tides come about every 12 hours 25 minutes, not every 12 hours?" + expect: "The Moon moves along its orbit in the same direction Earth turns, so Earth needs about 24 h 50 min to face the Moon again; two bulges give a high tide every half of that." + - q: "When are tides largest, and why?" + expect: "Spring tides, at new and full Moon, when the Sun and Moon are in line and their tidal effects add (the Sun's is about half the Moon's)." + - q: "If the Moon stayed fixed in the sky instead of moving along its orbit, how often would high tides come?" + expect: "Every 12 hours exactly: Earth turns once in 24 hours under two fixed bulges." + kind: predict + + - id: bloom-filter + split: dev + q: "How does a Bloom filter answer 'have I seen this before?' with little memory, and how can it be wrong?" + misconception: "A Bloom filter stores the items, or their hashes, and compares them." + quiz: + - q: "What happens when an item is added, and when one is looked up?" + expect: "Add: k hash functions give k positions in a bit array; set those bits to 1. Look up: check the same k bits; if any is 0, the item was never added; if all are 1, it probably was." + - q: "Which kind of wrong answer can it give, and which can it never give? Why?" + expect: "False positives (other items may have set all k bits); never false negatives, because an added item's bits are never cleared." + - q: "Why can you not delete an item by clearing its bits?" + expect: "Other items may share those bits; clearing them would make those items look absent (false negatives)." + - q: "Every bit in a filter is 1. What does it answer for an item that was never added?" + expect: "'Probably seen': every lookup finds all its bits set, so every answer is a false positive." + kind: predict + + - id: heat-pump + split: dev + q: "How can a heat pump deliver three times more heat than the electricity it uses?" + misconception: "A heat pump creates energy, so it breaks the conservation of energy." + quiz: + - q: "Where does the extra heat come from?" + expect: "From the outside air or ground: the pump moves existing heat from cold to warm; the electricity drives the moving, so heat delivered = heat taken from outside + electrical work." + - q: "What does the refrigerant do in one cycle?" + expect: "It evaporates at low pressure and takes in heat outside; the compressor raises its pressure and temperature; it condenses inside and gives off heat; an expansion valve lowers its pressure again." + - q: "Why does a heat pump deliver less heat per unit of electricity on a colder day?" + expect: "It must lift heat across a larger temperature difference; the ideal limit, COP = T_hot / (T_hot − T_cold) in kelvin, falls as T_cold falls." + - q: "By the ideal limit, how does a heat pump that heats a house to 35 °C compare between a 5 °C day and a −15 °C day?" + expect: "COP limit 308/30 ≈ 10 at 5 °C and 308/50 ≈ 6 at −15 °C: the colder day gives less heat per unit of electricity." + kind: predict + + - id: quorum + split: dev + q: "Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1?" + misconception: "Adding a server always lets a cluster survive more failures." + quiz: + - q: "What rule decides whether a write is committed?" + expect: "A majority (quorum) of the servers must acknowledge it: 3 of 5, or 3 of 4." + - q: "Why a majority, and not any group of servers?" + expect: "Any two majorities share at least one server, so two sides of a split network cannot both commit conflicting writes (no split brain), and a committed write is always seen by the next majority." + - q: "How many failures does a 6-server cluster survive?" + expect: "2: a majority of 6 is 4, so 2 can fail; it is no better than 5." + - q: "A 5-server cluster splits into a group of 3 and a group of 2. Which group can still commit writes, and why?" + expect: "Only the group of 3: it is a majority; the 2 cannot form one, so the two groups cannot commit conflicting writes." + kind: predict + + - id: antibiotic-resistance + split: dev + q: "How do bacteria become resistant to an antibiotic?" + misconception: "Individual bacteria learn to resist, or adapt because they are exposed to the drug." + quiz: + - q: "Where does resistance come from before the drug is used?" + expect: "Variation already exists: random mutations, or resistance genes acquired from other bacteria." + - q: "What does the antibiotic itself do to the population?" + expect: "It kills the susceptible bacteria; the resistant ones survive and multiply, so the population becomes mostly resistant (selection)." + - q: "How can resistance pass between different species of bacteria?" + expect: "Horizontal gene transfer: plasmids and other mobile DNA carry resistance genes from one bacterium to another, also across species." + - q: "One bacterium in a billion in an infection is resistant. Why can a full course of antibiotics end with a resistant infection?" + expect: "The drug kills the susceptible ones; the resistant one survives and, with its competitors gone, multiplies into a resistant population." + kind: predict + + - id: float-sum + split: dev + q: "Why does 0.1 + 0.2 not equal 0.3 in most programming languages?" + misconception: "It is a bug in the language or the processor." + quiz: + - q: "Why can a computer not store 0.1 exactly?" + expect: "Floating-point numbers are binary fractions; 0.1 = 1/10 has no finite binary expansion (like 1/3 in decimal), so it is stored as the nearest representable double." + - q: "Why does the sum then differ from 0.3?" + expect: "The rounding errors of 0.1 and 0.2 do not cancel; their rounded sum is a different double from the one nearest 0.3 (it prints as 0.30000000000000004)." + - q: "What should a program do instead of testing a == 0.3?" + expect: "Compare with a tolerance (abs(a − 0.3) < epsilon), or use decimal or integer types (cents) where exact decimal values matter, such as money." + - q: "Is 0.5 + 0.25 == 0.75 true in the same languages? Why?" + expect: "True: 1/2, 1/4, and 3/4 have power-of-2 denominators, so all are stored exactly and no rounding error occurs." + kind: predict + + - id: greenhouse + split: dev + q: "How do greenhouse gases warm the Earth?" + misconception: "The hole in the ozone layer is what causes global warming." + quiz: + - q: "Why do greenhouse gases let sunlight in but hold back the heat the Earth gives off?" + expect: "Sunlight is mostly visible light, which they barely absorb; the Earth gives off infrared, which they absorb and emit again, partly back down." + - q: "Why does adding CO2 warm the surface, in terms of where the Earth radiates to space?" + expect: "Radiation escapes to space from higher in the atmosphere when there is more CO2; higher air is colder and radiates less, so the Earth takes in more than it gives off until the surface and lower air warm." + - q: "Why do nitrogen and oxygen, most of the air, not act as greenhouse gases?" + expect: "Molecules with two identical atoms have no vibration that changes their charge distribution (dipole), so they barely absorb infrared; CO2 and H2O do." + - q: "On a planet with no greenhouse gases, where would its heat radiate to space from, and what follows for its surface temperature?" + expect: "From the surface itself, so the surface settles at the temperature that balances sunlight (about 255 K for Earth, instead of the actual 288 K)." + kind: predict + + # ------------------------------------------------------------------ test + - id: jpeg + split: test + q: "How does JPEG make a photo about ten times smaller?" + misconception: "JPEG makes a photo smaller by lowering its resolution (removing pixels)." + quiz: + - q: "What does JPEG do to each small block of the image before it discards anything?" + expect: "It splits the image into 8×8 blocks and transforms each (discrete cosine transform) into frequency components: smooth average and gradual changes, then finer detail." + - q: "Which step loses information, and why do people barely notice?" + expect: "Quantization: it rounds the frequency components, coarsely for high frequencies; the eye is less sensitive to fine detail (and to color detail, which is also subsampled)." + - q: "Why does a heavily compressed JPEG show blocks and ringing near sharp edges?" + expect: "Each 8×8 block is quantized on its own, so neighboring blocks stop matching; sharp edges need high frequencies, which were discarded." + - q: "At the same quality setting, which compresses better as JPEG: a photo of a clear sky or a scan of printed text? Why?" + expect: "The sky: smooth areas are mostly low frequencies, which quantize to few nonzero values; text has sharp edges that need high frequencies (and shows artifacts)." + kind: predict + + - id: transistor-switch + split: test + q: "How does a transistor act as a switch in a computer chip?" + misconception: "Current flows into the gate and through the transistor to turn it on." + quiz: + - q: "What turns a MOSFET on?" + expect: "A voltage on the gate; it is insulated, so almost no current flows into it, but its electric field draws charge carriers into a channel between source and drain, which then conducts." + - q: "Why does a CMOS gate use almost no power when its inputs are not changing?" + expect: "Each gate pairs transistors so that one of each pair is off; there is no path from supply to ground. Power is spent mainly on charging and discharging capacitance when it switches." + - q: "How do transistors make a logic gate such as NOT?" + expect: "A p-type transistor to the supply and an n-type to ground share the input: a high input turns on the n-type and pulls the output low; a low input turns on the p-type and pulls the output high." + - q: "What happens to a CMOS chip's power use when its clock frequency doubles, the voltage unchanged?" + expect: "The switching power about doubles: the gates charge and discharge their capacitance twice as often." + kind: predict + + - id: kidney-filter + split: test + q: "How do the kidneys clean the blood without losing what the body needs?" + misconception: "The kidneys pick out waste molecules one by one and leave everything else in the blood." + quiz: + - q: "What happens in the first step of filtering?" + expect: "Blood pressure pushes water and every small molecule through a filter (the glomerulus) into the tubule; blood cells and large proteins stay. About 180 litres a day." + - q: "How does the body keep the water, glucose, and salt?" + expect: "The tubules take back about 99% of the water and nearly all glucose and much salt; about 1.5 litres a day leaves as urine." + - q: "Why does glucose appear in urine in untreated diabetes?" + expect: "The transporters that take glucose back can handle only so much; above a blood level of about 10 mmol/L (180 mg/dL) they are full, and the excess stays in the urine." + - q: "A drug blocks the transporters that take glucose back in the tubules. What happens to the urine and to blood sugar?" + expect: "Glucose stays in the urine, and blood sugar falls (this is how SGLT2-inhibitor drugs work)." + kind: predict + + - id: garbage-collection + split: test + q: "How does a garbage collector know which memory it can free?" + misconception: "A garbage collector frees objects the program will not use again." + quiz: + - q: "Which objects does a tracing collector keep?" + expect: "Objects reachable from the roots (stack variables, globals, registers) by following references; everything unreachable is garbage." + - q: "Why can reference counting alone fail to free memory?" + expect: "Objects in a cycle refer to each other, so their counts never reach zero even when nothing else can reach them." + - q: "How can a program with a garbage collector still leak memory?" + expect: "By keeping a reference to objects it no longer needs (a growing cache or list): they stay reachable, so the collector cannot free them." + - q: "Two objects refer to each other, and nothing else refers to either. Does a tracing collector free them? Does pure reference counting?" + expect: "Tracing: yes, they are unreachable from the roots. Reference counting: no, each keeps the other's count at 1." + kind: predict + + - id: liquid-core + split: test + q: "How do earthquakes show that the Earth has a liquid outer core?" + misconception: "Scientists have drilled to the core or taken samples of it." + quiz: + - q: "What is the difference between P waves and S waves that matters here?" + expect: "P waves are compression waves and travel through solids and liquids; S waves are shear waves and cannot travel through a liquid." + - q: "What do seismometers far from an earthquake record, and what does it show?" + expect: "No direct S waves beyond about 104° from the epicenter (the S-wave shadow zone): a layer deep inside stops them, so it must be liquid." + - q: "Why is there also a zone where P waves are missing?" + expect: "P waves slow down and bend (refract) when they enter the core, so between about 104° and 140° no direct P waves arrive." + - q: "If the outer core were solid, what would change in the records of seismometers on the far side of the Earth?" + expect: "Direct S waves would arrive there too: no S-wave shadow zone." + kind: predict + + - id: backprop + split: test + q: "How does a neural network learn from its errors?" + misconception: "The correct answer is sent backwards through the network and each neuron copies it." + quiz: + - q: "What does training change, and in which direction?" + expect: "The weights; each is moved a small step against the gradient of the loss (the error measure), the direction that lowers the loss fastest." + - q: "Why is backpropagation efficient, compared with nudging each weight to see what happens?" + expect: "The chain rule, applied from the output back, reuses each layer's derivatives, so all gradients cost about as much as one or two forward passes, not one pass per weight." + - q: "What goes wrong when the learning rate is too large, and too small?" + expect: "Too large: steps overshoot and the loss jumps around or diverges. Too small: learning is very slow." + - q: "A weight's gradient is zero. What happens to it in one training step, and why?" + expect: "It does not change: a small change to it does not change the loss (to first order), so the step is zero." + kind: predict + + - id: mountain-boiling + split: test + q: "Why does water boil at a lower temperature high on a mountain?" + misconception: "Water up there boils at a lower temperature because it is colder, so food cooks just as fast." + quiz: + - q: "What condition sets the temperature at which water boils?" + expect: "Water boils when its vapor pressure equals the pressure of the air above it." + - q: "About how much lower is the boiling point at 3,000 m, and why?" + expect: "About 90 °C instead of 100 °C: air pressure there is about 70% of sea level, and water reaches that vapor pressure at a lower temperature." + - q: "How does a pressure cooker change this?" + expect: "It raises the pressure (about 1 atmosphere extra), so water boils at about 120 °C, and food cooks faster." + - q: "At the top of Everest the air pressure is about a third of sea level's. Will an egg in boiling water cook faster or slower than at sea level? Why?" + expect: "Slower: water boils there at about 70 °C, so the egg sits in cooler water." + kind: predict + + - id: ssd-wear + split: test + q: "How does an SSD store data, and why does it wear out?" + misconception: "SSDs wear out when you read data from them." + quiz: + - q: "How does a flash memory cell store a bit?" + expect: "Charge trapped in an insulated layer (a floating gate or charge trap) shifts the voltage at which the cell's transistor turns on; reading senses that voltage." + - q: "What wears a cell out?" + expect: "Programming and erasing push electrons through a thin insulating oxide; each cycle damages it a little, so a cell lasts a limited number of program/erase cycles (about 1,000–3,000 for TLC). Reading does not wear it." + - q: "Why does an SSD spread writes across the whole drive?" + expect: "Data is written in pages but erased only in large blocks, and each block wears; wear leveling spreads erases so no block wears out early." + - q: "An SSD holds photos that are written once and read every day for years. Will it wear out from this use? Why?" + expect: "No: reading does not wear the cells; wear comes from program/erase cycles, and these photos use very few." + kind: predict + + - id: rent-cap + split: test + q: "Why do economists say that rent caps cause housing shortages, and how sure are they?" + misconception: "A rent cap lowers rents across the whole market with no side effects." + quiz: + - q: "What is the basic mechanism?" + expect: "A price held below the market level raises the quantity people want and lowers the quantity offered; the gap is a shortage." + - q: "Why does the effect grow over time?" + expect: "In the short run the housing stock is fixed; over years, landlords convert, sell, or stop maintaining units, and less is built, so supply falls more." + - q: "Who gains and who loses?" + expect: "Tenants already in capped units gain; people looking for a place lose (fewer vacancies, longer searches, often higher uncapped rents). The size of the effects is measured differently by different studies." + - q: "A city caps rents in existing buildings but exempts new construction. Which part of the mechanism does that weaken, and which part remains?" + expect: "It weakens the effect on new building (construction is not discouraged); the shortage in existing capped units, conversions, and lower mobility remain." + kind: predict + + - id: pagerank + split: test + q: "How does PageRank decide which web pages are important?" + misconception: "PageRank counts the links to a page; more incoming links always means a higher rank." + quiz: + - q: "What makes a page important under PageRank?" + expect: "Links from important pages: each page shares its rank among the pages it links to, so a link from an important page with few links counts most." + - q: "What is the 'random surfer' picture?" + expect: "A page's rank is the long-run fraction of time a surfer spends there who follows random links and, with probability about 0.15, jumps to a random page." + - q: "Why is the random jump needed?" + expect: "Without it, pages with no outgoing links and closed loops trap the surfer (rank sinks), and the ranks do not settle to one well-defined answer." + - q: "Page X gets one link from a page with 1,000 outgoing links; page Y gets one link from a page of equal rank with 2 outgoing links. Which gains more rank from its link?" + expect: "Y: a page shares its rank among its outgoing links, so Y gets half of that page's share and X a thousandth." + kind: predict + + - id: muscle-contraction + split: test + q: "How does a muscle contract?" + misconception: "The protein filaments in the muscle get shorter." + quiz: + - q: "What moves inside a muscle fiber when it contracts?" + expect: "Myosin heads bind actin filaments and pull them past the myosin filaments (sliding filaments); the filaments keep their length, the overlap grows." + - q: "What role does ATP play?" + expect: "ATP binding releases the myosin head from actin; splitting ATP re-cocks the head for the next stroke." + - q: "What starts and stops a contraction?" + expect: "A nerve signal releases calcium inside the fiber; calcium moves the blocking proteins (troponin, tropomyosin) off actin's binding sites. When calcium is pumped away, the sites are covered again." + - q: "After death, cells stop making ATP. What happens to the muscles, and why?" + expect: "They lock stiff (rigor mortis): without ATP the myosin heads cannot release from actin." + kind: predict + + - id: noise-cancelling + split: test + q: "How do noise-cancelling headphones cancel sound?" + misconception: "They cancel noise by sealing or absorbing the sound, like earplugs." + quiz: + - q: "What does the headphone do with the sound from its microphone?" + expect: "It plays the same sound with the pressure inverted (opposite phase), so the two waves add to nearly zero at the ear (destructive interference)." + - q: "Why does it work better on a steady low hum than on voices or high sounds?" + expect: "Any delay shifts the anti-noise wave; a fixed delay is a small fraction of a long low-frequency period but a large fraction of a short high-frequency one. High sounds are left to the ear cups' passive isolation." + - q: "What does a 0.1 ms delay do at 100 Hz and at 5 kHz?" + expect: "100 Hz: the period is 10 ms, so the shift is 1% of a cycle and the cancellation still works. 5 kHz: the period is 0.2 ms, so the shift is half a cycle and the anti-noise adds to the sound instead." + - q: "If the headphone played the microphone's sound without inverting it, what would the listener hear?" + expect: "The noise louder, about twice the amplitude: the waves add in step instead of cancelling." + kind: predict + + - id: sort-bound + split: test + q: "Why can no comparison sort sort n items in fewer than about n log n comparisons?" + misconception: "No sorting algorithm of any kind can be faster than n log n." + quiz: + - q: "Why does the number of possible orderings matter?" + expect: "The input can be any of n! orderings, and each needs a different sequence of moves, so the algorithm must tell all n! apart." + - q: "Why does that give about n log n comparisons?" + expect: "Each comparison has two outcomes, so k comparisons separate at most 2^k cases; 2^k ≥ n! needs k ≥ log2(n!) ≈ n log2 n." + - q: "How can counting sort be faster?" + expect: "It does not compare items; it uses each key as an index into counts, which works when keys are small integers in a known range (time about n + range)." + - q: "What is the smallest number of comparisons that always suffices to sort 3 items, and why can 2 not be enough?" + expect: "3: there are 3! = 6 orderings; 2 comparisons separate at most 4, and 3 separate up to 8." + kind: predict + + - id: lightning + split: test + q: "What causes lightning, and why do you hear thunder after the flash?" + misconception: "Lightning never strikes the same place twice." + quiz: + - q: "How does a thundercloud become charged?" + expect: "In the updrafts, collisions between small ice crystals and soft hail (graupel) move charge between them; the light crystals carry one charge up, the heavier graupel carries the other lower, so the cloud's top and middle get opposite charges." + - q: "What makes thunder?" + expect: "The stroke heats the air in its channel to about 30,000 K in microseconds; the air expands violently and makes a shock wave that becomes sound." + - q: "Why is the thunder late, and how can you use that?" + expect: "Light arrives almost at once; sound travels about 340 m/s, so about 3 s per kilometer: count the seconds and divide by 3 for the distance in km." + - q: "You see a flash and hear its thunder 9 seconds later. About how far away was the stroke?" + expect: "About 3 km (9 s ÷ 3 s per km)." + kind: predict + + - id: double-spend + split: test + q: "How does Bitcoin stop someone from spending the same coin twice without a bank?" + misconception: "A central server checks each transaction against the account balances." + quiz: + - q: "Where are the transactions recorded, and who checks them?" + expect: "In a public ledger of blocks, each holding the hash of the one before; every node checks that the coins spent are unspent in that chain." + - q: "What makes it hard to rewrite the ledger to undo a payment?" + expect: "Proof of work: each block needs a costly hash search, so an attacker must redo the work of that block and every later one, faster than the rest of the network, which needs most of the network's hashing power." + - q: "Why does a merchant wait for several confirmations?" + expect: "Each later block adds work an attacker must redo; the chance a minority attacker catches up falls exponentially with the number of blocks (six is a common convention)." + - q: "An attacker controls 60% of the network's hashing power. Do six confirmations protect a merchant? Why?" + expect: "No: with most of the hashing power the attacker's chain grows faster and eventually overtakes the honest one, however many blocks it starts behind." + kind: predict + + - id: screen-yellow + split: test + q: "How can a screen show yellow when it emits only red, green, and blue light?" + misconception: "Red and green light mix to brown, as paints do, so the screen must emit yellow light." + quiz: + - q: "What does the eye actually measure about color?" + expect: "Three kinds of cone respond to overlapping ranges of wavelength; the brain sees color from the ratio of their three responses, not from the wavelength itself." + - q: "Why does red plus green light look yellow?" + expect: "Together they stimulate the long- and medium-wavelength cones in about the same ratio as spectral yellow light (about 580 nm) does, so the two are indistinguishable (metamers)." + - q: "Why can a screen not show every color the eye can see?" + expect: "The cones' sensitivities overlap, so no light stimulates one cone type alone; mixes of three primaries cover only a triangle (gamut) inside the colors the eye can see, missing the most saturated ones." + - q: "A spectrometer measures a screen showing yellow and a sodium lamp (light near 589 nm). Do the spectra match? Why do the two look alike?" + expect: "No: the screen shows separate red and green peaks, the lamp one line near 589 nm; both give the cones about the same ratio of responses." + kind: predict + + - id: https-identity + split: test + q: "How does HTTPS prove that a website is the one it claims to be?" + misconception: "The padlock means the website is safe and trustworthy." + quiz: + - q: "What is a certificate?" + expect: "A document that binds a domain name to a public key, signed by a certificate authority." + - q: "Why does the browser trust the certificate authority's signature?" + expect: "It checks a chain of signatures up to a root certificate already installed in the browser or operating system." + - q: "Why can an attacker not just copy a site's certificate?" + expect: "The certificate is public; during the handshake the server must prove it has the matching private key (by signing), which the attacker does not have." + - q: "A phishing site has a valid certificate and a padlock. How is that possible?" + expect: "A certificate proves control of a domain name, not honesty; the attacker gets a certificate for their own look-alike domain." + kind: predict + + - id: skater-spin + split: test + q: "Why does a spinning skater speed up when pulling the arms in?" + misconception: "The skater speeds up because pulling the arms in reduces air resistance." + quiz: + - q: "What stays the same as the skater pulls the arms in?" + expect: "Angular momentum, L = I × ω (moment of inertia times spin rate), because nothing outside applies a torque (friction aside)." + - q: "Why does the spin rate go up, and by how much if the moment of inertia halves?" + expect: "Moment of inertia depends on mass × distance² from the axis, so pulling mass in lowers I; with L fixed, ω rises: half the I gives twice the spin rate." + - q: "Where does the extra kinetic energy come from?" + expect: "The muscles do work pulling the arms inward against their tendency to fly out; kinetic energy (½ I ω²) rises, here doubling." + - q: "A skater spinning at 2 turns per second pulls in until the moment of inertia is a third of what it was. What is the new spin rate?" + expect: "6 turns per second: L = Iω is constant, so ω triples." + kind: predict + + - id: bond-prices + split: test + q: "Why do bond prices fall when interest rates rise?" + misconception: "Bond prices fall because the issuer becomes less likely to pay." + quiz: + - q: "Why does an existing bond lose value when rates rise?" + expect: "Its payments are fixed; new bonds pay the higher rate, so the old bond sells only at a lower price, low enough that its return matches the new rate (its future payments are discounted at the higher rate)." + - q: "Why does a long bond fall more than a short one?" + expect: "More of its payments lie far in the future, and discounting at a higher rate shrinks distant payments more (higher duration); a 10-year 2% bond falls about 9% when rates go from 2% to 3%." + - q: "If you hold the bond to maturity, what do you lose?" + expect: "You still get every coupon and the face value; the loss is the market price if you sell early, and the higher return you could have had." + - q: "Rates rise by 1 point. Which falls more in price: a 2-year bond or a 30-year bond? Why?" + expect: "The 30-year bond: more of its payments are far in the future, where a higher discount rate shrinks them most." + kind: predict + + - id: cache-order + split: test + q: "Why does a loop over a 2D array run much faster row by row than column by column?" + misconception: "The two loops do different amounts of work." + quiz: + - q: "How is a 2D array laid out in memory in C or NumPy by default?" + expect: "Row-major: each row is stored contiguously, and the next row follows it." + - q: "Why does the order of access matter to the CPU?" + expect: "Memory moves into the cache in lines of about 64 bytes; row order uses every value in a line (8 doubles), while column order jumps a full row ahead each step and uses one value per line loaded." + - q: "How large is the cost difference between a cache hit and a trip to main memory?" + expect: "About 1 ns for L1 cache versus about 100 ns for main memory, so missing the cache on each access can make the loop many times slower." + - q: "The array is stored column-major instead (Fortran, or NumPy with order='F'). Which loop order is fast now? Why?" + expect: "Column by column: now each column is contiguous, so that order uses every value in each cache line." + kind: predict diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/conditions.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/conditions.json new file mode 100644 index 0000000..963c67d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/conditions.json @@ -0,0 +1,6 @@ +{ + "none": "none", + "principles": "principles", + "quick": "quick", + "published": "published" +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/antibiotic-resistance.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/antibiotic-resistance.json new file mode 100644 index 0000000..42e7806 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/antibiotic-resistance.json @@ -0,0 +1,63 @@ +{ + "key": { + "A": "published", + "B": "principles", + "C": "quick", + "D": "none" + }, + "grades": { + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "Expected resistant cells: 10¹⁰ × 1/10⁸ = **about 100**, present *before* the first dose. (This is the expected number of mutation events. Early mutants multiply, so the expected number of resistant cells is larger.)" + ] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "Expected resistant cells before any drug: 10^10 × 10^-9 = 10. (This counts mutation events, not resistant cells. Early mutants leave many descendants.)", + "Then every cell would have the same chance to survive. Instead, a rare fraction survives, and those cells were resistant before they met the drug. (A small uniform chance also predicts a rare surviving fraction. Luria-Delbrück's evidence was the variance between cultures, not the rarity of survivors.)", + "Change the target | The target protein changes shape, so the drug cannot bind | MRSA makes a cell-wall enzyme that penicillin-type drugs do not bind (MRSA acquires an extra enzyme, PBP2a, that beta-lactams bind poorly. No existing target changes shape.)" + ] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "published", + "principles", + "none", + "quick" + ], + "why_best": "A gives the reader every step in a short, error-free chain (pre-existing variation with worked numbers, selection with a regrowth calculation, a quantitative Luria-Delbrück refutation of the adaptation idea, and horizontal transfer across species) and defines each term once, without detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/bloom-filter.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/bloom-filter.json new file mode 100644 index 0000000..74eb256 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/bloom-filter.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "published", + "D": "principles" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "quick", + "published", + "principles", + "none" + ], + "why_best": "A gives every needed step in the fewest words: add and query on a tiny worked array, why a 'no' is certain, why deletion breaks that guarantee, and a concrete false positive, with correct numbers and no detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/float-sum.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/float-sum.json new file mode 100644 index 0000000..c8bbefa --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/float-sum.json @@ -0,0 +1,61 @@ +{ + "key": { + "A": "principles", + "B": "none", + "C": "quick", + "D": "published" + }, + "grades": { + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "The digits after the point (the *significand*) get a fixed budget: **53 significant binary digits**." + ] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "A double is a whole number of at most 53 binary digits, times a power of two. The other 11 bits hold the sign and the power." + ] + } + }, + "rank": [ + "principles", + "quick", + "none", + "published" + ], + "why_best": "A gives every step the reader needs: why 0.1 repeats in binary, the stored values, the tie-rounded sum, why it prints as 0.30000000000000004, and the exact power-of-2 case, all with exact grid arithmetic and without B's detours or D's dense unit bookkeeping." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/gps-relativity.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/gps-relativity.json new file mode 100644 index 0000000..e0e6b74 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/gps-relativity.json @@ -0,0 +1,59 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "principles", + "D": "published" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "Three spheres meet at a point. So three satellites should be enough." + ] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "published", + "principles", + "none", + "quick" + ], + "why_best": "D states that the receiver only listens and uses a small worked example on a line to make the shared clock error b concrete; it then shows why per-satellite clock errors cannot be absorbed, all in few words and without detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/greenhouse.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/greenhouse.json new file mode 100644 index 0000000..8354bc5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/greenhouse.json @@ -0,0 +1,62 @@ +{ + "key": { + "A": "none", + "B": "published", + "C": "principles", + "D": "quick" + }, + "grades": { + "none": { + "quiz": [ + 1, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 1, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 1, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "Above about 12 km, ozone absorbs sunlight and temperature *rises* with height." + ] + }, + "quick": { + "quiz": [ + 1, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "The surface, 5 km of lapse rate below, warms by the same amount: about 1.2 K before feedbacks. (contradicts its own chain of a 150 m rise × 6.5 K/km = 1 K)", + "\"240 W/m² absorbed\" by the Surface in the flowchart (the atmosphere absorbs part of the sunlight; the surface absorbs much less than 240 W/m²)" + ] + } + }, + "rank": [ + "principles", + "none", + "quick", + "published" + ], + "why_best": "C builds each step of the mechanism in order (energy balance, wavelength split, dipole reason, why the one-layer model fails, emission height and lapse rate, the isothermal test case), with worked numbers and prediction prompts, and adds nothing the reader does not need, while A covers the same ground at much greater length, D contradicts its own numbers, and B skips why N2 and O2 do not absorb." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/heat-pump.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/heat-pump.json new file mode 100644 index 0000000..a37e208 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/heat-pump.json @@ -0,0 +1,60 @@ +{ + "key": { + "A": "published", + "B": "principles", + "C": "quick", + "D": "none" + }, + "grades": { + "published": { + "quiz": [ + 2, + 2, + 2, + 1 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 1 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 1 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 1 + ], + "misconception": "corrected", + "errors": [ + "Air at 0 °C is at 273 K, which is 273 degrees above absolute zero, where molecular motion stops. (Motion does not stop at absolute zero: zero-point motion remains.)", + "The loop diagram labels the compressor-to-condenser line 'warm, high-pressure liquid' and the line into the expansion valve 'hot, high-pressure gas'. These labels are swapped: hot gas leaves the compressor, and warm liquid enters the valve." + ] + } + }, + "rank": [ + "principles", + "published", + "quick", + "none" + ], + "why_best": "B gives the energy balance, the pressure-dependent boiling trick, the four-step loop with numbers, and a short entropy derivation of the Carnot limit with a mild-versus-cold table, all without errors, detours or repetition." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/key-exchange.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/key-exchange.json new file mode 100644 index 0000000..426f15f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/key-exchange.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "none", + "B": "quick", + "C": "principles", + "D": "published" + }, + "grades": { + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + } + }, + "rank": [ + "quick", + "principles", + "published", + "none" + ], + "why_best": "B covers every quiz point, including the operation, why both keys match, the worked p=23 example, the man-in-the-middle attack with its fix, and why powers mod p are needed, in the fewest words with no detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/quorum.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/quorum.json new file mode 100644 index 0000000..581c0ad --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/quorum.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "published", + "D": "principles" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 1, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "principles", + "quick", + "published", + "none" + ], + "why_best": "D shows the simple rule failing with a concrete split-brain example, derives q > n/2 from the overlap requirement in one step, and refutes the 'more servers is always better' idea with a checked availability table, so the reader gets every needed step without detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/two-tides.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/two-tides.json new file mode 100644 index 0000000..eb4651f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/two-tides.json @@ -0,0 +1,60 @@ +{ + "key": { + "A": "principles", + "B": "none", + "C": "published", + "D": "quick" + }, + "grades": { + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 0, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 0, + 2 + ], + "misconception": "absent", + "errors": [ + "3.433e-5 (near-side pull; 4.90e12/(3.78029e8 m)² ≈ 3.432e-5)", + "3.213e-5 (far-side pull; ≈ 3.212e-5), both minor rounding slips" + ] + } + }, + "rank": [ + "principles", + "none", + "published", + "quick" + ], + "why_best": "A gives the whole causal chain (free fall, subtracting the centre's pull with worked numbers, Earth turning under two bulges, the lunar-day timing, and Sun–Moon spring/neap tides) in a compact form, and it explicitly refutes the spin-centrifugal misconception without the length and stray interactive-page residue of B." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/vaccine-memory.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/vaccine-memory.json new file mode 100644 index 0000000..b037bd5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/grades/vaccine-memory.json @@ -0,0 +1,64 @@ +{ + "key": { + "A": "none", + "B": "principles", + "C": "published", + "D": "quick" + }, + "grades": { + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "\"the library contains millions of different shapes\": this understates the repertoire, which is commonly estimated at 10^8 to 10^10 distinct receptors per person.", + "\"Across the body's hundreds of billions of lymphocytes\": the usual estimate is about 10^12 lymphocytes, so this is low." + ] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "\"The cause is a changed antigen, not memory cells that wear off.\": this overstates the case for influenza, where waning antibody also lowers protection within and across seasons." + ] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "\"Many have a better fit, because the clone mutates its receptors during expansion\": this is applied to all lymphocytes, but only B cells undergo somatic hypermutation and affinity maturation; T cell receptors do not mutate." + ] + } + }, + "rank": [ + "quick", + "published", + "principles", + "none" + ], + "why_best": "D gives every link in the chain (random repertoire, selection, expansion, memory, race, and limits from antigen change) in the fewest words, with defined terms and a single comparison table, so this reader gets the full model with no detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/report.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/report.md new file mode 100644 index 0000000..3da646f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/report.md @@ -0,0 +1,44 @@ +# Artifact eval, 2026-10-04 (dev, 10 subjects) + +Builder, reader, and grader: fresh `claude -p` agents (reader: --model haiku). The reader saw only the renderings; an answer counts only when its quote is found in them (grounded). The grader saw each subject's explanations under shuffled labels. + +Conditions: **none** = `none`; **principles** = `principles`; **quick** = `quick`; **published** = `published` + +| condition | mean rank | ranked best | quiz (grounded) | predict | quiz (raw) | ungrounded | misconception taught | corrected | errors | stops / expl. | excess / expl. | words | forms | cost / expl. | min / expl. | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| none | 3.3 | 0/10 | 95% | 85% | 98% | 1 | 0 | 7 | 3 | 9.0 | 9.2 | 1689 | prose 5, prose+html 4, html 1 | $0.28 | 1.5 | +| principles | 1.7 | 5/10 | 98% | 95% | 98% | 0 | 0 | 7 | 4 | 9.2 | 8.5 | 1006 | prose 10 | $0.21 | 1.0 | +| quick | 2.5 | 3/10 | 94% | 95% | 94% | 0 | 0 | 5 | 9 | 7.9 | 6.9 | 606 | prose 8, diagram 2 | $0.46 | 1.6 | +| published | 2.5 | 2/10 | 95% | 95% | 95% | 0 | 0 | 6 | 2 | 8.5 | 8.5 | 1016 | prose 9, diagram 1 | $2.81 | 11.8 | + +Mean rank: the grader's ranking of each subject's 4 explanations by the reader's effort (1 is best). Quiz: fixed questions per subject, 0–2 each; predict: the last question of each, a case the explanation need not mention. Stops and excess: the reader's lists (at most 10 each). Words: visible words in all renderings. Cost: the builder and the reviews it ran; minutes: the builder. + +## Per subject (grounded quiz score, of 8) + +| subject | none | principles | quick | published | +|---|---|---|---|---| +| gps-relativity | 8 (1500 w) | 8 (1158 w) | 8 (676 w) | 8 (999 w) | +| vaccine-memory | 8 (1953 w) | 8 (1067 w) | 8 (600 w) | 8 (992 w) | +| key-exchange | 8 (1822 w) | 8 (931 w) | 8 (506 w) | 8 (1086 w) | +| two-tides | 8 (1929 w) | 8 (884 w) | 6 (599 w) | 6 (948 w) | +| bloom-filter | 8 (1439 w) | 8 (1083 w) | 8 (599 w) | 8 (1059 w) | +| heat-pump | 7 (1339 w) | 7 (784 w) | 7 (814 w) | 7 (1437 w) | +| quorum | 6 (1072 w) | 8 (777 w) | 7 (502 w) | 8 (896 w) | +| antibiotic-resistance | 8 (2511 w) | 8 (1194 w) | 8 (600 w) | 8 (949 w) | +| float-sum | 8 (1056 w) | 8 (661 w) | 8 (412 w) | 8 (721 w) | +| greenhouse | 7 (2271 w) | 7 (1522 w) | 7 (749 w) | 7 (1072 w) | + +✗: the explanation teaches the misconception. + +## Why the best is best + +- **gps-relativity**: published > principles > none > quick. D states that the receiver only listens and uses a small worked example on a line to make the shared clock error b concrete; it then shows why per-satellite clock errors cannot be absorbed, all in few words and without detours. +- **vaccine-memory**: quick > published > principles > none. D gives every link in the chain (random repertoire, selection, expansion, memory, race, and limits from antigen change) in the fewest words, with defined terms and a single comparison table, so this reader gets the full model with no detours. +- **key-exchange**: quick > principles > published > none. B covers every quiz point, including the operation, why both keys match, the worked p=23 example, the man-in-the-middle attack with its fix, and why powers mod p are needed, in the fewest words with no detours. +- **two-tides**: principles > none > published > quick. A gives the whole causal chain (free fall, subtracting the centre's pull with worked numbers, Earth turning under two bulges, the lunar-day timing, and Sun–Moon spring/neap tides) in a compact form, and it explicitly refutes the spin-centrifugal misconception without the length and stray interactive-page residue of B. +- **bloom-filter**: quick > published > principles > none. A gives every needed step in the fewest words: add and query on a tiny worked array, why a 'no' is certain, why deletion breaks that guarantee, and a concrete false positive, with correct numbers and no detours. +- **heat-pump**: principles > published > quick > none. B gives the energy balance, the pressure-dependent boiling trick, the four-step loop with numbers, and a short entropy derivation of the Carnot limit with a mild-versus-cold table, all without errors, detours or repetition. +- **quorum**: principles > quick > published > none. D shows the simple rule failing with a concrete split-brain example, derives q > n/2 from the overlap requirement in one step, and refutes the 'more servers is always better' idea with a checked availability table, so the reader gets every needed step without detours. +- **antibiotic-resistance**: published > principles > none > quick. A gives the reader every step in a short, error-free chain (pre-existing variation with worked numbers, selection with a regrowth calculation, a quantitative Luria-Delbrück refutation of the adaptation idea, and horizontal transfer across species) and defines each term once, without detours. +- **float-sum**: principles > quick > none > published. A gives every step the reader needs: why 0.1 repeats in binary, the stored values, the tie-rounded sum, why it prints as 0.30000000000000004, and the exact power-of-2 case, all with exact grid arithmetic and without B's detours or D's dense unit bookkeeping. +- **greenhouse**: principles > none > quick > published. C builds each step of the mechanism in order (energy balance, wavelength split, dipole reason, why the one-layer model fails, emission height and lapse rate, the isothermal test case), with worked numbers and prediction prompts, and adds nothing the reader does not need, while A covers the same ground at much greater length, D contradicts its own numbers, and B skips why N2 and O2 do not absorb. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/scores.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/scores.json new file mode 100644 index 0000000..efcb2df --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/scores.json @@ -0,0 +1,1492 @@ +{ + "none": { + "subjects": [ + { + "id": "gps-relativity", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 8, + "excess": 7, + "words": 1500, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.17838859999999995, + "seconds": 57, + "failed": false + }, + { + "id": "vaccine-memory", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 10, + "excess": 10, + "words": 1953, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.16533019999999998, + "seconds": 62, + "failed": false + }, + { + "id": "key-exchange", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 10, + "words": 1822, + "forms": [ + "prose", + "html" + ], + "rank": 4, + "cost": 0.3186072, + "seconds": 96, + "failed": false + }, + { + "id": "two-tides", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 10, + "excess": 10, + "words": 1929, + "forms": [ + "prose", + "html" + ], + "rank": 2, + "cost": 0.4116578, + "seconds": 131, + "failed": false + }, + { + "id": "bloom-filter", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 10, + "words": 1439, + "forms": [ + "html" + ], + "rank": 4, + "cost": 0.40253520000000004, + "seconds": 118, + "failed": false + }, + { + "id": "heat-pump", + "raw": [ + 2, + 2, + 2, + 1 + ], + "score": [ + 2, + 2, + 2, + 1 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [ + "Air at 0 °C is at 273 K, which is 273 degrees above absolute zero, where molecular motion stops. (Motion does not stop at absolute zero: zero-point motion remains.)", + "The loop diagram labels the compressor-to-condenser line 'warm, high-pressure liquid' and the line into the expansion valve 'hot, high-pressure gas'. These labels are swapped: hot gas leaves the compressor, and warm liquid enters the valve." + ], + "stops": 10, + "excess": 10, + "words": 1339, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.1563, + "seconds": 49, + "failed": false + }, + { + "id": "quorum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 0 + ], + "grounded": [ + true, + true, + true, + false + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 5, + "excess": 5, + "words": 1072, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.0993212, + "seconds": 32, + "failed": false + }, + { + "id": "antibiotic-resistance", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 9, + "excess": 10, + "words": 2511, + "forms": [ + "prose", + "html" + ], + "rank": 3, + "cost": 0.502619, + "seconds": 162, + "failed": false + }, + { + "id": "float-sum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [ + "The digits after the point (the *significand*) get a fixed budget: **53 significant binary digits**." + ], + "stops": 8, + "excess": 10, + "words": 1056, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.1607036, + "seconds": 49, + "failed": false + }, + { + "id": "greenhouse", + "raw": [ + 1, + 2, + 2, + 2 + ], + "score": [ + 1, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 10, + "words": 2271, + "forms": [ + "prose", + "html" + ], + "rank": 2, + "cost": 0.3816954, + "seconds": 124, + "failed": false + } + ] + }, + "principles": { + "subjects": [ + { + "id": "gps-relativity", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 10, + "words": 1158, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.28801780000000005, + "seconds": 89, + "failed": false + }, + { + "id": "vaccine-memory", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "\"the library contains millions of different shapes\": this understates the repertoire, which is commonly estimated at 10^8 to 10^10 distinct receptors per person.", + "\"Across the body's hundreds of billions of lymphocytes\": the usual estimate is about 10^12 lymphocytes, so this is low." + ], + "stops": 8, + "excess": 8, + "words": 1067, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.2302082, + "seconds": 68, + "failed": false + }, + { + "id": "key-exchange", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 8, + "excess": 10, + "words": 931, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.16838920000000002, + "seconds": 43, + "failed": false + }, + { + "id": "two-tides", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 10, + "words": 884, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.1563002, + "seconds": 42, + "failed": false + }, + { + "id": "bloom-filter", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 10, + "words": 1083, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.1830298, + "seconds": 48, + "failed": false + }, + { + "id": "heat-pump", + "raw": [ + 2, + 2, + 2, + 1 + ], + "score": [ + 2, + 2, + 2, + 1 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 8, + "words": 784, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.2534982, + "seconds": 67, + "failed": false + }, + { + "id": "quorum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 5, + "words": 777, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.18852040000000003, + "seconds": 52, + "failed": false + }, + { + "id": "antibiotic-resistance", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [ + "Expected resistant cells: 10¹⁰ × 1/10⁸ = **about 100**, present *before* the first dose. (This is the expected number of mutation events. Early mutants multiply, so the expected number of resistant cells is larger.)" + ], + "stops": 10, + "excess": 8, + "words": 1194, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.1721862, + "seconds": 49, + "failed": false + }, + { + "id": "float-sum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 8, + "words": 661, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.2104636, + "seconds": 55, + "failed": false + }, + { + "id": "greenhouse", + "raw": [ + 1, + 2, + 2, + 2 + ], + "score": [ + 1, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "Above about 12 km, ozone absorbs sunlight and temperature *rises* with height." + ], + "stops": 10, + "excess": 8, + "words": 1522, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.29882359999999997, + "seconds": 89, + "failed": false + } + ] + }, + "quick": { + "subjects": [ + { + "id": "gps-relativity", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "Three spheres meet at a point. So three satellites should be enough." + ], + "stops": 8, + "excess": 6, + "words": 676, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.44009420000000005, + "seconds": 91, + "failed": false + }, + { + "id": "vaccine-memory", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "\"Many have a better fit, because the clone mutates its receptors during expansion\": this is applied to all lymphocytes, but only B cells undergo somatic hypermutation and affinity maturation; T cell receptors do not mutate." + ], + "stops": 10, + "excess": 10, + "words": 600, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.4694588000000001, + "seconds": 94, + "failed": false + }, + { + "id": "key-exchange", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 6, + "excess": 7, + "words": 506, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.36011959999999993, + "seconds": 80, + "failed": false + }, + { + "id": "two-tides", + "raw": [ + 2, + 2, + 0, + 2 + ], + "score": [ + 2, + 2, + 0, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "3.433e-5 (near-side pull; 4.90e12/(3.78029e8 m)² ≈ 3.432e-5)", + "3.213e-5 (far-side pull; ≈ 3.212e-5), both minor rounding slips" + ], + "stops": 8, + "excess": 8, + "words": 599, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.5251108, + "seconds": 114, + "failed": false + }, + { + "id": "bloom-filter", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 7, + "excess": 6, + "words": 599, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.5246917999999999, + "seconds": 99, + "failed": false + }, + { + "id": "heat-pump", + "raw": [ + 2, + 2, + 2, + 1 + ], + "score": [ + 2, + 2, + 2, + 1 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 4, + "words": 814, + "forms": [ + "diagram" + ], + "rank": 3, + "cost": 0.47279319999999997, + "seconds": 98, + "failed": false + }, + { + "id": "quorum", + "raw": [ + 2, + 1, + 2, + 2 + ], + "score": [ + 2, + 1, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 6, + "excess": 4, + "words": 502, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.41601699999999997, + "seconds": 71, + "failed": false + }, + { + "id": "antibiotic-resistance", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [ + "Expected resistant cells before any drug: 10^10 × 10^-9 = 10. (This counts mutation events, not resistant cells. Early mutants leave many descendants.)", + "Then every cell would have the same chance to survive. Instead, a rare fraction survives, and those cells were resistant before they met the drug. (A small uniform chance also predicts a rare surviving fraction. Luria-Delbrück's evidence was the variance between cultures, not the rarity of survivors.)", + "Change the target | The target protein changes shape, so the drug cannot bind | MRSA makes a cell-wall enzyme that penicillin-type drugs do not bind (MRSA acquires an extra enzyme, PBP2a, that beta-lactams bind poorly. No existing target changes shape.)" + ], + "stops": 8, + "excess": 10, + "words": 600, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.430636, + "seconds": 104, + "failed": false + }, + { + "id": "float-sum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 6, + "excess": 4, + "words": 412, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.4691284, + "seconds": 83, + "failed": false + }, + { + "id": "greenhouse", + "raw": [ + 1, + 2, + 2, + 2 + ], + "score": [ + 1, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "The surface, 5 km of lapse rate below, warms by the same amount: about 1.2 K before feedbacks. (contradicts its own chain of a 150 m rise × 6.5 K/km = 1 K)", + "\"240 W/m² absorbed\" by the Surface in the flowchart (the atmosphere absorbs part of the sunlight; the surface absorbs much less than 240 W/m²)" + ], + "stops": 10, + "excess": 10, + "words": 749, + "forms": [ + "diagram" + ], + "rank": 3, + "cost": 0.507285, + "seconds": 120, + "failed": false + } + ] + }, + "published": { + "subjects": [ + { + "id": "gps-relativity", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 7, + "excess": 7, + "words": 999, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 2.1912534, + "seconds": 567, + "failed": false + }, + { + "id": "vaccine-memory", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "\"The cause is a changed antigen, not memory cells that wear off.\": this overstates the case for influenza, where waning antibody also lowers protection within and across seasons." + ], + "stops": 10, + "excess": 10, + "words": 992, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 2.1222304000000003, + "seconds": 543, + "failed": false + }, + { + "id": "key-exchange", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 10, + "excess": 9, + "words": 1086, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 4.4839193999999996, + "seconds": 1162, + "failed": false + }, + { + "id": "two-tides", + "raw": [ + 2, + 2, + 0, + 2 + ], + "score": [ + 2, + 2, + 0, + 2 + ], + "grounded": [ + true, + true, + false, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 5, + "excess": 5, + "words": 948, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 2.838019, + "seconds": 705, + "failed": false + }, + { + "id": "bloom-filter", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 7, + "excess": 6, + "words": 1059, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 2.3029966, + "seconds": 553, + "failed": false + }, + { + "id": "heat-pump", + "raw": [ + 2, + 2, + 2, + 1 + ], + "score": [ + 2, + 2, + 2, + 1 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 10, + "words": 1437, + "forms": [ + "diagram" + ], + "rank": 2, + "cost": 3.399423, + "seconds": 845, + "failed": false + }, + { + "id": "quorum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 6, + "excess": 8, + "words": 896, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 2.3876656, + "seconds": 589, + "failed": false + }, + { + "id": "antibiotic-resistance", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 10, + "words": 949, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 2.7889748, + "seconds": 686, + "failed": false + }, + { + "id": "float-sum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "A double is a whole number of at most 53 binary digits, times a power of two. The other 11 bits hold the sign and the power." + ], + "stops": 10, + "excess": 10, + "words": 721, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 2.377155, + "seconds": 607, + "failed": false + }, + { + "id": "greenhouse", + "raw": [ + 2, + 2, + 1, + 2 + ], + "score": [ + 2, + 2, + 1, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 10, + "excess": 10, + "words": 1072, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 3.234709, + "seconds": 804, + "failed": false + } + ] + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/explainers/antibiotic-resistance/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/explainers/antibiotic-resistance/explanation.md new file mode 100644 index 0000000..3b651b3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/explainers/antibiotic-resistance/explanation.md @@ -0,0 +1,139 @@ +# How bacteria become resistant to an antibiotic + +## The short answer + +Individual bacteria don't learn to resist a drug, and they don't adapt because they've been exposed to it. What happens is this: + +1. **Variation already exists.** A large bacterial population almost always contains a few cells that differ genetically from the rest. They differ because of random mutations, or because they picked up genes from other bacteria. A few of those differences happen to make a cell harder to kill with a particular drug. +2. **The antibiotic does the selecting.** The drug kills or stops the susceptible cells. The resistant few survive, and they now have food and space to themselves. +3. **The survivors multiply.** Bacteria divide fast, so within hours or days the population is mostly made of the resistant type. + +This is natural selection, and it runs fast because bacteria have huge populations and short generation times. The rest of this document explains each step, with rough numbers so you can see why it is so reliable. + +--- + +## 1. How antibiotics work, briefly + +An antibiotic is a molecule that interferes with something a bacterium needs and that our own cells either don't have or have in a different form. Common targets: + +| Target | Example drugs | What goes wrong for the bacterium | +|---|---|---| +| Cell wall construction | penicillins, cephalosporins (β-lactams); vancomycin | The wall is built badly, and the cell bursts under its own internal pressure | +| Ribosome (protein synthesis) | tetracyclines, macrolides, aminoglycosides | The cell can't make proteins, or makes faulty ones | +| DNA copying (DNA gyrase/topoisomerase) | fluoroquinolones such as ciprofloxacin | DNA gets tangled and broken | +| RNA polymerase | rifampicin | Genes can't be read | +| Folate synthesis (a metabolic pathway) | sulfonamides, trimethoprim | The cell can't make the building blocks for DNA | + +Every one of these works the same basic way: the drug molecule has to **get into or onto the cell** and then **bind its target tightly enough** to jam it. That gives the bacterium several possible ways to resist. + +## 2. What resistance looks like at the molecular level + +A resistant bacterium has some trait that breaks that chain. There are five main strategies: + +- **Destroy or disable the drug.** Some bacteria make enzymes that chemically inactivate the antibiotic. The best-known are the **β-lactamases**, which cut open the ring structure that penicillin-type drugs need in order to work. Other enzymes stick chemical groups onto aminoglycosides so they no longer bind the ribosome. +- **Change the target.** If the target's shape changes slightly, the drug no longer fits, but the target can still do its normal job. Examples: a single amino-acid change in RNA polymerase gives rifampicin resistance. A few changes in DNA gyrase give fluoroquinolone resistance. MRSA (methicillin-resistant *Staphylococcus aureus*) carries an extra cell-wall-building enzyme, PBP2a, that β-lactams barely bind. +- **Pump the drug out.** **Efflux pumps** are membrane proteins that push molecules out of the cell. If a cell makes more of them, or makes a version that recognises the drug, the drug's concentration inside the cell stays too low to work. +- **Keep the drug out.** Some bacteria (the "Gram-negative" group, such as *E. coli*) have an outer membrane with channels called porins. Losing or narrowing those channels slows the drug's entry. +- **Bypass the blocked step.** A cell can gain an alternative enzyme that does the same job but ignores the drug. Some trimethoprim-resistance genes work this way: they encode a different version of the enzyme the drug inhibits. + +Resistance is usually a **matter of degree**. It is measured as the **MIC (minimum inhibitory concentration)**: the lowest drug concentration that stops the bacteria from growing in a lab test. A resistant strain has a higher MIC, sometimes slightly higher and sometimes thousands of times higher. In a clinic, a strain is labelled "resistant" when its MIC is above what can be safely reached in a patient's body. + +## 3. Where the variation comes from + +### 3a. Mutation (passed from parent to offspring) + +Every time a bacterium copies its DNA, errors creep in. A typical rate is about 1 error per 10⁹–10¹⁰ base pairs copied. That is very accurate, but consider the numbers: + +- A particular resistance-causing change, such as one specific letter in one specific gene, might arise roughly once per **10⁸–10⁹ cell divisions**. The exact figure varies a lot by gene and drug. +- A serious infection can contain **10⁹–10¹¹** bacteria. One millilitre of a dense lab culture has around 10⁹. + +So a large population **very likely already contains** a few resistant mutants before any drug is given. The drug doesn't create them. It reveals them, by removing everyone else. + +> Quick estimate: if 10¹⁰ cells exist and 1 in 10⁸ carries a resistance mutation, about 10¹⁰ / 10⁸ = **100 resistant cells** are present at the start of treatment. + +This was shown directly in classic experiments in the 1940s and 50s (Luria–Delbrück, and the Lederbergs' replica plating). Resistant mutants turned up in cultures that had **never touched** the selecting agent. Their numbers also fluctuated from culture to culture in the way you'd expect if the mutations happened randomly at earlier times, rather than as a response to exposure. + +Some antibiotics, and stress in general, can raise mutation rates somewhat. That speeds things up, but the basic logic of random variation followed by selection is unchanged. + +### 3b. Horizontal gene transfer (getting genes from other bacteria) + +Bacteria can also acquire **ready-made resistance genes** from other bacteria, even from different species. This is a major reason resistance spreads so fast in hospitals. There are three main routes: + +- **Conjugation.** One cell builds a bridge to another and passes over a **plasmid**: a small, circular piece of DNA, separate from the main chromosome, that can copy itself. Plasmids often carry several resistance genes together. +- **Transformation.** Some bacteria can take up loose DNA from their surroundings, for example DNA released by dead cells, and stitch it into their own genome. +- **Transduction.** Viruses that infect bacteria (**bacteriophages**) sometimes accidentally package a piece of bacterial DNA and deliver it to the next cell they infect. + +Within a genome, **transposons** ("jumping genes") and **integrons** (gene-capture systems) let resistance genes move between chromosome and plasmid and gather into clusters. As a result, one transfer event can make a cell resistant to several drug classes at once. + +If you program, here is a loose analogy. Mutation is like random bit-flips in your own source code. Horizontal transfer is like importing a tested library someone else already wrote. + +## 4. Selection: why the resistant cells take over + +With no drug present, resistant and susceptible cells grow at similar rates. Resistant ones are often slightly *slower* (see section 6). Add the drug and the picture flips. + +### A toy calculation + +Suppose: + +- Start of treatment: 10¹⁰ susceptible cells and 100 resistant cells. +- The drug kills susceptible cells quickly, so their numbers fall about 10-fold every hour or two. +- Resistant cells keep dividing every ~30 minutes, which means doubling twice per hour. + +After 10 hours: + +- Susceptible: 10¹⁰ shrinks to roughly 10⁵ or fewer. +- Resistant: 100 × 2²⁰ ≈ 100 × 10⁶ = **10⁸**. + +The population has gone from 1 resistant cell in 10⁸ to almost entirely resistant in less than half a day. It doesn't matter much how rare the resistant cells were to start with. Exponential growth makes up the difference quickly. + +In a real infection, the immune system also kills bacteria and nutrients run out, so growth is slower than this. Often the immune system mops up the small number of resistant survivors. Resistance becomes a clinical problem when that doesn't happen: in severe infections, in people with weakened immune systems, or when resistant bacteria settle in and spread to other people. + +### The "selection window" + +Drug concentration matters, and in a way that isn't obvious. Let MIC_S be the susceptible strain's MIC and MIC_R the resistant strain's MIC. + +- **Below MIC_S:** both types grow. The drug barely affects them, though even low concentrations can give resistant cells a slight edge. +- **Between MIC_S and MIC_R:** susceptible cells die and resistant cells grow. This range is the **mutant selection window**, and it is where resistance is selected most strongly. +- **Above MIC_R:** both types are suppressed. Resistance gains a foothold only if a cell gets a *second*, bigger resistance step. + +Inside a patient, drug levels rise and fall between doses and differ from one tissue to another. So bacteria somewhere often spend time inside the window. A low dose that nevertheless kills the susceptible cells is the textbook recipe for selecting resistance. + +The interactive page (`index.html`) lets you move the drug concentration through these zones and watch the effect. + +## 5. Stepwise resistance + +Often a single mutation gives only a small rise in MIC. Each extra mutation pushes the MIC higher: for example, one change in DNA gyrase, then a second, then a change that switches on an efflux pump. Each step lets the bacteria survive in a slightly higher concentration, where the next mutation can then be selected. Repeated or prolonged treatment can therefore climb a ladder from slightly resistant to highly resistant. + +## 6. Why resistance doesn't simply disappear + +Resistance often has a **fitness cost**. An altered target may work a bit less well, and running pumps or making extra enzymes uses energy. With no drug around, resistant cells may then be slowly outcompeted by susceptible ones. + +In practice, though, resistance tends to stick around: + +- **Compensatory mutations.** Later mutations can restore the lost growth rate *without* removing the resistance, so the cost shrinks to nearly zero. +- **Co-selection.** If a plasmid carries genes for resistance to drug A, drug B and a heavy metal, then using *any* of those keeps the whole plasmid around. +- **Low-cost mechanisms.** Some resistance genes cost very little from the start. +- **Ongoing exposure.** Antibiotics are used widely in medicine and in farming, and they turn up at low levels in the environment, so selection pressure rarely stops completely. + +## 7. Related ideas that are *not* resistance + +- **Tolerance.** The bacteria die more *slowly* under the drug, but the MIC doesn't change. Slow-growing or dormant cells are often tolerant, because many antibiotics attack processes that are only busy in actively growing cells. +- **Persistence.** A small fraction of genetically identical cells happen to be dormant when the drug arrives. They survive, then regrow into a population that is just as susceptible as before. Persisters can lead to infections that keep coming back, and they give resistance mutations more time and more chances to appear. +- **Biofilms.** Communities of bacteria stuck to a surface inside a slimy matrix. Drugs penetrate them poorly and many of the cells grow slowly, so biofilms survive treatment even without resistance genes. + +## 8. What this implies + +The mechanism suggests these strategies: + +- **Use antibiotics only when they're needed.** Every use selects for resistance somewhere, including among the harmless bacteria in your gut, which can later pass genes to harmful ones. +- **Use enough drug, for a suitable time.** The goal is to get above the selection window and clear the infection. The best course length is still being researched. For many infections, shorter courses turn out to work as well as longer ones, and the old blanket rule "always finish the course" is now debated. Follow the prescriber's advice for your situation. +- **Combine drugs, sometimes.** If resistance to drug A arises in roughly 1 in 10⁸ cells, and resistance to drug B in roughly 1 in 10⁸, then a cell resistant to both by chance is about 1 in 10¹⁶, more cells than are in the infection. This is why tuberculosis and HIV are treated with combinations. +- **Stop transmission.** Hand hygiene, infection control and sanitation stop resistant strains spreading from person to person, which matters as much as how the drugs are used. + +## Summary + +- Bacterial populations are huge and varied. Random mutation and gene swapping mean a few cells are already resistant before any drug is given. +- Resistance mechanisms work in one of five ways: break the drug, change the target, pump the drug out, keep the drug out, or bypass the blocked step. +- The antibiotic doesn't cause resistance. It **selects** for it, by removing the competition. Exponential growth then turns a handful of survivors into the whole population. +- Drug concentrations between the susceptible and resistant MICs select resistance most strongly. +- Fitness costs, compensatory mutations and co-selection decide whether resistance fades or persists once the drug is gone. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/explainers/antibiotic-resistance/index.html b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/explainers/antibiotic-resistance/index.html new file mode 100644 index 0000000..7dbab9e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/explainers/antibiotic-resistance/index.html @@ -0,0 +1,375 @@ + + + + + +Antibiotic resistance: a selection simulator + + + + +

Watching resistance get selected

+

A small simulation of one bacterial population with two kinds of cells: susceptible (S) and resistant (R). +At the start there are no resistant cells at all. They appear only through random mutation as susceptible cells divide. Then the antibiotic decides who survives.

+ +
+
+ + +
Shown as a multiple of the susceptible strain's MIC. The resistant strain's MIC is × higher.
+
+
+ + +
The drug starts at hour 8 and is held at a constant level.
+
+
+ + +
Chance that a single cell division produces a resistant cell.
+
+
+ + +
How much slower resistant cells grow when no drug is present.
+
+
+ +
+ + + + + +
+ + +
+ Susceptible cells + Resistant cells + Drug present + Vertical axis is logarithmic: each gridline is 10× more cells. +
+ +

+ +

Why concentration matters: net growth rate vs. drug level

+

This chart shows how fast each type grows (above zero) or dies (below zero) at each drug concentration, ignoring crowding. +The shaded band is the mutant selection window: concentrations high enough to kill susceptible cells but too low to stop resistant ones. +The dashed line marks the concentration you've chosen above.

+ + +

What to try

+
    +
  • No drug: resistant mutants appear but stay a tiny minority. With a fitness cost, they're outcompeted.
    • +
  • In the window: the susceptible population collapses and the resistant line, which started from a handful of cells, climbs to take over. Press "Run again" a few times. When the first mutants appear, and whether they survive, changes from run to run.
    • +
  • Above both MICs: both types die. (Real drugs can't always reach this level safely in every part of the body. Bacteria can also climb out of this regime one mutation step at a time.)
    • +
  • Short treatment in the window: the drug stops before the susceptible cells are gone, and both types regrow. The resistant fraction is now far higher than before treatment.
    • +
  • Lower the mutation rate to 10⁻¹⁰: in many runs, no resistant cell exists when the drug arrives, and treatment succeeds. Resistance is a numbers game: (population size) × (mutation rate).
    • +
+ +
+How the model works +

Time advances in steps of 0.02 hours. Each type i has a birth rate g_i × (1 − N/K), where N is the total population and K = 10¹⁰ is the carrying capacity. It also has a drug-caused death rate

+

kill_i(c) = (g_max − g_min) · x^κ / (x^κ − g_min/g_max),   with x = c / MIC_i

+

This is a standard pharmacodynamic curve, chosen so that net growth is exactly zero at the MIC. The parameters are g_max = 1/hour for S (about one doubling every 42 minutes), g_min = −3/hour, κ = 2. The resistant type's g is reduced by the fitness cost, and its MIC is 64× higher.

+

Births and deaths are drawn at random (Poisson) while a type has fewer than 10,000 cells, so small resistant lineages can die out by chance. Above that size the model switches to the deterministic average. In each step, every susceptible birth has probability μ of producing a resistant cell instead. There is no immune system and no horizontal gene transfer, so this is deliberately the simplest case.

+
+ + + + diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/read.json new file mode 100644 index 0000000..65bb7c0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/read.json @@ -0,0 +1,49 @@ +{ + "answers": { + "1": { + "answer": "Resistance originates from random mutations in bacterial DNA and from horizontal gene transfer (acquiring genes from other bacteria). These create genetic variation so that a large population almost always contains a few cells that happen to be resistant before any drug is used.", + "quote": "A large bacterial population almost always contains a few cells that differ genetically from the rest. They differ because of random mutations, or because they picked up genes from other bacteria." + }, + "2": { + "answer": "The antibiotic kills or stops the susceptible cells, while resistant cells survive and multiply. By removing the competition, the drug allows resistant cells to reproduce exponentially with abundant food and space.", + "quote": "The drug kills or stops the susceptible cells. The resistant few survive, and they now have food and space to themselves." + }, + "3": { + "answer": "Bacteria can acquire ready-made resistance genes from other bacteria through three mechanisms: conjugation (direct DNA transfer via a bridge), transformation (taking up loose DNA), and transduction (viruses accidentally transfer bacterial DNA between cells).", + "quote": "Bacteria can also acquire ready-made resistance genes from other bacteria, even from different species." + }, + "4": { + "answer": "Exponential growth. Resistant cells double roughly every 30 minutes while susceptible cells die within hours. Starting from just 100 resistant cells among 10 billion susceptible ones, exponential growth produces 10 million resistant cells within 10 hours, overwhelming the population.", + "quote": "The population has gone from 1 resistant cell in 10⁸ to almost entirely resistant in less than half a day. Exponential growth makes up the difference quickly." + } + }, + "stops": [ + "β-lactamases, which cut open the ring structure that penicillin-type drugs need", + "Some bacteria (the \"Gram-negative\" group, such as E. coli) have an outer membrane with channels called porins", + "Efflux pumps are membrane proteins that push molecules out of the cell", + "In a clinic, a strain is labelled \"resistant\" when its MIC is above what can be safely reached in a patient's body", + "transposons (\"jumping genes\") and integrons (gene-capture systems) let resistance genes move between chromosome and plasmid", + "Later mutations can restore the lost growth rate without removing the resistance, so the cost shrinks to nearly zero", + "Between MIC_S and MIC_R: susceptible cells die and resistant cells grow. This range is the mutant selection window", + "A small fraction of genetically identical cells happen to be dormant when the drug arrives", + "Carrying capacity K = 10¹⁰ is the carrying capacity; this is deliberately the simplest case" + ], + "excess": [ + "Cell wall construction | penicillins, cephalosporins (β-lactams); vancomycin | The wall is built badly", + "Some bacteria make enzymes that chemically inactivate the antibiotic. The best-known are the β-lactamases", + "This was shown directly in classic experiments in the 1940s and 50s (Luria–Delbrück, and the Lederbergs' replica plating)", + "If you program, here is a loose analogy. Mutation is like random bit-flips in your own source code", + "Use antibiotics only when they're needed. Every use selects for resistance somewhere, including among harmless bacteria in your gut", + "This is natural selection, and it runs fast because bacteria have huge populations and short generation times", + "This is why tuberculosis and HIV are treated with combinations", + "the old blanket rule \"always finish the course\" is now debated", + "Tolerance. The bacteria die more slowly under the drug, but the MIC doesn't change", + "Biofilms. Communities of bacteria stuck to a surface inside a slimy matrix" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/run.json new file mode 100644 index 0000000..c8219e2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/none/run.json @@ -0,0 +1,13 @@ +{ + "result": "I made `explainers/antibiotic-resistance/explanation.md` and `explainers/antibiotic-resistance/index.html`. The prose covers how resistance works and where it comes from, with rough numbers. The page is a small simulation where you change the drug dose and watch resistant cells take over from almost nothing, which this reader can explore and check against the stated model better than text alone. I haven't opened the page in a browser.", + "error": null, + "cost": 0.502619, + "seconds": 162, + "turns": 3, + "files": [ + "explainers/antibiotic-resistance/explanation.md", + "explainers/antibiotic-resistance/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/explainers/antibiotic-resistance/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/explainers/antibiotic-resistance/explanation.md new file mode 100644 index 0000000..4f26c3f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/explainers/antibiotic-resistance/explanation.md @@ -0,0 +1,90 @@ +# How bacteria become resistant to an antibiotic + +**Short answer.** A single bacterium does not learn to resist a drug. A population becomes resistant through selection. A few cells already carry a gene that protects them. The antibiotic kills the other cells, and the protected cells multiply to replace them. The protective genes come from two sources: random copying errors (mutation) and genes passed from other bacteria (gene transfer). + +This matters in practice. An infection can improve on treatment and then return, and the drug no longer works on the bacteria that come back. + +The rest of this file builds that answer in four steps. It shows how a drug kills a cell, how a cell can block the drug, where the blocking genes come from, and how the drug itself makes those genes common. + +## 1. How an antibiotic kills a cell + +An antibiotic is a molecule that binds to a **target**: a part of the bacterial cell that the cell needs to live or divide. Two examples: + +- Penicillin binds the enzymes that build the cell wall. The wall weakens, and the cell bursts. +- Streptomycin binds the ribosome, the machine that makes proteins. The cell makes faulty proteins and dies. + +Human cells do not have these exact targets, so the drug harms bacteria much more than it harms the patient. + +To work, the drug must do three things: get into the cell, stay intact, and bind its target. + +## 2. How a cell blocks the drug + +Each of those three requirements can fail. Each failure is a resistance **mechanism**, and a gene controls each one. + +| Requirement | How the cell blocks it | Example | +|---|---|---| +| Get in | Make fewer entry channels in the outer membrane | Fewer porin channels in *E. coli* | +| Get in (and stay in) | Pump the drug out faster than it enters (an **efflux pump**) | Tetracycline pumps | +| Stay intact | Make an enzyme that destroys the drug | Beta-lactamase cuts penicillin | +| Bind the target | Change the target shape so the drug no longer fits | A changed ribosome protein blocks streptomycin | + +Resistance is usually a matter of degree, not all or nothing. Microbiologists measure it as the **MIC** (minimum inhibitory concentration): the lowest drug concentration that stops a strain from growing. A low MIC, such as 1 mg/L, means a little drug stops the strain. A high MIC, such as 64 mg/L, means the strain grows at drug levels that a patient's blood may never safely reach. "Resistant" means the MIC is above the level that treatment can achieve. + +## 3. Where the resistance genes come from + +### Mutation + +Each time a bacterium copies its DNA to divide, it makes rare errors. Most errors do nothing or cause harm. A few change a target or switch on a pump, and these give resistance. + +Worked example (illustrative numbers; real rates vary by drug and species from about 1 in a million to 1 in 10 billion per division): + +- A serious infection contains about 10¹⁰ bacteria (estimate). +- To grow from one cell to 10¹⁰ cells takes about 10¹⁰ cell divisions. +- Suppose a specific resistance mutation occurs once per 10⁸ divisions. +- Expected resistant cells: 10¹⁰ × 1/10⁸ = **about 100**, present *before* the first dose. + +So in a large population, resistant cells usually already exist. The drug does not create them. + +The obvious alternative idea is that the drug makes bacteria change to survive. Luria and Delbrück tested this in 1943. They grew many separate cultures and then exposed them to a killer (a virus, in their case). If the exposure caused resistance, every culture would have a similar, small number of survivors. Instead, most cultures had none and a few had hundreds. That pattern only fits mutations that happened at random times *before* exposure: an early mutation has many generations to multiply, a late one has few. + +### Gene transfer + +Bacteria can also receive resistance genes that evolved in other bacteria. This is **horizontal gene transfer**: genes move between living cells, not only from parent to offspring. There are three routes: + +- **Conjugation:** one cell passes a **plasmid** (a small, separate DNA circle) to another through direct contact. +- **Transformation:** a cell takes up free DNA from a dead cell nearby. +- **Transduction:** a virus that infects bacteria carries a piece of DNA from one cell to the next. + +Gene transfer is the main reason resistance spreads fast in hospitals. One plasmid can carry genes for several mechanisms, so one transfer event can make a cell resistant to several drugs at once. Transfer also crosses species: a harmless gut bacterium can give a gene to a dangerous one. + +## 4. How the drug makes resistance common + +Return to the infection with 10¹⁰ bacteria, of which about 100 are resistant. That is 1 resistant cell in 100 million. + +Treatment starts. The drug kills the susceptible cells. Suppose the immune system and the drug together reduce them to about 10⁴ within two days. The 100 resistant cells are not affected by the drug. Suppose they double once per hour in the body (estimate). After one day, which is 24 doublings: + +100 × 2²⁴ ≈ 100 × 1.7 × 10⁷ ≈ **1.7 × 10⁹ resistant cells** + +The resistant fraction has gone from 1 in 100 million to nearly 100%. No cell changed. The drug removed the competition. This is natural selection, and it runs fast because bacteria divide in hours, not years. + +### Dose controls which cells survive + +Suppose the susceptible strain has an MIC of 1 mg/L and a partly resistant mutant has an MIC of 8 mg/L (illustrative). + +- At **16 mg/L**, the drug stops both strains. Nothing is selected. +- At **4 mg/L**, the drug stops the susceptible strain but not the mutant. This concentration selects for resistance most strongly. +- At **0.5 mg/L**, the drug stops neither. It selects only weakly. + +The range between the two MICs is the **mutant selection window**. Low doses, missed doses, and drug levels that fall slowly after a course all keep bacteria in this window. This is one reason a prescribed course has a set dose and length. It is also why some treatments use two drugs at once: a cell would need two independent mutations, about 1 in 10⁸ × 1 in 10⁸ = 1 in 10¹⁶ divisions, which is larger than the whole infection. + +### What happens when the drug is removed + +Resistance often has a cost. A pump uses energy, and a changed target can work less well. Suppose a resistant strain grows 5% slower per generation than the susceptible strain. Without the drug, after 100 generations its share falls by a factor of 0.95¹⁰⁰ ≈ 0.006, about 170 times. In this case, stopping use of the drug makes resistance rare again. + +That reversal often fails in practice. Resistant strains acquire further **compensatory mutations** that remove most of the cost but keep the resistance. Some resistance genes cost almost nothing from the start. Then resistance stays in the population for years after the drug is no longer used. + +## Summary + +Bacteria become resistant to an antibiotic in two steps. First, variation: a few cells carry a gene that keeps the drug out, pumps it out, destroys it, or changes its target. These genes arise from random mutation or arrive by gene transfer from other bacteria. Second, selection: the drug kills the cells without the gene, and the cells with it multiply until they make up most of the population. + +**Not covered here:** *tolerance* and *persistence* (cells that survive a drug by growing slowly, without a resistance gene), biofilms, and the spread of resistant bacteria between people and animals. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/read.json new file mode 100644 index 0000000..35354f4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/read.json @@ -0,0 +1,48 @@ +{ + "answers": { + "1": { + "answer": "Resistance comes from two sources: random mutations during bacterial DNA copying, and genes transferred from other bacteria. Resistant cells already exist before treatment begins.", + "quote": "The protective genes come from two sources: random copying errors (mutation) and genes passed from other bacteria (gene transfer)." + }, + "2": { + "answer": "The antibiotic kills the susceptible cells in the population while resistant cells remain unaffected, allowing them to multiply and dominate as competition is removed.", + "quote": "The antibiotic kills the other cells, and the protected cells multiply to replace them." + }, + "3": { + "answer": "Resistance passes between different species through horizontal gene transfer via three mechanisms: conjugation (direct contact with a plasmid), transformation (uptake of free DNA), and transduction (transfer by virus).", + "quote": "Transfer also crosses species: a harmless gut bacterium can give a gene to a dangerous one." + }, + "4": { + "answer": "Even one resistant bacterium per billion can dominate because the drug eliminates all susceptible competitors. The rare resistant cells then multiply rapidly, going from negligible to nearly 100% of the population.", + "quote": "The resistant fraction has gone from 1 in 100 million to nearly 100%. No cell changed. The drug removed the competition. This is natural selection, and it runs fast because bacteria divide in hours, not years." + } + }, + "stops": [ + "Line 22: 'resistance mechanism' introduced as term but what distinguishes one from another unclear until table shown", + "Line 31: MIC definition complex—'above the level that treatment can achieve' requires understanding of pharmacokinetics", + "Line 37: How does mutation 'switch on' a pump? Mechanism not explained—'A few change a target or switch on a pump'", + "Lines 39-44: Heavy scientific notation (10¹⁰, 10⁸) and rapid calculation steps unfamiliar to reader new to subject", + "Line 48: Luria-Delbrück logic condensed—why does pattern 'most had none and a few had hundreds' prove pre-exposure mutation?", + "Line 52: Three gene transfer types (conjugation, transformation, transduction) listed with minimal explanation each", + "Line 58: How bacteria transfer genes 'across species' needs more context—'Transfer also crosses species: a harmless gut bacterium can give a gene to a dangerous one'", + "Line 62: 'Return to the infection with 10¹⁰ bacteria'—unclear if same example from line 40 or new scenario", + "Line 75: 'mutant selection window' introduced without explanation of why called 'window' or what defines boundaries", + "Line 78: Probability multiplication (1 in 10⁸ × 1 in 10⁸) not explained—'about 1 in 10⁸ × 1 in 10⁸ = 1 in 10¹⁶ divisions'" + ], + "excess": [ + "Line 16: 'Human cells do not have these exact targets'—explains drug safety, not resistance mechanism", + "Line 39: '(illustrative numbers; real rates vary by drug and species from about 1 in a million to 1 in 10 billion)'—complicates simplified example", + "Lines 48-49: Luria-Delbrück experiment details redundant—'They grew many separate cultures and then exposed them to a killer'", + "Line 59: 'Gene transfer is the main reason resistance spreads fast in hospitals'—real-world context, off the core mechanism", + "Lines 72-76: Dose optimization and selection windows—tangential to how resistance arises and dominates", + "Line 82: 'Suppose a resistant strain grows 5% slower per generation'—specific illustration adds complexity", + "Lines 82-84: 'What happens when the drug is removed'—about reversal, not about dominance during treatment", + "Line 90: 'Not covered here: tolerance and persistence'—merely notes scope, not essential to explanation" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/run.json new file mode 100644 index 0000000..7cf1bbe --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/antibiotic-resistance/explanation.md` as prose only, because the answer is a short chain of variation then selection that worked numbers show well, and a diagram or interactive page would not make it easier to follow.", + "error": null, + "cost": 0.1721862, + "seconds": 49, + "turns": 2, + "files": [ + "explainers/antibiotic-resistance/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/explanation.md new file mode 100644 index 0000000..0daf3f3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/explanation.md @@ -0,0 +1,73 @@ +# How do bacteria become resistant to an antibiotic? + + + +**The drug does not change the bacteria. In a large infection, a few bacteria already carry a resistance gene, made by a random copying error or received from another bacterium. The drug removes the susceptible bacteria around them, and the resistant bacteria multiply.** The bacteria become resistant, not the patient's body. + +## What a cell must block + +An antibiotic is a molecule that kills bacteria or stops their growth. It does this by binding a **target**: a molecule the cell needs. For example, penicillin binds the enzymes that build the cell wall. + +A **resistance gene** is a gene, changed by mutation or received from another bacterium, that blocks this. It works in one of four ways: + +| Way | Example | +|---|---| +| Destroy the drug | A beta-lactamase enzyme cuts penicillin before penicillin reaches its target. | +| Change the target | One DNA letter change alters the target protein, so rifampicin no longer binds. | +| Pump the drug out | An efflux pump, a membrane protein, exports tetracycline faster than it enters. | +| Keep the drug out | The cell loses a porin, an entry channel in its surface, so less drug gets in. | + +Each way means the cell needs more drug to stop it. The lowest concentration that stops a strain's growth is its **MIC** (minimum inhibitory concentration). Suppose a drug reaches 8 mg/L in the blood. A strain with an MIC of 0.5 mg/L is stopped: it is **susceptible**. A strain with an MIC of 32 mg/L grows: it is **resistant**. (Illustrative values.) + +## Resistant cells are there before the drug + +A given resistance mutation (a mutation that turns a normal gene into a resistance gene) happens about once per 10⁸ cell divisions. A serious infection can hold 10¹⁰ cells. Each division adds one cell, so growing to that size took about 10¹⁰ divisions. Expect + +10¹⁰ × 10⁻⁸ = **100** resistance mutations, before anyone gives a drug. + +Early mutants have divided since, so there are at least 100 resistant cells. Compare a small infection of 10⁶ cells: it expects 0.01 mutations, so there is only about a 1% chance that even one resistant cell exists. + +## The drug selects them + + +1. The drug stops the susceptible cells: it kills them or halts their growth, and the immune system clears the halted cells. +2. The resistant cells keep dividing. The immune system removes some, but alone it cannot keep up with a population that keeps doubling. The drug did most of the work. +3. 100 cells need **27** doublings to reach 10¹⁰ again (100 × 2²⁷ ≈ 100 × 1.3 × 10⁸ ≈ 10¹⁰). At a lab doubling time of 30 minutes, that is 13 hours. +4. The infection returns, and almost every cell is resistant. + +This is **selection**: the drug did not make resistant cells, it removed the susceptible cells. + +**How do we know the drug did not cause the change?** If exposure caused resistance, each cell would have the same small chance when it met the drug. Equal cultures would then give similar counts. Luria and Delbrück tested this in 1943. They used a virus that kills bacteria; the test asks only whether resistance arose before contact, so it applies to drugs too. They grew 20 equal cultures of bacteria, then exposed each one and counted the resistant colonies. The mean was about 11. For independent rare events with a mean of 11, the random scatter is about √11 ≈ 3.3, so the "drug causes it" idea predicts most cultures between 5 and 18. They found 11 cultures with 0 and one with **107**. Random mutation before exposure explains this spread: a mutant that appears early leaves many resistant descendants, and a late one leaves few. + +## Genes from other bacteria + + +A cell does not have to wait for its own rare mutation. A **plasmid** is a small circle of DNA, separate from the chromosome. In **conjugation**, two bacteria touch and one copies a plasmid into the other. This is **horizontal gene transfer**: a gene from another bacterium, not from the parent cell. + +Plasmids carry complete working genes, such as the beta-lactamase gene, which one letter change could never build. The same plasmid gene turns up in different species, which shows that the gene moved between them. One plasmid often carries several resistance genes at once. + +## Without the drug, resistance usually costs + + +Resistant cells take over only while the drug is present. Without the drug, resistance often has a **fitness cost**: the extra protein, or the changed target, makes the cell grow more slowly. + +Suppose resistant cells grow 5% slower: each time the susceptible cells double, the resistant cells grow only 0.95 times as much. So each doubling multiplies the ratio of resistant to susceptible cells by 0.95. The effect compounds: after 100 doublings the ratio is 0.95¹⁰⁰ ≈ **0.006** of its start value. + +But resistance rarely vanishes, for two reasons: + +- A **compensatory mutation** is a second change, in another gene, that restores growth; the first change and the resistance stay. If it cuts the cost to 1%, then 0.99¹⁰⁰ ≈ 0.37 of the ratio remains after 100 doublings. +- Plasmids persist. One plasmid often carries resistance genes for several drugs, so any one of those drugs still selects the whole plasmid. + +## What this predicts + +- Large infections are the risk, because they hold resistant cells before treatment. +- Two drugs with independent targets need two mutations in one cell. The two are independent events, so their chances multiply: 10⁻⁸ × 10⁻⁸ = 10⁻¹⁶. In 10¹⁰ cells, expect 10⁻⁶ such cells: a one-in-a-million chance. The treatment clears the infection. This fails if one event gives resistance to both drugs, for example one plasmid with both genes, or one pump that exports both molecules. Then about 100 cells survive again. + +**In short:** resistance genes come first, by chance or from other bacteria. The drug removes the susceptible cells, and the resistant cells multiply. + +## Not covered + +- Persister cells, which survive a drug without a resistance gene ("antibiotic persistence"). +- Other routes of gene transfer: transformation and transduction. +- Drugs that raise the mutation rate ("SOS response"). +- Course length, prescribing policy, farm use, biofilms. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.md new file mode 100644 index 0000000..328cfa5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.md @@ -0,0 +1,158 @@ +# Semantic model: how bacteria become resistant to an antibiotic + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How does a population of bacteria that an antibiotic used to kill become a population that the antibiotic no longer kills? + +## Audience and prior knowledge + +A technical reader: school mathematics and science, basic programming, new to microbiology. Knows: cells, DNA and genes (a gene codes for a protein), enzymes, that a mutation is a change in DNA, exponential growth and doubling, powers of ten, expected value, the idea of natural selection in outline. Does not know: how antibiotics act on a cell, what a plasmid is, horizontal gene transfer, minimum inhibitory concentration (MIC), fitness cost, the Luria–Delbrück result. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Bacterium | A single-cell organism that copies its DNA and divides in two. | fact | +| Antibiotic | A molecule that kills bacteria or stops their growth by binding to a target in the cell. | fact | +| Target | The molecule in the bacterium that the antibiotic binds to, usually a protein the cell needs (for example, the enzymes that build the cell wall). | fact | +| Resistance gene | A gene, changed by mutation or gained from another bacterium, that lets the cell grow at a drug concentration that stops normal cells. | fact | +| Susceptible cell | A cell that the drug stops at the concentration reached in the patient. | definition | +| Resistant cell | A cell that grows at the concentration reached in the patient. | definition | +| Mutation | A random copying error in DNA, made at a division, independently of the drug. | fact | +| Plasmid | A small circle of DNA, separate from the chromosome, that a bacterium can copy and pass to another bacterium. | fact | +| Horizontal gene transfer | A bacterium gets a gene from another bacterium, not from its parent cell. Main route: conjugation (two cells touch and one copies a plasmid into the other). | fact | +| Selection | The drug removes susceptible cells; resistant cells survive and divide, so their fraction in the population rises. | fact | +| MIC (minimum inhibitory concentration) | The lowest drug concentration that stops a strain's growth, measured in mg/L. | definition | +| Fitness cost | A resistant cell often grows more slowly than a susceptible cell when there is no drug, because it makes an extra protein or its changed target works less well. | fact (size varies) | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Mutation | makes (at random, before any drug) | resistance gene | about 1 in 10^8 divisions for one given resistance mutation | +| Horizontal gene transfer | gives | resistance gene | in one step, can cross species | +| Plasmid | carries | resistance genes | often several at once | +| Antibiotic | stops | susceptible cell | | +| Resistance gene | blocks | antibiotic action | four ways: destroy, change target, pump out, keep out | +| Antibiotic | selects | resistant cell | by removing its competitors | +| Fitness cost | slows | resistant cell, when there is no drug | | + +## Resistance mechanisms (what the resistance gene does) + +| Way | Example | +|---|---| +| Destroy the drug | A beta-lactamase enzyme cuts penicillin before it reaches the cell-wall enzymes. | +| Change the target | One DNA letter change in the rpoB gene changes the target protein, so rifampicin no longer binds. | +| Pump the drug out | An efflux pump (a membrane protein) exports tetracycline faster than it enters. | +| Keep the drug out | Loss of a porin (an entry channel in the outer membrane) lets less drug in. | + +Each way raises the MIC. Illustrative: beta-lactamase raises a strain's MIC from 0.5 to 32 mg/L. + +## Dependencies (learning order) + +1. An antibiotic works by binding a target in the cell. +2. A resistance gene stops that binding or its effect (four ways, with examples). +3. Resistance genes appear at random (mutation) or arrive from other bacteria (horizontal gene transfer), before the drug. +4. The drug removes the susceptible cells; the resistant ones remain and multiply (selection). +5. Without the drug, the fitness cost works against resistant cells. + +## Causal chain + +```text +random variation (mutation or gene transfer) → a few resistant cells among many susceptible ones +→ antibiotic stops the susceptible cells → resistant cells grow into the space and nutrients left +→ the population is now resistant → resistant cells spread to other hosts, and plasmids spread to other species +``` + +## Temporal sequence + +1. Before treatment: the infection grows to 10^10 cells. About 10^10 divisions made it, so about 100 resistance mutations happened. Early mutants have divided since, so resistant cells number at least 100, typically several hundred. +2. Treatment: the drug stops the susceptible cells, and the immune system clears them; the resistant cells keep dividing. +3. Regrowth: 100 cells double about 27 times to reach 10^10 again (2^27 ≈ 1.3 × 10^8 ≈ 10^10 / 100). Assumption: the immune system removes some cells but does not keep up with a population that doubles. Weakened patients clear fewer. +4. The infection returns, and the same drug no longer works on it. + +## Quantities + +- Rate of one given resistance mutation: about 10^-8 per division (order of magnitude; real values span about 10^-6 to 10^-10). +- Doubling: every cell in the population divides once. Growing from 1 cell to N cells takes about N divisions. +- Cells in a serious infection: 10^9 to 10^10 or more (estimate). Example uses 10^10. +- Expected resistance mutations before treatment: 10^10 × 10^-8 = 100. Resistant cells ≥ 100, because early mutants divide (mean about μN ln N ≈ 2,300; typical several hundred; heavy spread between infections). +- Small infection, 10^6 cells: 10^6 × 10^-8 = 0.01 expected mutations. Probability of at least one resistant cell = 1 − e^-0.01 ≈ 1%. For 10^10 cells, probability ≈ 1. +- Regrowth: 100 → 10^10 needs log2(10^8) ≈ 26.6, so about 27 doublings. At 30 min per doubling (laboratory, ideal) that is about 13 hours; in a patient, doublings are slower. +- MIC example (illustrative): susceptible strain 0.5 mg/L, resistant strain 32 mg/L, concentration reached in blood 8 mg/L. 8 > 0.5: the drug stops the first strain. 8 < 32: the second strain grows. +- Fitness cost (illustrative): resistant cells grow 5% slower, no drug: each time the susceptible cells double, the resistant cells grow 0.95 times as much. The resistant-to-susceptible ratio is multiplied by 0.95 each doubling, because both groups compound: after 10 doublings 0.95^10 ≈ 0.60, after 100 doublings 0.95^100 ≈ 0.006. (Not 1 − 100 × 0.05, which goes negative.) +- Compensatory mutation: a second mutation, in another gene, that cuts the cost from 5% to 1% without losing resistance: 0.99^100 ≈ 0.37 after 100 doublings. +- Plasmid persistence: one plasmid often carries resistance genes for several drugs, so any one of those drugs still selects the whole plasmid. +- Luria–Delbrück scatter: independent rare events with mean 11 have a spread of about √11 ≈ 3.3; mean ± 2 spreads ≈ 5 to 18. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Drug present | resistant fraction rises from about 10^-8 to near 1 | susceptible cells are removed | +| Drug absent | resistant fraction falls slowly, or stays | fitness cost; compensatory mutations reduce it | +| Small population (10^6) | about 1% chance of any resistant cell; drug clears it | 10^6 × 10^-8 = 0.01 | +| Two drugs with independent targets | needs two independent mutations: 10^-16 | 10^10 × 10^-16 = 10^-6 expected double-resistant cells | +| Two drugs, one plasmid or pump resists both | one event, about 10^-8 | 10^10 × 10^-8 = 100: the combination fails | +| Plasmid gene | one transfer gives a whole resistance gene, even to another species | not a rare copying error | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Explain resistance as selection of variants that existed before the drug | "The drug makes the bacteria change" | Luria and Delbrück (1943) grew 20 parallel cultures and plated each on a killing agent (a virus, not an antibiotic; same logic). Mean about 11 resistant colonies. If the agent caused resistance, every culture would show about 11 (most between 5 and 18). Observed: 11 cultures had 0, one had 107. That spread fits mutations at random times before exposure: an early mutant leaves many resistant descendants, a late one few. | +| Count horizontal gene transfer as a second source | "Every resistance must be a new mutation in that lineage" | A beta-lactamase is a complete enzyme; one letter change does not build a new enzyme. Bacteria get such genes on plasmids, and the same plasmid gene turns up in different species. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| Resistant cells exist before treatment (mechanism) | 10^10 cells × 10^-8 = 100 mutations: at least 100 resistant cells | 10^6 cells × 10^-8 = 0.01: about a 1% chance of any; the drug clears the infection | +| Resistance takes over only when the drug is present (guarantee) | With drug: fraction about 10^-8 → near 1 after regrowth (100 cells → 10^10 in 27 doublings) | Without drug, 5% cost: ratio × 0.95^100 ≈ 0.006 after 100 generations | +| Two drugs together prevent selection (guarantee, assumes independent resistance) | Independent: 10^-8 × 10^-8 = 10^-16; 10^10 cells give 10^-6 expected double-resistant cells | One plasmid or pump resists both: 10^10 × 10^-8 = 100 survivors | + +## Terms + +- **Antibiotic:** a molecule that kills bacteria or stops their growth by binding a target. +- **Target:** the molecule the antibiotic binds to. +- **Resistance gene:** a gene, changed by mutation or gained from another bacterium, that lets the cell grow in the drug. +- **Resistant / susceptible:** grows / is stopped at the concentration the drug reaches in the patient. Resistance is a property of the bacteria, not of the patient. +- **Mutation:** a random copying error in DNA at a division. +- **Plasmid:** a small circle of DNA, separate from the chromosome, that can pass between bacteria. +- **Horizontal gene transfer:** getting a gene from another bacterium, not from the parent. +- **Conjugation:** two bacteria touch, and one copies a plasmid into the other. +- **Selection:** the drug removes the susceptible cells, so the resistant fraction rises. +- **MIC:** the lowest concentration that stops a strain's growth. Low MIC (0.5 mg/L, below the 8 mg/L reached): the drug works. High MIC (32 mg/L, above it): the drug fails. +- **Fitness cost:** slower growth of a resistant cell when no drug is present. +- **Doubling:** every cell divides once. "Division" is one cell splitting. +- Mutation rate 10^-8 per division: one in a hundred million divisions makes this mutation. At 10^-6, resistance appears even in small populations; at 10^-10, only in very large ones. + +## Epistemic status + +- **Observed / established:** the four resistance mechanisms; mutations arise before exposure (Luria–Delbrück 1943); plasmids carry and transfer resistance genes; selection by antibiotic use. +- **Mathematical consequence:** expected counts (N × rate); doublings needed; product of independent rates. +- **Assumed:** the two resistances in combination therapy are independent; constant doubling time; the immune system does not keep up with growth. +- **Estimated / illustrative:** the 10^-8 rate, 10^10 cell count, 30 min doubling, 5% cost and MIC values are orders of magnitude. + +## Confusion points + +- "The patient becomes resistant" → the bacteria are resistant; the patient's body does not change. +- "The drug makes bacteria mutate / bacteria learn" → the variants exist before the drug; the drug removes their competitors. +- "Each cell slowly gets used to the drug" → a cell's resistance comes from its genes. The population changes because the fraction of resistant cells changes. Resistance can rise in steps (a first mutation raises MIC from 0.5 to 2 mg/L, a second to 16 mg/L), but each step is a new gene change, selected in turn. +- "Resistance disappears when we stop the drug" → it falls slowly at best; compensatory mutations reduce the cost, and plasmids persist. + +## Scope + +- **Out of scope:** tolerance and persister cells (survive without being resistant) — search "antibiotic persistence". +- **Out of scope:** transformation and transduction (other routes of gene transfer) — "horizontal gene transfer" in any microbiology text. +- **Out of scope:** stress-induced mutagenesis (some drugs raise the mutation rate via the SOS response) — search "SOS response antibiotic". +- **Out of scope:** course length, stewardship policy, agricultural use, biofilms. + +## Representation decision + +- **Stage:** 1 +- **Reason:** the subject is a causal sequence (variation → selection → regrowth → spread) with one table of mechanisms and short arithmetic. Nothing needs a parameter explored or motion seen. +- **Levels:** L1 resistance is selection of variants that existed before the drug / L2 the four mechanisms and the two sources / L3 numbers: N × rate, doublings, cost. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.yaml new file mode 100644 index 0000000..230d7fe --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.yaml @@ -0,0 +1,99 @@ +# Machine-readable half of model.md. `explainer check antibiotic-resistance` holds every rendering to it. + +values: + expected_mutations_large: 100 # 10^10 cells × 10^-8 per division + small_infection_chance_pct: 1 # 1 − e^-0.01 ≈ 1% + regrowth_doublings: 27 # log2(10^8) ≈ 26.6 + two_pow_27: 130000000 # 2^27 ≈ 1.34 × 10^8 + lab_regrowth_hours: 13 # 27 × 30 min + doubling_minutes: 30 + mic_susceptible: 0.5 # mg/L + mic_resistant: 32 # mg/L + blood_concentration: 8 # mg/L + cost_pct: 5 + cost_generations: 100 + ratio_after_100_generations: 0.006 # 0.95^100 + compensated_cost_pct: 1 + compensated_ratio: 0.37 # 0.99^100 + ld_cultures: 20 + ld_mean: 11 + ld_zero_cultures: 11 + ld_max: 107 + ld_poisson_low: 5 + ld_poisson_high: 18 + ld_year: 1943 + ld_scatter: 3.3 # √11 + +allow: [1, 2, 4, 6, 8, 10, 16, -6, -8, -16, 0.95, 0.99, 2.0] + +require: + - path: expected_mutations_large + in: [prose] + - path: regrowth_doublings + in: [prose] + - path: ratio_after_100_generations + in: [prose] + - path: ld_max + in: [prose] + +claims: + - id: preexisting + kind: mechanism + statement: "Resistant cells exist before the drug arrives, made by random mutation at about 10^-8 per division; a large infection already holds some." + example: "10^10 cells × 10^-8 = 100 resistance mutations before treatment; a 10^6-cell infection has about a 1% chance of any." + ask: "An infection of 10^10 bacteria has never met drug X. A given resistance mutation happens once per 10^8 divisions. Does the infection contain resistant cells, and roughly how many?" + - id: selection-not-induction + kind: why + statement: "The drug does not cause resistance; it removes the susceptible cells, so resistant cells that were already there take over." + alternative: "The drug makes the bacteria change, as an adaptation." + counterexample: "Luria and Delbrück 1943: 20 parallel cultures, mean about 11 resistant colonies. Induction predicts about 11 in each (most 5 to 18); observed 11 cultures with 0 and one with 107." + ask: "Why do scientists say the antibiotic selects resistance rather than causes it? What evidence separates the two ideas?" + - id: mechanisms + kind: definition + statement: "A resistance gene blocks the drug in one of four ways: destroy it, change its target, pump it out, or keep it out; each raises the MIC." + - id: hgt + kind: why + statement: "Bacteria also get complete resistance genes from other bacteria, often on plasmids, even across species." + alternative: "Every resistance is a new mutation in that lineage." + counterexample: "A beta-lactamase is a complete enzyme that one letter change cannot build; the same plasmid gene appears in different species." + ask: "How can a species become resistant to a drug in one step, without any mutation of its own?" + - id: takeover-needs-drug + kind: guarantee + statement: "Resistant cells take over only while the drug is present; without the drug a fitness cost makes their fraction fall." + example: "With drug: 100 survivors regrow to 10^10 in about 27 doublings (about 13 h at 30 min per doubling)." + counterexample: "Without drug, 5% slower growth: resistant-to-susceptible ratio × 0.95^100 ≈ 0.006 after 100 doublings; with a compensatory mutation (1% cost), 0.99^100 ≈ 0.37; plasmids with genes for several drugs stay selected by any of them." + ask: "If doctors stop using an antibiotic, what happens to the resistant fraction, and why might it not disappear?" + - id: mic + kind: definition + statement: "MIC is the lowest concentration that stops a strain's growth; a strain is resistant when its MIC is above the concentration the drug reaches in the patient." + example: "Blood concentration 8 mg/L: MIC 0.5 mg/L strain stopped, MIC 32 mg/L strain grows." + - id: combination + kind: guarantee + statement: "Two drugs given together stop selection when resistance to each needs its own independent change." + example: "10^-8 × 10^-8 = 10^-16; 10^10 cells give 10^-6 expected double-resistant cells." + counterexample: "If one plasmid or pump resists both drugs, one event at about 10^-8 suffices: 10^10 × 10^-8 = 100 survivors." + +accept_unjustified: + "√": "the scatter of independent rare-event counts is √mean (Poisson); stated as a fact the reader can use, deriving it is off the main line" + ln: "model.md only: the Luria–Delbrück mean μN ln N, a background check that resistant cells exceed 100; no rendering shows it" + +terms: + - term: resistant + means: "a property of the bacteria, not of the patient or the body" + avoid: [patient becomes resistant, body becomes resistant, become immune to] + - term: selection + means: "the drug removes susceptible cells; it does not cause the change" + avoid: [bacteria learn, drug causes mutations, adapt to the drug] + +budget: {prose: 950, reason: "two sources of resistance genes, four mechanisms in a table, and the selection guarantee with its no-drug case and the Luria–Delbrück evidence"} + +quiz: + - q: "A patient's infection has 10^10 bacteria. A resistance mutation to drug X happens once in 10^8 divisions. The patient takes drug X; it clears symptoms for a day, then the infection returns and drug X no longer works. Explain what happened." + expect: "About 100 resistant cells existed before treatment (10^10 × 10^-8). The drug removed the susceptible cells; the resistant cells survived and regrew (about 27 doublings to 10^10). The bacteria population is now resistant." + misconception: "The drug caused the bacteria to mutate or adapt, or the patient's body became resistant." + - q: "Would you expect the same outcome if the infection had only 10^6 bacteria? Why?" + expect: "Probably not: expected resistant mutations 10^6 × 10^-8 = 0.01, about a 1% chance any resistant cell exists, so the drug most likely clears it." + misconception: "Same outcome, because resistance develops during treatment regardless of population size." + - q: "A hospital stops using drug X for years. Does resistance to X vanish?" + expect: "It usually falls, because resistant cells often grow slower without the drug (fitness cost), but slowly and not always to zero: compensatory mutations reduce the cost and plasmids persist." + misconception: "Yes, it disappears quickly, or no, nothing changes because resistance is permanent." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/narrative.md new file mode 100644 index 0000000..4354736 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/narrative.md @@ -0,0 +1,59 @@ +# Narrative: antibiotic-resistance + +> Pass 3. `model.md` says what is true. This file says the path a first-time reader takes to it. + +## 1. Reader before and after + +- **Before:** a technical reader (school science and mathematics, basic programming) who knows cells, genes, mutation, exponential growth, and natural selection in outline, and nothing of how antibiotics act or how bacteria exchange genes. +- **After:** "Resistance genes appear at random or arrive from other bacteria before the drug; the drug removes the susceptible cells, so the resistant ones are left to multiply." + +## 2. Question and motive + +- **Question:** How does an infection that a drug used to clear become one it cannot clear? Result in words first: the drug does not change the bacteria. A few bacteria already carry a resistance gene; the drug removes their competitors. +- **Why care:** the common picture ("bacteria get used to the drug") predicts the wrong things: that resistance appears in every infection, that it needs the drug to arise, and that it is in the patient. +- **Why this approach:** start from what the drug does to one cell, because resistance is a way to block that. Then count: how many cells could already have a resistance gene? The count is the surprise. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| target | the molecule the antibiotic binds | penicillin binds the enzymes that build the cell wall | 2 | +| resistance gene | gene, mutated or gained, that lets the cell grow in the drug | beta-lactamase gene | 3 | +| beta-lactamase, efflux pump, porin, rpoB | the four mechanism examples | one row each in the table | 3 | +| MIC | lowest concentration that stops a strain's growth | 0.5 vs 32 mg/L against 8 mg/L in blood | 3 | +| resistant / susceptible | grows / stopped at the concentration reached | the two MIC strains | 3 | +| mutation rate per division | chance that one division makes this mutation | 10^-8 | 4 | +| selection | drug removes susceptible cells, resistant fraction rises | 100 survivors regrow | 5 | +| doubling | every cell divides once | 100 → 10^10 in 27 doublings | 5 | +| Luria–Delbrück | the experiment that separates selection from induction | 20 cultures, 0 to 107 | 6 | +| plasmid, horizontal gene transfer, conjugation | gene from another bacterium | beta-lactamase gene on a plasmid in different species | 7 | +| fitness cost, compensatory mutation | slower growth without drug; a second mutation that reduces it | 0.95^100 ≈ 0.006; 0.99^100 ≈ 0.37 | 8 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How does an infection that a drug used to clear become one it cannot clear? | — | the question as title, bold answer | The drug does not change the bacteria. A few bacteria already carry a resistance gene, by chance or from other bacteria; the drug removes their competitors, and they multiply. The bacteria are resistant, not the patient's body. | the answer | +| 2 | mechanism | What does a drug do to a cell? | so, to resist, a cell must block this | — | An antibiotic binds a target, a molecule the cell needs: penicillin binds the enzymes that build the cell wall. | target | +| 3 | mechanism | How can a cell stop that? | therefore | table: four ways, each with one example in words | Destroy the drug (beta-lactamase enzyme cuts penicillin); change the target so the drug no longer fits (one letter in rpoB, rifampicin); pump it out (efflux pump); keep it out (lose a porin, an entry channel). Each way means a higher concentration is needed to stop growth. That concentration is the MIC, minimum inhibitory concentration. Blood reaches about 8 mg/L: a strain with MIC 0.5 is stopped (susceptible), a strain with MIC 32 grows (resistant). | mechanisms, MIC, resistant | +| 4 | instance | Where does a resistance gene come from in an infection that never met the drug? | but the drug was never there… | arithmetic | A copying error at one division makes a given resistance mutation about once per 10^8 divisions. An infection of 10^10 cells took about 10^10 divisions to grow, so expect 10^10 × 10^-8 = 100 mutations. Early mutants have divided since, so resistant cells number at least 100. An infection of 10^6 cells expects 0.01 mutations: about a 1% chance that even one resistant cell exists. | variants exist before the drug | +| 5 | mechanism | What does the drug do to that mix? | so | numbered steps | The drug stops the susceptible cells; the immune system clears those, but it does not keep up with a population that doubles. The 100 resistant cells divide: 27 doublings (every cell divides once) take them back to 10^10, about 13 hours at a lab doubling time of 30 min. This is selection. | selection | +| 6 | objection | How do we know the drug did not cause the change? | but | the numbers of one experiment | If the drug caused the change (induction), each cell would have the same small chance when exposed, so equal cultures would show similar counts. Luria and Delbrück (1943) grew 20 equal cultures and then exposed them; mean 11 resistant colonies, so induction predicts most cultures between 5 and 18. They found 11 cultures with 0 and one with 107. An early mutant leaves many descendants, a late one few: that spread means the mutations happened before exposure. | evidence | +| 7 | generalize | Must every lineage wait for its own rare mutation? | but | — | A plasmid is a small circle of DNA, separate from the chromosome. In conjugation, two bacteria touch and one copies a plasmid into the other. So a cell gets a complete working gene, such as the beta-lactamase gene, in one step, even from another species. One letter change cannot build a whole new enzyme; this is how such genes spread. | second source | +| 8 | test | Does resistance stay after the drug stops? | therefore: takeover needs the drug | numbers | Resistance often has a fitness cost: the extra protein or the changed target makes the cell grow slower when there is no drug. With 5% slower growth, the ratio of resistant to susceptible cells is multiplied by 0.95 each generation: after 100 generations, 0.006. But a compensatory mutation (a second change that cuts the cost to 1%) gives 0.99^100 ≈ 0.37, so resistance can persist. | takeover needs the drug, and its limit | +| 9 | payoff | What does this predict? | so | two lines | Large infections are the risk, by the count in beat 4. Two drugs with independent targets need two mutations in one cell: 10^-8 × 10^-8 = 10^-16, so 10^10 cells expect 10^-6. This fails if one plasmid or pump resists both: 100 survivors again. | predictions | +| 10 | close | — | — | — | Resistance genes come first, by chance or from other bacteria; the drug removes the competitors; the resistant cells multiply. A short "not covered" list follows, outside the argument. | — | + +## 5. Concrete to symbol + +Expected count = cells × rate: "here 10^10 cells and 10^-8 per division give 100". Checked at 10^6 → 0.01. + +## 6. Links between representations + +The MIC numbers (0.5, 8, 32) appear in the table caption and in the definition of resistant. + +## 7. Close the loop + +- **Answer:** resistance genes come first, by chance or from other bacteria; the drug removes the competitors; the resistant cells multiply. +- **General argument:** any heritable variant that grows under the drug while others are stopped takes over while the drug is present. +- **Payoff:** predicts when resistance appears (large populations, single drugs, long use) and why it persists. Scope: persisters, other transfer routes, SOS mutagenesis, policy. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/decisions.yaml new file mode 100644 index 0000000..4cfc2fc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/decisions.yaml @@ -0,0 +1,221 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T151610-probe#1: + decision: adopted +2026-10-04T151610-probe#10: + decision: adopted +2026-10-04T151610-probe#11: + decision: adopted +2026-10-04T151610-probe#12: + decision: declined +2026-10-04T151610-probe#13: + decision: adopted +2026-10-04T151610-probe#14: + decision: declined +2026-10-04T151610-probe#15: + decision: adopted +2026-10-04T151610-probe#2: + decision: adopted +2026-10-04T151610-probe#3: + decision: adopted +2026-10-04T151610-probe#4: + decision: adopted +2026-10-04T151610-probe#5: + decision: adopted +2026-10-04T151610-probe#6: + decision: adopted +2026-10-04T151610-probe#7: + decision: adopted +2026-10-04T151610-probe#8: + decision: adopted +2026-10-04T151610-probe#9: + decision: adopted +2026-10-04T151845-coldread-narrative#1: + decision: adopted +2026-10-04T151845-coldread-narrative#10: + decision: adopted +2026-10-04T151845-coldread-narrative#11: + decision: adopted +2026-10-04T151845-coldread-narrative#12: + decision: adopted +2026-10-04T151845-coldread-narrative#13: + decision: adopted +2026-10-04T151845-coldread-narrative#14: + decision: adopted +2026-10-04T151845-coldread-narrative#15: + decision: adopted +2026-10-04T151845-coldread-narrative#16: + decision: adopted +2026-10-04T151845-coldread-narrative#17: + decision: adopted +2026-10-04T151845-coldread-narrative#18: + decision: adopted +2026-10-04T151845-coldread-narrative#19: + decision: adopted +2026-10-04T151845-coldread-narrative#2: + decision: adopted +2026-10-04T151845-coldread-narrative#20: + decision: adopted +2026-10-04T151845-coldread-narrative#21: + decision: adopted +2026-10-04T151845-coldread-narrative#22: + decision: adopted +2026-10-04T151845-coldread-narrative#23: + decision: adopted +2026-10-04T151845-coldread-narrative#24: + decision: adopted +2026-10-04T151845-coldread-narrative#25: + decision: adopted +2026-10-04T151845-coldread-narrative#26: + decision: adopted +2026-10-04T151845-coldread-narrative#27: + decision: adopted +2026-10-04T151845-coldread-narrative#28: + decision: adopted +2026-10-04T151845-coldread-narrative#29: + decision: adopted +2026-10-04T151845-coldread-narrative#3: + decision: adopted +2026-10-04T151845-coldread-narrative#30: + decision: adopted +2026-10-04T151845-coldread-narrative#31: + decision: adopted +2026-10-04T151845-coldread-narrative#32: + decision: adopted +2026-10-04T151845-coldread-narrative#33: + decision: adopted +2026-10-04T151845-coldread-narrative#34: + decision: adopted +2026-10-04T151845-coldread-narrative#35: + decision: adopted +2026-10-04T151845-coldread-narrative#36: + decision: adopted +2026-10-04T151845-coldread-narrative#37: + decision: adopted +2026-10-04T151845-coldread-narrative#38: + decision: adopted +2026-10-04T151845-coldread-narrative#4: + decision: adopted +2026-10-04T151845-coldread-narrative#5: + decision: declined +2026-10-04T151845-coldread-narrative#6: + decision: declined +2026-10-04T151845-coldread-narrative#7: + decision: adopted +2026-10-04T151845-coldread-narrative#8: + decision: adopted +2026-10-04T151845-coldread-narrative#9: + decision: adopted +2026-10-04T152038-coldread-prose#1: + decision: adopted +2026-10-04T152038-coldread-prose#10: + decision: adopted +2026-10-04T152038-coldread-prose#11: + decision: adopted +2026-10-04T152038-coldread-prose#12: + decision: adopted +2026-10-04T152038-coldread-prose#13: + decision: declined +2026-10-04T152038-coldread-prose#14: + decision: declined +2026-10-04T152038-coldread-prose#2: + decision: adopted +2026-10-04T152038-coldread-prose#3: + decision: adopted +2026-10-04T152038-coldread-prose#4: + decision: adopted +2026-10-04T152038-coldread-prose#5: + decision: adopted +2026-10-04T152038-coldread-prose#6: + decision: adopted +2026-10-04T152038-coldread-prose#7: + decision: adopted +2026-10-04T152038-coldread-prose#8: + decision: adopted +2026-10-04T152038-coldread-prose#9: + decision: declined +2026-10-04T152206-quiz-prose#1: + decision: adopted +2026-10-04T152206-quiz-prose#10: + decision: declined +2026-10-04T152206-quiz-prose#11: + decision: adopted +2026-10-04T152206-quiz-prose#12: + decision: adopted +2026-10-04T152206-quiz-prose#13: + decision: adopted +2026-10-04T152206-quiz-prose#14: + decision: adopted +2026-10-04T152206-quiz-prose#15: + decision: adopted +2026-10-04T152206-quiz-prose#16: + decision: declined +2026-10-04T152206-quiz-prose#17: + decision: declined +2026-10-04T152206-quiz-prose#18: + decision: declined +2026-10-04T152206-quiz-prose#19: + decision: declined +2026-10-04T152206-quiz-prose#2: + decision: adopted +2026-10-04T152206-quiz-prose#20: + decision: adopted +2026-10-04T152206-quiz-prose#21: + decision: adopted +2026-10-04T152206-quiz-prose#22: + decision: declined +2026-10-04T152206-quiz-prose#3: + decision: adopted +2026-10-04T152206-quiz-prose#4: + decision: declined +2026-10-04T152206-quiz-prose#5: + decision: adopted +2026-10-04T152206-quiz-prose#6: + decision: adopted +2026-10-04T152206-quiz-prose#7: + decision: adopted +2026-10-04T152206-quiz-prose#8: + decision: declined +2026-10-04T152206-quiz-prose#9: + decision: declined +2026-10-04T152433-quiz-prose#1: + decision: declined +2026-10-04T152433-quiz-prose#10: + decision: declined +2026-10-04T152433-quiz-prose#11: + decision: declined +2026-10-04T152433-quiz-prose#12: + decision: declined +2026-10-04T152433-quiz-prose#13: + decision: declined +2026-10-04T152433-quiz-prose#14: + decision: declined +2026-10-04T152433-quiz-prose#15: + decision: declined +2026-10-04T152433-quiz-prose#16: + decision: declined +2026-10-04T152433-quiz-prose#17: + decision: declined +2026-10-04T152433-quiz-prose#18: + decision: declined +2026-10-04T152433-quiz-prose#19: + decision: declined +2026-10-04T152433-quiz-prose#2: + decision: declined +2026-10-04T152433-quiz-prose#20: + decision: declined +2026-10-04T152433-quiz-prose#21: + decision: declined +2026-10-04T152433-quiz-prose#3: + decision: declined +2026-10-04T152433-quiz-prose#4: + decision: declined +2026-10-04T152433-quiz-prose#5: + decision: declined +2026-10-04T152433-quiz-prose#6: + decision: declined +2026-10-04T152433-quiz-prose#7: + decision: adopted +2026-10-04T152433-quiz-prose#8: + decision: declined +2026-10-04T152433-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151610-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151610-probe.json new file mode 100644 index 0000000..e2de577 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151610-probe.json @@ -0,0 +1,230 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:16:10", + "model": "default", + "prompt_sha": "e5129749a45f", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "'About 100 carry a resistance mutation' (N × rate) counts mutation events but calls them resistant cells. Descendants are left out.", + "why_it_matters": "The file's own Luria–Delbrück row says an early mutation leaves many resistant descendants. So the number of resistant cells is larger than N × rate, and it varies a lot from culture to culture. The causal chain and the temporal sequence both rely on this number, and the step from mutation events to resistant cells is left implicit.", + "suggested_addition": "Keep two separate names: 'mutation events ≈ N × μ = 10^10 × 10^-8 = 100' and 'resistant cells'. Then add: mutants that arose in earlier generations have divided since, so the expected number of resistant cells is about μN·ln(N) ≈ 10^-8 × 10^10 × 23 ≈ 2,300, and the median is about μN·ln(μN) ≈ 100 × 4.6 ≈ 460. State that 100 is a lower bound." + }, + { + "rule": 2, + "severity": "main", + "about": "The model never says why the roughly 100 surviving cells are not cleared by the host. It assumes nothing kills resistant cells after treatment.", + "why_it_matters": "A learner will expect the immune system to mop up 100 cells. Without this, step 3 (regrowth to 10^10) looks unjustified, and the 'small population is cleared' case seems to rest only on the drug.", + "suggested_addition": "State the assumption: host defences remove some cells but cannot keep up with a population that doubles. Give an example: if immune killing removes 50% of cells per doubling time, net growth is zero; if it removes 20%, 100 cells still reach 10^10 in about 27/log2(1.6) ≈ 40 doublings. Say that weakened patients (ICU, chemotherapy) clear fewer cells." + }, + { + "rule": 4, + "severity": "main", + "about": "The four resistance mechanisms (destroy, change target, pump out, keep out) are only listed. None has a concrete example, and the learning order puts this as step 2.", + "why_it_matters": "The audience explicitly does not know how antibiotics act on a cell. 'A change in DNA can stop that binding' stays abstract with no instance.", + "suggested_addition": "Give one line each. Destroy: a beta-lactamase enzyme cuts penicillin's beta-lactam ring before it reaches the wall-building enzymes. Change target: one base change in the rpoB gene stops rifampicin binding. Pump out: an efflux pump exports tetracycline. Keep out: loss of a porin channel cuts drug entry. Link one example to MIC numbers, e.g. beta-lactamase raises MIC from 0.5 to 32 mg/L (64-fold)." + }, + { + "rule": 2, + "severity": "main", + "about": "The 'breaks' column for combination therapy compares with one drug instead of removing the stated assumption, which is that the two resistances are independent.", + "why_it_matters": "The file itself says plasmids carry several resistance genes at once, and a single efflux pump can export both drugs. Either way the 10^-16 product fails. The claim also silently assumes both drugs are given together and that each drug alone kills cells resistant to the other.", + "suggested_addition": "Breaks case: if one plasmid or pump gives resistance to both drugs at about 10^-8, then expected double-resistant cells = 10^10 × 10^-8 = 100. Second breaks case: drug A alone leaves about 100 A-resistant cells. They regrow to 10^10 and then drug B is added: 10^10 × 10^-8 = 100 cells resistant to both." + }, + { + "rule": 3, + "severity": "main", + "about": "Several concepts have two names or two definitions. 'Resistance gene / resistance mutation' is one entity. 'Resistant' is defined both as 'survives the dose used' and as 'higher MIC'. Antibiotic 'kills or stops growth' (Entities) vs 'kills' (Terms). MIC 'stops growth' but the example says 8 mg/L 'kills'. Division / doubling / generation are used for the same thing.", + "why_it_matters": "A reader cannot tell whether resistance is relative to a dose or to a strain. They also cannot tell whether exceeding the MIC kills cells or only stops their growth.", + "suggested_addition": "Use one term, 'resistance determinant (mutation or acquired gene)'. Define resistant as 'MIC above the concentration reachable in the patient (e.g. MIC 32 > 8 mg/L)'. Say MIC measures growth inhibition, not killing. Pick one word, 'doubling', for one division of every cell." + }, + { + "rule": 5, + "severity": "edge", + "about": "'Low MIC ratio means the drug still works' uses an undefined ratio and gives no threshold value.", + "why_it_matters": "The reader cannot judge what number counts as resistant.", + "suggested_addition": "Define the ratio as reachable concentration ÷ MIC. 8/0.5 = 16 means the drug works; 8/32 = 0.25 means it fails. Give the cutoff (a clinical breakpoint), e.g. ratio ≥ about 1 to 4 counts as susceptible (illustrative)." + }, + { + "rule": 1, + "severity": "edge", + "about": "The fitness cost compounds multiplicatively (0.95^100). There is no justification against the linear alternative.", + "why_it_matters": "A learner may compute 1 − 0.05 × 100 = −4 (or 0) and get confused.", + "suggested_addition": "Each generation the resistant/susceptible ratio is multiplied by 0.95, because both lineages compound. So after 100 generations the ratio is 0.95^100 ≈ 0.006, not 1 − 5 = negative. After 10 generations it is ≈ 0.60." + }, + { + "rule": 1, + "severity": "edge", + "about": "The 'many 0, a few hundreds' result of Luria–Delbrück has no concrete numbers or comparison statistic.", + "why_it_matters": "The argument against 'the drug induces resistance' depends on variance versus mean, and that is never shown.", + "suggested_addition": "Example: 20 parallel cultures, mean about 10 resistant colonies. Induction (Poisson) predicts most counts between about 4 and 16, with variance ≈ mean ≈ 10. Observed counts were 11 zeros, several 1–5, and one of 107, so variance ≫ mean (illustrative; recompute)." + }, + { + "rule": 4, + "severity": "edge", + "about": "Plasmid transfer and spread to other hosts and species have no mechanism (conjugation) and no numbers.", + "why_it_matters": "'No waiting for a rare error' implies transfer is fast, but no rate is given to compare with 10^-8.", + "suggested_addition": "Name conjugation: cell-to-cell contact through a pilus copies the plasmid. Give an illustrative rate of 10^-4 to 10^-2 transfers per donor per hour in dense gut populations, versus 10^-8 per division for mutation." + }, + { + "rule": 3, + "severity": "edge", + "about": "'Compensatory mutations' appears in the confusion points without definition or numbers.", + "why_it_matters": "It explains why resistance persists after the drug is stopped, which is a likely follow-up question.", + "suggested_addition": "Definition: a second mutation that restores growth rate without losing resistance. Example: cost goes from 5% to 1%, so after 100 generations the fraction factor is 0.99^100 ≈ 0.37 instead of 0.006." + }, + { + "rule": 2, + "severity": "edge", + "about": "'A cell has the resistance gene or not' conflicts with stepwise resistance, where MIC can rise in steps.", + "why_it_matters": "The binary claim breaks for low-level mutations that accumulate.", + "suggested_addition": "Add the case: a first mutation raises MIC from 0.5 to 2 mg/L (still killed at 8), and a second raises it to 16 mg/L (survives). Each step is selected by sub-MIC or low exposure." + }, + { + "rule": 5, + "severity": "edge", + "about": "In-patient regrowth time is described only as 'slower' with no value.", + "why_it_matters": "The reader cannot tell when the relapse occurs.", + "suggested_addition": "Illustrative: a 2–4 h doubling in tissue gives 27 doublings in about 2–4.5 days, versus 13 h in the lab." + }, + { + "rule": 5, + "severity": "edge", + "about": "The 0.01 expected count is not turned into a probability.", + "why_it_matters": "'Usually none' is vague.", + "suggested_addition": "P(at least one resistant cell) = 1 − e^-0.01 ≈ 1% for 10^6 cells. For 10^10 cells (expected 100), P ≈ 1." + } + ], + "excess": [ + "Representation decision section (Stage, Levels): this is production metadata, not content the reader needs", + "Epistemic 'Disputed: shortened course' line: it is out of scope and not needed for the central question" + ], + "next_questions": [ + { + "q": "Why doesn't my immune system kill the 100 resistant cells that survive?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "What physically happens inside the cell to block the drug, e.g. what does a beta-lactamase do?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Does stopping the antibiotic course early cause resistance?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "How does a plasmid get from one bacterium to another, and how fast?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If resistance has a fitness cost, why doesn't it disappear when we stop using the drug?", + "answered_by_model": true, + "declared_out_of_scope": false + } + ] + }, + "usage": { + "cost": 0.1673884, + "seconds": 62 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "'About 100 carry a resistance mutation' (N × rate) counts mutation events but calls them resistant cells. Descendants are left out. → Keep two separate names: 'mutation events ≈ N × μ = 10^10 × 10^-8 = 100' and 'resistant cells'. Then add: mutants that arose in earlier generations have divided since, so the expected number of resistant cells is about μN·ln(N) ≈ 10^-8 × 10^10 × 23 ≈ 2,300, and the median is about μN·ln(μN) ≈ 100 × 4.6 ≈ 460. State that 100 is a lower bound." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "The model never says why the roughly 100 surviving cells are not cleared by the host. It assumes nothing kills resistant cells after treatment. → State the assumption: host defences remove some cells but cannot keep up with a population that doubles. Give an example: if immune killing removes 50% of cells per doubling time, net growth is zero; if it removes 20%, 100 cells still reach 10^10 in about 27/log2(1.6) ≈ 40 doublings. Say that weakened patients (ICU, chemotherapy) clear fewer cells." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 4", + "text": "The four resistance mechanisms (destroy, change target, pump out, keep out) are only listed. None has a concrete example, and the learning order puts this as step 2. → Give one line each. Destroy: a beta-lactamase enzyme cuts penicillin's beta-lactam ring before it reaches the wall-building enzymes. Change target: one base change in the rpoB gene stops rifampicin binding. Pump out: an efflux pump exports tetracycline. Keep out: loss of a porin channel cuts drug entry. Link one example to MIC numbers, e.g. beta-lactamase raises MIC from 0.5 to 32 mg/L (64-fold)." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 2", + "text": "The 'breaks' column for combination therapy compares with one drug instead of removing the stated assumption, which is that the two resistances are independent. → Breaks case: if one plasmid or pump gives resistance to both drugs at about 10^-8, then expected double-resistant cells = 10^10 × 10^-8 = 100. Second breaks case: drug A alone leaves about 100 A-resistant cells. They regrow to 10^10 and then drug B is added: 10^10 × 10^-8 = 100 cells resistant to both." + }, + { + "n": 5, + "severity": "main", + "kind": "rule 3", + "text": "Several concepts have two names or two definitions. 'Resistance gene / resistance mutation' is one entity. 'Resistant' is defined both as 'survives the dose used' and as 'higher MIC'. Antibiotic 'kills or stops growth' (Entities) vs 'kills' (Terms). MIC 'stops growth' but the example says 8 mg/L 'kills'. Division / doubling / generation are used for the same thing. → Use one term, 'resistance determinant (mutation or acquired gene)'. Define resistant as 'MIC above the concentration reachable in the patient (e.g. MIC 32 > 8 mg/L)'. Say MIC measures growth inhibition, not killing. Pick one word, 'doubling', for one division of every cell." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 5", + "text": "'Low MIC ratio means the drug still works' uses an undefined ratio and gives no threshold value. → Define the ratio as reachable concentration ÷ MIC. 8/0.5 = 16 means the drug works; 8/32 = 0.25 means it fails. Give the cutoff (a clinical breakpoint), e.g. ratio ≥ about 1 to 4 counts as susceptible (illustrative)." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 1", + "text": "The fitness cost compounds multiplicatively (0.95^100). There is no justification against the linear alternative. → Each generation the resistant/susceptible ratio is multiplied by 0.95, because both lineages compound. So after 100 generations the ratio is 0.95^100 ≈ 0.006, not 1 − 5 = negative. After 10 generations it is ≈ 0.60." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 1", + "text": "The 'many 0, a few hundreds' result of Luria–Delbrück has no concrete numbers or comparison statistic. → Example: 20 parallel cultures, mean about 10 resistant colonies. Induction (Poisson) predicts most counts between about 4 and 16, with variance ≈ mean ≈ 10. Observed counts were 11 zeros, several 1–5, and one of 107, so variance ≫ mean (illustrative; recompute)." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 4", + "text": "Plasmid transfer and spread to other hosts and species have no mechanism (conjugation) and no numbers. → Name conjugation: cell-to-cell contact through a pilus copies the plasmid. Give an illustrative rate of 10^-4 to 10^-2 transfers per donor per hour in dense gut populations, versus 10^-8 per division for mutation." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 3", + "text": "'Compensatory mutations' appears in the confusion points without definition or numbers. → Definition: a second mutation that restores growth rate without losing resistance. Example: cost goes from 5% to 1%, so after 100 generations the fraction factor is 0.99^100 ≈ 0.37 instead of 0.006." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 2", + "text": "'A cell has the resistance gene or not' conflicts with stepwise resistance, where MIC can rise in steps. → Add the case: a first mutation raises MIC from 0.5 to 2 mg/L (still killed at 8), and a second raises it to 16 mg/L (survives). Each step is selected by sub-MIC or low exposure." + }, + { + "n": 12, + "severity": "edge", + "kind": "rule 5", + "text": "In-patient regrowth time is described only as 'slower' with no value. → Illustrative: a 2–4 h doubling in tissue gives 27 doublings in about 2–4.5 days, versus 13 h in the lab." + }, + { + "n": 13, + "severity": "edge", + "kind": "rule 5", + "text": "The 0.01 expected count is not turned into a probability. → P(at least one resistant cell) = 1 − e^-0.01 ≈ 1% for 10^6 cells. For 10^10 cells (expected 100), P ≈ 1." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Representation decision section (Stage, Levels): this is production metadata, not content the reader needs" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Epistemic 'Disputed: shortened course' line: it is out of scope and not needed for the central question" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151845-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151845-coldread-narrative.json new file mode 100644 index 0000000..a85dd69 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151845-coldread-narrative.json @@ -0,0 +1,417 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:18:45", + "model": "default", + "prompt_sha": "3c4725ae6cae", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Said: 'Selection of variants that existed before the drug; not the drug changing them, not the patient.' Shown: bold answer.", + "unresolved": [ + "'not the patient': the reader hasn't been told the common belief this rejects (that the patient's body becomes resistant), so the denial means little" + ], + "unsaid": [], + "leap": [ + "The answer comes before any question is said. The reader can only guess that the topic is how resistance happens." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (mechanism)", + "text": "Said: 'binds a target the cell needs'. Shown: nothing.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Said: 'four ways; each raises MIC; resistant = MIC above concentration reached'. Shown: table of four ways, MIC example.", + "unresolved": [ + "'MIC': the abbreviation is never expanded or defined in the Said column", + "'concentration reached': it doesn't say where (in the blood? at the infection site?)" + ], + "unsaid": [ + "The names of the four ways appear only in the table, and none is explained in words", + "The numbers in the MIC example (two strains against the concentration reached) are shown but not walked through" + ], + "leap": [ + "It doesn't say how each of the four ways blocks the 'binds a target' step from beat 2, or why that 'raises MIC'" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (instance)", + "text": "Said: '10^10 × 10^-8 = 100; at least 100 resistant cells; 10^6 gives about 1%'. Shown: arithmetic.", + "unresolved": [ + "'10^10': not said to be the number of cells in an infection", + "'10^-8': not said to be a mutation rate per cell division" + ], + "unsaid": [], + "leap": [ + "'at least 100': 10^10 × 10^-8 is an expected count of new mutations. 'At least' needs a reason, such as earlier mutants having divided, and none is given", + "'10^6 gives about 1%': 10^6 × 10^-8 = 0.01 is an expected count. Calling it '1%' leaves unclear whether this is a 1% chance of one resistant cell or 1% of the cells" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Said: 'removes susceptible; 100 regrow in 27 doublings, about 13 h in the lab; immune system does not keep up'. Shown: steps.", + "unresolved": [], + "unsaid": [], + "leap": [ + "'immune system does not keep up' is asserted with no reason", + "'about 13 h' assumes a doubling time (about 29 min) that is never stated" + ], + "excess": [ + "'about 13 h in the lab': a lab figure that the argument doesn't need" + ], + "purpose_unclear": false + }, + { + "at": "Beat 6 (objection)", + "text": "Said: '20 cultures, mean 11, induction predicts 5–18 each; observed 11 zeros and a 107'. Shown: LD numbers.", + "unresolved": [ + "'induction': never defined as the drug causing the change", + "'LD': a label that is never expanded (Luria–Delbrück)", + "'cultures': it doesn't say what was grown, what was counted, or that it was counted after exposure to the drug" + ], + "unsaid": [ + "The 'LD numbers' display isn't tied to a named experiment in the Said column" + ], + "leap": [ + "It doesn't say why induction predicts 5–18 for each culture (similar counts everywhere)", + "It doesn't say why 11 zeros and a 107 point to mutations that existed before the drug. Mutants arising early and multiplying into jackpots is never given as the reasoning" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (generalize)", + "text": "Said: 'plasmids, conjugation, across species; whole enzymes'. Shown: nothing.", + "unresolved": [ + "'plasmids'", + "'conjugation'", + "'whole enzymes': unclear what this refers to" + ], + "unsaid": [], + "leap": [ + "It doesn't say how a gene moves from one bacterium to another, or why that removes the wait for each lineage's own mutation" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (test)", + "text": "Bridge: 'the guarantee needs the drug'. Said: 'cost 5%: 0.006; compensatory 1%: 0.37'. Shown: numbers.", + "unresolved": [ + "'cost': fitness cost is never defined", + "'compensatory': never defined", + "'the guarantee': no earlier beat named a guarantee" + ], + "unsaid": [ + "The exponent 100 (generations) that produces 0.006 and 0.37 is never said", + "It doesn't say what 0.006 and 0.37 are fractions of" + ], + "leap": [ + "It doesn't say why resistance costs growth when the drug is absent" + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 9 (payoff)", + "text": "Said: 'small infections rarely; two independent drugs 10^-16; breaks if one plasmid covers both'. Shown: two lines.", + "unresolved": [], + "unsaid": [], + "leap": [ + "10^-16 is presumably 10^-8 squared, but the multiplication isn't said" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 10 (close)", + "text": "Said: 'answer in the opening words; scope list'.", + "unresolved": [ + "'scope list': its contents aren't in the script" + ], + "unsaid": [], + "leap": [], + "excess": [ + "Restating the opening answer word for word adds nothing new unless the general argument is added" + ], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 4: '10^10 × 10^-8 = 100'. The two numbers aren't identified as cells and mutation rate, so the count that the whole argument rests on can't be followed", + "Beat 6: 'induction predicts 5–18 each; observed 11 zeros and a 107'. 'Induction' is undefined and the reasoning from the spread of counts to 'existed before the drug' is missing, so the key evidence doesn't land", + "Beat 7: 'plasmids, conjugation'. Both are undefined for a reader who knows nothing of gene exchange", + "Beat 8: 'cost 5%: 0.006; compensatory 1%: 0.37'. Cost, compensatory, the 100 generations and 'the guarantee' are all undefined" + ], + "cuts": [ + "Beat 5: 'about 13 h in the lab'. Cutting it removes an unexplained conversion", + "Beat 3: name and explain one mechanism (for example, an enzyme that destroys the drug) instead of a table of four unexplained names", + "Beat 10: replace the scope list with the one-sentence general argument" + ], + "question": { + "text": "How does an infection that a drug once cleared become one it can't clear? The question is never said in the Said column; the reader infers it from the hook's answer in Beat 1.", + "known_at": "Beat 1, only by inference", + "result_stated": "Yes, Beat 1: 'Selection of variants that existed before the drug; not the drug changing them, not the patient.'" + }, + "motive": "Missing from the script. The plan's reasons to care (the 'bacteria get used to the drug' picture predicts the wrong things) and the reason for the approach (start from what the drug does, then count) never appear in the Said column.", + "close": "Partly. Beat 10 repeats 'the answer in the opening words', but the general argument ('any heritable variant that grows under the drug while others are stopped takes over') is never said in the beats.", + "recap": "Resistant bacteria are already there by chance, or they get the gene from other bacteria; the drug just kills their competitors so they take over.", + "strongest": "Beat 4: one multiplication turns 'resistance must be caused by the drug' into 'about 100 resistant cells are already present', which is the surprise the explanation is built around.", + "weakest": "Beat 6: the Luria–Delbrück evidence is given as bare numbers, with 'induction' undefined and no reasoning from the jackpot pattern to the conclusion.", + "suggestions": [ + "Say in Beat 4: 'an infection holds about 10^10 bacteria; each division makes this mutation about once in 10^8 times'", + "Define induction in Beat 6, then say: if the drug caused it, every culture would have similar counts; if mutations came first, a mutant that arose early multiplies into a jackpot, so most cultures have none and a few have many", + "Define a plasmid as a small, separate ring of DNA that bacteria copy and pass to each other through contact (conjugation)", + "In Beat 8, say that the resistant cells grow 5% slower per generation, so after 100 generations they are 0.95^100 ≈ 0.6% of what they would otherwise be", + "State the motive in Beat 1: the 'bacteria get used to it' picture predicts the wrong things" + ] + }, + "usage": { + "cost": 0.12325699999999998, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: '10^10 × 10^-8 = 100'. The two numbers aren't identified as cells and mutation rate, so the count that the whole argument rests on can't be followed" + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 6: 'induction predicts 5–18 each; observed 11 zeros and a 107'. 'Induction' is undefined and the reasoning from the spread of counts to 'existed before the drug' is missing, so the key evidence doesn't land" + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 7: 'plasmids, conjugation'. Both are undefined for a reader who knows nothing of gene exchange" + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 8: 'cost 5%: 0.006; compensatory 1%: 0.37'. Cost, compensatory, the 100 generations and 'the guarantee' are all undefined" + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 5: 'about 13 h in the lab'. Cutting it removes an unexplained conversion" + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 3: name and explain one mechanism (for example, an enzyme that destroys the drug) instead of a table of four unexplained names" + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 10: replace the scope list with the one-sentence general argument" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): 'not the patient': the reader hasn't been told the common belief this rejects (that the patient's body becomes resistant), so the denial means little" + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): The answer comes before any question is said. The reader can only guess that the topic is how resistance happens." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): 'MIC': the abbreviation is never expanded or defined in the Said column" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): 'concentration reached': it doesn't say where (in the blood? at the infection site?)" + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): The names of the four ways appear only in the table, and none is explained in words" + }, + { + "n": 13, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): The numbers in the MIC example (two strains against the concentration reached) are shown but not walked through" + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): It doesn't say how each of the four ways blocks the 'binds a target' step from beat 2, or why that 'raises MIC'" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (instance): '10^10': not said to be the number of cells in an infection" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (instance): '10^-8': not said to be a mutation rate per cell division" + }, + { + "n": 17, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (instance): 'at least 100': 10^10 × 10^-8 is an expected count of new mutations. 'At least' needs a reason, such as earlier mutants having divided, and none is given" + }, + { + "n": 18, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (instance): '10^6 gives about 1%': 10^6 × 10^-8 = 0.01 is an expected count. Calling it '1%' leaves unclear whether this is a 1% chance of one resistant cell or 1% of the cells" + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 'immune system does not keep up' is asserted with no reason" + }, + { + "n": 20, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 'about 13 h' assumes a doubling time (about 29 min) that is never stated" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (objection): 'induction': never defined as the drug causing the change" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (objection): 'LD': a label that is never expanded (Luria–Delbrück)" + }, + { + "n": 23, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (objection): 'cultures': it doesn't say what was grown, what was counted, or that it was counted after exposure to the drug" + }, + { + "n": 24, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (objection): The 'LD numbers' display isn't tied to a named experiment in the Said column" + }, + { + "n": 25, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (objection): It doesn't say why induction predicts 5–18 for each culture (similar counts everywhere)" + }, + { + "n": 26, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (objection): It doesn't say why 11 zeros and a 107 point to mutations that existed before the drug. Mutants arising early and multiplying into jackpots is never given as the reasoning" + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (generalize): 'plasmids'" + }, + { + "n": 28, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (generalize): 'conjugation'" + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (generalize): 'whole enzymes': unclear what this refers to" + }, + { + "n": 30, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (generalize): It doesn't say how a gene moves from one bacterium to another, or why that removes the wait for each lineage's own mutation" + }, + { + "n": 31, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (test): 'cost': fitness cost is never defined" + }, + { + "n": 32, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (test): 'compensatory': never defined" + }, + { + "n": 33, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (test): 'the guarantee': no earlier beat named a guarantee" + }, + { + "n": 34, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (test): The exponent 100 (generations) that produces 0.006 and 0.37 is never said" + }, + { + "n": 35, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (test): It doesn't say what 0.006 and 0.37 are fractions of" + }, + { + "n": 36, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (test): It doesn't say why resistance costs growth when the drug is absent" + }, + { + "n": 37, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (payoff): 10^-16 is presumably 10^-8 squared, but the multiplication isn't said" + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 10 (close): 'scope list': its contents aren't in the script" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152038-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152038-coldread-prose.json new file mode 100644 index 0000000..e3a83bf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152038-coldread-prose.json @@ -0,0 +1,328 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:20:38", + "model": "default", + "prompt_sha": "fbc694b7c598", + "reply": { + "paragraphs": [ + { + "at": "Lead (bold paragraph under the title)", + "text": "The drug does not change the bacteria. In a large infection, a few bacteria already carry a resistance gene... The bacteria become resistant, not the patient's body.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What a cell must block, para 1", + "text": "An antibiotic is a molecule that kills bacteria or stops their growth... penicillin binds the enzymes that build the cell wall.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What a cell must block, para 2", + "text": "A resistance gene is a gene, changed by mutation or received from another bacterium, that blocks this. It works in one of four ways:", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Table of four ways", + "text": "Destroy the drug / Change the target / Pump the drug out / Keep the drug out", + "unresolved": [ + "rpoB gene", + "outer membrane" + ], + "leap": [], + "excess": [ + "One DNA letter change in the rpoB gene" + ], + "purpose_unclear": false + }, + { + "at": "What a cell must block, MIC paragraph", + "text": "Each way means the cell needs more drug to stop it. The lowest concentration ... is its MIC ... (Illustrative values.)", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Resistant cells are there before the drug, para 1", + "text": "A given resistance mutation happens about once per 10⁸ cell divisions ... Growing to that size took about 10¹⁰ divisions, so expect", + "unresolved": [], + "leap": [ + "Growing to that size took about 10¹⁰ divisions: the reason (going from 1 cell to N cells takes about N−1 divisions) is not given" + ], + "excess": [ + "(order of magnitude; real values range from about 10⁻⁶ to 10⁻¹⁰)" + ], + "purpose_unclear": false + }, + { + "at": "Resistant cells..., equation line", + "text": "10¹⁰ × 10⁻⁸ = 100 resistance mutations, before anyone gives a drug.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Resistant cells..., para 3", + "text": "Early mutants have divided since, so there are at least 100 resistant cells. Compare a small infection of 10⁶ cells...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The drug selects them, item 1", + "text": "The drug stops the susceptible cells, and the immune system clears them.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The drug selects them, item 2", + "text": "The resistant cells keep dividing... (A weakened patient clears fewer.)", + "unresolved": [], + "leap": [], + "excess": [ + "(A weakened patient clears fewer.)" + ], + "purpose_unclear": false + }, + { + "at": "The drug selects them, item 3", + "text": "A doubling is one round in which every cell divides once. 100 cells need 27 doublings to reach 10¹⁰ again (2²⁷ ≈ 1.3 × 10⁸)...", + "unresolved": [], + "leap": [ + "(2²⁷ ≈ 1.3 × 10⁸): the reader has to multiply by the starting 100 cells to get 10¹⁰, and the text never says so" + ], + "excess": [ + "A doubling is one round in which every cell divides once." + ], + "purpose_unclear": false + }, + { + "at": "The drug selects them, item 4", + "text": "The infection returns, and now almost every cell is resistant.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The drug selects them, para after list", + "text": "This is selection: the drug did not make resistant cells, it removed everything else.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The drug selects them, Luria–Delbrück paragraph", + "text": "How do we know the drug did not cause the change? ... They found 11 cultures with 0 and one with 107...", + "unresolved": [ + "with a virus as the killing agent" + ], + "leap": [ + "so the \"drug causes it\" idea predicts most cultures between 5 and 18: the text never says where the 5–18 range comes from" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Genes from other bacteria, para 1", + "text": "A cell does not have to wait for its own rare mutation. A plasmid is a small circle of DNA... horizontal gene transfer...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Genes from other bacteria, para 2", + "text": "Plasmids carry complete working genes... The same plasmid gene turns up in different species, and one plasmid often carries several resistance genes at once.", + "unresolved": [], + "leap": [ + "The same plasmid gene turns up in different species: the text doesn't say this is evidence of transfer, or what it is meant to show" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Without the drug, para 1", + "text": "Resistant cells take over only while the drug is present. Without the drug, resistance often has a fitness cost...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Without the drug, para 2", + "text": "Suppose resistant cells grow 5% slower... Plasmids also persist.", + "unresolved": [], + "leap": [ + "Plasmids also persist.: no reason is given" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What this predicts, bullet 1", + "text": "Large infections are the risk, because they hold resistant cells before treatment.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What this predicts, bullet 2", + "text": "Two drugs with independent targets need two mutations in one cell... This fails if one plasmid or one pump resists both drugs: then about 100 cells survive again.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "In short", + "text": "resistance genes come first, by chance or from other bacteria. The drug removes the competitors, and the resistant cells multiply.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered", + "text": "Persister cells... transformation and transduction... SOS response... biofilms.", + "unresolved": [ + "transformation and transduction", + "biofilms" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Resistant cells..., para 1: drop the parenthetical \"(order of magnitude; real values range from about 10⁻⁶ to 10⁻¹⁰)\" or shorten it", + "Selects them, item 3: drop the definition \"A doubling is one round in which every cell divides once.\" because this reader knows exponential growth", + "Selects them, item 2: drop \"(A weakened patient clears fewer.)\"", + "Table: shorten \"One DNA letter change in the rpoB gene\" to \"One DNA letter change\"" + ], + "question": { + "text": "How do bacteria come to resist an antibiotic: does the drug cause the change, or does it select bacteria that were already resistant?", + "known_at": "Title and the bold lead paragraph", + "result_stated": "Yes, in the lead: \"The drug does not change the bacteria... The drug removes their competitors, and the resistant bacteria multiply.\"" + }, + "motive": "Neither motive is stated outright. A reason to care is only implied, by \"The infection returns, and now almost every cell is resistant.\" There is no stated reason for the counting approach, though the arithmetic in \"10¹⁰ × 10⁻⁸ = 100 resistance mutations, before anyone gives a drug\" makes the case by example.", + "close": "Yes: \"In short: resistance genes come first, by chance or from other bacteria. The drug removes the competitors, and the resistant cells multiply.\"", + "recap": "A big infection already holds about a hundred resistant bacteria before treatment, from random mutation or genes passed between bacteria, and the drug just clears away everything else so they take over.", + "strongest": "'Resistant cells are there before the drug': the 10¹⁰ × 10⁻⁸ = 100 calculation, together with the Luria–Delbrück paragraph that tests the idea.", + "weakest": "'Genes from other bacteria', para 2: the facts in it (the same gene in different species, several genes on one plasmid) are stated without saying what they show.", + "suggestions": [ + "Say where 5–18 comes from (Poisson spread: about mean ± 2√mean).", + "Explain that a virus can infect and kill bacteria (a bacteriophage).", + "Add one sentence on why resistance matters: treatment fails and the patient relapses.", + "Say why plasmids persist without the drug, or cut that sentence.", + "Write the doubling arithmetic as 100 × 2²⁷ ≈ 1.3 × 10¹⁰." + ] + }, + "usage": { + "cost": 0.13425979999999998, + "seconds": 44 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Resistant cells..., para 1: drop the parenthetical \"(order of magnitude; real values range from about 10⁻⁶ to 10⁻¹⁰)\" or shorten it" + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Selects them, item 3: drop the definition \"A doubling is one round in which every cell divides once.\" because this reader knows exponential growth" + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Selects them, item 2: drop \"(A weakened patient clears fewer.)\"" + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "Table: shorten \"One DNA letter change in the rpoB gene\" to \"One DNA letter change\"" + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Table of four ways: rpoB gene" + }, + { + "n": 6, + "severity": "edge", + "kind": "unresolved", + "text": "Table of four ways: outer membrane" + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Resistant cells are there before the drug, para 1: Growing to that size took about 10¹⁰ divisions: the reason (going from 1 cell to N cells takes about N−1 divisions) is not given" + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "The drug selects them, item 3: (2²⁷ ≈ 1.3 × 10⁸): the reader has to multiply by the starting 100 cells to get 10¹⁰, and the text never says so" + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "The drug selects them, Luria–Delbrück paragraph: with a virus as the killing agent" + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "The drug selects them, Luria–Delbrück paragraph: so the \"drug causes it\" idea predicts most cultures between 5 and 18: the text never says where the 5–18 range comes from" + }, + { + "n": 11, + "severity": "edge", + "kind": "leap", + "text": "Genes from other bacteria, para 2: The same plasmid gene turns up in different species: the text doesn't say this is evidence of transfer, or what it is meant to show" + }, + { + "n": 12, + "severity": "edge", + "kind": "leap", + "text": "Without the drug, para 2: Plasmids also persist.: no reason is given" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: transformation and transduction" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: biofilms" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152206-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152206-quiz-prose.json new file mode 100644 index 0000000..1fab995 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152206-quiz-prose.json @@ -0,0 +1,220 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:21:51", + "model": "default", + "prompt_sha": "57d2fb9e88c1", + "reply": { + "answers": { + "1": "The main objects are: antibiotics (molecules that kill bacteria or stop their growth); targets (molecules the cell needs, such as the enzymes that build the cell wall); resistance genes and resistance mutations; and bacterial cells, both susceptible and resistant. Also involved are the MIC (minimum inhibitory concentration) and the drug concentration in blood, plasmids, conjugation and horizontal gene transfer, the immune system, fitness cost, compensatory mutations, and combination therapy.", + "2": "An antibiotic works by binding a target. A resistance gene blocks this in one of four ways: it destroys the drug (beta-lactamase), changes the target (rifampicin), pumps the drug out (efflux pump), or keeps the drug out (loss of a porin). Each of these raises the strain's MIC. If the MIC is above the drug level reached in blood, the strain is resistant; if below, it is susceptible. Resistance genes come from random copying errors during division or from other bacteria via plasmids passed by conjugation. The drug, with help from the immune system, removes susceptible cells, and the resistant ones multiply. When the drug is absent, a fitness cost works against resistant cells, but compensatory mutations can reduce that cost.", + "3": "In a large population, random mutation or gene transfer has already produced a few resistant cells before treatment. The drug removes their susceptible competitors, so the resistant cells grow back exponentially and the infection returns, now resistant.", + "4": "The key variable is population size. Expected resistant mutants are roughly N × 10⁻⁸. At 10¹⁰ cells about 100 mutants already exist, so treatment fails. At 10⁶ cells there is only about a 1% chance that even one exists, so treatment usually works. Using two drugs with independent targets multiplies the rates to 10⁻¹⁶, giving an expected 10⁻⁶ double mutants in 10¹⁰ cells, so the infection clears. That protection is lost if one plasmid or pump resists both drugs.", + "5": "Assumption or simplification: each division adds one cell, so reaching 10¹⁰ cells took about 10¹⁰ divisions. The mutation rate of 10⁻⁸ is an order of magnitude, and the MIC values are labelled illustrative. Observation: in Luria and Delbrück's 1943 experiment, 11 of 20 cultures had 0 resistant colonies and one had 107, far more spread than random scatter around a mean of 11 would give.", + "6": "Antibiotics kill bacteria by binding a target the cell needs. Resistance genes block this by destroying the drug, changing the target, pumping the drug out, or keeping it out, which raises the MIC above the drug level the body reaches. In a large infection, random mutation (about 1 per 10⁸ divisions) or plasmids received from other bacteria have already produced some resistant cells before any drug is given. The drug removes the susceptible cells, and the resistant ones regrow within hours to days. Without the drug, resistance often slows growth, but compensatory mutations let it persist.", + "7": "The Luria–Delbrück paragraph was hardest. The text does not show where the range '5 to 18' comes from, and it does not explain why a virus counts as a 'killing agent' or how it relates to a drug. A short note that ordinary random scatter is about ±2×√11, plus a picture of early versus late mutant lineages, would make it easier to follow.", + "8": "Growing to 10¹⁰ cells took about 10¹⁰ divisions, so about 100 resistant mutants were already present before treatment. Drug X and the immune system cleared the susceptible majority, so symptoms improved. The roughly 100 resistant cells kept dividing; about 27 doublings bring them back to 10¹⁰, which takes about 13 hours at a lab doubling time and longer in a patient. The returning infection is almost entirely resistant. The drug selected these cells; it did not create them.", + "9": "No, not usually. 10⁶ × 10⁻⁸ = 0.01 expected mutants, so there is only about a 1% chance that a resistant cell exists, and the drug would usually clear the infection. The exception is if a resistance gene arrives on a plasmid, which does not depend on the population's own mutations.", + "10": "Not necessarily. If resistance costs 5% growth, the resistant-to-susceptible ratio falls to about 0.006 of its starting value after 100 generations, so it shrinks a lot but does not reach zero. If a compensatory mutation cuts the cost to 1%, the ratio is still about 0.37 after 100 generations. The explanation gives no timescale linking 'years' to generations, and it does not say whether plasmid-borne resistance carries a cost.", + "11": "Yes. About 100 resistance mutations are expected (10¹⁰ × 10⁻⁸), and because early mutants have divided since, there are at least 100 resistant cells, possibly more.", + "12": "If the drug caused resistance, every cell would have the same small chance when it met the drug, so equal cultures would give similar counts with only ordinary scatter (here, mostly 5 to 18 around a mean of 11). Luria and Delbrück instead found 11 cultures with 0 and one with 107. This fits random mutation before exposure: a mutant that arose early leaves many resistant descendants, and a late one leaves few. Separately, the regrowth story shows the drug only removes the competitors.", + "13": "Through horizontal gene transfer. In conjugation, one bacterium copies a plasmid carrying a complete working resistance gene (such as beta-lactamase) into another bacterium, which can be of a different species. One plasmid often carries several resistance genes at once.", + "14": "Without the drug, the fitness cost makes the resistant fraction shrink: with a 5% cost the ratio is multiplied by 0.95 each generation, giving about 0.006 after 100 generations. It may not disappear because a compensatory mutation can cut the cost to about 1% (0.37 remains after 100 generations). Also, the decline is gradual rather than to zero." + }, + "audit": { + "terms": [ + "The same thing is counted in three units: 'divisions' (lines 24, 26), 'doublings' (line 35) and 'generations' (line 54). The reader must work out that they relate.", + "'resistance gene' (line 11), 'resistance mutation' (lines 24, 26) and 'One DNA letter change' / 'One letter change' (lines 15, 47) are used for overlapping ideas. A mutation counts as a gene in one place and is contrasted with whole plasmid genes in another.", + "'kills bacteria or stops their growth' (line 9) vs 'stops' (lines 20, 33) vs 'removes' (line 38) vs 'killing agent' (line 40): it is unclear whether stopping growth and killing are treated as the same thing.", + "'Genes from other bacteria' (heading, line 42), 'received from another bacterium' (lines 5, 11), 'conjugation' and 'horizontal gene transfer' (line 45): the relationship between conjugation and HGT is given only once.", + "'competitors' (lines 5, 61) vs 'susceptible cells' (line 33) vs 'everything else' (line 38) all name the same group." + ], + "unexplained": [ + "'most cultures between 5 and 18' (line 40): the text does not say how this range follows from a mean of 11.", + "'5% slower' becomes 'multiplies the ratio … by 0.95' each generation (line 54): it does not say whose generation, or what a 5% growth difference means per unit of time.", + "'in a patient it takes longer' (line 35): no figure is given, so the reader cannot judge the timescale of relapse.", + "'at least 100 resistant cells' (line 28): it does not say how many more could exist when early mutants have divided.", + "'cut the cost to 1%' (line 54): no indication of how typical this is or how fast a compensatory mutation appears." + ], + "missing_why": [ + "Line 54: why a growth-rate penalty multiplies the ratio by (1 − cost) each generation.", + "Line 59: why independent targets let the probabilities multiply, and why a single pump can defeat two different drugs.", + "Line 34: why the immune system 'does not keep up' with the resistant cells when it did clear the susceptible ones.", + "Line 40: why a virus is equivalent to a drug for the logic of the experiment.", + "Line 54: why a compensatory mutation reduces the cost without losing the resistance." + ], + "excess": [ + "Line 3: an HTML comment with process metadata ('Stage 1, rendered from model.md in STE-80').", + "Lines 63–68: the 'Not covered' list names topics the reader cannot use." + ] + }, + "gaps": [ + "It does not say how fast a plasmid spreads through a population once it arrives, or whether plasmid-borne resistance also carries a fitness cost.", + "There is no timescale linking 'generations' to real time, so the reader cannot judge what 'years without the drug' means (Q10).", + "It does not say what keeps the immune system from clearing about 100 resistant cells.", + "The source of the Luria–Delbrück 'expected scatter' range is not given.", + "It does not explain how resistance spreads between patients or within a hospital, though Q10 implies it." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 1, + "11": 2, + "12": 2, + "13": 2, + "14": 2 + }, + "reasons": { + "10": "Correctly says resistance falls through fitness cost (0.95^100 ≈ 0.006), slowly and not to zero, and that compensatory mutations reduce the cost (0.99^100 ≈ 0.37). It omits the expected point that plasmids persist, and instead treats plasmid behaviour as unexplained. That is one of the two listed reasons resistance need not vanish, so the answer is partial." + }, + "pass": false + }, + "usage": { + "cost": 0.20742880000000002, + "seconds": 53 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same thing is counted in three units: 'divisions' (lines 24, 26), 'doublings' (line 35) and 'generations' (line 54). The reader must work out that they relate." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'resistance gene' (line 11), 'resistance mutation' (lines 24, 26) and 'One DNA letter change' / 'One letter change' (lines 15, 47) are used for overlapping ideas. A mutation counts as a gene in one place and is contrasted with whole plasmid genes in another." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'kills bacteria or stops their growth' (line 9) vs 'stops' (lines 20, 33) vs 'removes' (line 38) vs 'killing agent' (line 40): it is unclear whether stopping growth and killing are treated as the same thing." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'Genes from other bacteria' (heading, line 42), 'received from another bacterium' (lines 5, 11), 'conjugation' and 'horizontal gene transfer' (line 45): the relationship between conjugation and HGT is given only once." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "'competitors' (lines 5, 61) vs 'susceptible cells' (line 33) vs 'everything else' (line 38) all name the same group." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'most cultures between 5 and 18' (line 40): the text does not say how this range follows from a mean of 11." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'5% slower' becomes 'multiplies the ratio … by 0.95' each generation (line 54): it does not say whose generation, or what a 5% growth difference means per unit of time." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'in a patient it takes longer' (line 35): no figure is given, so the reader cannot judge the timescale of relapse." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'at least 100 resistant cells' (line 28): it does not say how many more could exist when early mutants have divided." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "'cut the cost to 1%' (line 54): no indication of how typical this is or how fast a compensatory mutation appears." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Line 54: why a growth-rate penalty multiplies the ratio by (1 − cost) each generation." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "Line 59: why independent targets let the probabilities multiply, and why a single pump can defeat two different drugs." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "Line 34: why the immune system 'does not keep up' with the resistant cells when it did clear the susceptible ones." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "Line 40: why a virus is equivalent to a drug for the logic of the experiment." + }, + { + "n": 15, + "severity": "gap", + "kind": "missing_why", + "text": "Line 54: why a compensatory mutation reduces the cost without losing the resistance." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "Line 3: an HTML comment with process metadata ('Stage 1, rendered from model.md in STE-80')." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "Lines 63–68: the 'Not covered' list names topics the reader cannot use." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "It does not say how fast a plasmid spreads through a population once it arrives, or whether plasmid-borne resistance also carries a fitness cost." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "There is no timescale linking 'generations' to real time, so the reader cannot judge what 'years without the drug' means (Q10)." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "It does not say what keeps the immune system from clearing about 100 resistant cells." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "The source of the Luria–Delbrück 'expected scatter' range is not given." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "It does not explain how resistance spreads between patients or within a hospital, though Q10 implies it." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152433-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152433-quiz-prose.json new file mode 100644 index 0000000..bca5bce --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152433-quiz-prose.json @@ -0,0 +1,226 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:24:20", + "model": "default", + "prompt_sha": "57d2fb9e88c1", + "reply": { + "answers": { + "1": "The main objects are: bacteria (susceptible and resistant cells and strains); the antibiotic, a molecule that kills bacteria or stops their growth; its target, a molecule the cell needs, such as the cell-wall enzymes penicillin binds; resistance genes; mutations (random copying errors); plasmids, which are small circles of DNA separate from the chromosome; conjugation and horizontal gene transfer; MIC (minimum inhibitory concentration); fitness cost; compensatory mutations; the immune system; and drug concentration in the blood.", + "2": "An antibiotic binds a target the cell needs. A resistance gene blocks this in one of four ways: it destroys the drug, changes the target, pumps the drug out, or keeps it out. Any of these raises the strain's MIC. If the MIC is above the drug level reached in the blood, the strain is resistant; if below, it is susceptible. A cell gets a resistance gene by mutation, which is rare (about 1 per 10^8 divisions), or by receiving a plasmid from another bacterium through conjugation. The drug removes susceptible cells, so resistant cells multiply. Without the drug, resistance often carries a fitness cost, so resistant cells lose ground. A compensatory mutation can cancel that cost, and a plasmid carrying genes for several drugs can be kept by any one of those drugs.", + "3": "Resistant cells already exist in a large population before treatment, from random mutation or gene transfer. The drug removes the susceptible cells, and the surviving resistant cells regrow until they make up nearly the whole infection.", + "4": "Population size is the key variable. Expected resistant mutants = number of divisions × mutation rate. At 10^10 cells about 100 mutants are expected, so treatment fails. At 10^6 cells only 0.01 are expected, a 1% chance that even one exists, so treatment usually succeeds. Combining two independent drugs multiplies the two chances (10^-16 per cell), so even 10^10 cells expect about 10^-6 doubly resistant cells. That fails if a single event, such as one plasmid or one pump, gives resistance to both drugs.", + "5": "Assumptions and simplifications: the MIC values are marked illustrative; the 30-minute lab doubling time is applied to an infection; the 5% fitness cost is a supposed value; and 10^10 cells is taken to mean about 10^10 divisions. Established observation: Luria and Delbrück (1943) grew 20 equal cultures. Eleven had 0 resistant colonies and one had 107, against a mean of about 11. That spread fits mutation before exposure, not resistance caused by the drug.", + "6": "Antibiotics act on a target molecule, and bacteria resist by destroying the drug, changing the target, pumping the drug out, or keeping it out, which raises the MIC above the drug level in the blood. In a large infection, rare random mutations (about 1 per 10^8 divisions) mean about 100 resistant cells exist before any drug is given, and plasmids passed between bacteria can deliver whole resistance genes in one step. The drug kills or halts the susceptible cells, and the resistant ones regrow to the full infection size in about 27 doublings. This is selection, not induction, as Luria and Delbrück showed: resistant counts across equal cultures varied far too much for resistance to be caused by exposure. Without the drug, resistance usually costs growth and fades, but compensatory mutations and multi-drug plasmids keep it around; combining two independent drugs makes double resistance extremely unlikely.", + "7": "The Luria–Delbrück paragraph was hardest. It uses √11 as 'random scatter' and a 5–18 range without saying why scatter equals the square root of the mean, or why the range is about two scatters wide. It also does not show the expected distribution under the mutation-first idea. A short note on the Poisson rule, or a small simulated comparison of the two ideas, would make it easier. The 0.95-per-doubling ratio step is also a little loose.", + "8": "About 10^10 divisions × 10^-8 means roughly 100 or more cells were already resistant to X before treatment. Drug X killed or halted the susceptible majority, so symptoms cleared. The resistant cells kept dividing, and 100 cells reach 10^10 in about 27 doublings, about 13 hours at a 30-minute doubling time. The returning infection is almost entirely resistant, so X no longer works. The drug selected these cells; it did not create them.", + "9": "Probably not. With 10^6 cells the expected number of resistance mutations is 0.01, about a 1% chance that even one resistant cell exists, so drug X would usually clear the infection. The exception is if a resistance plasmid arrives from another bacterium, which the size argument does not cover.", + "10": "Not necessarily. If resistance has a fitness cost (for example 5%), the ratio of resistant to susceptible cells shrinks by 0.95 each doubling, to about 0.006 of its starting value after 100 doublings. But a compensatory mutation can cut the cost (at 1%, about 0.37 remains after 100 doublings). Also, if the gene sits on a plasmid with resistance genes for other drugs, using any of those drugs keeps selecting it. So resistance usually declines but rarely vanishes. The explanation does not give real time scales in years.", + "11": "Yes. About 10^10 × 10^-8 = 100 resistance mutations are expected, and because early mutants have divided since, there are at least about 100 resistant cells, possibly many more.", + "12": "Because the drug removes susceptible cells; it does not create resistant ones. If exposure caused resistance, each cell would have the same small chance on contact, so equal cultures would give similar counts scattered by about √mean (about 5–18 around a mean of 11). Luria and Delbrück found 11 cultures with 0 resistant colonies and one with 107. That huge spread fits random mutations before exposure: an early mutant leaves many descendants, a late one few.", + "13": "By horizontal gene transfer. Through conjugation, another bacterium, even one of a different species, copies a plasmid into the cell. The plasmid carries a complete working resistance gene, such as a beta-lactamase, often along with several other resistance genes.", + "14": "The resistant fraction usually declines because of the fitness cost, compounding each doubling (0.95^100 ≈ 0.006 at a 5% cost). It may not disappear: compensatory mutations restore growth while keeping the resistance (0.99^100 ≈ 0.37 at a 1% cost), and multi-drug plasmids are still selected whenever any one of their drugs is used." + }, + "audit": { + "terms": [ + "Mutation has several names: 'random copying error' (line 5), 'mutation' (lines 11, 24), 'One DNA letter change' (table, line 16), 'one letter change' (line 47). The reader can probably map these, but the link is never stated.", + "Gene transfer has several names: 'received from another bacterium' (lines 5, 11), 'Genes from other bacteria' (heading, line 42), 'conjugation' and 'horizontal gene transfer' (line 45). Conjugation is one route of horizontal transfer, and that is only hinted at in 'Not covered' (line 71).", + "'resistance gene' (line 11) and 'resistance mutation' (line 24) are used for close but different things: a gene versus the event that makes it.", + "Growth being stopped has three names: 'stops their growth' (line 9), 'halts their growth' (line 33), 'inhibitory' in MIC (line 20)." + ], + "unexplained": [ + "'At a lab doubling time of 30 minutes, that is 13 hours' (line 35): it does not say whether bacteria in a patient double this fast, or how a slower rate would change the regrowth time.", + "'after 100 doublings' (lines 54, 58): it does not say how long 100 doublings takes in real time, which is needed for the 'hospital stops using drug X for years' case.", + "'In 10¹⁰ cells, expect 10⁻⁶ such cells' (line 64): it is not translated into about a one-in-a-million chance that even one doubly resistant cell exists.", + "'about once per 10⁸ cell divisions' (line 24): it does not say whether this rate is typical or varies across genes and drugs, or how a rate of 10⁻⁶ or 10⁻¹⁰ would change the conclusion.", + "'0.95¹⁰⁰ ≈ 0.006' (line 54): this is a ratio relative to the start value, but it does not say what resistant fraction that leaves if resistance started common or rare." + ], + "missing_why": [ + "'the random scatter is about √11 ≈ 3.3, so ... predicts most cultures between 5 and 18' (line 40): no reason is given for why scatter is the square root of the mean, or why the range is about ±2 scatters.", + "'each time the susceptible cells double, the resistant cells grow only 0.95 times as much' (line 54): it does not say why a 5% slower growth translates to a 0.95 factor per doubling.", + "'A compensatory mutation is a second change that restores growth ... so the first change, and the resistance, stay' (line 58): it does not say why the cell gains a second mutation rather than losing the first one.", + "'Early mutants have divided since, so there are at least 100 resistant cells' (line 28): this mixes an expected count of mutation events with a minimum number of cells, and does not say how many more there may be.", + "'the test asks only whether resistance arose before contact, so it applies to drugs too' (line 40): the transfer from virus to antibiotic is asserted rather than argued." + ], + "excess": [ + "'The bacteria become resistant, not the patient's body.' (line 5): this is a common misconception, but the questions and the later argument never use it.", + "The 'Not covered' list (lines 68–73) names topics (persisters, SOS response, biofilms) the reader cannot use. It is harmless but adds no understanding." + ] + }, + "gaps": [ + "The explanation does not give real time scales for decline in a hospital after a drug is withdrawn, so 'for years' cannot be answered quantitatively.", + "It does not say whether plasmids themselves carry a fitness cost, or how fast they spread between cells or species.", + "It does not say whether bacteria divide at the lab rate inside a patient; the 13-hour regrowth depends on this.", + "It does not describe the expected distribution under the mutation-before-exposure idea, only that it 'explains this spread'.", + "It does not explain how mutation and transfer compare in importance in practice." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2, + "14": 2 + }, + "reasons": { + "1": "No points lost: lists the model's entities with correct definitions (antibiotic, target, plasmid, MIC, fitness cost, and others).", + "2": "No points lost: covers target binding, the four mechanisms, MIC versus blood level, both sources, selection, fitness cost, compensatory mutations and multi-drug plasmids.", + "3": "No points lost: states that resistant variants exist before treatment and the drug selects them.", + "4": "No points lost: gives N × rate with 10^10 versus 10^6, and the two-drug 10^-16 case with its plasmid/pump exception.", + "5": "No points lost: separates the illustrative assumptions from the established Luria–Delbrück observation with correct numbers.", + "6": "No points lost: a correct, specific summary of the full causal chain, selection versus induction, cost, compensation and combination therapy.", + "7": "No points lost: names a specific difficulty (the unexplained √mean scatter and the 5–18 range) and proposes a concrete fix.", + "8": "No points lost: about 100 pre-existing resistant cells, susceptible cells removed, regrowth in about 27 doublings, and explicitly selected, not created.", + "9": "No points lost: 0.01 expected mutations, about a 1% chance, drug likely clears it; the plasmid caveat is consistent with the model.", + "10": "No points lost: declines through fitness cost (0.95^100 ≈ 0.006) but persists through compensatory mutation (0.37) and multi-drug plasmids.", + "11": "No points lost: 10^10 × 10^-8 = 100, with at least about 100 resistant cells because early mutants divide.", + "12": "No points lost: the drug removes susceptible cells, and the Luria–Delbrück evidence is given with the induction prediction (5–18) against the observation (11 zeros, one 107).", + "13": "No points lost: horizontal gene transfer by conjugation, a plasmid carrying a complete gene such as beta-lactamase, across species.", + "14": "No points lost: the fraction falls through compounding cost, and compensatory mutations and multi-drug plasmids keep it from disappearing, with correct numbers." + }, + "pass": true + }, + "usage": { + "cost": 0.2225246, + "seconds": 57 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "Mutation has several names: 'random copying error' (line 5), 'mutation' (lines 11, 24), 'One DNA letter change' (table, line 16), 'one letter change' (line 47). The reader can probably map these, but the link is never stated." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Gene transfer has several names: 'received from another bacterium' (lines 5, 11), 'Genes from other bacteria' (heading, line 42), 'conjugation' and 'horizontal gene transfer' (line 45). Conjugation is one route of horizontal transfer, and that is only hinted at in 'Not covered' (line 71)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'resistance gene' (line 11) and 'resistance mutation' (line 24) are used for close but different things: a gene versus the event that makes it." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "Growth being stopped has three names: 'stops their growth' (line 9), 'halts their growth' (line 33), 'inhibitory' in MIC (line 20)." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'At a lab doubling time of 30 minutes, that is 13 hours' (line 35): it does not say whether bacteria in a patient double this fast, or how a slower rate would change the regrowth time." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'after 100 doublings' (lines 54, 58): it does not say how long 100 doublings takes in real time, which is needed for the 'hospital stops using drug X for years' case." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'In 10¹⁰ cells, expect 10⁻⁶ such cells' (line 64): it is not translated into about a one-in-a-million chance that even one doubly resistant cell exists." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'about once per 10⁸ cell divisions' (line 24): it does not say whether this rate is typical or varies across genes and drugs, or how a rate of 10⁻⁶ or 10⁻¹⁰ would change the conclusion." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'0.95¹⁰⁰ ≈ 0.006' (line 54): this is a ratio relative to the start value, but it does not say what resistant fraction that leaves if resistance started common or rare." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'the random scatter is about √11 ≈ 3.3, so ... predicts most cultures between 5 and 18' (line 40): no reason is given for why scatter is the square root of the mean, or why the range is about ±2 scatters." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'each time the susceptible cells double, the resistant cells grow only 0.95 times as much' (line 54): it does not say why a 5% slower growth translates to a 0.95 factor per doubling." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'A compensatory mutation is a second change that restores growth ... so the first change, and the resistance, stay' (line 58): it does not say why the cell gains a second mutation rather than losing the first one." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "'Early mutants have divided since, so there are at least 100 resistant cells' (line 28): this mixes an expected count of mutation events with a minimum number of cells, and does not say how many more there may be." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "'the test asks only whether resistance arose before contact, so it applies to drugs too' (line 40): the transfer from virus to antibiotic is asserted rather than argued." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "'The bacteria become resistant, not the patient's body.' (line 5): this is a common misconception, but the questions and the later argument never use it." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "The 'Not covered' list (lines 68–73) names topics (persisters, SOS response, biofilms) the reader cannot use. It is harmless but adds no understanding." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The explanation does not give real time scales for decline in a hospital after a drug is withdrawn, so 'for years' cannot be answered quantitatively." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "It does not say whether plasmids themselves carry a fitness cost, or how fast they spread between cells or species." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "It does not say whether bacteria divide at the lab rate inside a patient; the 13-hour regrowth depends on this." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "It does not describe the expected distribution under the mutation-before-exposure idea, only that it 'explains this spread'." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "It does not explain how mutation and transfer compare in importance in practice." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/understanding.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/understanding.md new file mode 100644 index 0000000..e695efc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/understanding.md @@ -0,0 +1,15 @@ +# Review: antibiotic-resistance (published tier, Stage 1 prose) + +## Runs + +| Run | Tool | Result | Action | +|---|---|---|---| +| 2026-10-04T151610-probe | probe | 5 main, 8 edge, 2 excess | Adopted 1–11, 13, 15: mutation events vs resistant cells, immune assumption, one example per mechanism, combination break case, one term per concept, MIC threshold, compounding cost, Luria–Delbrück numbers, conjugation, compensatory mutation, stepwise MIC, 1% probability. Declined 12 (in-patient doubling time: no reliable value) and 14 (representation section is template-required). | +| 2026-10-04T151845-coldread-narrative | cold read | 4 blocking | Rewrote every beat's Said column in full. Declined cut of the 13 h figure (now tied to a stated 30 min doubling), cut of the mechanisms table, removal of scope. | +| 2026-10-04T152038-coldread-prose | cold read | 0 blocking | Applied the short edge and excess fixes. Declined: virus agent (later explained), transformation/transduction and biofilms in Not covered (pointers only). | +| 2026-10-04T152206-quiz-prose | blind test | FAIL (Q10 scored 1) | Cause: the cold-read cut of "plasmids also persist" removed a rubric point. Restored with its reason (a plasmid with genes for several drugs stays selected by any of them). Also: one word "doubling" for the cost arithmetic, why the immune system cannot clear the survivors, √11 scatter for 5–18, why probabilities multiply, how one pump resists two drugs. | +| 2026-10-04T152433-quiz-prose | blind test | PASS, all 14 items 2 | Adopted the one-in-a-million wording. Declined the rest: real-time scales for hospital decline and in-patient growth, plasmid cost and spread rate, mutation vs transfer importance — each adds length past the 950-word budget for questions this reader need not answer to grasp the mechanism. | + +## Rule learned + +A cold-read "leap: no reason given" on a sentence that carries a quiz answer should be fixed by adding the reason, not by cutting the sentence. Check the rubric before you cut a claim the cold read questions. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/read.json new file mode 100644 index 0000000..ca19420 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "From random copying errors (mutations) during cell division and from receiving resistance genes from other bacteria through horizontal gene transfer via plasmids.", + "quote": "a few bacteria already carry a resistance gene, made by a random copying error or received from another bacterium" + }, + "2": { + "answer": "The antibiotic kills or halts the susceptible bacteria, eliminating them from the population while allowing resistant cells to continue dividing and multiplying.", + "quote": "The drug removes the susceptible bacteria around them, and the resistant bacteria multiply" + }, + "3": { + "answer": "Through plasmids—small circles of DNA that can be copied from one bacterium to another during conjugation, allowing genes to pass between different species.", + "quote": "In conjugation, two bacteria touch and one copies a plasmid into the other" + }, + "4": { + "answer": "Even though there are only a few resistant cells initially, they multiply exponentially while susceptible cells are killed. The immune system alone cannot stop this exponential growth, so resistant cells repopulate the infection.", + "quote": "The resistant cells keep dividing. The immune system removes some, but alone it cannot keep up with a population that keeps doubling" + } + }, + "stops": [ + "target: a molecule the cell needs", + "MIC (minimum inhibitory concentration)", + "fitness cost: the extra protein, or the changed target, makes the cell grow more slowly", + "A compensatory mutation is a second change, in another gene, that restores growth", + "efflux pump, a membrane protein, exports tetracycline faster than it enters", + "One letter change could never build", + "The cell loses a porin, an entry channel in its surface", + "their chances multiply: 10⁻⁸ × 10⁻⁸ = 10⁻¹⁶", + "the random scatter is about √11 ≈ 3.3", + "expect 10¹⁰ divisions" + ], + "excess": [ + "At a lab doubling time of 30 minutes, that is 13 hours", + "They grew 20 equal cultures of bacteria, then exposed each one and counted the resistant colonies", + "For independent rare events with a mean of 11, the random scatter is about √11 ≈ 3.3", + "the test asks only whether resistance arose before contact, so it applies to drugs too", + "The same plasmid gene turns up in different species, which shows that the gene moved between them", + "If exposure caused resistance, each cell would have the same small chance when it met the drug", + "Early mutants have divided since, so there are at least 100 resistant cells", + "the immune system removes some, but alone it cannot keep up", + "One plasmid often carries resistance genes for several drugs, so any one of those drugs still selects the whole plasmid", + "(Illustrative values.)" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/run.json new file mode 100644 index 0000000..94063c2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/antibiotic-resistance/explanation.md` as prose only, because the subject is a chain of causes (random variation, then selection by the drug, regrowth and spread) carried by a few worked numbers, so a diagram or interactive page wouldn't add anything. It passed the probe, the cold reads (nothing blocking) and the final blind test (all 14 items at full marks; the first run failed one item, and I fixed that); details are in `review/understanding.md`.", + "error": null, + "cost": 1.9340748, + "seconds": 686, + "turns": 81, + "files": [ + "explainers/antibiotic-resistance/explanation.md" + ], + "review_cost": 0.8549, + "reviews": 5 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/explanation.md new file mode 100644 index 0000000..8435c4e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/explanation.md @@ -0,0 +1,51 @@ +# How do bacteria become resistant to an antibiotic? + +**A few bacteria in a large population already carry a gene that protects them, by random chance. The antibiotic kills the other bacteria, and the protected ones divide to replace them. The bacteria do not learn; the drug selects.** + +## What an antibiotic does + +An antibiotic is a molecule that binds a **target**: a protein or structure that the bacterium needs. Penicillin, for example, blocks the enzymes that build the cell wall, so the cell bursts. A cell is **susceptible** if the drug kills it or stops its growth. A cell is **resistant** if it survives a dose that kills susceptible cells. The bacteria become resistant, not the patient. + +## Step 1: random variation makes a few resistant cells + +Resistance needs a **resistance gene**: a version of a gene whose protein stops the drug from working on that cell. A cell gets one in two ways. + +1. **Mutation.** Each time a cell copies its DNA, it makes rare random errors. A specific error that changes the target has a chance of about 10^-9 per division: once per billion divisions. +2. **Gene transfer.** Bacteria pass DNA to each other, often on a **plasmid**: a small circle of DNA, separate from the chromosome, that cells copy and hand over. One plasmid can carry several resistance genes and move between species, so one transfer can give resistance to several drugs. + +## Step 2: the antibiotic selects the resistant cells + +Worked numbers (illustrative): + +- A serious infection holds about 10^10 bacteria. +- Expected resistant cells before any drug: 10^10 × 10^-9 = 10. +- The antibiotic kills the susceptible cells. The 10 resistant cells survive. +- Each survivor divides. 10 × 2^30 ≈ 10^10, so 30 doublings rebuild the population. At 30 minutes per doubling (a lab rate; slower in a body), that takes about 15 hours. + +The new population is almost all resistant. This process is **selection**: the drug removes one kind of cell, and the remaining kind fills the space. + +Predict before you read on: what happens with a small population of 10^7 cells? Expected resistant cells = 10^7 × 10^-9 = 0.01. Usually no resistant cell exists, so the drug clears the infection. + +## Why selection, and not adaptation + + +The simplest alternative idea is that each cell meets the drug and adapts. Then every cell would have the same chance to survive. Instead, a rare fraction survives, and those cells were resistant before they met the drug. Luria and Delbrück showed this in 1943: the number of resistant bacteria varied between cultures in a way that only fits mutations that occurred before exposure. So the antibiotic does not cause resistance. It reveals and enriches resistance that random variation already made. + +## How a resistance gene protects the cell + + +Each resistance gene works in one of four ways: + +| Mechanism | What the cell does | Example | +|---|---|---| +| Destroy the drug | An enzyme cuts or changes the drug | Beta-lactamase cuts penicillin | +| Change the target | The target protein changes shape, so the drug cannot bind | MRSA makes a cell-wall enzyme that penicillin-type drugs do not bind | +| Pump the drug out | A membrane pump removes the drug faster than it enters | Efflux pumps against tetracycline | +| Keep the drug out | The cell makes fewer entry channels in its outer membrane | Fewer porins against several drugs | + +## What follows from this + +- **Two drugs at once help.** A cell needs two independent rare events: 10^-9 × 10^-9 = 10^-18 per cell. In 10^10 cells, that is 10^-8 expected cells, so effectively none. This fails if one plasmid carries both resistance genes. +- **Resistance is often a cost.** Without the drug, resistant cells often grow slower, so they stay rare. Every use of the drug gives them the advantage. + +**Not covered here:** persister cells, which survive a drug without a resistance gene; and stress responses that raise mutation rates. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.md new file mode 100644 index 0000000..a3ff463 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.md @@ -0,0 +1,100 @@ +# Semantic model: antibiotic resistance + +> Source of truth. Every rendering derives from this file. + +## Central question + +How do bacteria become resistant to an antibiotic? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (cells, DNA, genes code for proteins, exponential growth), basic programming. New to microbiology. Does not know: what an antibiotic targets, mutation rates, plasmids, horizontal gene transfer, selection as a mechanism. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Antibiotic | A molecule that kills bacteria or stops their growth by binding a target that bacteria need. | fact | +| Target | The bacterial protein or structure that the antibiotic binds (for penicillin: the enzymes that build the cell wall). | fact | +| Population | The bacteria in one infection: about 10^10 cells, all descended from a few cells. | estimate | +| Resistance gene | A version of a gene whose protein stops the antibiotic from working on that cell. | fact | +| Mutation | A random copying error in DNA when a cell divides. | fact | +| Gene transfer | One bacterium passes DNA to another, usually on a plasmid (a small circle of DNA). | fact | +| Selection | The antibiotic kills the susceptible cells, so the resistant cells make up the next population. | fact | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| Mutation, gene transfer | create | resistance gene in a few cells | before or during treatment, at random | +| Resistance gene | blocks | antibiotic action | four mechanisms | +| Antibiotic | kills | susceptible cells | not resistant cells | +| Selection | enriches | resistant cells | they divide into the space left | + +## Causal chain + +```text +random variation (mutation or gene transfer) → a few resistant cells +→ antibiotic kills susceptible cells → resistant cells survive and divide +→ resistant population +``` + +## Quantities + +- Mutation rate for one specific resistance mutation: about 10^-9 per cell division (illustrative; real values range about 10^-6 to 10^-10). +- Population in a serious infection: about 10^10 cells. +- Expected resistant cells before any drug: 10^10 × 10^-9 = 10 cells. +- Regrowth: 10 cells × 2^30 ≈ 10^10 cells. 30 doublings at 30 min each ≈ 15 hours (lab rate; slower in a body). +- Two drugs with independent targets: chance per cell 10^-9 × 10^-9 = 10^-18; expected cells in 10^10: 10^-8, so effectively none. + +## Alternative states + +| State | What changes | Why | +|---|---|---| +| No antibiotic | resistant cells stay rare | resistance often costs growth rate; no selection | +| Antibiotic present | resistant cells take over | selection | +| Two drugs together | resistance very unlikely | a cell needs two independent rare events | +| Resistance gene on a plasmid | spreads between cells and species | gene transfer, not only division | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Resistance comes from pre-existing random variation, then selection | Each cell "learns" or adapts when it meets the drug | If cells adapted on contact, all 10^10 would survive or none would; observed: a fixed rare fraction (about 10 in 10^10) survives, and those cells existed before exposure (Luria–Delbrück, 1943) | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| Selection enriches resistance | 10^10 cells with drug: 10 resistant survive, regrow to 10^10 in about 30 doublings | Small population of 10^7 cells: expected resistant cells = 0.01, so the drug usually clears it | +| Combination therapy prevents resistance | Two independent drugs: 10^-18 per cell, 10^-8 expected | A plasmid carrying both resistance genes gives both at once, so the multiplication no longer applies | + +## Terms + +- **Susceptible:** the antibiotic kills or stops this cell. +- **Resistant:** the cell survives a dose that kills susceptible cells. +- **Plasmid:** a small DNA circle, separate from the chromosome, that cells copy and pass to each other. +- **Selection:** see Entities. + +## Epistemic status + +- **Established:** four mechanisms; mutation and gene transfer as sources; resistance arises before exposure (Luria–Delbrück). +- **Illustrative:** the specific numbers 10^-9, 10^10, 30 min. + +## Confusion points + +- "The antibiotic causes the mutation" → the mutation is random and usually exists already; the antibiotic only selects. (Some antibiotics raise mutation rates a little; out of scope.) +- "The patient becomes resistant" → the bacteria do, not the person. +- "Resistant cells are stronger in general" → they often grow slower without the drug. + +## Scope + +- **Out of scope:** tolerance and persister cells (survive without a resistance gene) — see "antibiotic persistence". +- **Out of scope:** stress-induced mutagenesis (SOS response). +- **Out of scope:** clinical policy (stewardship, dosing) beyond one line. + +## Representation decision + +- **Stage:** 1 +- **Reason:** a short causal sequence plus a four-row list of mechanisms; prose with one table and worked numbers carries it. +- **Levels:** L1 answer / L2 the two steps / L3 why it is selection, not adaptation / L4 mechanisms. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.yaml new file mode 100644 index 0000000..f33a8ba --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.yaml @@ -0,0 +1,57 @@ +# Machine-readable half of model.md. `explainer check antibiotic-resistance` holds every rendering to it. + +values: + mutation_rate: "10^-9" + population: "10^10" + resistant_before: 10 + doublings: 30 + doubling_minutes: 30 + regrowth_hours: 15 + small_population: "10^7" + small_expected: 0.01 + combo_rate: "10^-18" + combo_expected: "10^-8" + luria_year: 1943 + +# exponents of the power-of-ten values above (the checker reads 10^-9 as 10 and -9) +allow: [2, 4, -9, 7, -18, -8] + +require: + - path: resistant_before + in: [prose] + - path: combo_rate + in: [prose] + +claims: + - id: selection-not-adaptation + kind: why + about: [selection] + statement: "Resistance comes from rare random variants that exist before the drug; the drug selects them, it does not create them." + alternative: "Each cell adapts when it meets the drug." + counterexample: "If cells adapted on contact, every cell would have the same chance; observed: about 10 of 10^10 survive, and they were resistant before exposure (Luria–Delbrück, 1943)." + ask: "Does the antibiotic cause the resistance mutation?" + - id: selection-worked + kind: mechanism + statement: "The drug kills susceptible cells; the resistant cells regrow the population." + example: "10^10 cells × 10^-9 = 10 resistant cells; 10 × 2^30 ≈ 10^10 after 30 doublings, about 15 hours." + - id: sources + kind: mechanism + statement: "Resistance genes arrive by mutation or by gene transfer on plasmids." + example: "A plasmid can carry several resistance genes and move between species." + - id: mechanisms + kind: mechanism + statement: "A resistance gene works in one of four ways: destroy the drug, change the target, pump the drug out, or keep it out." + example: "Beta-lactamase cuts penicillin before it reaches its target." + - id: combination + kind: guarantee + statement: "Two independent drugs make resistance far less likely." + example: "10^-9 × 10^-9 = 10^-18 per cell; 10^-8 expected cells in 10^10." + counterexample: "A plasmid with both resistance genes gives both at once." + ask: "Why does giving two drugs at once help?" + +terms: + - term: resistant + means: "the bacterial cell survives the drug; the patient is not resistant" + avoid: [patient becomes resistant, immune to the antibiotic] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/read.json new file mode 100644 index 0000000..5068d3f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/read.json @@ -0,0 +1,48 @@ +{ + "answers": { + "1": { + "answer": "From random mutations or gene transfer between bacteria before the drug is used.", + "quote": "A few bacteria in a large population already carry a gene that protects them, by random chance." + }, + "2": { + "answer": "It kills the susceptible bacteria and leaves the resistant ones to survive and repopulate.", + "quote": "This process is selection: the drug removes one kind of cell, and the remaining kind fills the space." + }, + "3": { + "answer": "Through plasmids—small circles of DNA that can move between different bacterial species.", + "quote": "One plasmid can carry several resistance genes and move between species, so one transfer can give resistance to several drugs." + }, + "4": { + "answer": "Even though only 1 in a billion bacteria are resistant, a large infection of 10^10 bacteria contains about 10 resistant bacteria (10^10 × 10^-9 = 10). When the antibiotic kills all susceptible bacteria, these 10 resistant survivors repopulate the cleared space.", + "quote": "Expected resistant cells before any drug: 10^10 × 10^-9 = 10. The antibiotic kills the susceptible cells. The 10 resistant cells survive." + } + }, + "stops": [ + "Efflux pumps against tetracycline", + "Fewer porins against several drugs", + "A specific error that changes the target has a chance of about 10^-9 per division: once per billion divisions.", + "the number of resistant bacteria varied between cultures in a way that only fits mutations that occurred before exposure", + "A cell needs two independent rare events: 10^-9 × 10^-9 = 10^-18 per cell", + "Without the drug, resistant cells often grow slower, so they stay rare.", + "10 × 2^30 ≈ 10^10, so 30 doublings rebuild the population.", + "This fails if one plasmid carries both resistance genes." + ], + "excess": [ + "The bacteria become resistant, not the patient.", + "Predict before you read on: what happens with a small population of 10^7 cells?", + "Luria and Delbrück showed this in 1943: the number of resistant bacteria varied between cultures in a way that only fits mutations that occurred before exposure.", + "At 30 minutes per doubling (a lab rate; slower in a body), that takes about 15 hours.", + "Not covered here: persister cells, which survive a drug without a resistance gene; and stress responses that raise mutation rates.", + "Beta-lactamase cuts penicillin", + "Resistance is often a cost.", + "Every use of the drug gives them the advantage.", + "The cell makes fewer entry channels in its outer membrane", + "MRSA makes a cell-wall enzyme that penicillin-type drugs do not bind" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/run.json new file mode 100644 index 0000000..f9b0e31 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/antibiotic-resistance/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/antibiotic-resistance/explanation.md`, which is prose with worked numbers and one table, plus its `model.md` and `model.yaml`; I chose prose because the answer is a short two-step story (random variation, then the drug kills the rest), so a diagram or interactive page wouldn't add anything, and `explainer check` passes.", + "error": null, + "cost": 0.430636, + "seconds": 104, + "turns": 21, + "files": [ + "explainers/antibiotic-resistance/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/explainers/bloom-filter/index.html b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/explainers/bloom-filter/index.html new file mode 100644 index 0000000..a473b2e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/explainers/bloom-filter/index.html @@ -0,0 +1,456 @@ + + + + + +Bloom filters: "have I seen this before?" in a few bits + + + +
+ +

Bloom filters: answering "have I seen this before?" with very little memory

+

A Bloom filter is a row of bits plus a few hash functions. It can tell you that something is definitely new or probably seen before. It uses a small, fixed amount of memory no matter how long the items are. The cost is that the "probably" answer is sometimes wrong. This page shows how it works, lets you try one, and explains how to work out how often it will be wrong.

+ +

1. The problem

+

Lots of programs need to ask the same question over and over: is this item in my set? Has this user name been taken? Has the web crawler already visited this URL? Is the row I'm looking for in this file on disk, or should I skip the file?

+

The obvious answer is to store every item, say in a hash set, and look it up. That gives exact answers, but you pay for the items themselves. A billion URLs of about 60 bytes each comes to roughly 60 GB before any overhead. Often you can't afford that much fast memory, or the full set lives somewhere slow (on disk, across the network), and you just want a cheap check before you go there.

+

A Bloom filter gives up a little accuracy in return for a large saving in memory. At a 1% error rate it needs about 10 bits per item, whether the item is a 6-letter word or a 2 KB document. For those billion URLs that's about 1.2 GB.

+ +

2. The machinery: bits and hash functions

+

You need two ingredients:

+
    +
  • An array of m bits, all starting at 0.
    • +
  • k different hash functions. A hash function turns any input (such as a string) into a number that looks random but is always the same for the same input. We take each number modulo m, so it picks a position in the array.
    • +
+

So each item is linked to k positions in the array. You can think of them as the item's "fingerprint" of bit positions.

+
+

Add an item: compute its k positions and set all of those bits to 1. (If a bit is already 1, it stays 1.)

+

Check an item: compute its k positions and look at the bits.

+
    +
  • If any of them is 0, the item was definitely never added. Adding it would have set that bit to 1, and bits never go back to 0.
    • +
  • If all of them are 1, the item was probably added. Those bits might have been set by other items.
    • +
+
+

The filter never stores the item itself. It only stores the bits that items have switched on. That's why it's small, and also why it can be fooled.

+ +

3. Try it

+

This filter has m = 32 bits. Add a few words, then check some words you added and some you didn't. The outlined squares are the positions the last word hashed to: orange means the bit was 1, red means it was 0.

+ +
+
+ + + + +
+
+ + + +
+
+
bit is 0   bit is 1   probed, was 1   probed, was 0
+
Type a word and press Add or Check.
+
+
Words actually added (the filter itself does not keep this list; it's shown here only so you can see which answers are wrong):
+
+
+ +

Things to notice:

+
    +
  • A word you added always comes back "probably yes". There are no false negatives.
    • +
  • When the filter is nearly empty, most new words land on at least one 0 and are rejected straight away.
    • +
  • As you add more words, the array fills up with 1s. Eventually a word you never added lands only on 1s that other words set, and the filter says "probably yes". That's a false positive. The Hunt button tries made-up words until it finds one.
    • +
  • Different words sometimes share positions. That sharing is what lets the filter use so few bits, and it's also the cause of false positives.
    • +
+ +

4. How it can be wrong

+

There are four possible outcomes:

+ + + + +
Filter says "definitely not"Filter says "probably yes"
Item was addedImpossible: its bits were set to 1 and never clearedCorrect
Item was not addedCorrectFalse positive: other items happened to cover all its bits
+

So the error only goes one way. "No" is always trustworthy, and "yes" means "check properly if it matters". That fits the usual way Bloom filters are used, as a cheap guard in front of an expensive exact lookup:

+
    +
  • A database keeps a Bloom filter for each file on disk. Before reading a file to look for a key, it asks the filter. "Definitely not" saves a disk read. "Probably yes" means it reads the file, and now and then finds nothing there, which costs a wasted read but never gives a wrong answer.
    • +
  • A web cache only stores a page the second time it's requested. A Bloom filter of "URLs seen once" answers "have I seen this before?" cheaply, and the occasional false positive just means a page gets cached a little early.
    • +
+

Two more limits come from the same design:

+
    +
  • You can't delete. Setting an item's bits back to 0 might erase bits that other items also depend on, and then those items would get false negatives. (A variant called a counting Bloom filter stores a small counter in each slot instead of a single bit. Adding increments the counters and deleting decrements them, at several times the memory cost.)
    • +
  • You can't list the contents. The items themselves were never stored, only bits.
    • +
+ +

5. How often is it wrong? The arithmetic

+

Here's how to estimate the false positive rate. Assume the hash functions behave like independent, fair random choices of position. Real hash functions are close enough to this.

+ +

Step 1: chance that a particular bit is still 0. A single hash picks a particular bit with probability 1/m, so it misses that bit with probability 1 − 1/m. Adding n items performs kn hashes in total, and the bit stays 0 only if every one of them misses it:

+P(bit still 0) = (1 − 1/m)kn ≈ e−kn/m +

(The approximation uses the standard fact that (1 − 1/m)m ≈ 1/e when m is large.)

+ +

Step 2: chance of a false positive. Now check an item that was never added. Its k positions are effectively random, and each one is 1 with probability 1 − e−kn/m. The filter wrongly says "yes" only if all k are 1:

+p ≈ (1 − e−kn/m)k +

Put another way, if a fraction f of the bits are 1, the false positive rate is about fk. The demo above shows this estimate under the bit array.

+ +

Step 3: choosing k. The number of hash functions pulls in two directions:

+
    +
  • More hashes per check means a stranger has to get lucky more times, which lowers the error.
    • +
  • More hashes per insertion fills the array with 1s faster, which raises the error.
    • +
+

The best balance turns out to be when about half the bits are 1. That happens when

+k = (m/n) · ln 2 ≈ 0.69 · (m/n) +

With that choice, each extra hash halves the error, so p ≈ (1/2)k. Rearranging gives the number of bits needed per item for a target error rate p:

+m/n = −ln p / (ln 2)2 ≈ 1.44 · log2(1/p) + + + + + + + +
Target false positive rateBits per itemBest k
10%4.83
1%9.67
0.1%14.410
0.01%19.213
+

Two things in this table are worth noticing. The item size doesn't appear anywhere. And each tenfold improvement in accuracy costs only about 4.8 more bits per item.

+ +

6. Explore the trade-off

+

Move the sliders to see how memory per item (m/n) and the number of hashes (k) affect the false positive rate. The curve plots the rate against k for the chosen memory budget. Its lowest point is the best k.

+
+ + +
+ +
+

Try overfilling the filter: keep k fixed and slide the bits per item down. This is what happens when you build a filter for a million items and then put ten million in it. The error rate doesn't creep up slowly; it rises steeply toward 100% as the array fills with 1s. Bloom filters need to be sized for the largest number of items you expect.

+ +

7. Summary

+
    +
  • A Bloom filter is m bits plus k hash functions. Adding an item sets its k bits. Checking an item tests them.
    • +
  • Any 0 means definitely not seen. All 1s means probably seen.
    • +
  • It can be wrong in only one direction: false positives, which happen when other items have already set all of a new item's bits. False negatives can't happen.
    • +
  • The false positive rate is about (1 − e−kn/m)k. With the best k, about 10 bits per item gives 1%, and every extra ~4.8 bits per item cuts the rate tenfold.
    • +
  • It saves memory because it stores no items, which is also why it can't delete items or list them.
    • +
+ +
+ + + + diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/read.json new file mode 100644 index 0000000..00055c3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "When adding: compute k positions and set all those bits to 1. When checking: compute k positions and look at the bits. If any is 0, definitely not added; if all are 1, probably added.", + "quote": "Add an item: compute its k positions and set all of those bits to 1. Check an item: compute its k positions and look at the bits." + }, + "2": { + "answer": "Can give false positives (saying yes when no), never false negatives (saying no when yes), because bits are set to 1 and never go back to 0.", + "quote": "So the error only goes one way. 'No' is always trustworthy, and 'yes' means 'check properly if it matters'." + }, + "3": { + "answer": "Clearing bits might erase bits that other items depend on, causing those items to be wrongly rejected (false negatives).", + "quote": "Setting an item's bits back to 0 might erase bits that other items also depend on, and then those items would get false negatives." + }, + "4": { + "answer": "It answers 'probably added' (yes), which is wrong—a false positive—since all k positions will be 1 regardless of whether the item was actually added.", + "quote": "If all of them are 1, the item was probably added." + } + }, + "stops": [ + "We take each number modulo m, so it picks a position in the array.", + "Assume the hash functions behave like independent, fair random choices of position.", + "(1 − 1/m)^(kn) ≈ e^(−kn/m) — where does this formula come from?", + "the standard fact that (1 − 1/m)^m ≈ 1/e when m is large.", + "a stranger has to get lucky more times, which lowers the error.", + "The best balance turns out to be when about half the bits are 1.", + "k = (m/n) · ln 2 — derivation not shown", + "p ≈ (1 − e^(−kn/m))^k — how does this follow from previous step?", + "bits never go back to 0 — why is this always true?", + "That sharing is what lets the filter use so few bits, and it's also the cause of false positives." + ], + "excess": [ + "A billion URLs of about 60 bytes each comes to roughly 60 GB before any overhead.", + "A database keeps a Bloom filter for each file on disk. Before reading a file to look for a key, it asks the filter.", + "A web cache only stores a page the second time it's requested. A Bloom filter of 'URLs seen once' answers 'have I seen this before?' cheaply.", + "A variant called a counting Bloom filter stores a small counter in each slot instead of a single bit.", + "Target false positive rate 10% needs 4.8 bits per item; 1% needs 9.6; 0.1% needs 14.4.", + "Two things in this table are worth noticing. The item size doesn't appear anywhere.", + "This filter has m = 32 bits. Add a few words, then check some words you added and some you didn't.", + "A word you added always comes back 'probably yes'. There are no false negatives.", + "the occasional false positive just means a page gets cached a little early.", + "It saves memory because it stores no items, which is also why it can't delete items or list them." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/run.json new file mode 100644 index 0000000..dd13086 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/bloom-filter/index.html`, a single interactive page. You learn a Bloom filter fastest by adding words, watching bits turn on and finding a false positive yourself, and the page puts the prose and the false-positive arithmetic right next to the hands-on filter and a trade-off chart (I haven't opened it in a browser yet).", + "error": null, + "cost": 0.40253520000000004, + "seconds": 118, + "turns": 3, + "files": [ + "explainers/bloom-filter/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/explainers/bloom-filter/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/explainers/bloom-filter/explanation.md new file mode 100644 index 0000000..dff85e6 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/explainers/bloom-filter/explanation.md @@ -0,0 +1,111 @@ +# Bloom filters: "have I seen this before?" in about 10 bits per item + +**The short answer.** A Bloom filter does not store the items. It stores a row of bits. To add an item, it hashes the item to a few bit positions and sets those bits to 1. To check an item, it hashes the item the same way and looks at the same positions. If any of those bits is 0, the item was never added. If all of them are 1, the item was *probably* added. The bits for that item could also have been set by other items. That last case is the only way the filter is wrong. It can say "yes" to an item that it never saw. It never says "no" to an item that it did see. + +**Why you would want this.** Suppose a database keeps keys on disk. A disk read is slow, and many lookups ask for keys that do not exist. A small Bloom filter in memory answers "definitely not here" for most of those lookups, so the database skips the disk read. A "probably here" answer costs one real disk read, and that read gives the true answer. So a rare wrong "yes" costs a little time. It never gives an incorrect result. Databases such as LevelDB, RocksDB, and Cassandra use Bloom filters in this way. + +## The parts + +- **Bit array**: `m` bits, all 0 at the start. +- **Hash function**: a function that turns an item into a number that looks random. The same item always gives the same number. Here, each hash gives a position from 0 to `m − 1`. +- **`k`**: the number of hash functions. Each item maps to `k` positions. +- **`n`**: the number of items added so far. + +There are two operations: + +1. **Add(x)**: compute the `k` positions for `x`. Set each bit to 1. +2. **Query(x)**: compute the `k` positions for `x`. If any bit is 0, answer "definitely not seen". If all bits are 1, answer "probably seen". + +## A worked example + +Use `m = 16` bits and `k = 3`. The hash outputs below are invented for the example. A real filter gets them from real hash functions. + +| Step | Positions | Bits that are 1 after the step | +|---|---|---| +| Add "cat" | 1, 6, 11 | 1, 6, 11 | +| Add "dog" | 3, 6, 14 | 1, 3, 6, 11, 14 | + +"cat" and "dog" share bit 6. That sharing is normal, and it is the source of every error. + +Now query three items: + +| Query | Positions | Bits at those positions | Answer | Correct? | +|---|---|---|---|---| +| "cat" | 1, 6, 11 | 1, 1, 1 | probably seen | yes, "cat" was added | +| "fox" | 2, 6, 11 | **0**, 1, 1 | definitely not seen | yes | +| "owl" | 1, 3, 14 | 1, 1, 1 | probably seen | **no: false positive** | + +"owl" was never added. Its three bits were set by "cat" (bit 1) and "dog" (bits 3 and 14). The filter has no record of *which* item set a bit, so it cannot tell the difference. + +## Why it never misses an item it saw + +Add only changes bits from 0 to 1. Nothing changes a bit back to 0. When you add `x`, its `k` bits become 1, and they stay 1. A later query for `x` computes the same `k` positions, finds all 1s, and answers "probably seen". So a **false negative** (a "no" for an item that was added) cannot happen. + +**Where this guarantee breaks.** The guarantee depends on "nothing clears a bit". Suppose you try to delete "cat" by clearing bits 1, 6, and 11. Bit 6 also belongs to "dog". A query for "dog" now finds bit 6 = 0 and answers "definitely not seen", which is wrong. For this reason a basic Bloom filter does not support delete. (A *counting Bloom filter* stores a small counter in place of each bit, so it can delete. It uses about 4 times the memory.) + +## Why so little memory + +A normal set stores each item. One million URLs of about 50 bytes each need about 50 MB, plus the overhead of the set structure. A Bloom filter stores only bits, and the number of bits does not depend on the size of an item. With 1 million items and a 1% false-positive target, the filter needs about 9.6 bits per item: about 1.2 MB in total. A long URL and a short URL cost the same. + +The price is that the filter can only answer the membership question. It cannot list its items, count them, or give back an item. + +## How often it is wrong + +Each false positive happens when all `k` bits of a new item are already 1. So the error rate depends on how full the array is. + +**Assumption:** each hash position is uniform and independent of the others. Good hash functions come close to this. + +One hash sets one specific bit with probability `1/m`. After `n` adds, there were `k·n` bit-sets. The chance that a given bit is still 0 is + +``` +(1 − 1/m)^(k·n) ≈ e^(−k·n/m) +``` + +The query for an item that was never added checks `k` bits. It is a false positive if all `k` of them are 1: + +``` +false-positive rate p ≈ (1 − e^(−k·n/m))^k +``` + +(This is a mathematical consequence of the assumption above, and it is a very close approximation for large `m`.) + +**Example.** `m/n = 10` bits per item, `k = 7`. Then `k·n/m = 0.7`, and `e^(−0.7) = 0.497`. So about half the bits are 1. Then `p ≈ 0.503^7 ≈ 0.0082`, which is 0.82%: about 1 wrong "yes" in 120 queries for absent items. + +### Why use several hash functions, not one + +The simplest design uses one hash (`k = 1`). With 10 bits per item, that gives `p = 1 − e^(−0.1) ≈ 9.5%`. More hashes make each item's "fingerprint" harder to fake by accident. But each hash also sets more bits, so the array fills faster. These two effects pull in opposite directions: + +| `k` (10 bits per item) | 1 | 2 | 4 | 7 | 10 | 20 | +|---|---|---|---|---|---|---| +| false-positive rate | 9.5% | 3.3% | 1.2% | **0.82%** | 1.0% | 5.5% | + +The best value is `k = (m/n)·ln 2`, which is about 7 here. At that value, about half of the bits are 1. The array then carries as much information as possible: a bit is equally likely to be 0 or 1. + +### What happens when the filter fills up + +The size `m` is fixed when you create the filter. If you add more items than planned, the error rate climbs fast. Take `m` = 10 million bits (1.25 MB) and `k = 7`: + +| items added `n` | 0.5 million | 1 million | 2 million | 5 million | +|---|---|---|---|---| +| false-positive rate | 0.02% | 0.82% | 14% | 81% | + +At 5 million items, almost every bit is 1, so almost every query answers "probably seen". The filter then gives almost no information. The bits cannot be resized, because the filter does not keep the items it would need to hash again. So choose `m` from the largest `n` you expect. + +### Sizing rule + +For a target rate `p`, you need + +``` +bits per item m/n = −ln(p) / (ln 2)² ≈ 1.44 · log2(1/p) +``` + +| target `p` | 10% | 1% | 0.1% | +|---|---|---|---| +| bits per item | 4.8 | 9.6 | 14.4 | +| best `k` | 3 | 7 | 10 | + +Each 10× drop in the error rate costs about 4.8 more bits per item. + +## Summary + +A Bloom filter maps each item to `k` positions in a bit array and sets those bits. A query checks the same positions. A 0 proves the item is absent. All 1s mean "probably present", because other items can set the same bits. That shared use of bits is what makes the filter small, and it is also what makes it wrong sometimes. The error is one-sided (false "yes", never false "no"). Its rate depends only on bits per item and `k`, and about 10 bits per item gives about 1%. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/read.json new file mode 100644 index 0000000..965c9f7 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "When adding an item, the filter hashes it to k bit positions and sets those bits to 1. When looking up an item, it hashes it the same way and checks those positions—if any bit is 0, the item was definitely not added; if all are 1, it was probably added.", + "quote": "To add an item, it hashes the item to a few bit positions and sets those bits to 1. To check an item, it hashes the item the same way and looks at the same positions. If any of those bits is 0, the item was never added. If all of them are 1, the item was *probably* added." + }, + "2": { + "answer": "It can give false positives (answering yes to items never added) but never false negatives (answering no to items that were added). This is because adding only changes bits from 0 to 1, never back to 0, so bits once set stay set.", + "quote": "It can say \"yes\" to an item that it never saw. It never says \"no\" to an item that it did see. Add only changes bits from 0 to 1. Nothing changes a bit back to 0." + }, + "3": { + "answer": "Because multiple items can share the same bits. Clearing an item's bits would also clear bits set by other items, causing their queries to incorrectly return 'not seen'.", + "quote": "Bit 6 also belongs to \"dog\". A query for \"dog\" now finds bit 6 = 0 and answers \"definitely not seen\", which is wrong." + }, + "4": { + "answer": "It answers 'probably seen' because when all bits in the filter are 1, any query will find all its k positions set to 1.", + "quote": "If all bits are 1, answer \"probably seen\"." + } + }, + "stops": [ + "a function that turns an item into a number that looks random", + "Assumption: each hash position is uniform and independent of the others", + "(1 − 1/m)^(k·n) ≈ e^(−k·n/m) — mathematical approximation not justified", + "A *counting Bloom filter* stores a small counter in place of each bit", + "The chance that a given bit is still 0 is (1 − 1/m)^(k·n) — derivation not shown", + "Each hash gives a position from 0 to `m − 1` — why this range?", + "false-positive rate p ≈ (1 − e^(−k·n/m))^k — how does this follow?", + "The array then carries as much information as possible: a bit is equally likely to be 0 or 1", + "bits per item m/n = −ln(p) / (ln 2)² — origin of formula not explained", + "The query for an item that was never added checks `k` bits" + ], + "excess": [ + "Databases such as LevelDB, RocksDB, and Cassandra use Bloom filters in this way", + "A normal set stores each item. One million URLs of about 50 bytes each need about 50 MB", + "The price is that the filter can only answer the membership question. It cannot list its items", + "Why use several hash functions, not one — optimization discussion section", + "More hashes make each item's fingerprint harder to fake by accident", + "The best value is `k = (m/n)·ln 2`, which is about 7 here", + "At that value, about half of the bits are 1. The array then carries as much information as possible", + "If you add more items than planned, the error rate climbs fast", + "The bits cannot be resized, because the filter does not keep the items it would need to hash again", + "Each 10× drop in the error rate costs about 4.8 more bits per item" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/run.json new file mode 100644 index 0000000..daf0d60 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/bloom-filter/explanation.md`, a prose explanation, because the mechanism is a short sequence of steps; worked tables (a 16-bit example, error rates by `k` and by fill level) show the trade-offs without an interactive page; `explainer check` was not run and I made no `model.md`/`model.yaml`, but I calculated all numbers by hand.", + "error": null, + "cost": 0.1830298, + "seconds": 48, + "turns": 2, + "files": [ + "explainers/bloom-filter/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/explanation.md new file mode 100644 index 0000000..3403260 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/explanation.md @@ -0,0 +1,88 @@ +# Bloom filters: "have I seen this before?" in a few bits per item + + + +**A Bloom filter does not keep the items. It keeps a row of bits, and each item turns on a few of them. If any of an item's bits is still off, the answer is "definitely not seen". If all of them are on, the answer is "probably seen", because other items can turn on the same bits. So the filter can be wrong in one direction only: it can say "probably seen" for an item it never saw.** + + +The motive is memory. A hash set of 1 million URLs, at about 50 bytes per URL, takes about 50 MB. A Bloom filter for the same URLs takes 1.25 MB (10 bits per URL). The price for the smaller memory is error: the filter gives a wrong "probably seen" for 0.82% of URLs it never saw. The memory does not depend on how long the items are, because the filter never keeps them. + +## How it works: add and query + + +Start with a **bit array** of m bits, all 0. In this small example, m = 16, with positions 0 to 15. + +Choose k different hash functions; here k = 3. Take each hash output mod m to get a **position**. For example, a hash output of 3,141,593 gives position 3,141,593 mod 16 = 9. Each item thus gets k positions, one from each hash function. + +To **add** an item, set its k positions to 1. Nothing else is kept. Add two items: + +- cat → positions 1, 6, 11 +- dog → positions 3, 9, 14 + +| Position | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| Bit | 0 | **1** | 0 | **1** | 0 | 0 | **1** | 0 | 0 | **1** | 0 | **1** | 0 | 0 | **1** | 0 | +| Set by | | cat | | dog | | | cat | | | dog | | cat | | | dog | | + +To **query** an item, compute the same k positions and read those bits: + +- If any bit is 0, the answer is **definitely not seen**. +- If all bits are 1, the answer is **probably seen**. + +Query fox → positions 2, 6, 14. Bit 2 is 0, so fox is definitely not seen. That answer is right. + +## How it can be wrong: false positives + + +Query owl → positions 3, 6, 11. Owl was never added. But bit 3 came from dog, and bits 6 and 11 came from cat. All three bits are 1, so the filter says "probably seen". A **false positive** is this error: "probably seen" for an item that was never added. The filter cannot tell owl from an added item, because it kept only bits, and bits do not record which item set them. + +## Why it is never wrong the other way + + +The other error, a **false negative**, is "definitely not seen" for an item that was added. A false negative cannot happen. Adding cat set bits 1, 6 and 11, and nothing ever sets a bit back to 0. So any later query for cat finds all three bits at 1. + +This guarantee depends on bits never going back to 0. Suppose you add owl for real, and then try to delete cat by clearing bits 1, 6 and 11. Owl also uses bits 6 and 11, so they are now 0, and the filter says owl is definitely not seen. That is a false negative. Bits are shared between items, so a Bloom filter has no delete operation. + +## How often it is wrong + + +Let n be the number of items added. Assume that each hash position is a uniform random choice, independent of the others. Then: + +1. Each time an add sets a bit, it misses a given bit with probability 1 − 1/m. +2. n adds set kn bits in total. So a given bit is still 0 with probability (1 − 1/m)^(kn). Because (1 − 1/m)^m ≈ 1/e for large m, this is about e^(−kn/m). +3. So the **fill** f, the fraction of bits that are 1, is about 1 − e^(−kn/m). +4. A never-added item gets k random positions. Each one is 1 with probability f. The **false-positive rate** p, the chance that a never-added item reads "probably seen", is the chance that all k are 1: **p ≈ f^k**. + +Take n = 1 million items and m = 10 million bits: 10 bits per item (m/n = 10). With k = 7: kn/m = 0.7, e^(−0.7) = 0.497, so f = 0.503 and p = 0.503^7 = 0.0082, or 0.82%. That is about 1 wrong answer in 120 queries for never-added items. + + +Why 7 hash functions? The number of hash functions k pulls in two directions, at 10 bits per item: + +| k | Fill f | False-positive rate p | Why | +|---|---|---|---| +| 1 | 0.095 | 9.5% | Most bits are 0, but a query checks only one bit, so p = f. | +| 7 | 0.503 | 0.82% | About half the bits are 1; a query checks 7 of them. | +| 20 | 0.865 | 5.5% | Each add sets 20 bits, so almost every bit is 1. | + +More hash functions make each query check more bits, but they also fill the array faster. Below half full, a query checks too few bits; above half full, almost every bit it checks is 1. The best balance is near f = 0.5, which the formula gives at k = (m/n)·ln 2, about 0.69 times the bits per item. Here 10 × 0.69 = 6.9, so k = 7. At the best k, p ≈ 0.5^k. Each extra bit per item adds 0.69 to k, so it multiplies p by 0.5^0.69 ≈ 0.62. Thus each extra 4.8 bits per item cuts p by about 10 times. + +## Overfilling + + +The rate p stays at 0.82% only while n stays near the **capacity**: the number of items the filter was sized for. Keep the same filter (10 million bits, k = 7) and add more items. At 2 million items, p is 13.8%. At 3 million items, kn/m = 2.1, f = 1 − e^(−2.1) = 0.88, and p = 0.88^7 = 40%. The filter never forgets an added item, but its "probably seen" answer stops telling you much. + +## What it is good for + + +Put the filter in front of a slow exact lookup, such as a database on disk. If the filter says "definitely not seen", skip the lookup; that is safe, because there are no false negatives. If the filter says "probably seen", do the lookup, which gives the true answer. A false positive thus costs one wasted lookup, never a wrong final answer. Of 1,000,000 queries for never-added items, about 8,200 do a wasted lookup at p = 0.82%; at p = 40%, about 400,000 do. + +## Summary + +A Bloom filter answers "have I seen this before?" by remembering only the bits each item turned on. A 0 bit proves an item is new. All 1s mean "probably seen", which is wrong at a rate p ≈ f^k that you choose with bits per item, and that rises when you add more items than the capacity. + +## Not covered here + +- **Deletion:** counting Bloom filters keep a small counter per position instead of a bit. Cuckoo filters are another option. +- **Growing:** changing m changes every position (hash mod m), so you cannot resize a filter in place. See "scalable Bloom filter". +- **Sizing:** for n items and a target p, m = −n·ln p / (ln 2)². +- **Cheap hash functions:** real code makes k positions from two hash values ("double hashing"). diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/model.md new file mode 100644 index 0000000..7d83da9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/model.md @@ -0,0 +1,162 @@ +# Semantic model: Bloom filter + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How does a Bloom filter answer "have I seen this before?" with little memory, and how can it be wrong? + +## Audience and prior knowledge + +A technical reader: school mathematics (powers, probability of independent events, e ≈ 2.718), basic programming (arrays, bits, functions, what a hash set is). New to Bloom filters. Knows that a hash function turns any input into a number that looks random, and that the same input always gives the same number; does not know how a Bloom filter uses several of them. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| bit array | A row of m bits, all 0 at the start; the only storage the filter has. | fact | +| hash function | A function that maps an item to one position in the bit array; the same item always gets the same position. | fact | +| k | The number of hash functions; each item gets k positions. | fact | +| m | The number of bits in the bit array. | fact | +| n | The number of items added so far. | fact | +| add | Set the item's k bits to 1. | fact | +| query | Look at the item's k bits; any 0 → "definitely not seen"; all 1 → "probably seen". | fact | +| false positive | The filter says "probably seen" for an item that was never added. | fact | +| false negative | The filter says "definitely not seen" for an item that was added. Impossible while bits only go 0 → 1. | mathematical consequence | +| false-positive rate p | The chance that a query for an item never added gives "probably seen". | mathematical consequence (under the random-hash assumption) | +| bits per item m/n | Memory spent per added item. | fact | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| add | sets to 1 | k bits | never sets a bit to 0 | +| query | reads | the same k bits | same hash functions as add | +| other items' bits | can cover | all k bits of a new item | cause of false positives | +| n / m | raises | fraction of bits that are 1 | fuller array → more false positives | +| k | raises the bits each query checks, and raises the fill | p | two opposite effects → a best k | + +## Dependencies (learning order) + +1. Hash function gives a fixed position per item. +2. Bit array, add (k bits set to 1). +3. Query; the 0 → "definitely not" rule. +4. Why a 0 proves absence (bits never go back to 0). +5. False positive: all k bits set by other items. +6. Rate p, its dependence on m/n and k; best k. +7. What breaks: overfilling; deleting. + +## Causal chain + +```text +add item → its k bits become 1 → bits never become 0 + → a later query on the item finds all 1 → "probably seen" is always right for added items (no false negatives) +other items fill bits → a never-added item's k bits can all be 1 already → false positive +more items per bit → more 1s → higher p +``` + +## Temporal sequence + +1. Start: m bits, all 0. +2. Add each item: compute k positions, set them to 1. +3. Query: compute k positions, read them. + +## Quantities + +- Toy: m = 16, k = 3. Positions are stipulated (stand-ins for hash outputs), numbered 0–15. + - "cat" → 1, 6, 11. "dog" → 3, 9, 14. After both: bits 1, 3, 6, 9, 11, 14 are 1 (6 of 16). + - Query "fox" → 2, 6, 14: bit 2 is 0 → definitely not seen. Correct. + - Query "owl" → 3, 6, 11: all 1 (3 from dog; 6, 11 from cat) → probably seen. Wrong: false positive. + - Estimate from the observed fill: (6/16)³ ≈ 0.053, about 5%. (The formula's average fill for m = 16, n = 2, k = 3 is 0.31, giving 3.1%; the toy positions were picked with no collisions, so the toy is fuller than average.) +- Hash output → position: a hash function gives a large number; take it mod m. Illustrative: h1("cat") = 3,141,593; 3,141,593 mod 16 = 9. +- Fill f: the fraction of bits that are 1. +- Formula (assumption: hash positions behave like independent uniform random choices): + 1. One bit-set misses a given bit with probability 1 − 1/m. + 2. n adds make kn bit-sets, so the bit is still 0 with probability (1 − 1/m)^(kn) ≈ e^(−kn/m). (Toy check: (15/16)^6 = 0.679 vs e^(−0.375) = 0.687.) + 3. So f ≈ 1 − e^(−kn/m). + 4. A never-added item gives a false positive when all k of its positions land on 1 bits: p ≈ f^k. + - Main example m/n = 10, k = 7: kn/m = 0.7, e^(−0.7) = 0.497, f = 0.503, p = 0.503^7 = 0.0082. +- Use: put the filter in front of a slow exact lookup (disk, database, network). "Definitely not seen" skips the lookup; "probably seen" does the lookup, which settles the answer. A false positive costs one wasted lookup, never a wrong final answer. 1,000,000 queries for absent items at p = 0.82% → about 8,200 wasted lookups; at p = 40% → about 400,000. +- At m/n = 10 bits per item: + - k = 1: p ≈ 9.5% + - k = 7: p ≈ 0.82% (best; (m/n)·ln 2 = 6.9) + - k = 20: p ≈ 5.5% +- Best k = (m/n)·ln 2. At best k, f ≈ 0.5. Why half: at k = 1, f = 0.095, but the query checks only one bit; at k = 20, f = 0.865, so almost every bit is 1 and a query rarely finds a 0. Between, more checks and a fuller array balance near f = 0.5. At best k, p ≈ 0.5^k = 0.6185^(m/n); this gives "each ~4.8 bits per item cuts p by 10×" (ln 10 / (ln 2)² = 4.79). +- Sizing: m = −n·ln p / (ln 2)². n = 1 M, p = 0.82% → m ≈ 10 M bits. +- Bits per item meaning: 5 bits/item (k = 3) → p ≈ 9.2%; 10 → 0.82%; 20 (k = 14) → about 0.0067%. Each extra ~4.8 bits per item cuts p by about 10×. +- Memory, 1 million items: 10 bits each = 10 million bits = 1.25 MB. Storing 1 million URLs of about 50 bytes each = about 50 MB (estimate; the URL length is assumed), before hash-set overhead. +- Overfill: filter sized for 1 million items (m = 10 million, k = 7). With 2 million items p ≈ 13.8%; with 3 million, p ≈ 40%. +- The memory does not depend on item size: a 2 KB document and a 5-byte word each cost the same k bit-sets. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Overfilled (n above plan) | p rises from 0.82% to 40% at 3× | most bits become 1 | +| Delete by clearing bits | false negatives appear | a cleared bit can belong to another item | +| k too small | p high (k = 1: 9.5%) | each query checks only one bit | +| k too large | p high (k = 20: 5.5%) | each add sets many bits, the array fills | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| store bits, not items | a hash set of the items | 1 million URLs ≈ 50 MB vs 1.25 MB for the filter at 0.82% | +| k > 1 hash functions | one hash function (k = 1) | at 10 bits/item: 9.5% false positives vs 0.82% at k = 7 | +| not too many hash functions | k = 20 | 5.5%: each add sets 20 bits, so the array fills up | +| never clear bits (no delete) | clear the k bits to delete | add owl too, delete cat by clearing 1, 6, 11 → owl's bits 6, 11 are 0 → "definitely not seen" for owl, which is in the filter | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| "definitely not seen" is always correct (no false negatives) | cat → 1, 6, 11 set; query cat later finds all 1 | delete allowed: clearing cat's bits makes owl (added) read as absent | +| p ≈ (1 − e^(−kn/m))^k stays small | 1 M items, 10 M bits, k = 7: 0.82% | 3 M items in the same filter: 40% | +| mechanism: add/query | toy: cat, dog, fox, owl | — | + +## Terms + +- **bit array**: the m bits. +- **position**: an index into the bit array, 0 to m − 1, given by a hash function. +- **add**: set the k positions to 1. (Not "insert" or "store": the item itself is not kept.) +- **query**: read the k positions. +- **definitely not seen** / **probably seen**: the two answers. +- **false positive**: "probably seen" for an item never added. +- **false negative**: "definitely not seen" for an item that was added. +- **false-positive rate p**: low 0.0067% (20 bits/item) means almost never wrong; 0.82% (10 bits/item) about 1 in 120 never-added queries; 40% (overfilled) nearly useless. +- **bits per item m/n**: 10 in the main example. +- **fill f**: fraction of bits that are 1; 0.503 in the main example; near 0 the filter wastes memory, near 1 nearly every query says "probably seen". +- **capacity**: the n used to choose m and k (1 million in the main example). Overfilling = adding more than capacity. + +## Epistemic status + +- **Established:** the add/query rules; no false negatives without deletion. +- **Mathematical consequence (under an assumption):** p formula and best k, assuming hash positions act as independent uniform random choices. The formula is an approximation; the true rate is close for large m. +- **Estimated:** 50 bytes per URL. + +## Confusion points + +- "The filter stores the items" → it stores only bits; you cannot list or recover items. +- "It can say no wrongly" → no; a 0 bit proves the item was never added. +- "More hash functions is always better" → no; past (m/n)·ln 2 the array fills and p rises. +- "p is the chance an added item is reported wrongly" → p applies only to items never added. +- "Remove an item by clearing its bits" → causes false negatives. + +## Scope + +- **Out of scope:** counting Bloom filters (counters instead of bits, allow deletion) — search "counting Bloom filter". +- **Out of scope:** cuckoo filters, quotient filters — alternatives that support deletion. +- **Out of scope:** deriving k from two hashes (double hashing, Kirsch–Mitzenmacher) — how real code gets k positions cheaply. +- **Out of scope:** the exact (non-approximate) false-positive formula. +- **Out of scope:** growing a filter when n is unknown — changing m changes every position (hash mod m), so a resized filter gives false negatives; see "scalable Bloom filter". + +## Why this form (representation) + +Prose with one small bit-array table. The mechanism is a short sequence (add, query) and the failure is one case; a table of the toy array carries the picture. The parameter trade-off has three values, so a small table is enough; a slider page would add little. + +## Representation decision + +- **Stage:** 1 +- **Reason:** sequential mechanism + one counterexample + a 3-row trade-off; a table in prose shows all of it. +- **Levels:** L1 what it answers / L2 add and query / L3 why no false negatives, why false positives, the rate / L4 sizing and limits. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/model.yaml new file mode 100644 index 0000000..c486ed8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/model.yaml @@ -0,0 +1,135 @@ +# Machine-readable half of model.md. `explainer check bloom-filter` holds every rendering to it. + +values: + toy_m: 16 + toy_k: 3 + toy_last_position: 15 + toy_cat: [1, 6, 11] + toy_dog: [3, 9, 14] + toy_fox: [2, 6, 14] + toy_owl: [3, 6, 11] + hash_example_output: 3141593 + hash_example_position: 9 + items_million: 1 + bits_per_item: 10 + m_bits_million: 10 + filter_mb: 1.25 + url_bytes: 50 + urls_mb: 50 + k_best: 7 + k_best_exact: 6.9 + kn_over_m: 0.7 + minus_kn_over_m: -0.7 + e_term: 0.497 + fill: 0.503 + p_percent: 0.82 + p_decimal: 0.0082 + one_in: 120 + queries: 1000000 + wasted_lookups: 8200 + k1_p_percent: 9.5 + k1_fill: 0.095 + k20: 20 + k20_p_percent: 5.5 + k20_fill: 0.865 + overfill_2x_million: 2 + overfill_2x_p_percent: 13.8 + overfill_3x_million: 3 + overfill_3x_p_percent: 40 + overfill_3x_wasted: 400000 + overfill_3x_kn_over_m: 2.1 + overfill_3x_minus_kn_over_m: -2.1 + overfill_3x_fill: 0.88 + best_k_fill: 0.5 + per_bit_factor: 0.62 + ten_x_bits: 4.8 + ln2: 0.69 + +allow: [0, 1, 2, 3, 4, 5, 8, 10, 12, 13, 100] + +require: + - path: p_percent + in: [prose] + - path: filter_mb + in: [prose] + - path: overfill_3x_p_percent + in: [prose] + - path: k1_p_percent + in: [prose] + +claims: + - id: mechanism-add-query + kind: mechanism + statement: "Add sets the item's k positions to 1. Query reads them: any 0 means definitely not seen; all 1 means probably seen." + example: "m = 16, k = 3. cat → 1, 6, 11; dog → 3, 9, 14. fox → 2, 6, 14: bit 2 is 0, definitely not seen." + ask: "A filter has bits 1, 3, 6, 9, 11, 14 set. A query's positions are 2, 6, 14. What does the filter answer, and can that answer be wrong?" + + - id: guarantee-no-false-negative + kind: guarantee + statement: "A 'definitely not seen' answer is always right, because add only sets bits to 1 and nothing sets them back to 0." + example: "cat sets 1, 6, 11; any later query for cat finds all three at 1." + counterexample: "Allow delete by clearing bits: add owl (3, 6, 11), then delete cat by clearing 1, 6, 11. Owl's bits 6 and 11 are now 0, so the filter says owl is definitely not seen, though owl was added." + ask: "Why can't you remove an item from a Bloom filter by setting its bits back to 0?" + + - id: mechanism-false-positive + kind: mechanism + statement: "A false positive happens when other items have already set all k positions of an item that was never added." + example: "owl → 3, 6, 11. Bit 3 came from dog, bits 6 and 11 from cat. The filter says probably seen; owl was never added." + + - id: why-bits-not-items + kind: why + statement: "The filter stores bits, not items, so memory depends on the number of items and the target error, not on item size." + alternative: "Store the items in a hash set." + counterexample: "1 million URLs of about 50 bytes = about 50 MB; a Bloom filter at 10 bits per item = 1.25 MB with 0.82% false positives." + + - id: why-several-hashes + kind: why + statement: "Several hash functions make a false positive need several 1 bits at once; too many fill the array." + alternative: "One hash function (k = 1), or many (k = 20)." + counterexample: "At 10 bits per item: k = 1 gives 9.5%; k = 7 gives 0.82%; k = 20 gives 5.5%." + ask: "At 10 bits per item, what happens to the false-positive rate if you use 20 hash functions instead of 7, and why?" + + - id: mechanism-rate + kind: mechanism + statement: "Each bit is still 0 with probability about e^(-kn/m), so the fill is f ≈ 1 − e^(-kn/m) and p ≈ f^k." + example: "m/n = 10, k = 7: e^(-0.7) = 0.497, f = 0.503, p = 0.503^7 = 0.0082." + + - id: limit-overfill + kind: guarantee + statement: "The false-positive rate stays small only while n stays near the capacity the filter was sized for." + example: "10 million bits, k = 7, 1 million items: 0.82%." + counterexample: "Same filter, 3 million items: 40%." + ask: "A filter sized for 1 million items gives 0.82% false positives. What happens if 3 million items are added?" + + - id: why-useful + kind: why + statement: "One-sided error is useful: put the filter before a slow exact lookup; 'definitely not seen' skips it, a false positive costs one wasted lookup." + alternative: "Do the slow lookup for every query." + counterexample: "1,000,000 queries for absent items: at 0.82%, about 8,200 lookups instead of 1,000,000." + +accept_unjustified: + ln: "ln 2 appears only in the best-k result (m/n)·ln 2, given as a result of the formula with its meaning (about half the bits are 1); deriving it needs calculus, out of scope for this reader." + +terms: + - term: add + means: "set the item's k positions to 1; the item itself is not kept" + avoid: [stores the item, stores the items, store the item] + - term: false positive + means: "'probably seen' for an item never added" + avoid: [false alarm, false hit] + - term: definitely not seen + means: "the answer when any position is 0" + avoid: [maybe not seen, probably not seen] + +budget: {prose: 1100, reason: "mechanism with a toy, two guarantees each with a breaking case, a derived rate with a worked number, and a use case"} + +quiz: + - q: "A Bloom filter says 'probably seen' for an item. Was the item added?" + expect: "Not necessarily: other items may have set all its bits (a false positive). Only 'definitely not seen' is certain." + misconception: "Yes, the item was certainly added." + - q: "You add more and more items to a fixed-size Bloom filter. What happens to its answers?" + expect: "More bits become 1, so false positives rise (0.82% at capacity to 40% at 3×). Items that were added still always read 'probably seen'." + misconception: "It starts forgetting items or gives false negatives." + - q: "Why does a Bloom filter use less memory than a hash set of the items?" + expect: "It keeps only about 10 bits per item, not the item; the price is a small chance of false positives." + misconception: "It compresses the items and can list them back." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/narrative.md new file mode 100644 index 0000000..4bbb524 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/narrative.md @@ -0,0 +1,60 @@ +# Narrative: bloom-filter + +## 1. Reader before and after + +- **Before:** a technical reader (school maths incl. probability of independent events and e; arrays, bits, hash functions, hash sets). Has never seen a Bloom filter. +- **After:** "A Bloom filter sets k bits per item and checks those k bits later; a 0 proves the item is new, but all 1s can come from other items, so 'yes' is only probable, and it gets less reliable as the array fills." + +## 2. Question and motive + +- **Question:** How can a program answer "have I seen this before?" for millions of items without storing them, and when is the answer wrong? +- **Result in words:** keep a row of bits; each item turns on a few of them. If any of an item's bits is off, it is definitely new. If all are on, it is probably not new, and the filter is wrong only in that direction, with a rate you choose by memory per item. +- **Why care:** 1 million URLs take about 50 MB; the filter takes 1.25 MB and is wrong 0.82% of the time, only toward "probably seen". +- **Why this approach:** a hash function already turns an item into a fixed number. Turn the number into a bit position and remember only that the bit was hit. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| bit array, m | row of m bits, all 0 at the start | m = 16, positions 0–15 | 2 | +| position | index in the bit array, from hash mod m | 3,141,593 mod 16 = 9 | 2 | +| k | number of hash functions per item | k = 3 | 2 | +| add | set the item's k positions to 1 | cat → 1, 6, 11 | 2 | +| query | read the item's k positions | fox → 2, 6, 14 | 3 | +| definitely not seen / probably seen | the two answers | fox / owl | 3, 4 | +| false positive | "probably seen" for an item never added | owl | 4 | +| false negative | "definitely not seen" for an added item | owl after clearing cat | 5 | +| n | number of items added | 1 million | 6 | +| fill f | fraction of bits that are 1 | 0.503 | 6 | +| p | false-positive rate | 0.82% | 6 | +| m/n, bits per item | memory per item | 10 | 6 | +| capacity | the n the filter was sized for | 1 million | 8 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | Why not just a hash set? | — | 50 MB vs 1.25 MB | "A hash set of 1 million URLs (about 50 bytes each) takes about 50 MB. A Bloom filter keeps no URLs: it keeps a row of bits that each URL turned on, 10 bits per URL, 1.25 MB. It answers 'definitely not seen' or 'probably seen', and only the second answer can be wrong." | what the filter promises | +| 2 | mechanism | What does it store? | So | 16-bit table after cat, dog | "Start with m = 16 bits, all 0, numbered 0–15. Use k = 3 different hash functions. Each gives a large number; take it mod 16 to get a position (3,141,593 mod 16 = 9). To add an item, set its 3 positions to 1. cat → 1, 6, 11. dog → 3, 9, 14." | add, m, k, position | +| 3 | mechanism | How does it answer? | Then | fox query | "To query an item, compute the same 3 positions and read them. If any bit is 0, the answer is 'definitely not seen'. If all are 1, the answer is 'probably seen'. fox → 2, 6, 14; bit 2 is 0, so fox is definitely not seen." | query, both answers | +| 4 | notice | Can it be wrong? | But | owl query | "owl → 3, 6, 11, never added. Bit 3 came from dog; bits 6 and 11 from cat. All are 1, so the filter says 'probably seen'. This wrong 'probably seen' is a false positive." | false positive | +| 5 | guarantee | Can it be wrong the other way? | Therefore | cat query; delete case | "The other error, a false negative, is 'definitely not seen' for an item that was added. It cannot happen, because add only sets bits to 1 and nothing sets them back. But suppose you try to delete cat by clearing 1, 6, 11, after adding owl for real. Owl's bits 6 and 11 are now 0, so owl reads 'definitely not seen': a false negative. Bits are shared, so a Bloom filter has no delete." | one-sided error and why | +| 6 | mechanism | How often is it wrong? | So | 4-step derivation, 10 bits/item | "Let n be the number of items added, and assume each hash position is a uniform random choice, independent of the others. One bit-set misses a given bit with probability 1 − 1/m. n adds make kn bit-sets, so the bit stays 0 with probability (1 − 1/m)^(kn) ≈ e^(−kn/m). So the fill f, the fraction of bits that are 1, is about 1 − e^(−kn/m). A never-added item is a false positive when all k of its positions are 1: p ≈ f^k. With n = 1 million, m = 10 million (10 bits per item, m/n = 10) and k = 7: kn/m = 0.7, f = 0.503, p = 0.503^7 = 0.82%." | n, f, p, m/n | +| 7 | approach | Why k = 7, not 1 or 20? | But | 3-row table | "At 10 bits per item: k = 1 gives f = 0.095 but checks only one bit, p = 9.5%. k = 20 makes f = 0.865, so nearly every bit is 1, p = 5.5%. k = 7 gives f ≈ 0.5, the balance, p = 0.82%. The best k is (m/n)·ln 2." | k trade-off | +| 8 | limit | What if I add too many? | And | 2× → 13.8%, 3× → 40% | "The filter was sized for a capacity of 1 million items. Add 2 million and p is 13.8%; 3 million, 40%. Added items still always read 'probably seen'." | capacity | +| 9 | payoff | What is it good for? | So | filter before disk | "Check the filter before a slow lookup on disk. 'Definitely not seen' skips the lookup, safely, because there are no false negatives. 'Probably seen' does the lookup, which gives the true answer. Of 1,000,000 queries for absent items, about 8,200 do a wasted lookup." | use | +| 10 | close | Answered? | — | — | "A Bloom filter remembers only the bits each item turned on. A 0 bit proves an item is new. All 1s mean 'probably seen', wrong at rate p, which rises as the array fills." | after-sentence | + +## 5. Concrete to symbol + +Toy first (m = 16, k = 3, two items), then the symbols m, k, n bound aloud, then the formula with m/n = 10, k = 7. + +## 6. Links between representations + +The toy table shows bit numbers above, 0/1 below, and which item set each bit. + +## 7. Close the loop + +- **Answer:** It keeps only bits each item turned on. A 0 proves "new"; all 1s mean "probably seen", wrong at rate p. +- **General argument:** bits only go from 0 to 1, so added items always pass; other items' bits can cover a new item, with probability f^k. +- **Payoff:** a cheap pre-check before slow lookups. Scope: deletion (counting Bloom filters), growing, k from two hashes. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/decisions.yaml new file mode 100644 index 0000000..b87b0fd --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/decisions.yaml @@ -0,0 +1,211 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T150533-probe#1: + decision: adopted +2026-10-04T150533-probe#10: + decision: declined +2026-10-04T150533-probe#11: + decision: declined +2026-10-04T150533-probe#12: + decision: declined +2026-10-04T150533-probe#2: + decision: adopted +2026-10-04T150533-probe#3: + decision: adopted +2026-10-04T150533-probe#4: + decision: declined +2026-10-04T150533-probe#5: + decision: declined +2026-10-04T150533-probe#6: + decision: adopted +2026-10-04T150533-probe#7: + decision: adopted +2026-10-04T150533-probe#8: + decision: adopted +2026-10-04T150533-probe#9: + decision: declined +2026-10-04T150725-coldread-narrative#1: + decision: adopted +2026-10-04T150725-coldread-narrative#10: + decision: adopted +2026-10-04T150725-coldread-narrative#11: + decision: adopted +2026-10-04T150725-coldread-narrative#12: + decision: adopted +2026-10-04T150725-coldread-narrative#13: + decision: adopted +2026-10-04T150725-coldread-narrative#14: + decision: adopted +2026-10-04T150725-coldread-narrative#15: + decision: adopted +2026-10-04T150725-coldread-narrative#16: + decision: adopted +2026-10-04T150725-coldread-narrative#17: + decision: adopted +2026-10-04T150725-coldread-narrative#18: + decision: adopted +2026-10-04T150725-coldread-narrative#19: + decision: adopted +2026-10-04T150725-coldread-narrative#2: + decision: adopted +2026-10-04T150725-coldread-narrative#20: + decision: adopted +2026-10-04T150725-coldread-narrative#21: + decision: adopted +2026-10-04T150725-coldread-narrative#22: + decision: adopted +2026-10-04T150725-coldread-narrative#23: + decision: adopted +2026-10-04T150725-coldread-narrative#24: + decision: adopted +2026-10-04T150725-coldread-narrative#25: + decision: adopted +2026-10-04T150725-coldread-narrative#26: + decision: adopted +2026-10-04T150725-coldread-narrative#27: + decision: adopted +2026-10-04T150725-coldread-narrative#28: + decision: adopted +2026-10-04T150725-coldread-narrative#29: + decision: adopted +2026-10-04T150725-coldread-narrative#3: + decision: adopted +2026-10-04T150725-coldread-narrative#30: + decision: adopted +2026-10-04T150725-coldread-narrative#31: + decision: adopted +2026-10-04T150725-coldread-narrative#32: + decision: adopted +2026-10-04T150725-coldread-narrative#33: + decision: adopted +2026-10-04T150725-coldread-narrative#34: + decision: adopted +2026-10-04T150725-coldread-narrative#35: + decision: adopted +2026-10-04T150725-coldread-narrative#36: + decision: declined +2026-10-04T150725-coldread-narrative#37: + decision: adopted +2026-10-04T150725-coldread-narrative#38: + decision: adopted +2026-10-04T150725-coldread-narrative#39: + decision: adopted +2026-10-04T150725-coldread-narrative#4: + decision: adopted +2026-10-04T150725-coldread-narrative#40: + decision: adopted +2026-10-04T150725-coldread-narrative#41: + decision: adopted +2026-10-04T150725-coldread-narrative#5: + decision: adopted +2026-10-04T150725-coldread-narrative#6: + decision: adopted +2026-10-04T150725-coldread-narrative#7: + decision: adopted +2026-10-04T150725-coldread-narrative#8: + decision: adopted +2026-10-04T150725-coldread-narrative#9: + decision: adopted +2026-10-04T150831-coldread-narrative#1: + decision: adopted +2026-10-04T150831-coldread-narrative#10: + decision: adopted +2026-10-04T150831-coldread-narrative#2: + decision: adopted +2026-10-04T150831-coldread-narrative#3: + decision: declined +2026-10-04T150831-coldread-narrative#4: + decision: adopted +2026-10-04T150831-coldread-narrative#5: + decision: adopted +2026-10-04T150831-coldread-narrative#6: + decision: adopted +2026-10-04T150831-coldread-narrative#7: + decision: adopted +2026-10-04T150831-coldread-narrative#8: + decision: adopted +2026-10-04T150831-coldread-narrative#9: + decision: declined +2026-10-04T151003-coldread-prose#1: + decision: adopted +2026-10-04T151003-coldread-prose#10: + decision: declined +2026-10-04T151003-coldread-prose#11: + decision: declined +2026-10-04T151003-coldread-prose#12: + decision: declined +2026-10-04T151003-coldread-prose#13: + decision: declined +2026-10-04T151003-coldread-prose#14: + decision: declined +2026-10-04T151003-coldread-prose#2: + decision: adopted +2026-10-04T151003-coldread-prose#3: + decision: adopted +2026-10-04T151003-coldread-prose#4: + decision: declined +2026-10-04T151003-coldread-prose#5: + decision: declined +2026-10-04T151003-coldread-prose#6: + decision: adopted +2026-10-04T151003-coldread-prose#7: + decision: adopted +2026-10-04T151003-coldread-prose#8: + decision: adopted +2026-10-04T151003-coldread-prose#9: + decision: adopted +2026-10-04T151043-quiz-prose#1: + decision: adopted +2026-10-04T151043-quiz-prose#10: + decision: adopted +2026-10-04T151043-quiz-prose#11: + decision: declined +2026-10-04T151043-quiz-prose#12: + decision: adopted +2026-10-04T151043-quiz-prose#13: + decision: adopted +2026-10-04T151043-quiz-prose#14: + decision: declined +2026-10-04T151043-quiz-prose#2: + decision: declined +2026-10-04T151043-quiz-prose#3: + decision: declined +2026-10-04T151043-quiz-prose#4: + decision: adopted +2026-10-04T151043-quiz-prose#5: + decision: adopted +2026-10-04T151043-quiz-prose#6: + decision: adopted +2026-10-04T151043-quiz-prose#7: + decision: adopted +2026-10-04T151043-quiz-prose#8: + decision: adopted +2026-10-04T151043-quiz-prose#9: + decision: declined +2026-10-04T151208-quiz-prose#1: + decision: adopted +2026-10-04T151208-quiz-prose#10: + decision: declined +2026-10-04T151208-quiz-prose#11: + decision: declined +2026-10-04T151208-quiz-prose#12: + decision: declined +2026-10-04T151208-quiz-prose#13: + decision: declined +2026-10-04T151208-quiz-prose#14: + decision: declined +2026-10-04T151208-quiz-prose#2: + decision: declined +2026-10-04T151208-quiz-prose#3: + decision: declined +2026-10-04T151208-quiz-prose#4: + decision: declined +2026-10-04T151208-quiz-prose#5: + decision: declined +2026-10-04T151208-quiz-prose#6: + decision: declined +2026-10-04T151208-quiz-prose#7: + decision: declined +2026-10-04T151208-quiz-prose#8: + decision: declined +2026-10-04T151208-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150533-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150533-probe.json new file mode 100644 index 0000000..e3725c0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150533-probe.json @@ -0,0 +1,185 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "bloom-filter", + "at": "2026-10-04T15:05:33", + "model": "default", + "prompt_sha": "7ab239541a1d", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "The derivation of p ≈ (1 − e^(−kn/m))^k skips steps and never connects the fill fraction to p", + "why_it_matters": "The reader is shown the final formula but not why. Three steps are missing. First, one add misses a given bit with probability 1 − 1/m. Second, there are kn independent tries, so the bit is still 0 with probability (1 − 1/m)^(kn). Third, (1 − 1/m)^(kn) ≈ e^(−kn/m), and the audience knows e but not this limit. The final step, p = (fraction of 1 bits)^k, also assumes a never-added item's k positions are independent of each other and may repeat. None of this is stated, so the formula appears without support.", + "suggested_addition": "Work through the main example. With m/n = 10 and k = 7: kn/m = 0.7 and e^(−0.7) ≈ 0.497, so the fraction of 1 bits is ≈ 0.503 and p ≈ 0.503^7 ≈ 0.0082. Check the approximation on the toy: (15/16)^6 ≈ 0.679 versus e^(−0.375) ≈ 0.687. Say that p is the chance that k independent uniform draws all land on 1 bits." + }, + { + "rule": 4, + "severity": "edge", + "about": "The toy 'rough rate' and the formula disagree, and the file doesn't say why", + "why_it_matters": "The toy uses the observed fill, 6/16 = 0.375, so p ≈ 5.3%. The formula with m = 16, n = 2, k = 3 predicts a fill of 1 − e^(−0.375) ≈ 0.31, so p ≈ 3.1%. The gap comes from the toy positions being chosen with no collisions. A careful reader will compute both numbers and get confused.", + "suggested_addition": "Say that (6/16)^3 ≈ 5.3% uses the bits actually set. The formula predicts the average fill, ≈ 0.31, which gives p ≈ 3.1%. Random hashes would often collide and set fewer than 6 bits." + }, + { + "rule": 1, + "severity": "edge", + "about": "Best k = (m/n)·ln 2 and 'about half the bits are 1' are stated without a reason", + "why_it_matters": "Without a reason, ln 2 looks arbitrary, and 'half full' isn't linked to it.", + "suggested_addition": "Intuition: when the array is half full, each bit tells you the most. At k = 7, m/n = 10, the fill is 0.503. At k = 1 the fill is 0.095: most bits are 0, so each check carries little information. At k = 20 the fill is 0.865: almost every bit is 1, so checks rarely find a 0. Optionally add that at best k, p ≈ 0.5^k = 0.6185^(m/n). That is where 'each ~4.8 bits per item cuts p by 10×' comes from, since ln 10 / ln(1/0.6185) ≈ 4.79." + }, + { + "rule": 2, + "severity": "edge", + "about": "The no-false-negatives guarantee also assumes the query uses the same m and the same hash functions as the add. Only deletion is shown as breaking it.", + "why_it_matters": "A reader may try to 'grow' a full filter by enlarging the array. That silently breaks the guarantee.", + "suggested_addition": "Breaking case: resize the toy array from m = 16 to m = 32 without re-adding. If the positions are hash mod m, cat's positions change (for example 1 → 17), the new bit is 0, and cat reads as 'definitely not seen'." + }, + { + "rule": 2, + "severity": "edge", + "about": "The formula's random-hash assumption has no breaking instance", + "why_it_matters": "Epistemic status names the assumption, but Concrete cases breaks the formula only by overfilling, never by violating the assumption itself.", + "suggested_addition": "Breaking case: all 7 hash functions are identical. Each item then sets 1 bit, the fill is ≈ 9.5% at m/n = 10, and p ≈ 9.5% instead of 0.82%. The formula's prediction is off by 11×." + }, + { + "rule": 5, + "severity": "main", + "about": "The file never says what a 'probably seen' answer costs, or why an error that runs in only one direction is useful", + "why_it_matters": "The central question is 'have I seen this before?' The reader needs to know that a false positive usually costs one wasted exact check, never a wrong final answer. Without that, 0.82% means nothing in practice.", + "suggested_addition": "Use case: check the filter before a disk or database lookup. With 1,000,000 queries for absent URLs at p = 0.82%, about 8,200 queries hit the disk unnecessarily, and about 991,800 are answered from 1.25 MB of RAM. At p = 40%, about 400,000 hit the disk." + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts have more than one name, and some are used before they are defined", + "why_it_matters": "The fill fraction is called 'fraction of bits that are 1', 'more 1s', 'fill' and 'fills' (in Relationships, Alternative states and Confusion points), but it is not in Terms. Relationships uses n/m while the rest of the file uses bits per item m/n. Planned capacity is called 'n above plan' and 'sized for 1 million items' but is never defined. 'Rate' means both the toy's observed-fill estimate and p. 'Position' and 'bit' are used interchangeably.", + "suggested_addition": "Define fill f = fraction of bits that are 1 (for example 0.503 at m/n = 10, k = 7). Use m/n everywhere. Define capacity n₀ = the n used to choose m and k (1,000,000 in the example). Call the toy figure 'estimate from the observed fill'." + }, + { + "rule": 4, + "severity": "edge", + "about": "The step from hash output to position has no worked example", + "why_it_matters": "The audience knows a hash gives a large number but not how it becomes an index from 0 to m − 1. The toy positions are only stipulated.", + "suggested_addition": "Example: h1(\"cat\") = 3,141,593 and 3,141,593 mod 16 = 9 (an illustrative value). Repeat with h2 and h3 to get k positions." + }, + { + "rule": 1, + "severity": "edge", + "about": "The file doesn't say why each slot is 1 bit rather than a byte or a boolean", + "why_it_matters": "A programmer's natural alternative is a bool array, which typically uses 1 byte per slot.", + "suggested_addition": "With 1 byte per slot, 10 M slots cost 10 MB instead of 1.25 MB: 8× the memory for no gain in accuracy, since each slot only ever holds 0 or 1." + } + ], + "excess": [ + "The 'Why this form (representation)' and 'Representation decision' sections (stage, levels) are rendering metadata, not content for the reader", + "The k = 20 example appears in three places (Quantities, Alternative states, Why this form). One is enough.", + "The 20 bits/item, p ≈ 0.0067% data point is not needed once the rule '~4.8 bits per item per 10×' is given" + ], + "next_questions": [ + { + "q": "What do I actually do with 'probably seen'? Where is a Bloom filter used, and what does a false positive cost?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Given my n and a target p, how do I choose m and k (for example m = −n·ln p / (ln 2)^2)?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How do real implementations get k independent hash functions cheaply?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "If I can't delete by clearing bits, how do I support deletion?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "What if I don't know n in advance? Can a filter grow without breaking the no-false-negatives guarantee?", + "answered_by_model": false, + "declared_out_of_scope": false + } + ] + }, + "usage": { + "cost": 0.1518542, + "seconds": 55 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "The derivation of p ≈ (1 − e^(−kn/m))^k skips steps and never connects the fill fraction to p → Work through the main example. With m/n = 10 and k = 7: kn/m = 0.7 and e^(−0.7) ≈ 0.497, so the fraction of 1 bits is ≈ 0.503 and p ≈ 0.503^7 ≈ 0.0082. Check the approximation on the toy: (15/16)^6 ≈ 0.679 versus e^(−0.375) ≈ 0.687. Say that p is the chance that k independent uniform draws all land on 1 bits." + }, + { + "n": 2, + "severity": "edge", + "kind": "rule 4", + "text": "The toy 'rough rate' and the formula disagree, and the file doesn't say why → Say that (6/16)^3 ≈ 5.3% uses the bits actually set. The formula predicts the average fill, ≈ 0.31, which gives p ≈ 3.1%. Random hashes would often collide and set fewer than 6 bits." + }, + { + "n": 3, + "severity": "edge", + "kind": "rule 1", + "text": "Best k = (m/n)·ln 2 and 'about half the bits are 1' are stated without a reason → Intuition: when the array is half full, each bit tells you the most. At k = 7, m/n = 10, the fill is 0.503. At k = 1 the fill is 0.095: most bits are 0, so each check carries little information. At k = 20 the fill is 0.865: almost every bit is 1, so checks rarely find a 0. Optionally add that at best k, p ≈ 0.5^k = 0.6185^(m/n). That is where 'each ~4.8 bits per item cuts p by 10×' comes from, since ln 10 / ln(1/0.6185) ≈ 4.79." + }, + { + "n": 4, + "severity": "edge", + "kind": "rule 2", + "text": "The no-false-negatives guarantee also assumes the query uses the same m and the same hash functions as the add. Only deletion is shown as breaking it. → Breaking case: resize the toy array from m = 16 to m = 32 without re-adding. If the positions are hash mod m, cat's positions change (for example 1 → 17), the new bit is 0, and cat reads as 'definitely not seen'." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 2", + "text": "The formula's random-hash assumption has no breaking instance → Breaking case: all 7 hash functions are identical. Each item then sets 1 bit, the fill is ≈ 9.5% at m/n = 10, and p ≈ 9.5% instead of 0.82%. The formula's prediction is off by 11×." + }, + { + "n": 6, + "severity": "main", + "kind": "rule 5", + "text": "The file never says what a 'probably seen' answer costs, or why an error that runs in only one direction is useful → Use case: check the filter before a disk or database lookup. With 1,000,000 queries for absent URLs at p = 0.82%, about 8,200 queries hit the disk unnecessarily, and about 991,800 are answered from 1.25 MB of RAM. At p = 40%, about 400,000 hit the disk." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts have more than one name, and some are used before they are defined → Define fill f = fraction of bits that are 1 (for example 0.503 at m/n = 10, k = 7). Use m/n everywhere. Define capacity n₀ = the n used to choose m and k (1,000,000 in the example). Call the toy figure 'estimate from the observed fill'." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 4", + "text": "The step from hash output to position has no worked example → Example: h1(\"cat\") = 3,141,593 and 3,141,593 mod 16 = 9 (an illustrative value). Repeat with h2 and h3 to get k positions." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 1", + "text": "The file doesn't say why each slot is 1 bit rather than a byte or a boolean → With 1 byte per slot, 10 M slots cost 10 MB instead of 1.25 MB: 8× the memory for no gain in accuracy, since each slot only ever holds 0 or 1." + }, + { + "n": 10, + "severity": "excess", + "kind": "model entry", + "text": "The 'Why this form (representation)' and 'Representation decision' sections (stage, levels) are rendering metadata, not content for the reader" + }, + { + "n": 11, + "severity": "excess", + "kind": "model entry", + "text": "The k = 20 example appears in three places (Quantities, Alternative states, Why this form). One is enough." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "The 20 bits/item, p ≈ 0.0067% data point is not needed once the rule '~4.8 bits per item per 10×' is given" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150725-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150725-coldread-narrative.json new file mode 100644 index 0000000..d11a5a4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150725-coldread-narrative.json @@ -0,0 +1,443 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "bloom-filter", + "at": "2026-10-04T15:07:25", + "model": "default", + "prompt_sha": "84de7cead935", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: 50 MB vs 1.25 MB. Said: answer first: bits, one-sided error", + "unresolved": [ + "one-sided error", + "bits (bits of what, set by what?)" + ], + "unsaid": [ + "50 MB vs 1.25 MB: the words never say what is being stored (how many items, which kinds) or which number belongs to the hash set and which to the filter" + ], + "leap": [ + "How can storing 'bits' replace storing the items? Nothing yet links an item to a bit." + ], + "excess": [ + "answer first: this is a stage direction, not something the reader can use" + ], + "purpose_unclear": true + }, + { + "at": "Beat 2 (mechanism)", + "text": "Shown: 16-bit table after cat, dog. Said: hash mod m → position; add sets k bits", + "unresolved": [ + "m", + "k", + "add" + ], + "unsaid": [ + "16-bit table: the words never say that m = 16 here, or which positions cat and dog set", + "how one item gets k different positions (k hash functions?) is never said" + ], + "leap": [ + "'hash mod m → position' gives one position, but 'add sets k bits' needs k of them. Where do the other k−1 come from?" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Shown: fox query. Said: any 0 → definitely not seen", + "unresolved": [], + "unsaid": [ + "fox's positions, and which of them is 0, are not said" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (notice)", + "text": "Shown: owl query. Said: bits from cat and dog cover owl → false positive", + "unresolved": [ + "false positive (used as a label, never defined as 'probably seen' for an item never added)" + ], + "unsaid": [ + "the answer 'probably seen' when all bits are 1 is never said as a rule. Only the 0 case was stated in beat 3", + "owl's positions and which bits came from cat and which from dog" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (guarantee)", + "text": "Shown: cat query; delete case. Said: bits never go to 0 → no false negatives; clearing breaks it", + "unresolved": [ + "false negatives", + "clearing" + ], + "unsaid": [ + "delete case: nothing in the words introduces deletion or says why it comes up here" + ], + "leap": [ + "'clearing breaks it': the reason (a cleared bit may be shared with another added item, such as owl's bit from cat) is not given" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Shown: 4-step derivation, 10 bits/item. Said: f ≈ 1 − e^(−kn/m), p ≈ f^k = 0.82%", + "unresolved": [ + "f", + "n", + "p", + "bits/item (as m/n)" + ], + "unsaid": [ + "the 4-step derivation is shown but none of its steps are spoken", + "the value of k behind 0.82% (k = 7) is not said", + "10 bits/item is not tied to m/n in the words" + ], + "leap": [ + "why f ≈ 1 − e^(−kn/m): the chance a given bit is missed by one hash, raised to the power kn, is not given", + "why p ≈ f^k: the independence assumption is not stated" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (approach)", + "text": "Shown: 3-row table. Said: too few checks vs too full; best k at f ≈ 0.5", + "unresolved": [ + "k = 7 (in the reader's question, but no beat has said k = 7)" + ], + "unsaid": [ + "the 3-row table's contents: which k values and which p values" + ], + "leap": [ + "why the best k lands at f ≈ 0.5 is asserted, not shown" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (limit)", + "text": "Shown: 2× → 13.8%, 3× → 40%. Said: p depends on n", + "unresolved": [ + "2×, 3× (twice and three times what? The intended n is never said aloud)", + "capacity (in 'Reader now knows' but never spoken)" + ], + "unsaid": [ + "the shown percentages are not mentioned in the words" + ], + "leap": [], + "excess": [ + "p depends on n: already plain from the beat 6 formula, so nothing new is said aloud" + ], + "purpose_unclear": false + }, + { + "at": "Beat 9 (payoff)", + "text": "Shown: filter before disk. Said: a false positive costs one wasted lookup", + "unresolved": [], + "unsaid": [ + "filter before disk: the setup (check the filter, then go to disk only on 'probably seen') is not said" + ], + "leap": [ + "why it is safe: no false negatives means a real item is never skipped. This is not linked back to beat 5" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 10 (close)", + "text": "Shown: —. Said: answer the question in its words", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": true + } + ], + "blocking": [ + "Beat 2: 'add sets k bits'. k is undefined, and one 'hash mod m' gives one position, not k, so the core mechanism can't be rebuilt", + "Beat 4: 'false positive'. The rule 'all 1s → probably seen' is never said, so the reader cannot tell why owl gets the wrong answer", + "Beat 6: 'f ≈ 1 − e^(−kn/m), p ≈ f^k = 0.82%'. f, n and p are undefined, the derivation is shown but not spoken, and k = 7 is missing", + "Beat 10: 'answer the question in its words'. A placeholder: the close says nothing" + ], + "cuts": [ + "Beat 1 'answer first:' and beat 10's placeholder: replace with the real sentences, or drop them", + "Beat 8 'p depends on n': already follows from beat 6. Say the 2×/3× numbers instead" + ], + "question": { + "text": "How can a program answer 'have I seen this before?' for many items without storing them, and when is the answer wrong?", + "known_at": "Only implied at beat 1 (reader's question 'Why not just a hash set?'). The Said column never asks it.", + "result_stated": "No, not in full words. Beat 1 gives only 'bits, one-sided error', and beat 10 is a placeholder." + }, + "motive": "Reason to care: only hinted by beat 1 Shown '50 MB vs 1.25 MB', with nothing said about what is stored. Reason for the approach (a hash already maps an item to a number, so keep one bit for it): missing from the beats.", + "close": "Not answered. Beat 10 Said is 'answer the question in its words' with nothing shown. The general argument (bits only go 0→1, so added items always pass; others' bits cover a new item with probability f^k) is never said at the end.", + "recap": "A Bloom filter turns on a few bits per item; any off bit means definitely new, all on means probably seen, and it gets less reliable as it fills.", + "strongest": "Beat 5: 'bits never go to 0 → no false negatives'. A clear one-line reason for the one-sided guarantee.", + "weakest": "Beat 6: the formula arrives with f, n and p undefined and the derivation unspoken. Beat 10 is a close close behind, as an empty placeholder.", + "suggestions": [ + "In beat 2, say 'k different hash functions, each mod m, give k positions; here m = 16, k = 3, and cat sets 1, 6, 11'.", + "In beat 3 or 4, state the rule 'all k bits 1 → probably seen' before showing owl.", + "In beat 6, define n, f and p aloud, speak the steps (chance one bit is missed = (1 − 1/m)^(kn) ≈ e^(−kn/m)), and say that k = 7 gives 0.82%.", + "In beat 1, say '1 million URLs: a hash set takes about 50 MB, the filter 1.25 MB, wrong 0.82% of the time, and only toward \"probably seen\"'.", + "In beat 5, give the reason clearing fails: a bit may be shared with another added item.", + "Write beat 10 using the plan's own answer and general argument from section 7." + ] + }, + "usage": { + "cost": 0.10809079999999999, + "seconds": 36 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'add sets k bits'. k is undefined, and one 'hash mod m' gives one position, not k, so the core mechanism can't be rebuilt" + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'false positive'. The rule 'all 1s → probably seen' is never said, so the reader cannot tell why owl gets the wrong answer" + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 6: 'f ≈ 1 − e^(−kn/m), p ≈ f^k = 0.82%'. f, n and p are undefined, the derivation is shown but not spoken, and k = 7 is missing" + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 10: 'answer the question in its words'. A placeholder: the close says nothing" + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 1 'answer first:' and beat 10's placeholder: replace with the real sentences, or drop them" + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 8 'p depends on n': already follows from beat 6. Say the 2×/3× numbers instead" + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): one-sided error" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): bits (bits of what, set by what?)" + }, + { + "n": 9, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): 50 MB vs 1.25 MB: the words never say what is being stored (how many items, which kinds) or which number belongs to the hash set and which to the filter" + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): How can storing 'bits' replace storing the items? Nothing yet links an item to a bit." + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): m" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): k" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): add" + }, + { + "n": 14, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): 16-bit table: the words never say that m = 16 here, or which positions cat and dog set" + }, + { + "n": 15, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): how one item gets k different positions (k hash functions?) is never said" + }, + { + "n": 16, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (mechanism): 'hash mod m → position' gives one position, but 'add sets k bits' needs k of them. Where do the other k−1 come from?" + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): fox's positions, and which of them is 0, are not said" + }, + { + "n": 18, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (notice): false positive (used as a label, never defined as 'probably seen' for an item never added)" + }, + { + "n": 19, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (notice): the answer 'probably seen' when all bits are 1 is never said as a rule. Only the 0 case was stated in beat 3" + }, + { + "n": 20, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (notice): owl's positions and which bits came from cat and which from dog" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (guarantee): false negatives" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (guarantee): clearing" + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (guarantee): delete case: nothing in the words introduces deletion or says why it comes up here" + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (guarantee): 'clearing breaks it': the reason (a cleared bit may be shared with another added item, such as owl's bit from cat) is not given" + }, + { + "n": 25, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): f" + }, + { + "n": 26, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): n" + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): p" + }, + { + "n": 28, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): bits/item (as m/n)" + }, + { + "n": 29, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): the 4-step derivation is shown but none of its steps are spoken" + }, + { + "n": 30, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): the value of k behind 0.82% (k = 7) is not said" + }, + { + "n": 31, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): 10 bits/item is not tied to m/n in the words" + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): why f ≈ 1 − e^(−kn/m): the chance a given bit is missed by one hash, raised to the power kn, is not given" + }, + { + "n": 33, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): why p ≈ f^k: the independence assumption is not stated" + }, + { + "n": 34, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (approach): k = 7 (in the reader's question, but no beat has said k = 7)" + }, + { + "n": 35, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (approach): the 3-row table's contents: which k values and which p values" + }, + { + "n": 36, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (approach): why the best k lands at f ≈ 0.5 is asserted, not shown" + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (limit): 2×, 3× (twice and three times what? The intended n is never said aloud)" + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (limit): capacity (in 'Reader now knows' but never spoken)" + }, + { + "n": 39, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (limit): the shown percentages are not mentioned in the words" + }, + { + "n": 40, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 9 (payoff): filter before disk: the setup (check the filter, then go to disk only on 'probably seen') is not said" + }, + { + "n": 41, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (payoff): why it is safe: no false negatives means a real item is never skipped. This is not linked back to beat 5" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150831-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150831-coldread-narrative.json new file mode 100644 index 0000000..2bea3a1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150831-coldread-narrative.json @@ -0,0 +1,209 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "bloom-filter", + "at": "2026-10-04T15:08:31", + "model": "default", + "prompt_sha": "84de7cead935", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Hash set of 1M URLs ≈ 50 MB; Bloom filter keeps 10 bits per URL, 1.25 MB; answers 'definitely not seen' / 'probably seen', only the second can be wrong.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (mechanism)", + "text": "m = 16 bits, k = 3 hash functions, hash mod 16 gives a position; add sets 3 positions; cat → 1, 6, 11; dog → 3, 9, 14.", + "unresolved": [], + "unsaid": [ + "The shown '16-bit table after cat, dog' may mark which item set each bit, but the words never say so." + ], + "leap": [ + "'(3,141,593 mod 16 = 9)' is not linked to any item. Position 9 later turns up under dog, so the reader may stop to check whether this was dog's hash." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Query reads the same 3 positions; any 0 → definitely not seen; all 1 → probably seen; fox → 2, 6, 14, bit 2 is 0.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (notice)", + "text": "owl → 3, 6, 11, never added, all bits set by dog and cat → false positive.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (guarantee)", + "text": "No false negatives because bits only go to 1; deleting cat after adding owl would cause one; so no delete.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Derivation: miss prob 1 − 1/m, (1 − 1/m)^(kn) ≈ e^(−kn/m), f ≈ 1 − e^(−kn/m), p ≈ f^k; with m/n = 10, k = 7 → p = 0.82%.", + "unresolved": [ + "'p' is never named as the false-positive rate. The beat says 'p ≈ f^k' straight after 'is a false positive when…', so the reader has to infer what p means." + ], + "unsaid": [], + "leap": [ + "'(1 − 1/m)^(kn) ≈ e^(−kn/m)': the step (1 − 1/m)^m ≈ 1/e is not stated.", + "'p ≈ f^k': it is not said that the fill fraction f is used as the probability that one random position is 1, or that the k positions are treated as independent for this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (approach)", + "text": "k = 1 → p = 9.5%, k = 20 → p = 5.5%, k = 7 → f ≈ 0.5, p = 0.82%; best k = (m/n)·ln 2.", + "unresolved": [], + "unsaid": [], + "leap": [ + "'The best k is (m/n)·ln 2' is stated with no reason, and the beat never says that 10·ln 2 ≈ 6.9 rounds to 7.", + "'f ≈ 0.5, the balance': it is not said why half-full is the optimum." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (limit)", + "text": "Sized for 1M; 2M → 13.8%, 3M → 40%; added items still read 'probably seen'.", + "unresolved": [], + "unsaid": [], + "leap": [ + "The beat does not say that m and k stay fixed (10M bits, k = 7) while n grows." + ], + "excess": [ + "'Added items still always read \"probably seen\"' repeats beat 5 with nothing new." + ], + "purpose_unclear": false + }, + { + "at": "Beat 9 (payoff)", + "text": "Filter before a disk lookup; 'definitely not seen' skips it; ≈ 8,200 wasted lookups per 1M absent queries.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [ + "'safely, because there are no false negatives' restates beat 5." + ], + "purpose_unclear": false + }, + { + "at": "Beat 10 (close)", + "text": "Remembers only bits; a 0 proves new; all 1s → probably seen, wrong at rate p, rising as the array fills.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Beat 8: drop 'Added items still always read \"probably seen\"' (repeats beat 5).", + "Beat 9: shorten 'safely, because there are no false negatives' to 'safely' (repeats beat 5)." + ], + "question": { + "text": "How can a program check 'have I seen this?' for millions of items without storing the items, and when is that answer wrong?", + "known_at": "Beat 1. It is implied by the 50 MB vs 1.25 MB contrast, but the words never ask it as a question.", + "result_stated": "Yes. Beat 1: 'It answers \"definitely not seen\" or \"probably seen\", and only the second answer can be wrong.' Beat 10: 'A 0 bit proves an item is new. All 1s mean \"probably seen\", wrong at rate p, which rises as the array fills.'" + }, + "motive": "Reason to care: beat 1, 'A hash set of 1 million URLs ... takes about 50 MB ... 10 bits per URL, 1.25 MB', and beat 9, 'Check the filter before a slow lookup on disk.' Reason for the approach: partly given in beat 1, 'keeps no URLs: it keeps a row of bits that each URL turned on'. The words never say why hashing to bit positions is the natural way to do this.", + "close": "Mostly. Beat 10: 'A Bloom filter remembers only the bits each item turned on. A 0 bit proves an item is new. All 1s mean \"probably seen\", wrong at rate p, which rises as the array fills.' It answers the question. It does not repeat the reason a 0 is proof (bits only ever go from 0 to 1), and it does not mention that the rate is set by memory per item.", + "recap": "A Bloom filter hashes each item to k bits and sets them; a 0 among an item's bits proves it is new, all 1s only means 'probably seen', and with about 10 bits per item it is wrong under 1% of the time, getting worse as it fills.", + "strongest": "Beats 4–5: the owl example produces a false positive from cat's and dog's bits, then reuses owl to show why deleting creates false negatives.", + "weakest": "Beat 7: 'The best k is (m/n)·ln 2' and 'f ≈ 0.5, the balance' are given with no reason. Beat 6's undefined 'p' is a close second.", + "suggestions": [ + "In beat 6, name p aloud: 'call the false-positive rate p'.", + "In beat 6, add one clause: (1 − 1/m)^m ≈ 1/e for large m.", + "In beat 7, say that 10·ln 2 ≈ 6.9, hence k = 7. Either give the one-line reason for the optimum or mark it as stated without proof.", + "In beat 2, tie the mod example to an item, e.g. 'dog's second hash is 3,141,593; mod 16 = 9'.", + "In beat 8, say that m and k stay fixed while n grows.", + "In beat 10, add 'because bits only ever go from 0 to 1' so the close carries the general argument." + ] + }, + "usage": { + "cost": 0.1236152, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Beat 8: drop 'Added items still always read \"probably seen\"' (repeats beat 5)." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Beat 9: shorten 'safely, because there are no false negatives' to 'safely' (repeats beat 5)." + }, + { + "n": 3, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): The shown '16-bit table after cat, dog' may mark which item set each bit, but the words never say so." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (mechanism): '(3,141,593 mod 16 = 9)' is not linked to any item. Position 9 later turns up under dog, so the reader may stop to check whether this was dog's hash." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): 'p' is never named as the false-positive rate. The beat says 'p ≈ f^k' straight after 'is a false positive when…', so the reader has to infer what p means." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): '(1 − 1/m)^(kn) ≈ e^(−kn/m)': the step (1 − 1/m)^m ≈ 1/e is not stated." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): 'p ≈ f^k': it is not said that the fill fraction f is used as the probability that one random position is 1, or that the k positions are treated as independent for this." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (approach): 'The best k is (m/n)·ln 2' is stated with no reason, and the beat never says that 10·ln 2 ≈ 6.9 rounds to 7." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (approach): 'f ≈ 0.5, the balance': it is not said why half-full is the optimum." + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (limit): The beat does not say that m and k stay fixed (10M bits, k = 7) while n grows." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151003-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151003-coldread-prose.json new file mode 100644 index 0000000..63bb141 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151003-coldread-prose.json @@ -0,0 +1,322 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "bloom-filter", + "at": "2026-10-04T15:10:03", + "model": "default", + "prompt_sha": "4717e3c3480f", + "reply": { + "paragraphs": [ + { + "at": "Title (line 1)", + "text": "Bloom filters: \"have I seen this before?\" in a few bits per item", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold lede (line 5)", + "text": "A Bloom filter does not keep the items. It keeps a row of bits...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Motive paragraph (line 8)", + "text": "The motive is memory. A hash set of 1 million URLs...", + "unresolved": [], + "leap": [ + "\"gives a wrong 'probably seen' for 0.82% of new URLs\": the 0.82% figure and the choice of 10 bits per URL are given without a source here. It is a forward claim that is only derived at line 56, and nothing says it will be derived later." + ], + "excess": [ + "\"(The 50 bytes per URL is an estimate.)\": this repeats the \"about\" already in the sentence." + ], + "purpose_unclear": false + }, + { + "at": "How it works, para 1 (line 13)", + "text": "Start with a bit array of m bits, all 0...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "How it works, para 2 (line 15)", + "text": "Choose k different hash functions; here k = 3...", + "unresolved": [], + "leap": [], + "excess": [ + "\"A hash function gives a large number, and the same item always gives the same number.\": this reader already knows hash functions." + ], + "purpose_unclear": false + }, + { + "at": "Add + list (lines 17-20)", + "text": "To add an item, set its k positions to 1... cat → 1, 6, 11; dog → 3, 9, 14", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Table (lines 22-25)", + "text": "Position / Bit / Set by", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Query rule (lines 27-30)", + "text": "To query an item... If any bit is 0... If all bits are 1...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Fox query (line 32)", + "text": "Query fox → positions 2, 6, 14...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "False positives (line 37)", + "text": "Query owl → positions 3, 6, 11...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "No false negatives, para 1 (line 42)", + "text": "The other error, a false negative...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "No false negatives, para 2 (line 44)", + "text": "This guarantee depends on bits never going back to 0...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "How often, intro (line 49)", + "text": "Let n be the number of items added. Assume...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Derivation list (lines 51-54)", + "text": "1. ...1 − 1/m. 2. ...e^(−kn/m). 3. fill f... 4. p ≈ f^k", + "unresolved": [], + "leap": [ + "Step 4, \"Each one is 1 with probability f\": the step from 'fraction of bits that are 1' to 'probability a random position is 1', and then multiplying k of these, uses the independence assumption only implicitly. The reader can manage this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Worked numbers (line 56)", + "text": "Take n = 1 million items and m = 10 million bits...", + "unresolved": [], + "leap": [], + "excess": [ + "\"That is 10 bits per item (m/n = 10).\": this says the same thing twice." + ], + "purpose_unclear": false + }, + { + "at": "Why 7, intro (line 59)", + "text": "Why 7 hash functions? The number of hash functions k pulls in two directions...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "k table (lines 61-65)", + "text": "k=1 / 7 / 20 with fill, p, why", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Best k (line 67)", + "text": "More hash functions make each query check more bits... best k is (m/n)·ln 2... Each extra 4.8 bits per item then cuts p by about 10 times.", + "unresolved": [], + "leap": [ + "\"The best k is (m/n)·ln 2\": stated with no reason.", + "\"At the best k, about half the bits are 1 (a mathematical result of the formula above)\": the reader is not shown why.", + "\"Each extra 4.8 bits per item then cuts p by about 10 times\": no reason given, and the reader cannot easily rebuild it." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Overfilling (line 72)", + "text": "The rate p stays at 0.82% only while n stays near the capacity...", + "unresolved": [], + "leap": [ + "\"the number of items used to choose m and k\": the text has not said how m is chosen from n. Only k = (m/n)·ln 2 was given, and the sizing formula appears later, in 'Not covered'." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What it is good for (line 77)", + "text": "Put the filter in front of a slow exact lookup...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Summary (line 81)", + "text": "A Bloom filter answers \"have I seen this before?\"...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered list (lines 85-88)", + "text": "Deletion / Growing / Sizing / Cheap hash functions", + "unresolved": [ + "\"counting Bloom filters\" (it gets a one-line gloss)", + "\"Cuckoo filters\"", + "\"scalable Bloom filter\"", + "\"double hashing\" (only glossed briefly)" + ], + "leap": [ + "\"m = −n·ln p / (ln 2)²\": given without derivation. That is acceptable for a pointer." + ], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Line 15: cut \"A hash function gives a large number, and the same item always gives the same number.\" because this reader already knows it.", + "Line 8: cut \"(The 50 bytes per URL is an estimate.)\"", + "Line 56: shorten \"That is 10 bits per item (m/n = 10).\" to a single statement." + ], + "question": { + "text": "How can you answer 'have I seen this item before?' using much less memory than a hash set, and what does it cost in accuracy?", + "known_at": "Title (line 1), made concrete in the lede (line 5) and the memory paragraph (line 8)", + "result_stated": "Yes. Line 5: \"So the filter can be wrong in one direction only: it can say 'probably seen' for an item it never saw.\" Line 8: \"takes 1.25 MB (10 bits per URL) and gives a wrong 'probably seen' for 0.82% of new URLs.\"" + }, + "motive": "Reason to care: \"The motive is memory. A hash set of 1 million URLs... takes about 50 MB. A Bloom filter... takes 1.25 MB\" (line 8), and the use case \"Put the filter in front of a slow exact lookup\" (line 77). Reason for the approach: \"The memory does not depend on how long the items are, because the filter never keeps them\" (line 8). The reason for using several hash functions is given at lines 59-67.", + "close": "Yes. Line 81: \"A Bloom filter answers 'have I seen this before?' by remembering only the bits each item turned on. A 0 bit proves an item is new. All 1s mean 'probably seen', which is wrong at a rate p ≈ f^k that you choose with bits per item, and that rises when you add more items than the capacity.\"", + "recap": "A Bloom filter stores only a few bits per item instead of the items themselves, so it can say 'definitely new' with certainty but only 'probably seen' otherwise; at about 10 bits per item that guess is wrong under 1% of the time.", + "strongest": "Lines 37-44: the owl example reuses cat's and dog's bits to produce a false positive. It then shows, with the same bits, why deleting cat would create a false negative. Both claims are made concrete with no new machinery.", + "weakest": "Line 67: three results arrive in a row with no reason given (best k = (m/n)·ln 2, half the bits are 1 at the best k, and 4.8 bits per item gives 10× lower p). This comes right after a step-by-step derivation that justified every step.", + "suggestions": [ + "At line 67, give a one-line reason: for fixed m/n, p = f^k is smallest when f = 1/2. Then k = (m/n)·ln 2, and p ≈ 0.5^k ≈ 0.6185^(m/n), which gives 4.8 bits per 10× reduction.", + "At line 8, add a short '(derived below)' after 0.82%.", + "In the Overfilling section, mention that m is chosen from the expected n and a target p, or move the sizing formula there." + ] + }, + "usage": { + "cost": 0.12415359999999999, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Line 15: cut \"A hash function gives a large number, and the same item always gives the same number.\" because this reader already knows it." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Line 8: cut \"(The 50 bytes per URL is an estimate.)\"" + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Line 56: shorten \"That is 10 bits per item (m/n = 10).\" to a single statement." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Motive paragraph (line 8): \"gives a wrong 'probably seen' for 0.82% of new URLs\": the 0.82% figure and the choice of 10 bits per URL are given without a source here. It is a forward claim that is only derived at line 56, and nothing says it will be derived later." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Derivation list (lines 51-54): Step 4, \"Each one is 1 with probability f\": the step from 'fraction of bits that are 1' to 'probability a random position is 1', and then multiplying k of these, uses the independence assumption only implicitly. The reader can manage this." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Best k (line 67): \"The best k is (m/n)·ln 2\": stated with no reason." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Best k (line 67): \"At the best k, about half the bits are 1 (a mathematical result of the formula above)\": the reader is not shown why." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Best k (line 67): \"Each extra 4.8 bits per item then cuts p by about 10 times\": no reason given, and the reader cannot easily rebuild it." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Overfilling (line 72): \"the number of items used to choose m and k\": the text has not said how m is chosen from n. Only k = (m/n)·ln 2 was given, and the sizing formula appears later, in 'Not covered'." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered list (lines 85-88): \"counting Bloom filters\" (it gets a one-line gloss)" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered list (lines 85-88): \"Cuckoo filters\"" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered list (lines 85-88): \"scalable Bloom filter\"" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered list (lines 85-88): \"double hashing\" (only glossed briefly)" + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "Not covered list (lines 85-88): \"m = −n·ln p / (ln 2)²\": given without derivation. That is acceptable for a pointer." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151043-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151043-quiz-prose.json new file mode 100644 index 0000000..2e3af51 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151043-quiz-prose.json @@ -0,0 +1,164 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "bloom-filter", + "at": "2026-10-04T15:10:34", + "model": "default", + "prompt_sha": "87e51d7b50fa", + "reply": { + "answers": { + "1": "The main objects are the bit array (m bits, all starting at 0), the k hash functions, the items, and each item's k positions (hash output mod m). The two operations are add and query. The answers are 'definitely not seen' and 'probably seen'. The errors are false positives and false negatives. The quantities are n (items added), the fill f, the false-positive rate p and the capacity.", + "2": "Each hash function maps an item to one position in the bit array, so each item gets k positions. Adding an item sets its k bits to 1. A query reads the same k bits. If any of them is 0 the answer is 'definitely not seen'; if all are 1 it is 'probably seen'. Different items can share bits, and a bit does not record which item set it. As n grows relative to m, the fill f rises. The false-positive rate p is about f^k.", + "3": "Adding items only ever turns bits on, so an added item always finds its bits at 1 and there are no false negatives. As more items are added, more bits are on by chance, so a never-added item can find all its k bits on. That is a false positive, with rate p ≈ (1 − e^(−kn/m))^k.", + "4": "The most important variable is the load: items per bit (n/m), or equivalently bits per item. If you add more items to a fixed m, the fill f = 1 − e^(−kn/m) rises and p = f^k rises sharply. With m = 10 million and k = 7, p is 0.82% at 1 million items, 13.8% at 2 million and 40% at 3 million. Giving each item more bits does the opposite: about 4.8 extra bits per item cuts p by 10 times (when k is chosen at its best value). k also matters: too few hashes means each query checks too few bits, and too many fills the array.", + "5": "Assumption: each hash position is a uniform random choice, independent of the others. The text also uses the approximation (1 − 1/m)^m ≈ 1/e. Established fact: false negatives cannot happen, because bits are never set back to 0. A worked fact: 10 bits per item with k = 7 gives p = 0.82%.", + "6": "A Bloom filter stores no items, only an m-bit array. Each item is hashed by k functions to k positions, and adding the item sets those bits to 1. A query checks the same positions: any 0 means definitely not seen, and all 1s means probably seen, because other items may have set those bits. With random hashing, the fill is f ≈ 1 − e^(−kn/m) and the false-positive rate is p ≈ f^k. The best k is (m/n)·ln 2, which gives 0.82% at 10 bits per item, so the filter is much smaller than a hash set and works well as a guard in front of a slow exact lookup. You cannot delete items, and adding many more items than the capacity makes p climb steeply.", + "7": "The hardest part was the optimum paragraph. The text says the best k is (m/n)·ln 2, that about half the bits are 1 at the best k, and that 4.8 extra bits per item cut p by 10 times. All three are stated as results with no derivation or intuition. A short line would have helped, for example: at f = 1/2, p = (1/2)^k and k = (m/n)·ln 2, so p = 2^(−(m/n)·ln 2); and ln 10 / (ln 2)² ≈ 4.8. The text also does not show how the 13.8% and 40% figures were computed.", + "8": "Not necessarily. The item was probably added, but it may be a false positive: all its bits could have been set by other items (like owl in the example).", + "9": "The fill rises and more queries for new items get 'probably seen'. With the example filter, the false-positive rate goes from 0.82% at 1 million items to 13.8% at 2 million and 40% at 3 million. Added items are still always reported as 'probably seen', but that answer stops telling you much.", + "10": "It keeps only a few bits per item (for example 10) and never stores the items themselves. Its memory therefore does not depend on how long the items are: 1.25 MB versus about 50 MB for 1 million URLs.", + "11": "Bit 2 is 0, so the filter answers 'definitely not seen'. That answer cannot be wrong: if the item had been added, bit 2 would have been set and never cleared.", + "12": "Bits are shared between items. Clearing an item's bits may clear bits that another added item needs, and that item would then get 'definitely not seen', which is a false negative. The example is deleting cat, which clears bits 6 and 11 that owl also uses.", + "13": "p rises from 0.82% to about 5.5%. Each add sets 20 bits, so the fill goes to about 0.865. Each query does check 20 bits, but nearly all of them are 1, so (0.865)^20 ≈ 5.5%.", + "14": "p rises to about 40%, which follows from f = 1 − e^(−2.1) ≈ 0.88 and p ≈ 0.88^7. About 400,000 of every 1,000,000 queries for new items would then do a wasted lookup. Added items are still never missed." + }, + "audit": { + "terms": [ + "The two answers are worded in two ways. Line 5 says the item is 'definitely new' / 'probably not new'; lines 29-30 and later say 'definitely not seen' / 'probably seen'.", + "'capacity' (line 72) and n, 'the number of items added' (line 49), are both used to describe the filter's design point. That the filter was 'sized for' n = 1 million is only implied.", + "The same error is called 'a wrong \"probably seen\"' (line 8), 'false positive' (line 37), 'false-positive rate p' (line 54) and 'wasted lookup' (line 77)." + ], + "unexplained": [ + "Line 67: 'Each extra 4.8 bits per item then cuts p by about 10 times.' The text does not say where 4.8 comes from.", + "Line 72: 'At 2 million items, p is 13.8%. At 3 million items, p is 40%.' No calculation is shown, unlike the worked 0.82% example.", + "Line 63, the k = 1 row: the text says p equals the fill (9.5%), but the reader has to work out themselves that with one hash p = f." + ], + "missing_why": [ + "Line 67: 'The best k is (m/n)·ln 2'. This is stated with no reason for why ln 2 appears.", + "Line 67: 'At the best k, about half the bits are 1 (a mathematical result of the formula above)'. The text asserts this without showing it.", + "Line 87: the sizing formula 'm = −n·ln p / (ln 2)²' is given with no derivation or link to p ≈ f^k." + ], + "excess": [ + "Line 15: 'A hash function gives a large number, and the same item always gives the same number.' This reader already knows hash functions.", + "Lines 85-88: the 'Not covered here' items (counting/cuckoo filters, scalable Bloom filters, double hashing) are not used by the argument or the questions." + ] + }, + "gaps": [ + "The text gives no derivation or intuition for the optimal k = (m/n)·ln 2, for the fill being one half at the optimum, or for the 4.8-bits-per-10× rule.", + "The overfill figures (13.8%, 40%) are given without the calculation.", + "The text does not explain why the formula is only approximate (repeated bits in step 2, independence of the hash positions) or how close it is for real hash functions." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 1, + "11": 2, + "12": 2, + "13": 2, + "14": 2 + }, + "reasons": { + "10": "Correctly says the filter keeps about 10 bits per item, not the items (1.25 MB vs about 50 MB). It omits the price the rubric expects: the memory saving comes with a small chance of false positives." + }, + "pass": false + }, + "usage": { + "cost": 0.172155, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The two answers are worded in two ways. Line 5 says the item is 'definitely new' / 'probably not new'; lines 29-30 and later say 'definitely not seen' / 'probably seen'." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'capacity' (line 72) and n, 'the number of items added' (line 49), are both used to describe the filter's design point. That the filter was 'sized for' n = 1 million is only implied." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "The same error is called 'a wrong \"probably seen\"' (line 8), 'false positive' (line 37), 'false-positive rate p' (line 54) and 'wasted lookup' (line 77)." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "Line 67: 'Each extra 4.8 bits per item then cuts p by about 10 times.' The text does not say where 4.8 comes from." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 72: 'At 2 million items, p is 13.8%. At 3 million items, p is 40%.' No calculation is shown, unlike the worked 0.82% example." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 63, the k = 1 row: the text says p equals the fill (9.5%), but the reader has to work out themselves that with one hash p = f." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 67: 'The best k is (m/n)·ln 2'. This is stated with no reason for why ln 2 appears." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 67: 'At the best k, about half the bits are 1 (a mathematical result of the formula above)'. The text asserts this without showing it." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 87: the sizing formula 'm = −n·ln p / (ln 2)²' is given with no derivation or link to p ≈ f^k." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "Line 15: 'A hash function gives a large number, and the same item always gives the same number.' This reader already knows hash functions." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "Lines 85-88: the 'Not covered here' items (counting/cuckoo filters, scalable Bloom filters, double hashing) are not used by the argument or the questions." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "The text gives no derivation or intuition for the optimal k = (m/n)·ln 2, for the fill being one half at the optimum, or for the 4.8-bits-per-10× rule." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "The overfill figures (13.8%, 40%) are given without the calculation." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The text does not explain why the formula is only approximate (repeated bits in step 2, independence of the hash positions) or how close it is for real hash functions." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151208-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151208-quiz-prose.json new file mode 100644 index 0000000..ab846b4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151208-quiz-prose.json @@ -0,0 +1,162 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "bloom-filter", + "at": "2026-10-04T15:12:02", + "model": "default", + "prompt_sha": "87e51d7b50fa", + "reply": { + "answers": { + "1": "The main objects are: a bit array of m bits; k hash functions, whose outputs are taken mod m to give positions; the items being added and queried; the two operations, add and query; the two answers, \"definitely not seen\" and \"probably seen\"; false positives and false negatives; and the quantities n (items added), f (fill, the fraction of bits that are 1), p (false-positive rate) and capacity.", + "2": "Each item is hashed by the k hash functions into k positions in the bit array. Adding an item sets those bits to 1. Querying an item reads the same k bits: any 0 means \"definitely not seen\", all 1s mean \"probably seen\". Because different items can share bits, an item that was never added can find all its bits set, which is a false positive. Bits never go back to 0, so an added item always finds its bits set, which means no false negatives. Fill f grows with kn/m, and p ≈ f^k. The filter is sized for a capacity n, and adding more items raises f and so raises p.", + "3": "Each added item turns on k shared bits and nothing is ever turned off. So a query that finds a 0 proves the item is new, and a query that finds all 1s may only be matching bits set by other items. That chance, p ≈ (1 − e^(−kn/m))^k, grows as the array fills.", + "4": "The key variable is items per bit, n/m (or its inverse, bits per item). If more items go into the same m, the fill f = 1 − e^(−kn/m) rises, so a never-added item is more likely to find all k bits at 1, and p rises sharply: 0.82% at 1M, 13.8% at 2M, 40% at 3M. More bits per item lowers p: at the best k, each extra 4.8 bits per item cuts p about tenfold. Changing k pulls two ways. More hashes mean more bits checked per query, but also faster filling. The best is near f = 0.5, at k ≈ 0.69·m/n.", + "5": "Assumption: each hash position is a uniform random choice, independent of the others. There is also the approximation (1 − 1/m)^m ≈ 1/e. Fact: nothing ever sets a bit back to 0, so a false negative is impossible. A worked figure: 1M items in 10M bits with k = 7 gives p ≈ 0.82%.", + "6": "A Bloom filter replaces a set of items with an array of m bits. Adding an item sets the k bits picked by k hash functions. A query answers \"definitely not seen\" if any of its k bits is 0, and \"probably seen\" if all are 1. Since bits are shared and never cleared, it can give false positives but never false negatives, and for the same reason it can't delete. With independent uniform hashes, the fill is f ≈ 1 − e^(−kn/m) and the false-positive rate is p ≈ f^k: about 0.82% at 10 bits per item with k = 7. The best k makes the array about half full, and p rises sharply once you add more items than the filter was sized for. That makes it a cheap pre-check before a slow exact lookup.", + "7": "The hardest part was the paragraph on choosing k. It says the optimum k = (m/n)·ln 2 \"which the formula gives\" without showing why f = 0.5 is the best point. It also jumps through 0.5^0.69 ≈ 0.62 to \"each 4.8 bits cuts p by 10\". Two things would help: a line showing that f = 0.5 means e^(−kn/m) = 1/2, so k = (m/n)·ln 2, and a line showing 0.62^4.8 ≈ 0.1.", + "8": "Not necessarily. All of its k bits are 1, which happens if it was added, but other items may have set those bits (a false positive, as with owl). It was probably added, but the filter can't confirm it.", + "9": "The fill f rises towards 1, so more never-added items find all their bits set, and the false-positive rate climbs: 0.82% at 1M, 13.8% at 2M, 40% at 3M for the 10M-bit, k = 7 filter. Added items are still always reported as \"probably seen\" (no false negatives), but \"probably seen\" stops meaning much.", + "10": "It doesn't store the items. It stores only a fixed number of bits per item (for example 10), whatever the item's length. A hash set stores the items themselves (for example about 50 bytes per URL). For 1M URLs that is 1.25 MB against about 50 MB, paid for with a small false-positive rate.", + "11": "Bit 2 is not set, so the answer is \"definitely not seen\". That answer can't be wrong: an added item would have set bit 2, and bits never go back to 0. (This is the fox example.)", + "12": "Bits are shared between items. Clearing one item's bits can clear bits another added item depends on. For example, deleting cat clears 6 and 11, which owl also uses, so owl would then read \"definitely not seen\". That is a false negative, and it breaks the guarantee.", + "13": "It gets worse: about 5.5% instead of 0.82%. Each add sets 20 bits, so the fill rises to about 0.865, and checking 20 bits that are almost all 1 doesn't filter well (0.865^20 ≈ 5.5%).", + "14": "kn/m = 2.1, so f = 1 − e^(−2.1) ≈ 0.88 and p ≈ 0.88^7 ≈ 40%. Added items are still never missed, but about 40% of queries for new items wrongly say \"probably seen\". In front of a database, that means about 400,000 wasted lookups per million queries for new items, against 8,200." + }, + "audit": { + "terms": [ + "The same concept appears under several names: \"never-added item\" (step 4, 'How often it is wrong'), \"new items\" / \"new URLs\" (intro, line 8; 'How often', line 56; 'What it is good for'), and \"an item that was never added\" ('false positives'). These are all the same population that p is measured over.", + "Capacity has two names: \"capacity\" ('Overfilling') and \"sized for\" ('Overfilling'; Q14). The summary also says p is chosen \"with bits per item\" rather than with capacity." + ], + "unexplained": [ + "'Overfilling': \"At 2 million items, p is 13.8%\" is given without the working that the 3-million case gets (kn/m = 1.4, f ≈ 0.75).", + "'How often it is wrong', final paragraph: \"Thus each extra 4.8 bits per item cuts p by about 10 times\". It doesn't show where 4.8 comes from (0.62^4.8 ≈ 0.1).", + "'Not covered here': \"m = −n·ln p / (ln 2)²\" is given with no meaning or example. The reader can't tell what m it gives for, say, p = 1%." + ], + "missing_why": [ + "'How often it is wrong': \"The best balance is near f = 0.5, which the formula gives at k = (m/n)·ln 2\". The reader isn't shown why the optimum is f = 0.5, or how setting e^(−kn/m) = 1/2 yields ln 2.", + "Step 4: \"p ≈ f^k\" relies on the k checked bits being independent, each 1 with probability f. That is only implied by the earlier assumption and isn't tied to this step.", + "'Not covered here': the sizing formula \"m = −n·ln p / (ln 2)²\" is stated with no reason for its form. It follows from p ≈ 0.5^k with k = (m/n)·ln 2, but the text doesn't say so." + ], + "excess": [ + "'How it works': \"a hash output of 3,141,593 gives position 3,141,593 mod 16 = 9\". This reader already knows hash functions and mod; the worked mod example adds nothing for them.", + "'Summary' restates the bold intro paragraph and the rate formula with nothing new." + ] + }, + "gaps": [ + "The 13.8% figure at 2M items has no derivation.", + "There is no derivation of why the optimal k gives f = 0.5, or of k = (m/n)·ln 2.", + "The 4.8-bits-per-tenfold claim is not shown.", + "The sizing formula in 'Not covered' is unexplained, though it is explicitly marked out of scope." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2, + "14": 2 + }, + "reasons": {}, + "pass": true + }, + "usage": { + "cost": 0.172995, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same concept appears under several names: \"never-added item\" (step 4, 'How often it is wrong'), \"new items\" / \"new URLs\" (intro, line 8; 'How often', line 56; 'What it is good for'), and \"an item that was never added\" ('false positives'). These are all the same population that p is measured over." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Capacity has two names: \"capacity\" ('Overfilling') and \"sized for\" ('Overfilling'; Q14). The summary also says p is chosen \"with bits per item\" rather than with capacity." + }, + { + "n": 3, + "severity": "gap", + "kind": "unexplained", + "text": "'Overfilling': \"At 2 million items, p is 13.8%\" is given without the working that the 3-million case gets (kn/m = 1.4, f ≈ 0.75)." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'How often it is wrong', final paragraph: \"Thus each extra 4.8 bits per item cuts p by about 10 times\". It doesn't show where 4.8 comes from (0.62^4.8 ≈ 0.1)." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'Not covered here': \"m = −n·ln p / (ln 2)²\" is given with no meaning or example. The reader can't tell what m it gives for, say, p = 1%." + }, + { + "n": 6, + "severity": "gap", + "kind": "missing_why", + "text": "'How often it is wrong': \"The best balance is near f = 0.5, which the formula gives at k = (m/n)·ln 2\". The reader isn't shown why the optimum is f = 0.5, or how setting e^(−kn/m) = 1/2 yields ln 2." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Step 4: \"p ≈ f^k\" relies on the k checked bits being independent, each 1 with probability f. That is only implied by the earlier assumption and isn't tied to this step." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "'Not covered here': the sizing formula \"m = −n·ln p / (ln 2)²\" is stated with no reason for its form. It follows from p ≈ 0.5^k with k = (m/n)·ln 2, but the text doesn't say so." + }, + { + "n": 9, + "severity": "excess", + "kind": "excess", + "text": "'How it works': \"a hash output of 3,141,593 gives position 3,141,593 mod 16 = 9\". This reader already knows hash functions and mod; the worked mod example adds nothing for them." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "'Summary' restates the bold intro paragraph and the rate formula with nothing new." + }, + { + "n": 11, + "severity": "gap", + "kind": "gap", + "text": "The 13.8% figure at 2M items has no derivation." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "There is no derivation of why the optimal k gives f = 0.5, or of k = (m/n)·ln 2." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "The 4.8-bits-per-tenfold claim is not shown." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The sizing formula in 'Not covered' is unexplained, though it is explicitly marked out of scope." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/understanding.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/understanding.md new file mode 100644 index 0000000..c7c0e5b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/explainers/bloom-filter/review/understanding.md @@ -0,0 +1,16 @@ +# Review: bloom-filter (prose, published tier) + +## Probe (2026-10-04T150533) +Adopted 1, 2, 3, 6, 7, 8: worked derivation of p ≈ f^k, toy-vs-formula note, why best k is near half full, use case (filter before a slow lookup), fill and capacity terms, hash → position example. Declined: resize and identical-hash breaking cases (resizing went to Scope), byte-vs-bit slots, and the excess notes. + +## Cold read, narrative +- Run 150725: 4 blocking (beats too terse: k undefined, "all 1 → probably seen" not said, derivation not spoken, close a placeholder). Rewrote every beat's "Said" as full sentences. +- Run 150831: no blocking. Adopted the short edge fixes (name p, (1 − 1/m)^m ≈ 1/e, 6.9 → 7, m and k fixed when overfilling, cut repeats). + +## Cold read, prose (151003) +No blocking. Cut the "hash functions are deterministic" sentence and the estimate note; added reasons for best k ≈ half full and the 4.8-bit rule; reworded capacity. Declined: glosses for the Not-covered pointers (Scope only). + +## Blind test, prose +- Run 151043: FAIL, 13/14 at 2; item 10 (why less memory) scored 1: the answer omitted the price. Fixed: "The price for the smaller memory is error" next to the memory numbers. Also unified the two answers' wording, added p = f for k = 1, and worked the 3× overfill number. +- Run 151208: PASS, all 14 items at 2. +- Remaining audit notes not applied: no derivation of the 2× (13.8%) figure, of the optimum f = 0.5 (needs calculus), or of the sizing formula (Scope pointer). Unified "new items" → "never-added items" after this run. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/read.json new file mode 100644 index 0000000..b91c006 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "When an item is added, its k positions (determined by hash functions) are set to 1. When looking up an item, the same k positions are read: if any bit is 0, the answer is 'definitely not seen'; if all are 1, the answer is 'probably seen'.", + "quote": "each item turns on a few of them. If any of an item's bits is still off, the answer is \"definitely not seen\". If all of them are on, the answer is \"probably seen\"." + }, + "2": { + "answer": "It can give false positives (saying 'probably seen' for items never added), but cannot give false negatives (saying 'definitely not seen' for added items). False positives occur because different items can set the same bits, but false negatives cannot happen because bits never reset to 0.", + "quote": "If all of them are on, the answer is \"probably seen\", because other items can turn on the same bits. So the filter can be wrong in one direction only: it can say \"probably seen\" for an item it never saw." + }, + "3": { + "answer": "Because bits are shared between items. Clearing the bits for one item also clears bits that other items use, causing those other items to incorrectly return 'definitely not seen', creating false negatives.", + "quote": "Suppose you add owl for real, and then try to delete cat by clearing bits 1, 6 and 11. Owl also uses bits 6 and 11, so they are now 0. Bits are shared between items." + }, + "4": { + "answer": "It answers 'probably seen'. When all bits are 1, any query will find all its k positions at 1, so every item (including never-added ones) gets answered as 'probably seen'.", + "quote": "If all bits are 1, the answer is probably seen." + } + }, + "stops": [ + "\"Choose k different hash functions\" – what hash functions are and how they work is not explained", + "\"Take each hash output mod m to get a position\" – the mod operator is not defined", + "\"each hash position is a uniform random choice\" – 'uniform random' is probability terminology not introduced", + "\"The number of hash functions k pulls in two directions\" – metaphor unclear", + "\"k = (m/n)·ln 2\" – formula appears without explanation of origin or derivation", + "\"while n stays near the capacity\" – 'capacity' used without clear definition", + "\"real code makes k positions from two hash values ('double hashing')\" – method mentioned but not explained" + ], + "excess": [ + "\"A hash set of 1 million URLs, at about 50 bytes per URL, takes about 50 MB\" – motivating example, not needed for mechanism", + "The table at lines 22-25 showing bit positions – visualizes what was already described in text", + "\"That answer is right.\" (line 32) – obvious conclusion the reader would draw", + "\"That is about 1 wrong answer in 120 queries\" – restates the same 0.82% figure already given", + "\"Each extra bit per item adds 0.69 to k, so it multiplies p by 0.5^0.69 ≈ 0.62\" – mathematical detail beyond basic understanding", + "The full \"Why 7 hash functions?\" table and discussion – interesting but possibly beyond scope of basic explanation" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/run.json new file mode 100644 index 0000000..408b753 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/bloom-filter/explanation.md`, prose only: the subject is a short add-then-query procedure and one failure case, and a 16-bit table plus a 3-row trade-off table show both without a diagram or interactive page; the blind test passed with all 14 items at 2 after one fix, and the cold read found nothing blocking.", + "error": null, + "cost": 1.4500966000000002, + "seconds": 553, + "turns": 54, + "files": [ + "explainers/bloom-filter/explanation.md" + ], + "review_cost": 0.8529, + "reviews": 6 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/explanation.md new file mode 100644 index 0000000..9d0e6a5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/explanation.md @@ -0,0 +1,67 @@ +# How does a Bloom filter remember what it has seen, and when does it get it wrong? + +**A Bloom filter does not store the items. It stores a row of bits, and each item switches on a few of them. To ask "have I seen this?", you check the item's bits. If any bit is off, the answer is a sure "no". If all are on, the answer is "probably yes", because other items can switch on the same bits.** + +## The parts + +- A **bit array**: m bits, all 0 at the start. +- k **hash functions**. A hash function turns an item into a position from 0 to m − 1. The same item always gives the same position; different items give positions that look random. + +## How it works + + +Take a toy filter with m = 10 bits and k = 2 hash functions. + +1. **Add** an item: compute its k positions and set those bits to 1. "cat" gives positions 1 and 6. "dog" gives positions 3 and 6. +2. **Query** an item: compute its k positions and look at those bits. If any bit is 0, answer "no". If all are 1, answer "probably yes". + +After adding cat and dog, the array is: + +| position | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | +|---|---|---|---|---|---|---|---|---|---|---| +| bit | 0 | **1** | 0 | **1** | 0 | 0 | **1** | 0 | 0 | 0 | + +Query "owl": its positions are 3 and 5. Bit 5 is 0, so owl was never added: a certain "no". + +## Why "no" is always right + + +Adding an item only changes bits from 0 to 1. Nothing changes a bit back to 0. So every item you added still has all its bits set, and it always gets "probably yes". + +This guarantee breaks if you try to delete. Suppose you remove cat by clearing bits 1 and 6. Dog also uses bit 6. Now a query for dog finds bit 6 at 0 and answers "no", which is wrong. For this reason, a plain Bloom filter does not support deletion. + +## How "yes" can be wrong + + +Query "fox": its positions are 1 and 3. Cat set bit 1 and dog set bit 3. Both bits are 1, so the filter answers "probably yes" for an item it never saw. This is a **false positive**. + +The filter cannot tell which item set a bit. As you add more items, more bits become 1, and an unseen item is more likely to find all its bits already on. + +## How much memory, and how often wrong + + +With n items in m bits and k hash functions, the **false-positive rate** p (the fraction of unseen items that get "probably yes") is about: + +p ≈ (1 − e^(−kn/m))^k + +This formula assumes the hash positions are independent and uniform. +Store 1,000,000 web addresses of about 50 bytes each: as a normal set, about 50 MB. A Bloom filter with 9,600,000 bits (1.2 MB, or 9.6 bits per item) and k = 7 gives p ≈ 1%. So 1 in 100 unseen addresses gets a wrong "probably yes". The filter size depends on n and the accepted rate, not on item length. + + +Why several hash functions? Each extra hash makes a false positive need one more lucky bit. But each extra hash also sets more bits to 1. At 9.6 bits per item: + +| k | false-positive rate | +|---|---| +| 1 | 9.9% | +| 7 | 1% | +| 20 | 7% | + +The best k is about (m/n) × ln 2, which is 9.6 × 0.69 ≈ 7 here. + +The rate also depends on how full the filter is. Put 3,000,000 items into the same 1.2 MB filter, and p rises to about 43%. Choose m from the largest n you expect. + +## When to use one + +Use a Bloom filter in front of a slow lookup, such as a disk or a network call. A "no" lets you skip the slow lookup. A "probably yes" sends you to the slow lookup, which gives the true answer. A false positive then costs one wasted lookup, not a wrong result. + +*Not covered: counting Bloom filters (they allow deletion by storing small counters instead of bits) and cuckoo filters.* diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/model.md new file mode 100644 index 0000000..a8f3b01 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/model.md @@ -0,0 +1,78 @@ +# Semantic model: Bloom filter + +## Central question + +How does a Bloom filter answer "have I seen this before?" with little memory, and how can it be wrong? + +## Audience and prior knowledge + +A technical reader: school mathematics (powers, e^x), basic programming (arrays, functions). Knows what a set is. Does not know what a hash function is in detail, or what a Bloom filter is. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| bit array | A row of m bits, all 0 at the start. | fact | +| hash function | A function that turns an item into a position 0..m−1; the same item always gives the same position, different items give positions that look random. | fact (assumed uniform) | +| add | Set the k bits at the item's k positions to 1. | fact | +| query | Check the item's k positions; "no" if any bit is 0, "probably yes" if all are 1. | fact | +| false positive | A "probably yes" for an item that was never added. | fact | + +## Causal chain + +```text +item → k hash functions → k positions → (add: set to 1) / (query: all 1? → probably yes; any 0 → no) +other items' bits overlap → all k positions 1 by chance → false positive +``` + +## Quantities + +- m = number of bits; n = number of items added; k = number of hash functions. +- False-positive rate p ≈ (1 − e^(−kn/m))^k (assumes independent uniform hashes). +- Best k ≈ (m/n)·ln 2. +- n = 1,000,000, m = 9,600,000 bits (1.2 MB), k = 7 → p ≈ 1%. About 9.6 bits per item. +- Same data as a set of 50-byte strings: about 50 MB. +- Same m and n, k = 1 → p ≈ 9.9%. k = 20 → p ≈ 7%. +- Same m, k = 7, n = 3,000,000 → p ≈ 43%. + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| k hashes, not one | k = 1 | at 9.6 bits per item, p ≈ 9.9% instead of 1% | +| not too many hashes | k = 20 | array fills with 1s; p ≈ 7% | +| store bits, not items | a set of the strings | 50 MB instead of 1.2 MB for 1,000,000 items | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| No false negatives | added "dog" → bits 3, 6 are 1 forever → query says probably yes | delete "cat" by clearing bits 1, 6 → bit 6 is 0 → "dog" gets "no" | +| False positives exist | "owl" → bits 3, 5; bit 5 is 0 → no | "fox" → bits 1, 3; both set by cat and dog → probably yes | + +Worked example: m = 10, k = 2. cat → 1, 6. dog → 3, 6. + +## Terms + +- bit array, hash function, position, add, query, false positive, false-positive rate (p: 1% means 1 in 100 unseen items gets "probably yes"; at 43% the filter is close to useless). + +## Epistemic status + +- **Mathematical consequence:** no false negatives (bits only go 0 → 1). +- **Estimate:** the p formula; it assumes independent uniform hashes and is a close approximation. + +## Confusion points + +- "Probably yes" means certain → no; it can be a false positive. +- You can remove items by clearing bits → no; that creates false negatives. +- More hashes are always better → no; there is a best k. + +## Scope + +- **Out of scope:** counting Bloom filters (allow deletion); cuckoo filters; how to build k hashes from two (double hashing). + +## Representation decision + +- **Stage:** 1 +- **Reason:** The mechanism is a short sequence of steps; one 10-bit worked example and one formula carry it. A static table shows the bit array. +- **Levels:** L1 what it answers / L2 add and query / L3 why errors are one-sided / L4 sizing. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/model.yaml new file mode 100644 index 0000000..e164d8e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/explainers/bloom-filter/model.yaml @@ -0,0 +1,70 @@ +values: + p_percent: 1 + bits_per_item: 9.6 + toy: + m: 10 + k: 2 + cat: [1, 6] + dog: [3, 6] + fox: [1, 3] + owl: [3, 5] + real: + n: 1000000 + m_bits: 9600000 + megabytes: 1.2 + k: 7 + p_percent: 1 + bits_per_item: 9.6 + set_megabytes: 50 + string_bytes: 50 + k1_p_percent: 9.9 + k20_p_percent: 7 + k_many: 20 + overfull_n: 3000000 + overfull_p_percent: 43 + ln2: 0.69 + +allow: [0, 1, 2, 3, 4, 8, 9, 100] + +require: + - path: p_percent + in: [prose] + - path: bits_per_item + in: [prose] + - path: overfull_p_percent + in: [prose] + +claims: + - id: mech-add-query + kind: mechanism + statement: "Add sets the item's k bits to 1; query answers no if any of them is 0." + example: "m=10, k=2: cat sets 1 and 6, dog sets 3 and 6; owl checks 3 and 5, bit 5 is 0, so no." + - id: no-false-negatives + kind: guarantee + statement: "An added item always gets 'probably yes', because bits only change from 0 to 1." + example: "dog's bits 3 and 6 stay 1, so dog always gets probably yes." + counterexample: "Delete cat by clearing 1 and 6: bit 6 becomes 0 and dog now gets no." + ask: "Why can you not remove an item by setting its bits back to 0?" + - id: false-positive + kind: limit + statement: "An item never added can find all its bits already set by other items." + example: "fox checks 1 and 3, set by cat and dog, so fox gets probably yes." + ask: "How can a Bloom filter say 'probably yes' for an item it never saw?" + - id: why-k + kind: why + statement: "Several hash functions lower the false-positive rate, up to a best k of about (m/n) ln 2." + alternative: "one hash function, or very many" + counterexample: "At 9.6 bits per item: k=1 gives 9.9%, k=7 gives 1%, k=20 gives 7%." + - id: why-bits + kind: why + statement: "The filter stores bits, not items, so its size does not depend on item length." + alternative: "store the items in a set" + counterexample: "1,000,000 items of 50 bytes: about 50 MB as a set, 1.2 MB as a filter with 1% false positives." + +accept_unjustified: + ln: "ln 2 in the best-k rule comes from minimizing the formula; the k table shows the minimum with numbers." + +terms: + - term: probably yes + means: "the answer when all k bits are 1; it can be a false positive" + avoid: [definitely yes] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/read.json new file mode 100644 index 0000000..0929347 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "When adding an item, the filter computes its k hash positions and sets those bits to 1. When looking up an item, it computes the same k positions and checks those bits; if any bit is 0, the answer is \"no\"; if all are 1, the answer is \"probably yes\".", + "quote": "Add an item: compute its k positions and set those bits to 1. Query an item: compute its k positions and look at those bits. If any bit is 0, answer \"no\". If all are 1, answer \"probably yes\"." + }, + "2": { + "answer": "It can give false positives (saying \"yes\" for items never added) but can never give false negatives (saying \"no\" for items that were added). This is because adding items only changes bits from 0 to 1; nothing changes them back to 0.", + "quote": "Adding an item only changes bits from 0 to 1. Nothing changes a bit back to 0. So every item you added still has all its bits set, and it always gets \"probably yes\"." + }, + "3": { + "answer": "You cannot delete an item by clearing its bits because multiple items share the same bits. If you clear a bit for one deleted item, you may clear a bit that another existing item still needs, causing that item to incorrectly answer \"no\".", + "quote": "Dog also uses bit 6. Now a query for dog finds bit 6 at 0 and answers \"no\", which is wrong. For this reason, a plain Bloom filter does not support deletion." + }, + "4": { + "answer": "It answers \"probably yes\", because every item would find all its bits set to 1, regardless of whether it was actually added.", + "quote": "As you add more items, more bits become 1, and an unseen item is more likely to find all its bits already on." + } + }, + "stops": [ + "Hash functions described as turning items into positions \"that look random\" - what does random mean here?", + "Line 38: \"The filter cannot tell which item set a bit\" - why is it important that the filter cannot distinguish?", + "The false-positive rate formula (line 45) with e^(−kn/m)^k - variables and the exponential are not explained.", + "Line 47: \"assumes the hash positions are independent and uniform\" - what does uniform mean for positions?", + "Why the specific formula for optimal k (line 59) works is not derived or justified.", + "Line 3: \"other items can switch on the same bits\" - this key insight appears before fully explaining its consequences.", + "Transition between theory and the toy example (line 13) could clarify why using 10 bits and 2 functions." + ], + "excess": [ + "The specific web address example with 50 MB and 1.2 MB sizes (lines 48-49) - illustrates but is not needed to understand the mechanism.", + "The table of false-positive rates for different k values (lines 53-57) - the concept is already explained in text.", + "The formula for optimal k as (m/n) × ln 2 (line 59) - detailed optimization beyond basic mechanism.", + "The case of putting 3,000,000 items into the filter (lines 61-62) - illustrative detail not essential.", + "The entire \"When to use one\" section (lines 63-66) - context about application rather than mechanism.", + "The note about counting Bloom filters and cuckoo filters (line 67) - explicitly marked as not covered." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/run.json new file mode 100644 index 0000000..a2067fa --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/bloom-filter/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/bloom-filter/explanation.md`, a prose explanation (plus its `model.md` and `model.yaml`; `explainer check` passes). I chose prose because the mechanism is a short series of steps: one 10-bit worked example, a small table of k values and one formula show the memory saving (1.2 MB instead of about 50 MB, at about 1% wrong answers) and both ways it can be wrong, so a diagram or interactive page would not add enough to justify it.", + "error": null, + "cost": 0.5246917999999999, + "seconds": 99, + "turns": 29, + "files": [ + "explainers/bloom-filter/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/explainers/float-sum/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/explainers/float-sum/explanation.md new file mode 100644 index 0000000..183334e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/explainers/float-sum/explanation.md @@ -0,0 +1,176 @@ +# Why 0.1 + 0.2 ≠ 0.3 + +Type this into Python, JavaScript, Java, C, Ruby, Go, or most other languages: + +``` +0.1 + 0.2 == 0.3 → false +0.1 + 0.2 → 0.30000000000000004 +``` + +This isn't a bug in the language or in your CPU. The computer is doing exactly what it was designed to do. The surprise comes from two facts that work together: + +1. The computer stores numbers in **binary**, and **0.1, 0.2 and 0.3 can't be written exactly in binary**. Each one gets rounded to the nearest number that *can* be stored. +2. Those three small rounding errors don't cancel out. + +The rest of this document explains both facts and then what to do about them. + +--- + +## 1. A familiar version of the problem: 1/3 in decimal + +Before binary, look at something you already know. In decimal, 1/3 has no finite form: + +``` +1/3 = 0.3333333333... (the 3s never end) +``` + +Say you could only keep 4 decimal places. You'd store 1/3 as `0.3333`. Then: + +``` +0.3333 + 0.3333 + 0.3333 = 0.9999 ≠ 1 +``` + +Nothing went wrong with the addition. The error was already there once you rounded 1/3 to fit the space you had. + +Which fractions end and which repeat depends on the **base**. In base 10, a fraction has a finite form only if its denominator (after simplifying) has no prime factors other than 2 and 5, the prime factors of 10. So 1/2, 1/4, 1/5 and 1/8 end, while 1/3, 1/6 and 1/7 repeat forever. + +## 2. The same problem in binary + +Base 2 has only one prime factor: 2. So in binary, a fraction has a finite form **only if its denominator is a power of 2**: 1/2, 1/4, 3/8, 5/16, and so on. + +Now look at our numbers as fractions: + +| Decimal | Fraction | Denominator a power of 2? | +|---|---|---| +| 0.5 | 1/2 | yes, exact in binary | +| 0.25 | 1/4 | yes, exact | +| 0.1 | 1/10 | no (10 = 2 × **5**) | +| 0.2 | 1/5 | no | +| 0.3 | 3/10 | no | + +So 0.1 repeats forever in binary, the same way 1/3 does in decimal: + +``` +0.1 (decimal) = 0.0001100110011001100110011... (binary, "0011" repeats forever) +``` + +You can check this by repeated doubling, which is how you convert a fraction to binary. Double the number. If the result is 1 or more, write a 1 and subtract 1. Otherwise write a 0. + +``` +0.1 × 2 = 0.2 → 0 +0.2 × 2 = 0.4 → 0 +0.4 × 2 = 0.8 → 0 +0.8 × 2 = 1.6 → 1 (keep 0.6) +0.6 × 2 = 1.2 → 1 (keep 0.2) +0.2 × 2 = 0.4 → 0 ← we've been at 0.2 before, so the pattern repeats +... +``` + +The digits come out `0.0001 1001 1001 1001 ...`, and they never stop. + +## 3. How much room the computer has + +Most languages store non-integer numbers in the **IEEE 754 double-precision** format (a "double", or `float` in Python and `number` in JavaScript). It's scientific notation in base 2: + +``` +value = ± (1.xxxxxxxx...) × 2^exponent +``` + +The digits after the point (the *significand*) get a fixed budget: **53 significant binary digits**. That's roughly 15–17 significant decimal digits. Whatever doesn't fit is rounded off, to the *nearest* number that can be stored. + +So "0.1" in your source code is never stored as 0.1. It's stored as the closest double to 0.1. Here are the exact stored values, written out in full decimal. Every double has a finite decimal expansion, because its denominator is a power of 2. + +| You wrote | What is actually stored | Error | +|---|---|---| +| `0.1` | 0.1000000000000000055511151231257827… | a little **too big** | +| `0.2` | 0.2000000000000000111022302462515654… | a little **too big** | +| `0.3` | 0.2999999999999999888977697537484345… | a little **too small** | + +That already explains most of the surprise. The 0.1 and 0.2 you add are both slightly too big. The 0.3 you compare against is slightly too small. They're about to land on different sides of the true 0.3. + +## 4. The addition itself + +The computer adds the two stored values. The exact mathematical sum is: + +``` + 0.1000000000000000055511151231257827… ++ 0.2000000000000000111022302462515654… += 0.3000000000000000166533453693773481… +``` + +That result also has to be stored as a double, so it gets rounded again. Near 0.3, the doubles are spaced about 5.55 × 10⁻¹⁷ apart. The two candidates on either side of our sum are: + +``` +lower: 0.29999999999999998889776975… (this is the double we call "0.3") +sum: 0.30000000000000001665334536… +upper: 0.30000000000000004440892098… +``` + +The sum sits **exactly halfway** between them: 2.775 × 10⁻¹⁷ from each. IEEE 754's tie rule is "round half to even": pick the neighbour whose last binary digit is 0. That's the upper one. So the result is: + +``` +0.1 + 0.2 → 0.3000000000000000444089209850062616… +``` + +That's a *different double* from the one `0.3` gives you. They're neighbours, one step apart, but they aren't equal, so `==` returns `false`. + +> For readers who like exact arithmetic: the double for 0.1 is exactly +> 3602879701896397 × 2⁻⁵⁵, and the double for 0.2 is twice that. Their sum is +> 10808639105689191 × 2⁻⁵⁵. Near 0.3 the stored values are whole multiples of +> 2⁻⁵⁴, and 10808639105689191 / 2 = 5404319552844595.5, an exact tie. Rounding to +> even gives 5404319552844596 × 2⁻⁵⁴. But `0.3` is stored as 5404319552844595 × 2⁻⁵⁴. + +## 5. Why it *prints* as 0.30000000000000004, while 0.1 prints as 0.1 + +If 0.1 isn't stored exactly, why does `print(0.1)` show `0.1`? + +Modern languages print the **shortest decimal string that maps back to the same double**. "0.1" is shorter than "0.1000000000000000055511…", and reading "0.1" back gives exactly the same stored value. So the language shows you "0.1". It hides the error, but nothing is lost. + +For the sum, "0.3" would map back to the *other* double (the lower neighbour). The shortest string that identifies the upper one takes 17 significant digits: `0.30000000000000004`. That trailing `4` isn't noise. It's the shortest way to tell the two doubles apart. + +Seeing the error at all is the exception. Most of the time the rounding stays hidden below the printed digits. For example, `0.1 + 0.7` happens to print `0.7999999999999999`, but `0.1 + 0.4` prints `0.5`, because the rounding happened to land on the double nearest 0.5. + +## 6. Why computers use binary floating point anyway + +- **Speed and hardware:** binary arithmetic is what the circuits do natively. Doubles are added and multiplied in a few nanoseconds by dedicated hardware. +- **Range:** the exponent lets one 64-bit format hold numbers from about 10⁻³⁰⁸ to 10³⁰⁸. +- **Bounded, predictable error:** IEEE 754 guarantees that each basic operation (+, −, ×, ÷, √) gives the exact result *correctly rounded* to the nearest double. The relative error per operation is at most about 1.1 × 10⁻¹⁶. That's far more precision than almost any physical measurement has. + +The cost is the one you've seen: decimal fractions that look simple to us are not simple in base 2. + +## 7. What to do about it + +**Don't compare floats with `==`** when they come out of a calculation. Check whether they are *close enough*: + +```python +import math +math.isclose(0.1 + 0.2, 0.3) # True (default relative tolerance 1e-9) +abs((0.1 + 0.2) - 0.3) < 1e-9 # True, a hand-written version +``` + +What counts as "close enough" depends on your problem. A relative tolerance (a fraction of the numbers' size) usually works better than a fixed absolute one. + +**For money and other exact decimal quantities, don't use binary floats.** Two common options: + +- Store **integers in the smallest unit** (cents rather than dollars): `10 + 20 == 30` is exact. +- Use a **decimal type**, which stores digits in base 10: Python's `decimal.Decimal`, Java's `BigDecimal`, C#'s `decimal`, SQL's `NUMERIC`. + +```python +from decimal import Decimal +Decimal("0.1") + Decimal("0.2") == Decimal("0.3") # True +``` + +(Create the Decimal from the *string* `"0.1"`. `Decimal(0.1)` would faithfully copy the already-rounded binary value.) + +**For exact rational arithmetic**, some languages provide fractions, such as Python's `fractions.Fraction(1, 10)`. They're slower, but never round. + +**Whole numbers are safe.** Every integer up to 2⁵³ (about 9 × 10¹⁵) is stored exactly in a double, so integer arithmetic within that range has no rounding at all. + +--- + +## Summary + +- Binary can only write fractions with power-of-2 denominators exactly. 0.1, 0.2 and 0.3 are tenths and fifths, so they repeat forever in binary, like 1/3 does in decimal. +- A double keeps 53 binary digits, so each of those numbers is rounded to the nearest storable value. 0.1 and 0.2 round up slightly, and 0.3 rounds down slightly. +- Adding the rounded 0.1 and 0.2, then rounding the sum, lands one step above the stored 0.3. The two are different numbers, so `==` is false. +- That's correct IEEE 754 behaviour, not a bug. Compare with a tolerance, or use integers or decimal types when you need exact decimal results. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/read.json new file mode 100644 index 0000000..e215575 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/read.json @@ -0,0 +1,48 @@ +{ + "answers": { + "1": { + "answer": "Because 0.1 equals 1/10, and the denominator 10 contains the factor 5. In binary, only fractions whose denominator is a power of 2 can be stored exactly. Since 5 is not a power of 2, 0.1 repeats forever in binary.", + "quote": "Base 2 has only one prime factor: 2. So in binary, a fraction has a finite form only if its denominator is a power of 2" + }, + "2": { + "answer": "The stored 0.1 and 0.2 are both slightly too large; the stored 0.3 is slightly too small. When the rounded 0.1 and 0.2 are added and the sum is rounded again, it lands on a different double (one step above) than the stored 0.3.", + "quote": "Adding the rounded 0.1 and 0.2, then rounding the sum, lands one step above the stored 0.3." + }, + "3": { + "answer": "Use a tolerance-based comparison instead, checking if the values are close enough rather than exactly equal. Functions like math.isclose() or checking the absolute difference against a small tolerance value should be used.", + "quote": "Don't compare floats with `==` when they come out of a calculation. Check whether they are *close enough*" + }, + "4": { + "answer": "Yes, 0.5 + 0.25 == 0.75 is true. Since 0.5 = 1/2, 0.25 = 1/4, and 0.75 = 3/4 all have power-of-2 denominators, they can be stored exactly in binary with no rounding error.", + "quote": "Binary can only write fractions with power-of-2 denominators exactly." + } + }, + "stops": [ + "IEEE 754 double-precision format—introduced as technical term without explanation", + "53 significant binary digits—why this specific number? What does significant mean?", + "round half to even: pick the neighbour whose last binary digit is 0—rule stated without explanation", + "value = ± (1.xxxxxxxx...) × 2^exponent—the notation and meaning unclear", + "relative tolerance (a fraction of the numbers' size)—term defined but concept may confuse", + "power of 2 denominators—why this property is special not immediately clear", + "2⁻⁵⁴ notation—mathematical notation possibly unclear to reader new to subject", + "exactly halfway between them—why this condition triggers special rounding behavior unexplained" + ], + "excess": [ + "For readers who like exact arithmetic: the double for 0.1 is exactly 3602879701896397 × 2⁻⁵⁵—marked as extra", + "Conversion algorithm: 0.1 × 2 = 0.2 → 0 through 0.2 × 2 = 0.4 → 0—thorough but excessive detail", + "Why computers use binary floating point: Speed, Range, Bounded error—interesting but not needed", + "For example, 0.1 + 0.7 prints as 0.7999999999999999—illustrates already-explained concept", + "Modern languages print the shortest decimal string that maps back—explains print behavior but tangential", + "Seeing the error at all is the exception. Most of the time rounding stays hidden—context but not essential", + "Python's fractions.Fraction(1, 10). They're slower, but never round—alternatives, not needed", + "Every integer up to 2⁵³ is stored exactly in a double—additional note about integers, not central", + "0.1 (decimal) = 0.0001100110011001100110011...(binary)—precise but possibly excessive", + "Create the Decimal from the string 0.1. Decimal(0.1) would faithfully copy—implementation detail" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/run.json new file mode 100644 index 0000000..40a7d09 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I wrote `explainers/float-sum/explanation.md`. Prose works best because the explanation is a step-by-step argument (1/3 in decimal, then 0.1 in binary, then the exact stored values, then the tie-breaking rounding) with small worked tables, and neither a diagram nor an interactive page would add enough to justify a second file.", + "error": null, + "cost": 0.1607036, + "seconds": 49, + "turns": 2, + "files": [ + "explainers/float-sum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/explanation.md new file mode 100644 index 0000000..c56e195 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/explanation.md @@ -0,0 +1,109 @@ +# Why 0.1 + 0.2 is not 0.3 + +In Python, JavaScript, Java, C, and most other languages: + +```python +>>> 0.1 + 0.2 +0.30000000000000004 +>>> 0.1 + 0.2 == 0.3 +False +``` + +**Short answer:** the computer cannot store 0.1, 0.2, or 0.3 exactly. It stores the nearest value it can hold. The sum of the stored 0.1 and the stored 0.2 lands on a different stored value than the stored 0.3. This is not a bug. It is how the standard number format works, and the same thing happens with decimals on paper. + +This matters because equality tests on decimal fractions fail silently, and money calculations drift by fractions of a cent. + +## 1. The same problem in decimal + +Imagine a calculator that keeps only 4 decimal digits. + +- 1/3 is stored as 0.3333 (rounded down). +- 2/3 is stored as 0.6667 (rounded up). +- 1/3 + 1/3 = 0.3333 + 0.3333 = 0.6666. + +So on this calculator, 1/3 + 1/3 ≠ 2/3. Each number was rounded on its own, and the rounding errors do not cancel. Computers have exactly this problem, but in base 2, and 1/10 plays the role of 1/3. + +## 2. Why 0.1 has no exact binary form + +A fraction has a finite expansion in base 10 only if its denominator (in lowest terms) has no prime factors other than 2 and 5. That is why 1/4 = 0.25 ends, but 1/3 repeats. + +In base 2, the only allowed prime factor is 2. The denominator of 1/10 is 2 × 5. The factor 5 makes the binary expansion repeat forever: + +``` +0.1 (decimal) = 0.0001100110011001100110011... (binary, "0011" repeats) +``` + +The computer must cut this off somewhere, so it must round. + +## 3. What the computer actually stores + +Most languages use the **double** format (IEEE 754 binary64), which the processor supports in hardware. A double holds a number as + +``` +(an integer of at most 53 bits) × (a power of 2) +``` + +So the values a double can hold form a grid. Between 0.25 and 0.5, the grid step is 2⁻⁵⁴ ≈ 5.55 × 10⁻¹⁷. Every literal you type is rounded to the nearest grid point: + +| You type | Stored as integer × power of 2 | Exact stored value | Error | +|---|---|---|---| +| `0.1` | 7205759403792794 × 2⁻⁵⁶ | 0.1000000000000000055511151231… | slightly high | +| `0.2` | 7205759403792794 × 2⁻⁵⁵ | 0.2000000000000000111022302462… | slightly high | +| `0.3` | 5404319552844595 × 2⁻⁵⁴ | 0.2999999999999999888977697537… | slightly low | + +Note that the stored 0.2 is exactly twice the stored 0.1. Doubling is exact in binary: only the power of 2 changes. + +The errors already point in different directions. 0.1 and 0.2 are a little too high. 0.3 is a little too low. + +## 4. The addition, step by step + +Write everything in units of 2⁻⁵⁴, the grid step near 0.3: + +- stored 0.1 = 1801439850948198.5 units +- stored 0.2 = 3602879701896397 units +- exact sum = **5404319552844595.5** units + +The exact sum is not a grid point. The processor must round it again. It sits exactly halfway between two grid points: + +``` + stored 0.3 result + 5404319552844595 exact sum 5404319552844596 + ───────────●─────────────────○─────────────────●─────────── + 0.29999999999999998890 0.30000000000000001665 0.30000000000000004441 + ←— half a step —→←— half a step —→ +``` + +For an exact tie, the IEEE 754 rule is **round half to even**: choose the grid point whose integer is even. 5404319552844596 is even, so the result goes up. + +The result is therefore one grid step above the stored 0.3. The two doubles differ, so `==` returns `False`. + +## 5. Why it prints as 0.30000000000000004 + +This step explains a common confusion: if 0.1 is not stored exactly, why does `print(0.1)` show `0.1`? + +Languages print the **shortest decimal string that rounds back to the same double**. For the stored 0.1, the string `0.1` is enough, because no other double is nearer to 0.1. This hides the error. + +For the result of the addition, `0.3` is not enough. The string `0.3` rounds to the other grid point, the stored 0.3. The shortest string that identifies the result is `0.30000000000000004`. The printed form reveals the error only because the error moved the value to a different grid point. + +## 6. When addition is exact + +The problem comes from denominators with factors other than 2. Fractions with power-of-2 denominators are stored exactly, and their sums are exact while they fit in 53 bits: + +```python +>>> 0.5 + 0.25 == 0.75 +True +``` + +Here 1/2, 1/4, and 3/4 are all grid points, so no rounding happens. + +## 7. What to do in code + +- **Compare with a tolerance**, not with `==`: `math.isclose(0.1 + 0.2, 0.3)` in Python, or `abs(a - b) < 1e-9` with a tolerance that suits your data. +- **Use integers for money**: store 10 cents as `10`, not `0.10`. +- **Use a decimal type** when the decimal digits must be exact: `decimal.Decimal("0.1")` in Python, `BigDecimal` in Java. These types store base-10 digits, so 0.1 is exact. They are slower than doubles. + +## Summary + +0.1, 0.2, and 0.3 each have a repeating binary expansion, so each is rounded to the nearest double. The stored 0.1 and 0.2 are slightly high and the stored 0.3 is slightly low. Their exact sum falls halfway between two doubles, and the tie rule rounds it up, one step above the stored 0.3. The values differ, so the comparison is false. + +Not covered: 32-bit floats, special values (NaN, infinity), and how errors grow across long chains of operations. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/model.md new file mode 100644 index 0000000..7a20360 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/model.md @@ -0,0 +1,30 @@ +# Model: why 0.1 + 0.2 != 0.3 + +Reader: technical, school maths, basic programming; new to floating point. + +Central question: why does `0.1 + 0.2 == 0.3` give false? + +Answer in one line: the computer stores each number as the nearest binary fraction, +and the three roundings (0.1, 0.2, and the sum) do not land on the same stored value as 0.3. + +Entities +- double (IEEE 754 binary64): a 53-bit integer significand times a power of two. +- grid: the set of values a double can hold; near 0.3 the spacing is 2^-54. +- rounding: each literal and each operation result goes to the nearest grid point; ties go to the even significand. +- printing: languages print the shortest decimal that maps back to the same double. + +Causal chain +1. 1/10 has a factor 5 in its denominator, so its binary expansion repeats (0.000110011...). +2. Storage rounds 0.1 and 0.2 slightly up; 0.3 is rounded slightly down. +3. The exact sum of the stored 0.1 and 0.2 lies exactly halfway between two grid points. +4. Round-half-to-even picks the upper point, one grid step above stored 0.3. +5. Shortest-round-trip printing shows that point as 0.30000000000000004. + +Analogy: 4-digit decimal, 1/3 + 1/3 -> 0.3333 + 0.3333 = 0.6666, but 2/3 is stored as 0.6667. + +Holds / breaks: 0.5 + 0.25 == 0.75 is exact (denominators are powers of 2). + +Confusion points: "0.1 prints as 0.1, so it is stored exactly" (false: printing hides it); +"it is a bug in the language" (false: it is the hardware number format). + +Scope: no float32, no NaN/infinity, no error analysis of long sums. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/model.yaml new file mode 100644 index 0000000..bad74c2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/explainers/float-sum/model.yaml @@ -0,0 +1,11 @@ +format: IEEE 754 binary64 +significand_bits: 53 +grid_step_near_0_3: "2^-54 ≈ 5.55e-17" +stored: + "0.1": {integer: 7205759403792794, scale: "2^-56", exact: "0.1000000000000000055511151231257827021181583404541015625"} + "0.2": {integer: 7205759403792794, scale: "2^-55", exact: "0.200000000000000011102230246251565404236316680908203125"} + "0.3": {integer: 5404319552844595, scale: "2^-54", exact: "0.299999999999999988897769753748434595763683319091796875"} +exact_sum_of_stored: {integer: 5404319552844595.5, scale: "2^-54", exact: "0.3000000000000000166533453693773481063544750213623046875"} +result: {integer: 5404319552844596, scale: "2^-54", exact: "0.3000000000000000444089209850062616169452667236328125", printed: "0.30000000000000004"} +tie_rule: round half to even +exact_example: "0.5 + 0.25 == 0.75" diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/read.json new file mode 100644 index 0000000..a851b01 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/read.json @@ -0,0 +1,48 @@ +{ + "answers": { + "1": { + "answer": "Because 0.1 has a repeating binary expansion. Its denominator (10 = 2 × 5) contains the factor 5, which is not allowed in base 2 where only factors of 2 can produce finite expansions.", + "quote": "The denominator of 1/10 is 2 × 5. The factor 5 makes the binary expansion repeat forever" + }, + "2": { + "answer": "Because 0.1 and 0.2 are stored slightly too high while 0.3 is stored slightly too low. Their sum lands between two grid points and rounds to a different point than the stored 0.3.", + "quote": "0.1 and 0.2 are a little too high. 0.3 is a little too low." + }, + "3": { + "answer": "Compare with a tolerance instead of testing for exact equality. Use math.isclose() or check if the absolute difference is smaller than an acceptable threshold.", + "quote": "Compare with a tolerance, not with ==: math.isclose(0.1 + 0.2, 0.3) in Python, or abs(a - b) < 1e-9" + }, + "4": { + "answer": "Yes, 0.5 + 0.25 == 0.75 is true in these languages. Because 0.5, 0.25, and 0.75 have only power-of-2 denominators (1/2, 1/4, 3/4), they are stored exactly with no rounding errors.", + "quote": "Here 1/2, 1/4, and 3/4 are all grid points, so no rounding happens." + } + }, + "stops": [ + "IEEE 754 binary64 technical term not defined: 'the **double** format (IEEE 754 binary64)'", + "Why specifically 53 bits unexplained: '(an integer of at most 53 bits) × (a power of 2)'", + "Notation 2⁻⁵⁴ and its formula unexplained: 'Between 0.25 and 0.5, the grid step is 2⁻⁵⁴'", + "Rounding rule 'round half to even' not explained: 'choose the grid point whose integer is even'", + "Mathematical term 'lowest terms' not defined: 'its denominator (in lowest terms) has no prime factors'", + "Phrase 'rounds back' unclear: 'shortest decimal string that rounds back to the same double'", + "Grid concept used but not initially defined: 'So the values a double can hold form a grid'", + "Why power-of-2 denominators are exact not explained: 'Fractions with power-of-2 denominators are stored exactly'", + "How money calculations 'drift' mechanism not shown: 'money calculations drift by fractions of a cent'", + "How computer decides where to cut binary expansion: 'The computer must cut this off somewhere, so it must round'" + ], + "excess": [ + "Language list not needed for understanding: 'In Python, JavaScript, Java, C, and most other languages:'", + "Large binary integers are technical noise: '7205759403792794 × 2⁻⁵⁶'", + "Supporting detail about doubling: 'Note that the stored 0.2 is exactly twice the stored 0.1. Doubling is exact in binary'", + "Decimal analogy illustrative but not essential: 'and the same thing happens with decimals on paper'", + "Supporting example about 1/4 and 1/3: 'That is why 1/4 = 0.25 ends, but 1/3 repeats'", + "Visual diagram redundant with text: 'stored 0.3 ... exact sum ... result [with spacing diagram]'", + "Meta-note about excluded topics: 'Not covered: 32-bit floats, special values (NaN, infinity)'", + "Calculator analogy numerical details: '1/3 + 1/3 = 0.3333 + 0.3333 = 0.6666'" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/run.json new file mode 100644 index 0000000..942bfb3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/float-sum/explanation.md`, along with `model.md` and `model.yaml` to hold the facts and numbers it relies on. I chose prose because the cause is one short chain: round each number to binary, add, round once more. A worked table and one small number-line sketch carry the numbers, so a diagram or interactive page would add little. I checked the binary arithmetic by hand, but I couldn't run `explainer check` because I have no shell here.", + "error": null, + "cost": 0.2104636, + "seconds": 55, + "turns": 4, + "files": [ + "explainers/float-sum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/explanation.md new file mode 100644 index 0000000..eaa51b2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/explanation.md @@ -0,0 +1,67 @@ +# Why does 0.1 + 0.2 not equal 0.3? + +In Python, JavaScript, and most other languages, `0.1 + 0.2 == 0.3` is `false`, and `0.1 + 0.2` prints `0.30000000000000004`. + +**The computer never holds 0.1, 0.2, or 0.3. It holds the nearest numbers it can store in binary. The addition is correct, but its result rounds to the next storable number above the one that 0.3 rounds to.** + +## What the computer holds + +A literal such as `0.1` becomes a **double**: the 64-bit binary format that most languages use for numbers with a decimal point. A double is a whole number of at most 53 binary digits, times a power of two. The other 11 bits hold the sign and the power. For example, 0.75 = 3 × 2^-2, so a double holds 0.75 exactly. + +## Why 0.1 must be rounded + +A whole number times a power of two is a fraction whose denominator is a power of two. A fraction like that ends in binary: 3/8 = 0.011. The denominator of 1/10 has a factor 5. No power of two is divisible by 5, so 1/10 never ends in binary: + + 1/10 = 0.000110011001100110011… (0011 repeats forever) + +This is the same as 1/3 = 0.3333… in decimal. A machine that keeps only a fixed number of digits must cut the expansion, so it rounds. With 4 decimal digits, 1/3 becomes 0.3333, and 0.3333 + 0.3333 = 0.6666, but 2/3 becomes 0.6667. The same thing happens with 0.1, 0.2, and 0.3 in binary. + +## The stored values + +The **stored value** of a literal is the double nearest to it. The **gap** is the distance between two neighboring doubles. + +The gap depends on the size of the number. Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54. The ends check: 2^52 × 2^-54 = 0.25, and 2^53 × 2^-54 = 0.5. Neighboring whole numbers differ by 1, so neighboring doubles here differ by u = 2^-54 ≈ 5.55e-17. Between 0.125 and 0.25 the multiplier is 2^-55, so the gap is u/2. Between 0.0625 and 0.125 the gap is u/4. + +To see the rounding, count each number in units of u. For example, 0.3 × 2^54 = 5404319552844595.2, so 0.3 is 5404319552844595.2 u. The nearest doubles are at 5404319552844595 u and 5404319552844596 u. The nearer one is …595 (short for 5404319552844595), so 0.3 rounds down. 0.1 is in the range where the gap is u/4, so its stored value is a multiple of u/4, such as …198.5. + +The **error** is the stored value minus the decimal: 0.1 and 0.2 round up, and 0.3 rounds down. + +| Number | Exact, in u | Gap here | Stored, in u | Stored, in decimal | Error | +|---|---|---|---|---|---| +| 0.1 | 1801439850948198.4 | u/4 | 1801439850948198.5 | 0.10000000000000000555 | +5.55e-18 | +| 0.2 | 3602879701896396.8 | u/2 | 3602879701896397 | 0.20000000000000001110 | +1.11e-17 | +| 0.3 | 5404319552844595.2 | u | 5404319552844595 | 0.29999999999999998890 | −1.11e-17 | +## What `+` does + +The `+` operation first finds the **exact sum** of the two stored values. Then it rounds that exact sum to a double, as every arithmetic operation does. The result is the **rounded sum**. + + exact sum = 1801439850948198.5 + 3602879701896397 = 5404319552844595.5 u + +The exact sum is exactly halfway between the doubles at …595 u and …596 u. This case is a **tie**. The IEEE 754 standard, which sets these rules, sends a tie to the neighbor whose count of u is even. 5404319552844596 is even, so the rounded sum is 5404319552844596 u, which is 0.30000000000000004441. + +Stored 0.3 is at …595 u. The rounded sum and stored 0.3 are neighboring doubles, one gap (5.55e-17) apart. They are different numbers, so `==` is false. + +The addition did not make a mistake. The rounded sum is the nearest double to the exact sum of its inputs. The inputs were not 0.1 and 0.2. + +## Why `print(0.1)` shows `0.1` + + +Languages print the shortest decimal that rounds back to the same double. The text "0.1" rounds back to stored 0.1, so one digit is enough. The text "0.3" rounds back to …595 u, not to the rounded sum at …596 u. The shortest decimal that rounds back to …596 u is 0.30000000000000004, with 17 significant digits. + +## When it works + + +`0.5 + 0.25 == 0.75` is true. All three are fractions with power-of-two denominators (1/2, 1/4, 3/4), so each is stored with error 0, and the exact sum is itself a double. The failure needs a decimal whose denominator has a factor 5, such as 0.1. + +A decimal type also avoids the problem. Python's `Decimal("0.1") + Decimal("0.2") == Decimal("0.3")` is true, because `Decimal` stores base-10 digits, and 1/10 ends in base 10. + +## What to do + + +Do not compare doubles with `==` when they come from decimal literals or from arithmetic. Compare with a tolerance instead: `abs((0.1 + 0.2) - 0.3) < 1e-9` is true, because the rounded sum and stored 0.3 are one gap (5.55e-17) apart. Choose a tolerance much larger than the gap, but smaller than any difference that matters to you; for values near 1, 1e-9 meets both conditions. For money, count whole cents as integers, or use a decimal type. + +## Not covered here + +- The sign and exponent bits of a double: see IEEE 754, or Goldberg, "What Every Computer Scientist Should Know About Floating-Point Arithmetic" (1991). +- How errors grow over many operations, such as adding 0.1 ten times: see a numerical analysis text. +- How to choose between an absolute and a relative tolerance: see the documentation for Python's `math.isclose`. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/model.md new file mode 100644 index 0000000..db99cce --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/model.md @@ -0,0 +1,130 @@ +# Semantic model: why 0.1 + 0.2 != 0.3 in floating point + +> Source of truth. Every rendering derives from this file. + +## Central question + +Why does `0.1 + 0.2 == 0.3` evaluate to false in most programming languages? + +## Audience and prior knowledge + +A technical reader: school mathematics and science, basic programming. Knows decimals, fractions, powers, that computers store numbers in binary, and what `==` does. Does not know how a float is stored, which fractions end in binary, what rounding to a fixed number of bits does, or how languages print floats. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| double | The 64-bit binary floating-point format (IEEE 754 binary64) that most languages use for a number with a decimal point. | fact | +| significant bits | A double keeps 53 binary digits: every double is a whole number below 2^53 times a power of two. | fact | +| gap | The distance between two neighboring doubles. Because the 53 bits are fixed, the gap halves each time the size of the number halves. | fact | +| stored value | The double nearest to the decimal you wrote; what the program holds for `0.1`. | fact | +| error | stored value − written decimal; + means the stored value is above the decimal. | definition | +| exact sum | The true sum of two stored values, before rounding. | definition | +| rounded sum | The exact sum after rounding to the nearest double; this is the result of `+`. | fact | +| rounding | Replacing an exact number with the nearest double. A tie (exactly halfway) goes to the neighbor whose whole number is even. | fact | +| printed value | The shortest decimal that rounds back to the same double; what `print` shows. | fact (Python, JavaScript, and most modern languages) | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| decimal 0.1 | is rounded to | stored value | 1/10 does not end in binary | +| stored 0.1, stored 0.2 | add to | exact sum | | +| exact sum | is rounded to | rounded sum | every arithmetic operation rounds its result | +| rounded sum | is one gap above | stored 0.3 | so `==` is false | +| printed value | hides | the error of stored 0.1 | | + +## Dependencies (learning order) + +1. A fixed number of digits cannot hold every fraction (decimal: 1/3 → 0.3333). +2. A fraction ends in binary only if its denominator (in lowest terms) is a power of two; 1/10 has a factor 5. +3. So 0.1, 0.2, 0.3 each become the nearest double, with their own error. +4. Measured in the gap near 0.3, the exact sum falls at a half; it rounds up, while 0.3 rounded down. +5. Printing shows the shortest decimal that picks out the double, so 0.1 looks exact and the sum does not. + +## Causal chain + +```text +1/10 has a factor 5 in its denominator → no finite binary expansion +→ 0.1 and 0.2 round up, 0.3 rounds down (errors +5.55e-18, +1.11e-17, −1.11e-17) +→ exact sum = stored 0.3 + half a gap → tie → rounds to the even neighbor, the double above +→ rounded sum and stored 0.3 are neighboring, different doubles → == is false +``` + +## Quantities + +Unit u = 2^-54 ≈ 5.55e-17, the gap between doubles from 0.25 to 0.5. From 0.125 to 0.25 the gap is u/2; from 0.0625 to 0.125 it is u/4. + +| Number | Exact, in u | Allowed steps | Stored, in u | Stored, in decimal | Error | +|---|---|---|---|---|---| +| 0.1 | 1801439850948198.4 | u/4 | 1801439850948198.5 | 0.10000000000000000555 | +5.55e-18 | +| 0.2 | 3602879701896396.8 | u/2 | 3602879701896397 | 0.20000000000000001110 | +1.11e-17 | +| exact sum | 5404319552844595.5 | — | — | 0.30000000000000001665 | +1.67e-17 | +| rounded sum | — | u | 5404319552844596 | 0.30000000000000004441 | +4.44e-17 | +| 0.3 | 5404319552844595.2 | u | 5404319552844595 | 0.29999999999999998890 | −1.11e-17 | + +- Check: 0.3 × 2^54 = 5404319552844595.2; 0.1 × 2^56 = 7205759403792793.6 → 7205759403792794 quarter-units = 1801439850948198.5 u. +- Exact sum: 1801439850948198.5 + 3602879701896397 = 5404319552844595.5. A tie between …595 and …596; …596 is even, so the rounded sum is …596. (Round half up would also give …596.) +- Difference rounded sum − stored 0.3 = 1 u ≈ 5.55e-17. Relative: 5.55e-17 / 0.3 ≈ 1.9e-16, so the first 16 significant digits agree and the difference shows in the 17th: 0.3000000000000000|4. +- Printing: "0.1" rounds to stored 0.1, so one digit is enough. For the rounded sum, "0.3" and every shorter decimal round to stored 0.3, the wrong double; the first that works is 0.30000000000000004 (17 significant digits). +- Binary of 1/10: 0.000110011001100110011… (0011 repeats). +- Gap meaning: near 0.3 the gap is 5.55e-17 (fine); near 10^16 it is 2, so 10^16 + 1 cannot be stored. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| 0.5 + 0.25 == 0.75 | true | 1/2, 1/4, 3/4 have power-of-two denominators: errors 0, exact sum is a double | +| decimal type (Python `Decimal("0.1")`) | 0.1 + 0.2 == 0.3 is true | digits stored in base 10, where 1/10 ends | +| 4-digit decimal calculator | 1/3 → 0.3333; 0.3333 + 0.3333 = 0.6666, but 2/3 → 0.6667 | the same effect in base 10 with 1/3 | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Fixed 53-bit binary storage | Store exact decimal digits | Exact storage of every fraction needs unbounded memory (1/3 has no finite decimal either); fixed-size binary is what the processor adds in one step. Decimal types exist and are slower. | +| Compare with a tolerance: abs(a − b) < 1e-9 | `==` | `0.1 + 0.2 == 0.3` is false though the difference is 5.55e-17. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here (numbers) | +|---|---|---| +| A sum of stored values equals the stored value of the decimal sum | 0.5 + 0.25 == 0.75: all errors 0 | 0.1 + 0.2: rounded sum …596 u, stored 0.3 …595 u | +| Only power-of-two denominators end in binary | 3/8 = 0.011 in binary | 1/10: 2^k / 10 is never whole, because no power of 2 is divisible by 5 | +| Fixed digits lose information (decimal analogy) | 1/4 = 0.25 | 1/3 → 0.3333; 0.3333 + 0.3333 = 0.6666 ≠ 0.6667 | + +## Terms + +- **double**: the 64-bit binary format for numbers with a decimal point. +- **gap**: the distance between neighboring doubles; u = 5.55e-17 near 0.3. Small gap = fine resolution; near 10^16 the gap is 2. +- **stored value**: the double the program holds for a literal like `0.1`. Never call it "0.1" alone. +- **error**: stored value − written decimal. +- **exact sum** vs **rounded sum**: before and after rounding. Never "the sum" alone. +- **tie**: an exact number exactly halfway between two doubles. +- **printed value**: the shortest decimal that rounds back to the same double. + +## Epistemic status + +- **Established:** IEEE 754 binary64, round half to even, the stored values above. +- **Mathematical consequence:** 1/10 has no finite binary expansion. +- **Observed:** Python and JavaScript print `0.30000000000000004` for `0.1 + 0.2`. + +## Confusion points + +- "The addition is buggy" → the addition is correct to the last bit; the inputs were never exactly 0.1 and 0.2. +- "0.1 is stored exactly, because it prints as 0.1" → printing shows the shortest decimal that picks out the double. +- "The result is random noise" → it is deterministic: the same on every IEEE 754 machine. +- "Floats are always inexact" → 0.5 + 0.25 == 0.75 is exact. + +## Scope + +- **Out of scope:** sign and exponent bits — IEEE 754; Goldberg, "What Every Computer Scientist Should Know About Floating-Point Arithmetic" (1991). +- **Out of scope:** error growth in long sums (adding 0.1 ten times) — numerical analysis texts. +- **Out of scope:** how to choose a tolerance (relative vs absolute) — `math.isclose` documentation. +- **Out of scope:** languages whose literals are not doubles by default (Raku uses exact fractions) and fused or extended-precision hardware. + +## Representation decision + +- **Stage:** 1 (prose) +- **Reason:** one sequential causal chain with worked numbers; nothing for the reader to explore or to see move. A table carries the numbers. +- **Levels:** L1 the answer / L2 stored values and the rounded sum / L3 why 1/10 does not end in binary / L4 bit layout (Scope). diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/model.yaml new file mode 100644 index 0000000..8cb1cf5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/model.yaml @@ -0,0 +1,99 @@ +# Machine-readable half of model.md. `explainer check float-sum` holds every rendering to it. + +values: + bits: 53 + printed_sum: 0.30000000000000004 + gap_u: 5.55e-17 + stored: + p1: 0.10000000000000000555 + p2: 0.20000000000000001110 + p3: 0.29999999999999998890 + exact_sum: 0.30000000000000001665 + rounded_sum: 0.30000000000000004441 + units: + p1_exact: 1801439850948198.4 + p1_stored: 1801439850948198.5 + p2_exact: 3602879701896396.8 + p2_stored: 3602879701896397 + p3_exact: 5404319552844595.2 + p3_stored: 5404319552844595 + exact_sum: 5404319552844595.5 + rounded_sum: 5404319552844596 + errors: + p1: 5.55e-18 + p2: 1.11e-17 + p3: -1.11e-17 + binary_tenth: 0.000110011001100110011 + exponent_u: -54 + sum_exact_u: 5404319552844595.5 + sum_rounded_u: 5404319552844596 + p3_stored_u: 5404319552844595 + +allow: ["0011", -2, 52, 54, -55, 198.5, 595, 596, 15, 0.25, 0.5, 0.75, 0.125, 0.0625, 1, 2, 3, 4, 5, 8, 10, 16, 17, 64, 754, 0.3333, 0.6666, 0.6667, 1e-9, 0.011] + +require: + - path: printed_sum + in: [prose] + - path: sum_exact_u + in: [prose] + - path: sum_rounded_u + in: [prose] + - path: p3_stored_u + in: [prose] + - path: gap_u + in: [prose] + +claims: + - id: tenth-not-binary + kind: mechanism + statement: "A fraction ends in binary only if its denominator is a power of two; 1/10 has a factor 5, so 0.1 has no finite binary expansion." + example: "3/8 = 0.011 in binary ends; 1/10 = 0.000110011… repeats, because 2^k / 10 is never a whole number." + ask: "Why can a double hold 0.75 exactly but not 0.1?" + - id: stored-values + kind: mechanism + statement: "Each literal becomes the nearest double; 0.1 and 0.2 round up, 0.3 rounds down." + example: "In units of u = 2^-54: 0.1 = 1801439850948198.4 → .5; 0.2 = 3602879701896396.8 → 3602879701896397; 0.3 = 5404319552844595.2 → 5404319552844595." + - id: sum-rounds-up + kind: mechanism + statement: "The exact sum of the stored values lands exactly halfway between two doubles and rounds to the upper one, one gap above stored 0.3." + example: "1801439850948198.5 + 3602879701896397 = 5404319552844595.5 → tie → 5404319552844596 (even); stored 0.3 is 5404319552844595." + ask: "The stored 0.1 and stored 0.2 are each very close to 0.1 and 0.2. Why is their sum not the same double as stored 0.3?" + - id: exact-when-binary + kind: guarantee + statement: "When every number is a binary fraction, float addition is exact and == behaves as in school arithmetic." + example: "0.5 + 0.25 == 0.75 is true." + counterexample: "0.1 + 0.2 == 0.3 is false: 1/10 is not a binary fraction." + - id: printing-hides + kind: misconception + correction: "The stored value of 0.1 is 0.10000000000000000555; printing hides the error." + statement: "print(0.1) shows 0.1 because languages print the shortest decimal that rounds back to the same double, not the exact stored value." + example: "\"0.1\" picks out stored 0.1, so one digit suffices; for the rounded sum, \"0.3\" picks out the wrong double, so 17 digits are needed: 0.30000000000000004." + ask: "If 0.1 is not stored exactly, why does print(0.1) show 0.1?" + - id: why-tolerance + kind: why + statement: "Compare floats with a tolerance, or use a decimal type, instead of ==." + alternative: "Compare with ==" + counterexample: "0.1 + 0.2 == 0.3 is false although the difference is only 5.55e-17; abs((0.1 + 0.2) - 0.3) < 1e-9 is true." + +accept_unjustified: {} + +terms: + - term: stored value + means: "the double held for a literal; never plain '0.1 is stored'" + avoid: [stored exactly as 0.1] + - term: rounded sum + means: "the result of +, after rounding; 'exact sum' is before rounding" + avoid: [the true result of +] + +budget: {prose: 900, reason: "the mechanism needs a five-row number table, the binary argument, and the printing step"} + +quiz: + - q: "Does 0.5 + 0.25 == 0.75 evaluate to true or false with doubles, and why?" + expect: "True: 1/2, 1/4, 3/4 have power-of-two denominators, so all are stored exactly and the sum is exact." + misconception: "False, because floats are always inexact." + - q: "Is the + operation itself making an error that the hardware could avoid? Where does the difference come from?" + expect: "No; + gives the nearest double to the exact sum of its inputs. The inputs were already not 0.1 and 0.2 (1/10 does not end in binary), and their small errors push the exact sum to a point that rounds to a different double than 0.3 does." + misconception: "The addition circuit is imprecise / it is a bug." + - q: "Would 0.1 + 0.2 == 0.3 be true using Python's Decimal type with Decimal('0.1') etc.? Why?" + expect: "Yes: Decimal stores base-10 digits, where 1/10 ends, so no rounding occurs." + misconception: "No, computers can never represent 0.1." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/narrative.md new file mode 100644 index 0000000..4024d15 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/narrative.md @@ -0,0 +1,53 @@ +# Narrative: float-sum + +## 1. Reader before and after + +- **Before:** A technical reader: school mathematics, basic programming. Knows binary integers, decimals, fractions, powers, and `==`. Has seen `0.1 + 0.2` print `0.30000000000000004` or heard of it. Does not know how a float is stored. +- **After:** "0.1 and 0.2 cannot be stored exactly in binary, so the computer adds two nearby numbers; their sum rounds to a different number than 0.3 does." + +## 2. Question and motive + +- **Question:** Why is `0.1 + 0.2 == 0.3` false? Result in words: the computer never holds 0.1, 0.2, or 0.3. It holds the nearest binary numbers. The addition is correct, but the nearest number to the sum is one step above the nearest number to 0.3. +- **Why care:** `==` on decimals silently fails in prices, loops, and tests. +- **Why this approach:** ask what number the computer actually holds for `0.1`; everything follows from that. + +## 3. Introduction ledger + +| Reference | Means | Grounded by | Beat | +|---|---|---|---| +| double | the 64-bit binary format for decimal-point numbers | `x = 0.1` in Python or JavaScript | 2 | +| 53 bits | a double is a whole number below 2^53 times a power of two | 0.75 = 3 × 2^-2 | 2 | +| binary fraction / ends in binary | denominator is a power of two | 3/8 = 0.011 | 3 | +| stored value | the nearest double to the literal | stored 0.1 = 0.10000000000000000555 | 4 | +| error | stored value − written decimal | +5.55e-18 for 0.1 | 4 | +| gap, u | distance between neighboring doubles; u = 2^-54 near 0.3 | 0.3 sits between …595 u and …596 u | 4 | +| exact sum / rounded sum | before / after rounding the result of + | …595.5 u / …596 u | 5 | +| tie, round half to even | exactly halfway → the even whole number | …595.5 → …596 | 5 | +| printed value | shortest decimal that rounds back to the same double | "0.1"; "0.30000000000000004" | 6 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | Why is it false? | — | `0.1 + 0.2 == 0.3` → false; prints 0.30000000000000004 | "The computer never holds 0.1, 0.2, or 0.3. It holds the nearest binary numbers. The addition is correct, but the sum rounds to the number one step above the one that 0.3 rounds to." | the result in words | +| 2 | instance | What does the computer hold for `0.1`? | so | 0.75 = 3 × 2^-2 | "In Python, JavaScript, and most languages, a literal like 0.1 becomes a double: a whole number below 2^53 times a power of two. The whole number has at most 53 binary digits." | the format | +| 3 | mechanism | Can 0.1 be written that way? | but | 1/3 = 0.3333… in decimal; 3/8 = 0.011; 1/10 = 0.000110011… in binary | "A whole number times a power of two is a fraction whose denominator is a power of two. 1/10 has a factor 5 in its denominator, so it repeats forever in binary, as 1/3 does in decimal. The computer must round it." | 0.1 must be rounded | +| 4 | mechanism | How far off is it, and which way? | therefore | table: exact and stored values counted in u; errors | "Neighboring doubles are one gap apart. Between 0.25 and 0.5 the gap is u = 2^-54 ≈ 5.55e-17; for numbers half as big the gap halves (u/2 for 0.2, u/4 for 0.1). Count each number in u: 0.3 is 5404319552844595.2 u, between doubles at …595 and …596; nearest is …595, so 0.3 rounds down. 0.1 and 0.2 round up. The error is stored value minus decimal." | the three stored values | +| 5 | mechanism | Then what does + do? | so | 1801439850948198.5 + 3602879701896397 = 5404319552844595.5 → 5404319552844596 | "+ first finds the exact sum of the two stored values, then rounds it to a double, like every arithmetic operation. The exact sum is exactly halfway between …595 and …596: a tie. The rule sends ties to the even whole number, …596. Stored 0.3 is …595, one gap below." | why == is false | +| 6 | objection | Why does print(0.1) show 0.1? | but | "0.1" vs "0.30000000000000004" | "Printing shows the shortest decimal that rounds back to the same double. '0.1' rounds back to stored 0.1. '0.3' rounds back to …595, not …596, so the rounded sum needs 17 digits." | printing hides the error | +| 7 | test | Does it always fail? | so | 0.5 + 0.25 == 0.75 → true; Python Decimal('0.1') + Decimal('0.2') == Decimal('0.3') → true | "Doubles are exact when every denominator is a power of two. Decimal types store base-10 digits, where 1/10 ends." | when it holds | +| 8 | close | What do I do? | therefore | `abs(a - b) < 1e-9`; money in whole cents | "Compare with a tolerance much larger than the gap but smaller than any difference you care about. For money, count whole cents as integers, or use a decimal type." | the fix | + +## 5. Concrete to symbol + +u is introduced as "the gap near 0.3, 2^-54 ≈ 5.55e-17" with the instance 0.3 × 2^54 = 5404319552844595.2. + +## 6. Links between representations + +Each row of the table pairs the value in u with the decimal value; the prose names the row. + +## 7. Close the loop + +- **Answer:** The computer adds the nearest doubles to 0.1 and 0.2, not 0.1 and 0.2. Their exact sum rounds to the double above 0.3, while 0.3 itself rounded to the double below. +- **General argument:** any decimal whose denominator has a factor 5 is rounded; sums of rounded numbers can round to a different neighbor than the rounded decimal sum. +- **Payoff:** predict when `==` is safe; use tolerance or decimal types. Scope pointers. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/decisions.yaml new file mode 100644 index 0000000..ff9a8b8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/decisions.yaml @@ -0,0 +1,189 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T151909-probe#1: + decision: adopted +2026-10-04T151909-probe#10: + decision: adopted +2026-10-04T151909-probe#11: + decision: adopted +2026-10-04T151909-probe#12: + decision: adopted +2026-10-04T151909-probe#13: + decision: adopted +2026-10-04T151909-probe#14: + decision: adopted +2026-10-04T151909-probe#15: + decision: adopted +2026-10-04T151909-probe#2: + decision: adopted +2026-10-04T151909-probe#3: + decision: adopted +2026-10-04T151909-probe#4: + decision: adopted +2026-10-04T151909-probe#5: + decision: declined +2026-10-04T151909-probe#6: + decision: adopted +2026-10-04T151909-probe#7: + decision: declined +2026-10-04T151909-probe#8: + decision: declined +2026-10-04T151909-probe#9: + decision: adopted +2026-10-04T152128-coldread-narrative#1: + decision: adopted +2026-10-04T152128-coldread-narrative#10: + decision: adopted +2026-10-04T152128-coldread-narrative#11: + decision: adopted +2026-10-04T152128-coldread-narrative#12: + decision: adopted +2026-10-04T152128-coldread-narrative#13: + decision: adopted +2026-10-04T152128-coldread-narrative#14: + decision: adopted +2026-10-04T152128-coldread-narrative#15: + decision: adopted +2026-10-04T152128-coldread-narrative#16: + decision: adopted +2026-10-04T152128-coldread-narrative#17: + decision: adopted +2026-10-04T152128-coldread-narrative#18: + decision: adopted +2026-10-04T152128-coldread-narrative#19: + decision: adopted +2026-10-04T152128-coldread-narrative#2: + decision: adopted +2026-10-04T152128-coldread-narrative#20: + decision: adopted +2026-10-04T152128-coldread-narrative#21: + decision: adopted +2026-10-04T152128-coldread-narrative#22: + decision: adopted +2026-10-04T152128-coldread-narrative#23: + decision: adopted +2026-10-04T152128-coldread-narrative#24: + decision: declined +2026-10-04T152128-coldread-narrative#25: + decision: declined +2026-10-04T152128-coldread-narrative#26: + decision: adopted +2026-10-04T152128-coldread-narrative#27: + decision: adopted +2026-10-04T152128-coldread-narrative#28: + decision: adopted +2026-10-04T152128-coldread-narrative#29: + decision: adopted +2026-10-04T152128-coldread-narrative#3: + decision: adopted +2026-10-04T152128-coldread-narrative#30: + decision: adopted +2026-10-04T152128-coldread-narrative#31: + decision: declined +2026-10-04T152128-coldread-narrative#32: + decision: adopted +2026-10-04T152128-coldread-narrative#33: + decision: adopted +2026-10-04T152128-coldread-narrative#34: + decision: adopted +2026-10-04T152128-coldread-narrative#35: + decision: adopted +2026-10-04T152128-coldread-narrative#36: + decision: adopted +2026-10-04T152128-coldread-narrative#37: + decision: adopted +2026-10-04T152128-coldread-narrative#38: + decision: adopted +2026-10-04T152128-coldread-narrative#39: + decision: adopted +2026-10-04T152128-coldread-narrative#4: + decision: adopted +2026-10-04T152128-coldread-narrative#40: + decision: adopted +2026-10-04T152128-coldread-narrative#5: + decision: adopted +2026-10-04T152128-coldread-narrative#6: + decision: adopted +2026-10-04T152128-coldread-narrative#7: + decision: adopted +2026-10-04T152128-coldread-narrative#8: + decision: adopted +2026-10-04T152128-coldread-narrative#9: + decision: adopted +2026-10-04T152332-coldread-prose#1: + decision: adopted +2026-10-04T152332-coldread-prose#10: + decision: declined +2026-10-04T152332-coldread-prose#11: + decision: adopted +2026-10-04T152332-coldread-prose#12: + decision: adopted +2026-10-04T152332-coldread-prose#13: + decision: adopted +2026-10-04T152332-coldread-prose#14: + decision: adopted +2026-10-04T152332-coldread-prose#2: + decision: adopted +2026-10-04T152332-coldread-prose#3: + decision: adopted +2026-10-04T152332-coldread-prose#4: + decision: adopted +2026-10-04T152332-coldread-prose#5: + decision: adopted +2026-10-04T152332-coldread-prose#6: + decision: adopted +2026-10-04T152332-coldread-prose#7: + decision: adopted +2026-10-04T152332-coldread-prose#8: + decision: adopted +2026-10-04T152332-coldread-prose#9: + decision: adopted +2026-10-04T152417-quiz-prose#1: + decision: adopted +2026-10-04T152417-quiz-prose#10: + decision: adopted +2026-10-04T152417-quiz-prose#11: + decision: declined +2026-10-04T152417-quiz-prose#12: + decision: adopted +2026-10-04T152417-quiz-prose#13: + decision: declined +2026-10-04T152417-quiz-prose#14: + decision: adopted +2026-10-04T152417-quiz-prose#15: + decision: adopted +2026-10-04T152417-quiz-prose#16: + decision: adopted +2026-10-04T152417-quiz-prose#17: + decision: adopted +2026-10-04T152417-quiz-prose#2: + decision: adopted +2026-10-04T152417-quiz-prose#3: + decision: adopted +2026-10-04T152417-quiz-prose#4: + decision: declined +2026-10-04T152417-quiz-prose#5: + decision: adopted +2026-10-04T152417-quiz-prose#6: + decision: adopted +2026-10-04T152417-quiz-prose#7: + decision: adopted +2026-10-04T152417-quiz-prose#8: + decision: adopted +2026-10-04T152417-quiz-prose#9: + decision: adopted +2026-10-04T152559-coldread-prose#1: + decision: adopted +2026-10-04T152559-coldread-prose#2: + decision: adopted +2026-10-04T152559-coldread-prose#3: + decision: adopted +2026-10-04T152559-coldread-prose#4: + decision: declined +2026-10-04T152559-coldread-prose#5: + decision: adopted +2026-10-04T152559-coldread-prose#6: + decision: adopted +2026-10-04T152559-coldread-prose#7: + decision: adopted +2026-10-04T152559-coldread-prose#8: + decision: adopted diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T151909-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T151909-probe.json new file mode 100644 index 0000000..71f2a93 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T151909-probe.json @@ -0,0 +1,218 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "float-sum", + "at": "2026-10-04T15:19:09", + "model": "default", + "prompt_sha": "22368ea1ea35", + "reply": { + "gaps": [ + { + "rule": 4, + "severity": "main", + "about": "The model gives the result of rounding 1/10 to 53 bits but never works through how the rounding is done.", + "why_it_matters": "The causal chain says '0.1, 0.2, 0.3 rounded to 53 bits' but never shows that step. Without it the long decimal expansions look like magic, and the reader cannot check where the +5.55e-18 error comes from.", + "suggested_addition": "Write each value as a whole number times a power of two. 0.1 × 2^55 = 3602879701896396.8, which rounds up to 3602879701896397, so stored 0.1 = 3602879701896397 / 2^55 with error +0.2/2^55 ≈ +5.55e-18. Stored 0.2 = 3602879701896397 / 2^54 (error +1.11e-17). 0.3 × 2^54 = 5404319552844595.2, which rounds down to 5404319552844595, so the error is −0.2/2^54 ≈ −1.11e-17." + }, + { + "rule": 2, + "severity": "main", + "about": "The claim 'the errors do not cancel to the error of 0.3' (Dependency 4) leaves its key step implicit: why 0.1 and 0.2 round up while 0.3 rounds down.", + "why_it_matters": "This sign difference is the whole reason the sum and stored 0.3 end up on different doubles. The file lists the signed errors but never adds them up or explains the signs.", + "suggested_addition": "Add the errors explicitly: +5.55e-18 + 1.11e-17 = +1.665e-17, while stored 0.3 has error −1.11e-17. The fractional parts .8 (0.1) and .2 (0.3) from the scaling above explain the signs. Contrast: for 0.5 + 0.25 all three errors are 0." + }, + { + "rule": 2, + "severity": "main", + "about": "The causal chain says 'exact sum slightly above 0.3 → rounded to the double above 0.3', which hides the tie.", + "why_it_matters": "Being above 0.3 does not decide the result. The exact sum 0.30000000000000001665 is exactly halfway between the two neighboring doubles, and only the tie rule picks the upper one. As written, the chain names the wrong reason.", + "suggested_addition": "Rewrite the chain step as: exact sum = lower neighbor + 2.776e-17 = exactly half a gap (5.55e-17 / 2) → tie → round half to even picks 0.30000000000000004440 (the significand ending ...334, which is even, over ...333)." + }, + { + "rule": 4, + "severity": "main", + "about": "The printed value ('shortest decimal that rounds back') has no worked example.", + "why_it_matters": "Confusion point 2 depends on it. The reader needs to see why stored 0.1 prints as '0.1' but the sum needs 17 digits.", + "suggested_addition": "'0.1' rounds to the stored 0.1, so 1 digit is enough. For the sum, '0.3' rounds to 0.29999999999999998889 (the wrong double), and so does every shorter candidate up to 0.3000000000000000. The first decimal that maps to 0.30000000000000004440 is 0.30000000000000004 (17 significant digits)." + }, + { + "rule": 4, + "severity": "edge", + "about": "The binary expansion of 1/10 is stated, not derived.", + "why_it_matters": "The audience does not know what a binary fraction is. A repeated-doubling example shows why the block 0011 repeats.", + "suggested_addition": "Double and take the integer part: 0.1→0.2 (0), 0.4 (0), 0.8 (0), 1.6 (1), 1.2 (1), 0.4 (0) ... The value 0.4 has come back, so the digits 0011 repeat forever: 0.0001100110011..." + }, + { + "rule": 2, + "severity": "edge", + "about": "The claim 'only power-of-two denominators end in binary' has no instance where it holds and no argument.", + "why_it_matters": "It is the mathematical core of the explanation, and the reason (the factor 5) appears only in the causal chain.", + "suggested_addition": "Holds: 3/8 = 0.011 in binary (3 × 2^3 / 8 is a whole number). Breaks: 1/10 × 2^k = 2^k/10 = 2^(k−1)/5 is never a whole number, because no power of 2 is divisible by 5." + }, + { + "rule": 1, + "severity": "edge", + "about": "The truncation row in 'Why this form' says 'with numbers' but gives none, and round half to even has no stated alternative.", + "why_it_matters": "The table promises numbers. The reader may also ask whether the tie rule is what causes the result.", + "suggested_addition": "Truncation: stored 0.1 would be 0.09999999999999999167. The sum still differs from stored 0.3 (it truncates to about 0.29999999999999993339), so truncation does not fix it. Round half up would also pick 0.30000000000000004 here. Half-to-even exists to avoid an upward bias across many ties, not to cause this result." + }, + { + "rule": 2, + "severity": "edge", + "about": "'Whole numbers below 2^53 are exact' and 'same result on every IEEE 754 machine' have no instance where the assumption is removed.", + "why_it_matters": "Each guarantee needs a breaking case.", + "suggested_addition": "2^53 + 1 = 9007199254740993 is stored as 9007199254740992. Determinism can break with x87 80-bit intermediates or with compiler fast-math/FMA contraction." + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts have more than one name, and some terms are used without definition.", + "why_it_matters": "In the file 'sum' means the exact sum, the rounded sum, and 'a double'. 'Significant bits', '53 bits' and 'fixed number of digits' name one idea. 'Stored value' is also called 'the double'. 'Error' and 'relative error' are used but never defined. 'Significant bits' and 'tie' are missing from Terms.", + "suggested_addition": "Define error = stored value − written decimal (for example +5.55e-18 for 0.1). Fix the names 'exact sum' (0.30000000000000001665) and 'rounded sum' (0.30000000000000004440). Add 'significant bits (53)' and 'tie' to Terms." + }, + { + "rule": 5, + "severity": "edge", + "about": "The signed errors and the relative error 1.9e-16 are not interpreted for the reader.", + "why_it_matters": "The reader should know what a + or − sign means, and that the difference only shows up around the 17th significant digit.", + "suggested_addition": "A + sign means the stored value is above the decimal. 1.9e-16 relative means the first 15 digits agree: 0.300000000000000 | 04, so you only see a difference in about the 17th digit." + }, + { + "rule": 6, + "severity": "edge", + "about": "The file says 'most programming languages' and lists Java under printed value without qualification.", + "why_it_matters": "The reader will ask which languages differ. Java's Double.toString was not always shortest before JDK 19.", + "suggested_addition": "Note that languages with default decimal or rational literals (for example Raku: 0.1 + 0.2 == 0.3 is True) differ. Qualify the Java status." + } + ], + "excess": [ + "Hex bit patterns (0x3FB999999999999A etc.) belong to L4 and are out of scope for this audience.", + "The full 55-digit exact expansions could be shortened to about 20 significant digits.", + "The list of decimal types in three languages (one example is enough).", + "Mention of Kahan summation." + ], + "next_questions": [ + { + "q": "How exactly does 0.1 become 0.1000000000000000055511...? What are the 53 bits?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why do 0.1 and 0.2 round up while 0.3 rounds down?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If 0.1 isn't stored exactly, why does print(0.1) show 0.1?", + "answered_by_model": true, + "declared_out_of_scope": false + }, + { + "q": "How should I compare floats then, and what tolerance should I use?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Does adding 0.1 ten times give exactly 1.0, and do errors grow in long sums?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.1529652, + "seconds": 57 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 4", + "text": "The model gives the result of rounding 1/10 to 53 bits but never works through how the rounding is done. → Write each value as a whole number times a power of two. 0.1 × 2^55 = 3602879701896396.8, which rounds up to 3602879701896397, so stored 0.1 = 3602879701896397 / 2^55 with error +0.2/2^55 ≈ +5.55e-18. Stored 0.2 = 3602879701896397 / 2^54 (error +1.11e-17). 0.3 × 2^54 = 5404319552844595.2, which rounds down to 5404319552844595, so the error is −0.2/2^54 ≈ −1.11e-17." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "The claim 'the errors do not cancel to the error of 0.3' (Dependency 4) leaves its key step implicit: why 0.1 and 0.2 round up while 0.3 rounds down. → Add the errors explicitly: +5.55e-18 + 1.11e-17 = +1.665e-17, while stored 0.3 has error −1.11e-17. The fractional parts .8 (0.1) and .2 (0.3) from the scaling above explain the signs. Contrast: for 0.5 + 0.25 all three errors are 0." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 2", + "text": "The causal chain says 'exact sum slightly above 0.3 → rounded to the double above 0.3', which hides the tie. → Rewrite the chain step as: exact sum = lower neighbor + 2.776e-17 = exactly half a gap (5.55e-17 / 2) → tie → round half to even picks 0.30000000000000004440 (the significand ending ...334, which is even, over ...333)." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 4", + "text": "The printed value ('shortest decimal that rounds back') has no worked example. → '0.1' rounds to the stored 0.1, so 1 digit is enough. For the sum, '0.3' rounds to 0.29999999999999998889 (the wrong double), and so does every shorter candidate up to 0.3000000000000000. The first decimal that maps to 0.30000000000000004440 is 0.30000000000000004 (17 significant digits)." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 4", + "text": "The binary expansion of 1/10 is stated, not derived. → Double and take the integer part: 0.1→0.2 (0), 0.4 (0), 0.8 (0), 1.6 (1), 1.2 (1), 0.4 (0) ... The value 0.4 has come back, so the digits 0011 repeat forever: 0.0001100110011..." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 2", + "text": "The claim 'only power-of-two denominators end in binary' has no instance where it holds and no argument. → Holds: 3/8 = 0.011 in binary (3 × 2^3 / 8 is a whole number). Breaks: 1/10 × 2^k = 2^k/10 = 2^(k−1)/5 is never a whole number, because no power of 2 is divisible by 5." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 1", + "text": "The truncation row in 'Why this form' says 'with numbers' but gives none, and round half to even has no stated alternative. → Truncation: stored 0.1 would be 0.09999999999999999167. The sum still differs from stored 0.3 (it truncates to about 0.29999999999999993339), so truncation does not fix it. Round half up would also pick 0.30000000000000004 here. Half-to-even exists to avoid an upward bias across many ties, not to cause this result." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 2", + "text": "'Whole numbers below 2^53 are exact' and 'same result on every IEEE 754 machine' have no instance where the assumption is removed. → 2^53 + 1 = 9007199254740993 is stored as 9007199254740992. Determinism can break with x87 80-bit intermediates or with compiler fast-math/FMA contraction." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts have more than one name, and some terms are used without definition. → Define error = stored value − written decimal (for example +5.55e-18 for 0.1). Fix the names 'exact sum' (0.30000000000000001665) and 'rounded sum' (0.30000000000000004440). Add 'significant bits (53)' and 'tie' to Terms." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 5", + "text": "The signed errors and the relative error 1.9e-16 are not interpreted for the reader. → A + sign means the stored value is above the decimal. 1.9e-16 relative means the first 15 digits agree: 0.300000000000000 | 04, so you only see a difference in about the 17th digit." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 6", + "text": "The file says 'most programming languages' and lists Java under printed value without qualification. → Note that languages with default decimal or rational literals (for example Raku: 0.1 + 0.2 == 0.3 is True) differ. Qualify the Java status." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "Hex bit patterns (0x3FB999999999999A etc.) belong to L4 and are out of scope for this audience." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "The full 55-digit exact expansions could be shortened to about 20 significant digits." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "The list of decimal types in three languages (one example is enough)." + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Mention of Kahan summation." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152128-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152128-coldread-narrative.json new file mode 100644 index 0000000..984e99e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152128-coldread-narrative.json @@ -0,0 +1,414 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "float-sum", + "at": "2026-10-04T15:21:28", + "model": "default", + "prompt_sha": "cce588df23ed", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: `0.1 + 0.2 == 0.3` → false; prints 0.30000000000000004. Said: The answer in three sentences.", + "unresolved": [ + "The answer in three sentences." + ], + "unsaid": [], + "leap": [ + "The 'Said' cell describes the narration instead of giving it. The reader never hears the three sentences, so the promised result is not delivered here." + ], + "excess": [ + "prints 0.30000000000000004: the reader has already seen or heard of this. It works as a hook, but it tells them nothing new." + ], + "purpose_unclear": false + }, + { + "at": "Beat 2 (instance)", + "text": "Shown: 0.75 = 3 × 2^-2. Said: A double is a whole number (53 bits) times a power of two.", + "unresolved": [ + "double", + "(53 bits)" + ], + "unsaid": [ + "The example does not connect 0.75 to 'double'. The words do not say that 3 is the 'whole number' and 2^-2 is the 'power of two'." + ], + "leap": [ + "The beat does not say why a float is stored this way, or that a double is what the computer holds for `0.1`. 'Double' is never tied to the code in Beat 1.", + "The beat does not say what the 53-bit limit means in practice: that the whole number must be below 2^53." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Shown: 1/3 = 0.3333… in decimal; 1/10 = 0.000110011… in binary. Said: Only power-of-two denominators end; 10 has a factor 5.", + "unresolved": [ + "end (meaning: the binary expansion terminates)" + ], + "unsaid": [ + "The beat never says aloud that 1/10 repeats forever in binary, the way 1/3 does in decimal. The analogy is shown but not spoken." + ], + "leap": [ + "The beat does not explicitly link a non-ending binary fraction to 'cannot be a whole number times a power of two, so it must be rounded'. The words never say the conclusion, 'must be rounded'." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Shown: table: exact and stored values in u; errors. Said: 0.1 and 0.2 round up, 0.3 rounds down.", + "unresolved": [ + "u", + "stored values", + "errors (sign convention)" + ], + "unsaid": [ + "The table's unit u is never defined in the words: not what it is, and not that it equals 2^-54.", + "The table's columns (exact, stored, error) are not walked through in the words.", + "The words never say what 'round up/down' is measured against: the nearest double." + ], + "leap": [ + "The beat does not say why the stored value is the nearest one, or that neighboring doubles are spaced one gap apart.", + "The beat does not say why 0.1 rounds up and 0.3 rounds down. The reader has to take it from the table." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: …198.5 + …397 = …595.5 → …596. Said: Exact sum is a tie; it rounds to the even neighbor, one gap above stored 0.3.", + "unresolved": [ + "…198.5 / …397 / …595.5 / …596 (the truncated digits and their units)", + "tie", + "even neighbor" + ], + "unsaid": [ + "The words do not say that these numbers are counts of u, or that …595 is stored 0.3.", + "The beat does not say why stored 0.1 can be a half-u (…198.5) when doubles were described as whole numbers. The explanation never says that the gap is smaller near 0.1." + ], + "leap": [ + "The beat does not give a reason for the rule that ties go to the even neighbor.", + "The beat does not explain why the sum is rounded at all: that the result of + is itself rounded to a double." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (objection)", + "text": "Shown: \"0.1\" vs \"0.30000000000000004\". Said: Printing shows the shortest decimal that picks out the double.", + "unresolved": [ + "picks out the double" + ], + "unsaid": [], + "leap": [ + "The beat does not explain why 0.30000000000000004 is the shortest decimal for …596 u. The reader needs to know that '0.3' would pick out …595 instead, and the beat does not say so." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (test)", + "text": "Shown: 0.5 + 0.25 == 0.75 → true; Decimal('0.1') + Decimal('0.2') == Decimal('0.3') → true. Said: Exact when denominators are powers of two, or when the base is 10.", + "unresolved": [ + "Decimal('0.1')", + "the base is 10" + ], + "unsaid": [ + "The beat never says that Decimal is a type that stores numbers in base 10." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (close)", + "text": "Shown: `abs(a - b) < 1e-9`; integer cents; Decimal. Said: Compare with a tolerance, or avoid binary for decimal money.", + "unresolved": [ + "tolerance" + ], + "unsaid": [ + "The beat does not say why the threshold is 1e-9.", + "The words do not explain 'integer cents'." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 1: 'The answer in three sentences.' The beat describes the narration instead of giving it, so the reader never actually receives the result.", + "Beat 4: 'u' is used in the table but never defined in the words. Everything numeric after this depends on it.", + "Beat 5: '…198.5 + …397 = …595.5 → …596'. The truncated numbers in u are not explained, and the half-u for 0.1 contradicts 'whole number times a power of two' from Beat 2.", + "Beat 5: 'tie' and 'even neighbor' are not introduced. This is the step that decides the result.", + "Beat 2: 'double' is never tied to what Python or JavaScript stores for `0.1`." + ], + "cuts": [ + "Beat 1's re-display of 0.30000000000000004 could be shortened, since the reader already knows it. This is a minor saving.", + "Beat 8: the three fixes could be reduced to one sentence. The integer-cents detail is not needed for the argument." + ], + "question": { + "text": "Why does `0.1 + 0.2 == 0.3` evaluate to false?", + "known_at": "Beat 1", + "result_stated": "Not in words. Beat 1 says only 'The answer in three sentences.' Pieces show up later: Beat 4 says '0.1 and 0.2 round up, 0.3 rounds down', and Beat 5 says 'one gap above stored 0.3'. The result is never stated as a whole." + }, + "motive": "Reason to care: missing from the beats. Beat 8's 'avoid binary for decimal money' hints at it only at the very end. Reason for the approach: missing. Beat 2's question, 'What does the computer hold?', appears only in the Reader's-question column, not in what is Said.", + "close": "No. Beat 8 gives fixes ('Compare with a tolerance, or avoid binary for decimal money') but does not restate the answer to the opening question. The general argument, that any decimal whose denominator has a factor 5 gets rounded and that sums of rounded numbers can land on a different neighbor, is never said in the beats.", + "recap": "0.1, 0.2 and 0.3 can't be stored exactly in binary, so the computer adds two slightly-high approximations, and the result rounds to the double just above the one it uses for 0.3.", + "strongest": "Beat 3: the parallel between 1/3 = 0.3333… in decimal and 1/10 = 0.000110011… in binary, plus '10 has a factor 5'. It is clear and fits what this reader already knows.", + "weakest": "Beat 5: the elided '…' numbers in an undefined unit u, a half-unit value, and an unexplained tie rule all arrive in the single step that delivers the answer.", + "suggestions": [ + "Write the actual three sentences in Beat 1's Said column.", + "Define u in Beat 4's words: the gap between neighboring doubles near 0.3, equal to 2^-54.", + "Explain that the gap is smaller near 0.1, which is why it can sit on a half-u, or express each value in its own gap.", + "Say in Beat 5 that + rounds its exact result to the nearest double, and give a one-line reason for round-half-to-even.", + "Add a sentence early on about why to care, such as prices, loops, or tests.", + "End Beat 8 by restating the answer and the general rule about factors of 5 before giving the fixes." + ] + }, + "usage": { + "cost": 0.0989208, + "seconds": 33 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 1: 'The answer in three sentences.' The beat describes the narration instead of giving it, so the reader never actually receives the result." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'u' is used in the table but never defined in the words. Everything numeric after this depends on it." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: '…198.5 + …397 = …595.5 → …596'. The truncated numbers in u are not explained, and the half-u for 0.1 contradicts 'whole number times a power of two' from Beat 2." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: 'tie' and 'even neighbor' are not introduced. This is the step that decides the result." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'double' is never tied to what Python or JavaScript stores for `0.1`." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 1's re-display of 0.30000000000000004 could be shortened, since the reader already knows it. This is a minor saving." + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 8: the three fixes could be reduced to one sentence. The integer-cents detail is not needed for the argument." + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): The answer in three sentences." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): The 'Said' cell describes the narration instead of giving it. The reader never hears the three sentences, so the promised result is not delivered here." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): double" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): (53 bits)" + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): The example does not connect 0.75 to 'double'. The words do not say that 3 is the 'whole number' and 2^-2 is the 'power of two'." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (instance): The beat does not say why a float is stored this way, or that a double is what the computer holds for `0.1`. 'Double' is never tied to the code in Beat 1." + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (instance): The beat does not say what the 53-bit limit means in practice: that the whole number must be below 2^53." + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): end (meaning: the binary expansion terminates)" + }, + { + "n": 16, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): The beat never says aloud that 1/10 repeats forever in binary, the way 1/3 does in decimal. The analogy is shown but not spoken." + }, + { + "n": 17, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): The beat does not explicitly link a non-ending binary fraction to 'cannot be a whole number times a power of two, so it must be rounded'. The words never say the conclusion, 'must be rounded'." + }, + { + "n": 18, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): u" + }, + { + "n": 19, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): stored values" + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): errors (sign convention)" + }, + { + "n": 21, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The table's unit u is never defined in the words: not what it is, and not that it equals 2^-54." + }, + { + "n": 22, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The table's columns (exact, stored, error) are not walked through in the words." + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The words never say what 'round up/down' is measured against: the nearest double." + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): The beat does not say why the stored value is the nearest one, or that neighboring doubles are spaced one gap apart." + }, + { + "n": 25, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): The beat does not say why 0.1 rounds up and 0.3 rounds down. The reader has to take it from the table." + }, + { + "n": 26, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): …198.5 / …397 / …595.5 / …596 (the truncated digits and their units)" + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): tie" + }, + { + "n": 28, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): even neighbor" + }, + { + "n": 29, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The words do not say that these numbers are counts of u, or that …595 is stored 0.3." + }, + { + "n": 30, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The beat does not say why stored 0.1 can be a half-u (…198.5) when doubles were described as whole numbers. The explanation never says that the gap is smaller near 0.1." + }, + { + "n": 31, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): The beat does not give a reason for the rule that ties go to the even neighbor." + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): The beat does not explain why the sum is rounded at all: that the result of + is itself rounded to a double." + }, + { + "n": 33, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (objection): picks out the double" + }, + { + "n": 34, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (objection): The beat does not explain why 0.30000000000000004 is the shortest decimal for …596 u. The reader needs to know that '0.3' would pick out …595 instead, and the beat does not say so." + }, + { + "n": 35, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): Decimal('0.1')" + }, + { + "n": 36, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): the base is 10" + }, + { + "n": 37, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (test): The beat never says that Decimal is a type that stores numbers in base 10." + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): tolerance" + }, + { + "n": 39, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (close): The beat does not say why the threshold is 1e-9." + }, + { + "n": 40, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (close): The words do not explain 'integer cents'." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152332-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152332-coldread-prose.json new file mode 100644 index 0000000..6f298bc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152332-coldread-prose.json @@ -0,0 +1,308 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "float-sum", + "at": "2026-10-04T15:23:32", + "model": "default", + "prompt_sha": "5a19de7d0d5c", + "reply": { + "paragraphs": [ + { + "at": "L3 (opening)", + "text": "In Python, JavaScript, and most other languages, `0.1 + 0.2 == 0.3` is `false`...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L5 (bold summary)", + "text": "The computer never holds 0.1, 0.2, or 0.3...", + "unresolved": [ + "one step above" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L9 What the computer holds", + "text": "A literal such as `0.1` becomes a **double**...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L13 Why 0.1 must be rounded", + "text": "A whole number times a power of two is a fraction whose denominator is a power of two...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L15 code block", + "text": "1/10 = 0.000110011001100110011…", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L17", + "text": "This is the same as 1/3 = 0.3333… in decimal...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L21 The stored values", + "text": "The **stored value** of a literal is the double nearest to it. The **gap**...", + "unresolved": [], + "leap": [ + "Between 0.25 and 0.5 the gap is u = 2^-54: no reason is given for this number. The reader has to work out that 53 digits starting at 2^-2 end at 2^-54.", + "\"Because a double has a fixed 53 binary digits\" conflicts with \"at most 53 binary digits\" in L9, and it isn't explained why a fixed digit count makes the gap shrink." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L23", + "text": "To see the rounding, count each number in units of u...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L25 table", + "text": "| Number | Exact, in u | Steps allowed here | ...", + "unresolved": [ + "Steps allowed here", + "Error" + ], + "leap": [ + "The table never says why 0.1 gets u/4 and 0.2 gets u/2. The reader has to match 0.1 to the range 0.0625–0.125 and 0.2 to the range 0.125–0.25 from L21." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L31", + "text": "The error is the stored value minus the decimal. 0.1 and 0.2 round up...", + "unresolved": [], + "leap": [], + "excess": [ + "0.1 and 0.2 round up (positive error). 0.3 rounds down (negative error)." + ], + "purpose_unclear": false + }, + { + "at": "L35 What + does", + "text": "The `+` operation first finds the **exact sum**...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L37 code block", + "text": "exact sum = 1801439850948198.5 + 3602879701896397 = 5404319552844595.5 u", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L39", + "text": "The exact sum is exactly halfway between …595 and …596. This case is a **tie**...", + "unresolved": [ + "The rule" + ], + "leap": [ + "It never says that …596 is the even one, so the reader can't see why the tie goes up here.", + "\"even whole number\": a whole number of what? The text doesn't say the count of u is what must be even." + ], + "excess": [ + "so ties do not always push results upward" + ], + "purpose_unclear": false + }, + { + "at": "L41", + "text": "The stored value of 0.3 is 5404319552844595 u...", + "unresolved": [], + "leap": [], + "excess": [ + "The stored value of 0.3 is 5404319552844595 u." + ], + "purpose_unclear": false + }, + { + "at": "L43", + "text": "The addition did not make a mistake...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L48 Why print(0.1) shows 0.1", + "text": "Languages print the shortest decimal that rounds back to the same double...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L53 When it works", + "text": "`0.5 + 0.25 == 0.75` is true...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L55", + "text": "A decimal type also avoids the problem...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L60 What to do", + "text": "Do not compare doubles with `==`...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L64-66 Not covered here", + "text": "- The sign and exponent bits of a double...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "L39: \"The rule sends a tie to the even whole number\". This is the turning point of the argument, but the text names no rule and never says …596 is the even choice. The reader can't confirm why the sum rounds up.", + "L21: \"Between 0.25 and 0.5 the gap is u = 2^-54\". Every later number is counted in u, but the text gives no reason for this value." + ], + "cuts": [ + "L39: \"so ties do not always push results upward\". It is an aside that pulls attention away from the case being explained.", + "L31: \"0.1 and 0.2 round up (positive error). 0.3 rounds down (negative error).\" The Error column already shows this; keep only the definition of error.", + "L41: \"The stored value of 0.3 is 5404319552844595 u.\" This repeats the table." + ], + "question": { + "text": "Why is 0.1 + 0.2 == 0.3 false, and why does it print 0.30000000000000004?", + "known_at": "Title and L3", + "result_stated": "Yes. L5: \"The computer never holds 0.1, 0.2, or 0.3. It holds the nearest binary numbers to them. The addition is correct, but the result rounds to the number one step above the number that 0.3 rounds to.\"" + }, + "motive": "Reason to care: it is implied by L3 (a surprising `false`) and made concrete in L60, \"Do not compare doubles with `==`...\". Reason for the approach: L23, \"To see the rounding, count each number in units of u.\" No sentence says why this method was chosen over others.", + "close": "Yes. L43: \"The addition did not make a mistake. The result is the nearest double to the exact sum of its inputs. The inputs were not 0.1 and 0.2.\" L53 then states the general condition: \"The failure needs a decimal whose denominator has a factor 5.\"", + "recap": "0.1 and 0.2 get stored slightly too high and 0.3 slightly too low, so their sum lands on the neighboring double just above stored 0.3.", + "strongest": "L17: the 4-digit 1/3 example (0.3333 + 0.3333 = 0.6666, but 2/3 becomes 0.6667) shows the same failure in decimal, where the reader can check it.", + "weakest": "L39: the tie paragraph leaves the even-rounding rule unexplained at the exact step where the result is decided.", + "suggestions": [ + "In L39, say: \"Ties go to the neighbor whose count of u is even; …596 is even, so the sum rounds up.\"", + "In L21, give the reason: numbers from 0.25 to 0.5 are 53-digit whole numbers times 2^-54, so neighbors are 2^-54 apart. Halving the range halves the gap.", + "Add \"in the range\" next to the u/4 and u/2 values in the table, or link each row to its range from L21.", + "Make L9's \"at most 53\" and L21's \"fixed 53\" agree." + ] + }, + "usage": { + "cost": 0.10868280000000001, + "seconds": 34 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "L39: \"The rule sends a tie to the even whole number\". This is the turning point of the argument, but the text names no rule and never says …596 is the even choice. The reader can't confirm why the sum rounds up." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "L21: \"Between 0.25 and 0.5 the gap is u = 2^-54\". Every later number is counted in u, but the text gives no reason for this value." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "L39: \"so ties do not always push results upward\". It is an aside that pulls attention away from the case being explained." + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "L31: \"0.1 and 0.2 round up (positive error). 0.3 rounds down (negative error).\" The Error column already shows this; keep only the definition of error." + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "L41: \"The stored value of 0.3 is 5404319552844595 u.\" This repeats the table." + }, + { + "n": 6, + "severity": "edge", + "kind": "unresolved", + "text": "L5 (bold summary): one step above" + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "L21 The stored values: Between 0.25 and 0.5 the gap is u = 2^-54: no reason is given for this number. The reader has to work out that 53 digits starting at 2^-2 end at 2^-54." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "L21 The stored values: \"Because a double has a fixed 53 binary digits\" conflicts with \"at most 53 binary digits\" in L9, and it isn't explained why a fixed digit count makes the gap shrink." + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "L25 table: Steps allowed here" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "L25 table: Error" + }, + { + "n": 11, + "severity": "edge", + "kind": "leap", + "text": "L25 table: The table never says why 0.1 gets u/4 and 0.2 gets u/2. The reader has to match 0.1 to the range 0.0625–0.125 and 0.2 to the range 0.125–0.25 from L21." + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "L39: The rule" + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "L39: It never says that …596 is the even one, so the reader can't see why the tie goes up here." + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "L39: \"even whole number\": a whole number of what? The text doesn't say the count of u is what must be even." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152417-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152417-quiz-prose.json new file mode 100644 index 0000000..0bf498b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152417-quiz-prose.json @@ -0,0 +1,195 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "float-sum", + "at": "2026-10-04T15:24:05", + "model": "default", + "prompt_sha": "008e69251529", + "reply": { + "answers": { + "1": "The main objects are: decimal literals (0.1, 0.2, 0.3); the double, a 64-bit binary format that holds a whole number of at most 53 binary digits times a power of two; the stored value, which is the double nearest a literal; the gap, which is the distance between neighboring doubles (u = 2^-54 between 0.25 and 0.5); the error, which is the stored value minus the decimal; the + operation, giving an exact sum and then a rounded sum; the tie and the round-half-to-even rule; `==`; printing as the shortest decimal that rounds back to the double; and the Decimal type.", + "2": "Each literal is rounded to its stored value, a double on a grid. The grid's gap depends on the size of the number. `+` adds the stored values exactly, then rounds the result onto the grid. The rule for ties decides which neighbor it lands on. `==` compares the two doubles bit for bit. Printing shows the shortest decimal that rounds back to that double, which hides small errors unless two doubles need to be told apart.", + "3": "0.1 and 0.2 cannot end in binary because 10 has a factor of 5, so both are stored slightly too high, while 0.3 is stored slightly too low. Their exact sum falls halfway between two doubles, the tie rounds up to the double just above stored 0.3, and so `==` returns false.", + "4": "The key variable is whether the decimal's denominator contains a factor of 5, meaning it is not a power-of-two fraction. If every value is a power-of-two fraction (0.5, 0.25, 0.75), the stored error is zero, the exact sum is already a double, and `==` is true. A second variable is the number of digits kept. With more digits the gap gets smaller, but a repeating binary fraction still has to be cut. The text implies this but does not discuss changing the digit count.", + "5": "Assumption or simplification: the text ignores the sign and exponent bits and treats a double as 'a whole number of at most 53 binary digits times a power of two'. Established fact: 0.1 + 0.2 prints 0.30000000000000004, and in units of u the stored 0.3 is 5404319552844595 while the sum rounds to …596.", + "6": "A double can only hold a binary fraction with at most 53 significant bits. 1/10 has a factor 5 in its denominator, so its binary expansion never ends and it has to be rounded, much as 1/3 is in decimal. Stored 0.1 and 0.2 are each slightly above their true values, and stored 0.3 is slightly below. Adding the stored 0.1 and 0.2 exactly gives a value halfway between two doubles, and the round-to-even rule picks the one just above stored 0.3, so the two differ by one gap and `==` is false. Printing shows the shortest decimal that identifies a double, which is why 0.1 prints as 0.1 but the sum needs 17 digits.", + "7": "The hardest part was the table and the units of u. Fractional values in u, such as 1801439850948198.5, are confusing until you link them to 'steps allowed here = u/4'. The text also never says why the gap is exactly 2^-54 in [0.25, 0.5), which is 2^-2 × 2^-52 from the 53 digits. In the tie step, it never says outright that …596 is the even one. One worked line on the gap, and the sentence '…596 is even, …595 is odd', would have made it easier.", + "8": "It is true. 1/2, 1/4 and 3/4 have power-of-two denominators, so each is stored with zero error, and the exact sum is itself a double, so rounding changes nothing.", + "9": "No. The `+` returns the nearest double to the exact sum of its inputs, which is correct. The difference comes from the inputs, which were already rounded versions of 0.1 and 0.2, and from the fact that 0.3 was rounded independently (downward). The final rounding of the sum, a tie broken to even, is the only rounding the operation does, and it follows the rule.", + "10": "Yes, it would be true. Decimal stores base-10 digits, and 1/10 ends in base 10, so no rounding happens.", + "11": "0.75 = 3 × 2^-2 is a whole number times a power of two, so its denominator is a power of two. 1/10 has a factor 5 in its denominator, no power of two is divisible by 5, so its binary expansion repeats forever and has to be cut.", + "12": "Both stored values err upward (+5.55e-18 and +1.11e-17), so their exact sum is 5404319552844595.5 u. That is a tie between …595 and …596, and ties go to even, so it rounds to …596. Stored 0.3 was rounded down, to …595. The two results are neighboring doubles one gap (5.55e-17) apart.", + "13": "Languages print the shortest decimal that rounds back to the same double. The text '0.1' already rounds back to stored 0.1, so one digit is enough and the error stays hidden." + }, + "audit": { + "terms": [ + "The neighbor spacing is called 'gap' (§The stored values), 'Steps allowed here' (table header), 'one step above' (bold summary line 5) and 'u' (line 21), which in fact is the gap only in [0.25, 0.5).", + "The result of `+` is called 'rounded sum' (line 35), 'the result' (line 43) and '0.1 + 0.2' (§What to do).", + "The rounded values are called 'nearest binary numbers' (line 5), 'stored value' (line 21) and 'stored 0.1' (line 48).", + "The deviation is called 'Error' (table, line 31) and 'difference' (line 48 and line 60, where it means the gap between the sum and stored 0.3)." + ], + "unexplained": [ + "'Steps allowed here' column with values u/4, u/2, u (table): the text does not say that this is why stored 0.1 can be …198.5, a value with a fractional part in u.", + "'u = 2^-54 ≈ 5.55e-17' (line 21): no derivation from the 53 digits is given, so the reader cannot say what the gap would be in another range.", + "'1e-9' tolerance (line 60): the text does not say why this value was chosen or what would go wrong with 1e-15 or 1e-3.", + "'1 part in 5 × 10^15 ... 17th digit' (line 48): the step from this ratio to the 17th digit is not shown." + ], + "missing_why": [ + "'Because a double has a fixed 53 binary digits, the gap shrinks as the number shrinks. Between 0.25 and 0.5 the gap is u = 2^-54' (line 21): the text does not explain why the exponent is -54.", + "'The rule sends a tie to the even whole number' (line 39): the text does not say why this rule exists, and does not state that 596 is even and 595 is odd.", + "'Languages print the shortest decimal that rounds back to the same double' (line 48): the text does not say why languages use this rule." + ], + "excess": [ + "'Decimal types are slower, because the processor adds binary numbers directly' (line 55): a detour that no question or step of the argument uses.", + "'Not covered here' list (lines 62-66): reference pointers that this reader does not need." + ] + }, + "gaps": [ + "The table's fractional 'in u' values (…198.4 rounding to …198.5) depend on the 'u/4 steps' idea, which needs one explicit sentence.", + "The text never states that …596 is even, so the reader has to work out the tie direction alone.", + "The value u = 2^-54 is given without derivation, which makes the whole 'count in units of u' method feel arbitrary.", + "The text gives no guidance on why 1e-9 is a reasonable tolerance." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "1": "Full credit. Covers double, 53 significant bits, stored value, gap with u = 2^-54, error, exact vs rounded sum, tie with half-to-even, printed value, and the Decimal type.", + "2": "Full credit. Covers literal to stored value, the gap scaling with magnitude, the exact sum rounded onto the grid, tie-breaking, == comparing doubles, and printing hiding errors.", + "3": "Full credit. The chain is correct: factor 5, 0.1 and 0.2 stored high and 0.3 low, a tie, rounding to the double above stored 0.3, == false. 'Rounds up' matches the model's outcome for this case.", + "4": "Full credit. Names the power-of-two denominator as the key variable, with 0.5 + 0.25 as the case. Precision (bits kept) is noted as a secondary variable.", + "5": "Full credit. Separates the simplification (sign and exponent bits ignored, which the model scopes out) from established facts (IEEE stored values, the observed 0.30000000000000004).", + "6": "Full credit. Gives a complete, correct explanation that includes the printing step.", + "7": "Full credit. A specific difficulty report naming the units of u, where the gap comes from, and which neighbor is even. This is a subjective item and is not checked against model.md.", + "8": "Full credit. True; the power-of-two denominators give zero error and an exact sum.", + "9": "Full credit. + is correctly rounded. The difference comes from the inputs, which were already rounded, and from 0.3 being rounded independently.", + "10": "Full credit. True; Decimal stores base-10 digits, where 1/10 ends.", + "11": "Full credit. States the power-of-two denominator rule and the factor 5, and explains that no power of 2 is divisible by 5.", + "12": "Full credit. The exact sum 5404319552844595.5 u is a tie, goes to the even …596, and stored 0.3 is …595, one gap below.", + "13": "Full credit. Print shows the shortest decimal that rounds back to the same double, and '0.1' suffices." + }, + "pass": true + }, + "usage": { + "cost": 0.17389939999999998, + "seconds": 44 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The neighbor spacing is called 'gap' (§The stored values), 'Steps allowed here' (table header), 'one step above' (bold summary line 5) and 'u' (line 21), which in fact is the gap only in [0.25, 0.5)." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The result of `+` is called 'rounded sum' (line 35), 'the result' (line 43) and '0.1 + 0.2' (§What to do)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "The rounded values are called 'nearest binary numbers' (line 5), 'stored value' (line 21) and 'stored 0.1' (line 48)." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "The deviation is called 'Error' (table, line 31) and 'difference' (line 48 and line 60, where it means the gap between the sum and stored 0.3)." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'Steps allowed here' column with values u/4, u/2, u (table): the text does not say that this is why stored 0.1 can be …198.5, a value with a fractional part in u." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'u = 2^-54 ≈ 5.55e-17' (line 21): no derivation from the 53 digits is given, so the reader cannot say what the gap would be in another range." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'1e-9' tolerance (line 60): the text does not say why this value was chosen or what would go wrong with 1e-15 or 1e-3." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'1 part in 5 × 10^15 ... 17th digit' (line 48): the step from this ratio to the 17th digit is not shown." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "'Because a double has a fixed 53 binary digits, the gap shrinks as the number shrinks. Between 0.25 and 0.5 the gap is u = 2^-54' (line 21): the text does not explain why the exponent is -54." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'The rule sends a tie to the even whole number' (line 39): the text does not say why this rule exists, and does not state that 596 is even and 595 is odd." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'Languages print the shortest decimal that rounds back to the same double' (line 48): the text does not say why languages use this rule." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "'Decimal types are slower, because the processor adds binary numbers directly' (line 55): a detour that no question or step of the argument uses." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "'Not covered here' list (lines 62-66): reference pointers that this reader does not need." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The table's fractional 'in u' values (…198.4 rounding to …198.5) depend on the 'u/4 steps' idea, which needs one explicit sentence." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "The text never states that …596 is even, so the reader has to work out the tie direction alone." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "The value u = 2^-54 is given without derivation, which makes the whole 'count in units of u' method feel arbitrary." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The text gives no guidance on why 1e-9 is a reasonable tolerance." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152559-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152559-coldread-prose.json new file mode 100644 index 0000000..b46c7cf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152559-coldread-prose.json @@ -0,0 +1,252 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "float-sum", + "at": "2026-10-04T15:25:59", + "model": "default", + "prompt_sha": "5a19de7d0d5c", + "reply": { + "paragraphs": [ + { + "at": "Title + line 3", + "text": "In Python, JavaScript, and most other languages, `0.1 + 0.2 == 0.3` is `false`...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 5 (bold summary)", + "text": "The computer never holds 0.1, 0.2, or 0.3...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What the computer holds, line 9", + "text": "A literal such as `0.1` becomes a **double**...", + "unresolved": [], + "leap": [ + "\"64-bit binary format\" but \"at most 53 binary digits\": the reader isn't told what the other 11 bits do until the 'Not covered' list at the end. This is an edge issue, not a stopper." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why 0.1 must be rounded, line 13 + expansion", + "text": "A whole number times a power of two is a fraction whose denominator is a power of two...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 17", + "text": "This is the same as 1/3 = 0.3333… in decimal...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The stored values, line 21", + "text": "The **stored value** of a literal is the double nearest to it. The **gap** is...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 23", + "text": "The gap depends on the size of the number. Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54...", + "unresolved": [], + "leap": [ + "Why the multiplier must be 2^-54 in this range. The text never links it back to the 53-digit limit from line 9 (a bigger whole number with a smaller power would need more than 53 digits). The reader may reread line 9 to work this out." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 25", + "text": "To see the rounding, count each number in units of u...", + "unresolved": [ + "\"…595\" / \"…198.5\": the ellipsis shorthand for long counts is used without saying what it means. It can be inferred." + ], + "leap": [ + "\"its stored value can be a quarter-multiple of u, such as …198.5\": the example is a half, which makes the reader stop to check that .5 counts as a multiple of 1/4. The text also doesn't show why .4 rounds to .5 rather than .25." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "table, lines 27-31", + "text": "| Number | Exact, in u | Gap here | Stored, in u | ...", + "unresolved": [ + "\"Error\" column: its sign convention is only defined in the next paragraph." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 33", + "text": "The error is the stored value minus the decimal. 0.1 and 0.2 round up; 0.3 rounds down.", + "unresolved": [], + "leap": [], + "excess": [ + "\"0.3 rounds down\" was already said in line 25, and the table's signs show both directions." + ], + "purpose_unclear": false + }, + { + "at": "What `+` does, line 37 + exact sum", + "text": "The `+` operation first finds the **exact sum**...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 41", + "text": "The exact sum is exactly halfway... **tie**... IEEE 754...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 43", + "text": "Stored 0.3 is at …595 u. The rounded sum and stored 0.3 are neighboring doubles...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 45", + "text": "The addition did not make a mistake...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why print(0.1) shows 0.1, line 50", + "text": "Languages print the shortest decimal that rounds back to the same double...", + "unresolved": [], + "leap": [ + "\"One gap is about 1 part in 5 × 10^15 of 0.3, so the two doubles agree in their first 16 digits\": the reader has to do the arithmetic (5.55e-17 / 0.3) to check this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "When it works, line 55", + "text": "`0.5 + 0.25 == 0.75` is true...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 57", + "text": "A decimal type also avoids the problem...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What to do, line 62", + "text": "Do not compare doubles with `==`... Compare with a tolerance instead...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered here, lines 66-68", + "text": "- The sign and exponent bits of a double...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "line 33: cut \"0.1 and 0.2 round up; 0.3 rounds down.\" The table's Error column already shows it, and line 25 already said it for 0.3. Moving the error definition above the table would also stop the reader puzzling over the column.", + "line 50: cut the last sentence (\"One gap is about 1 part in 5 × 10^15...\"). It asks for arithmetic, and the claim before it (\"with 17 significant digits\") already makes the point." + ], + "question": { + "text": "Why does 0.1 + 0.2 == 0.3 come out false, and why does the sum print as 0.30000000000000004?", + "known_at": "Title and line 3", + "result_stated": "Yes, in words at line 5: \"The computer never holds 0.1, 0.2, or 0.3. It holds the nearest numbers it can store in binary. The addition is correct, but its result rounds to the next storable number above the one that 0.3 rounds to.\"" + }, + "motive": "Reason to care: only implied. Line 3 presents the surprise, and line 62 makes it practical: \"Do not compare doubles with `==`...\". No early sentence says why the reader should care. Reason for the approach: line 25, \"To see the rounding, count each number in units of u\", together with line 23's demonstration that the gap is constant within a range. That is enough.", + "close": "Yes. Line 45 gives the general argument in words: \"The addition did not make a mistake. The rounded sum is the nearest double to the exact sum of its inputs. The inputs were not 0.1 and 0.2.\" Line 55 states the general condition: \"The failure needs a decimal whose denominator has a factor 5.\"", + "recap": "0.1 and 0.2 are each stored slightly too high and 0.3 slightly too low, and the correctly rounded sum lands one storable step above stored 0.3, so == says false.", + "strongest": "Lines 41-43: the exact sum works out to …595.5 u, the tie rule sends it to …596 u, and that is one gap above stored 0.3 at …595 u. The whole result rests on concrete counts the reader can check.", + "weakest": "Line 23-25: why the multiplier is 2^-54 is not tied back to the 53-digit limit, and \"quarter-multiple of u, such as …198.5\" uses a half as its example of a quarter. It is the one place the reader rereads.", + "suggestions": [ + "In line 23, add one clause such as \"53 binary digits reach up to 2^53, so to cover 0.25–0.5 the power must be 2^-54\" to close the leap.", + "In line 25, show the rounding for 0.1 explicitly: the neighbors are …198.25 and …198.5, and .4 is nearer to .5.", + "Define the Error column before the table." + ] + }, + "usage": { + "cost": 0.1279186, + "seconds": 41 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "line 33: cut \"0.1 and 0.2 round up; 0.3 rounds down.\" The table's Error column already shows it, and line 25 already said it for 0.3. Moving the error definition above the table would also stop the reader puzzling over the column." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "line 50: cut the last sentence (\"One gap is about 1 part in 5 × 10^15...\"). It asks for arithmetic, and the claim before it (\"with 17 significant digits\") already makes the point." + }, + { + "n": 3, + "severity": "edge", + "kind": "leap", + "text": "What the computer holds, line 9: \"64-bit binary format\" but \"at most 53 binary digits\": the reader isn't told what the other 11 bits do until the 'Not covered' list at the end. This is an edge issue, not a stopper." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "line 23: Why the multiplier must be 2^-54 in this range. The text never links it back to the 53-digit limit from line 9 (a bigger whole number with a smaller power would need more than 53 digits). The reader may reread line 9 to work this out." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "line 25: \"…595\" / \"…198.5\": the ellipsis shorthand for long counts is used without saying what it means. It can be inferred." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "line 25: \"its stored value can be a quarter-multiple of u, such as …198.5\": the example is a half, which makes the reader stop to check that .5 counts as a multiple of 1/4. The text also doesn't show why .4 rounds to .5 rather than .25." + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "table, lines 27-31: \"Error\" column: its sign convention is only defined in the next paragraph." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Why print(0.1) shows 0.1, line 50: \"One gap is about 1 part in 5 × 10^15 of 0.3, so the two doubles agree in their first 16 digits\": the reader has to do the arithmetic (5.55e-17 / 0.3) to check this." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152657-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152657-quiz-prose.json new file mode 100644 index 0000000..fa6e1d9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152657-quiz-prose.json @@ -0,0 +1,162 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "float-sum", + "at": "2026-10-04T15:26:49", + "model": "default", + "prompt_sha": "008e69251529", + "reply": { + "answers": { + "1": "The main objects are decimal literals (0.1, 0.2, 0.3) and the double, a 64-bit binary format. A double is a whole number of at most 53 binary digits times a power of two. The other objects are: the stored value (the nearest double to a literal); the gap, meaning the distance between neighbouring doubles, with u = 2^-54 for doubles between 0.25 and 0.5; the error (stored value minus the decimal); the exact sum; the rounded sum; the tie-to-even rule from IEEE 754; the shortest-decimal printing rule; the == comparison; Python's Decimal type; and a tolerance.", + "2": "Each literal is rounded to its stored value, a nearby double. The gap between doubles depends on the size of the number: u/4 near 0.1, u/2 near 0.2 and u near 0.3. So each stored value has its own small error. The + operation adds the stored values exactly, then rounds the result to a double, breaking ties toward an even count of u. == compares the rounded sum with stored 0.3. Printing shows the shortest decimal that rounds back to the same double.", + "3": "0.1 and 0.2 cannot end in binary, so both are stored slightly too high, while 0.3 is stored slightly too low. Their exact sum lands halfway between two doubles, and the tie rounds up to the double just above stored 0.3, so == is false.", + "4": "The most important variable is whether the decimal's denominator has a factor of 5. If every value is a fraction with a power-of-two denominator, such as 0.5, 0.25 or 0.75, each one is stored with error 0. The exact sum is then itself a double, so == is true. With a factor of 5, the value has to be rounded, and the errors can push the sum onto a different double. The explanation also implies that a larger number has a larger gap, so its rounding error is bigger in absolute terms.", + "5": "Simplification: the sign and exponent bits are left out, and growth of errors over many operations is not covered. Established fact: IEEE 754 rounds a tie to the neighbour with an even count, and stored 0.1 = 0.10000000000000000555.", + "6": "Languages store numbers as doubles: a whole number of up to 53 binary digits times a power of two. 1/10 has a factor of 5 in its denominator, so it repeats forever in binary and must be rounded to the nearest double. As a result, stored 0.1 and 0.2 are slightly high, and stored 0.3 is slightly low. Adding the stored values exactly gives a result halfway between two doubles, and tie-to-even rounds it to the double one gap above stored 0.3, so == is false even though + did its job correctly. Printing shows the shortest decimal that maps back to each double, so 0.1 prints as 0.1 but the sum prints as 0.30000000000000004.", + "7": "The hardest part was counting in units of u (line 23). It does not say why doubles between 0.25 and 0.5 must use whole numbers from 2^52 to 2^53 times 2^-54, rather than a smaller whole number with a larger multiplier. One sentence would help: the whole number always uses all 53 digits. Mixing u/4 steps with values written in u (for example ...198.5) also takes effort.", + "8": "True. 1/2, 1/4 and 3/4 have power-of-two denominators, so each is stored exactly. The exact sum, 0.75, is itself a double, so no rounding happens.", + "9": "No. + computes the exact sum of its inputs and rounds it to the nearest double, which is correct. The difference comes from the inputs: they were the stored values, not 0.1 and 0.2. The tie-breaking rounding of the sum then moves the result to a different double from the one 0.3 rounds to.", + "10": "Yes, it is true. Decimal stores base-10 digits, and 1/10 ends in base 10, so all three values are exact.", + "11": "0.75 = 3 × 2^-2 is a whole number times a power of two. 1/10 has a factor of 5 in its denominator, and no power of two is divisible by 5. So in binary 0.1 repeats forever (0011…) and has to be cut off and rounded.", + "12": "Their errors do not cancel the way 0.3's error needs. Both are rounded up (+5.55e-18 and +1.11e-17), while 0.3 is rounded down (−1.11e-17). The exact sum, ...595.5 u, sits exactly halfway between ...595 and ...596. Tie-to-even sends it to ...596, one gap (5.55e-17) above stored 0.3 at ...595.", + "13": "Printing shows the shortest decimal that rounds back to the same double. The text \"0.1\" rounds to stored 0.1, so one digit is enough. The sum's double, ...596, needs 0.30000000000000004 instead." + }, + "audit": { + "terms": [ + "The same idea has two names: 'the power' (line 9: 'hold the sign and the power') and 'exponent' (line 65: 'sign and exponent bits').", + "Three words name the same integer part of a double: 'whole number' (line 9), 'multiplier' is used for the power-of-two factor (line 23: 'the multiplier is 2^-55'), and 'count of u' (line 40). This risks confusing which part is meant.", + "'gap' and 'u' are used together and overlap: 'gap ... u = 2^-54' (line 23), but elsewhere the gap is 'u/4' or 'u/2'. Later, 'one gap (5.55e-17)' (lines 42 and 61) assumes the gap means u." + ], + "unexplained": [ + "The 'Error' column (line 29 table: +5.55e-18, +1.11e-17, −1.11e-17) is given in decimal while every other column is in u. The text never says that these errors are what decide which double the sum lands on.", + "Line 49, 'with 17 significant digits': nothing says why 17 matters or what it implies.", + "Line 9, '53 binary digits ... other 11 bits': nothing says what more or fewer digits would change, for example a smaller gap." + ], + "missing_why": [ + "Line 23, 'Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54': it does not explain why the whole number must fill all 53 digits, which is what fixes the multiplier.", + "Line 40, tie-to-even: it does not explain why ties go to an even count instead of always rounding up or down.", + "Line 25, '0.1 ... its stored value is a multiple of u/4, such as …198.5': it does not show that 198.5 is the nearest multiple of 0.25 to 198.4." + ], + "excess": [ + "Lines 63–67, the 'Not covered here' list of references, is not needed by this reader.", + "Line 61, the money advice ('count whole cents as integers'), is a detour from the argument." + ] + }, + "gaps": [ + "The explanation never says why the integer part always has exactly 53 bits within a range. That is the step that makes the gap depend on the size of the number.", + "The tie-to-even rule is stated with no reason behind it.", + "It does not show that the decimal-errors table and the counts in u say the same thing." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "9": "Full marks, narrowly. The answer gives the right reason, that the inputs were stored values and not 0.1 and 0.2, but does not say why (1/10 does not end in binary)." + }, + "pass": true + }, + "usage": { + "cost": 0.16206720000000002, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same idea has two names: 'the power' (line 9: 'hold the sign and the power') and 'exponent' (line 65: 'sign and exponent bits')." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Three words name the same integer part of a double: 'whole number' (line 9), 'multiplier' is used for the power-of-two factor (line 23: 'the multiplier is 2^-55'), and 'count of u' (line 40). This risks confusing which part is meant." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'gap' and 'u' are used together and overlap: 'gap ... u = 2^-54' (line 23), but elsewhere the gap is 'u/4' or 'u/2'. Later, 'one gap (5.55e-17)' (lines 42 and 61) assumes the gap means u." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "The 'Error' column (line 29 table: +5.55e-18, +1.11e-17, −1.11e-17) is given in decimal while every other column is in u. The text never says that these errors are what decide which double the sum lands on." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 49, 'with 17 significant digits': nothing says why 17 matters or what it implies." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 9, '53 binary digits ... other 11 bits': nothing says what more or fewer digits would change, for example a smaller gap." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 23, 'Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54': it does not explain why the whole number must fill all 53 digits, which is what fixes the multiplier." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 40, tie-to-even: it does not explain why ties go to an even count instead of always rounding up or down." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 25, '0.1 ... its stored value is a multiple of u/4, such as …198.5': it does not show that 198.5 is the nearest multiple of 0.25 to 198.4." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "Lines 63–67, the 'Not covered here' list of references, is not needed by this reader." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "Line 61, the money advice ('count whole cents as integers'), is a detour from the argument." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "The explanation never says why the integer part always has exactly 53 bits within a range. That is the step that makes the gap depend on the size of the number." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "The tie-to-even rule is stated with no reason behind it." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "It does not show that the decimal-errors table and the counts in u say the same thing." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/read.json new file mode 100644 index 0000000..0daeaac --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "Because 0.1 expressed as a fraction is 1/10, whose denominator has a factor of 5. Since no power of two is divisible by 5, the fraction 1/10 cannot terminate in binary and must be rounded.", + "quote": "The denominator of 1/10 has a factor 5. No power of two is divisible by 5, so 1/10 never ends in binary: 1/10 = 0.000110011001100110011… (0011 repeats forever)" + }, + "2": { + "answer": "The exact sum of the stored values (5404319552844595.5 u) falls exactly halfway between two representable numbers. The IEEE 754 standard rounds such ties to the neighboring value with an even count, which is 5404319552844596 u, representing a different value from stored 0.3.", + "quote": "The exact sum is exactly halfway between the doubles at …595 u and …596 u. The IEEE 754 standard sends a tie to the neighbor whose count of u is even." + }, + "3": { + "answer": "A program should compare floating-point numbers using a tolerance instead of testing exact equality. It should use code like abs((0.1 + 0.2) - 0.3) < 1e-9 to check if values are close enough.", + "quote": "Compare with a tolerance instead: `abs((0.1 + 0.2) - 0.3) < 1e-9` is true, because the rounded sum and stored 0.3 are one gap (5.55e-17) apart." + }, + "4": { + "answer": "Yes, 0.5 + 0.25 == 0.75 is true. Because 0.5 (1/2), 0.25 (1/4), and 0.75 (3/4) are all fractions with power-of-two denominators, each is stored with zero error and their sum is exact.", + "quote": "All three are fractions with power-of-two denominators (1/2, 1/4, 3/4), so each is stored with error 0, and the exact sum is itself a double." + } + }, + "stops": [ + "Unexplained technical term: 'A double is a whole number of at most 53 binary digits, times a power of two'", + "Ambiguous antecedent: 'A fraction like that ends in binary'", + "Unsupported mathematical claim: 'No power of two is divisible by 5'", + "Unexplained formula with specific ranges: 'Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54'", + "Undefined concept: 'To see the rounding, count each number in units of u'", + "Unclear which range is meant: '0.1 is in the range where the gap is u/4, so its stored value is a multiple of u/4'", + "Ambiguous phrasing: 'sends a tie to the neighbor whose count of u is even'", + "Unexplained how determined: 'The shortest decimal that rounds back to …596 u is 0.30000000000000004'", + "Unjustified claim: 'each is stored with error 0, and the exact sum is itself a double'", + "Unclear conversion step: '0.3 × 2^54 = 5404319552844595.2, so 0.3 is 5404319552844595.2 u'" + ], + "excess": [ + "Shows the problem but doesn't explain it: 'and `0.1 + 0.2` prints `0.30000000000000004`'", + "Example of exact storage unrelated to the core problem: 'For example, 0.75 = 3 × 2^-2, so a double holds 0.75 exactly'", + "Helpful but not necessary analogy: 'This is the same as 1/3 = 0.3333… in decimal. A machine that keeps only a fixed number of digits must cut the expansion'", + "Explains printing behavior, not the core problem: 'Why `print(0.1)` shows `0.1`'", + "Supporting printing detail not needed: 'Languages print the shortest decimal that rounds back to the same double'", + "Solution to the problem rather than explanation: 'A decimal type also avoids the problem. Python's `Decimal(\"0.1\") + Decimal(\"0.2\") == Decimal(\"0.3\")` is true'", + "Listed as not covered, not part of explanation: 'How errors grow over many operations, such as adding 0.1 ten times'", + "Listed as not covered, not part of explanation: 'How to choose between an absolute and a relative tolerance'", + "Listed as not covered, not part of explanation: 'The sign and exponent bits of a double'", + "Specific decimal values may be excessive detail: '| Number | Exact, in u | Gap here | Stored, in u | Stored, in decimal | Error |' (the table in lines 29-33)" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/run.json new file mode 100644 index 0000000..88d5ad3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/float-sum/explanation.md`, plain prose with one table of numbers, because the answer is a single chain of cause and effect with worked values (\"1/10 never ends in binary → 0.1, 0.2 and 0.3 get rounded → their sum lands one step above where 0.3 lands\"), with nothing to click through or watch move; it passes `explainer check`, the probe, two cold reads with no blocking findings, and the blind test (13/13), though code execution wasn't allowed here so I worked out all the numbers by hand.", + "error": null, + "cost": 1.552655, + "seconds": 607, + "turns": 60, + "files": [ + "explainers/float-sum/explanation.md" + ], + "review_cost": 0.8245, + "reviews": 6 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/explanation.md new file mode 100644 index 0000000..ec350e0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/explanation.md @@ -0,0 +1,55 @@ +# Why does 0.1 + 0.2 not equal 0.3? + + + +**The computer never holds 0.1, 0.2, or 0.3. It holds the nearest binary fractions, and those small errors add up to a result that is a different number from the stored 0.3.** In most languages, `0.1 + 0.2` prints `0.30000000000000004`, so `0.1 + 0.2 == 0.3` is false. + +## The same problem in base 10 + +Suppose a calculator keeps 3 decimal digits. It cannot hold 1/3, so it stores 0.333. Add two of them: 0.333 + 0.333 = 0.666. But the calculator stores 2/3 as 0.667. So "1/3 + 1/3 == 2/3" is false on this calculator. + +The calculator adds correctly. The inputs were already wrong, because 1/3 has no finite decimal form. + +## Why 0.1 has the same problem in base 2 + +Most languages store a decimal number as a **double**: a binary fraction with 53 significant bits (the IEEE 754 standard). A binary fraction is a sum of powers of 2: 1/2, 1/4, 1/8, and so on. + + +In base 2, 0.1 is like 1/3 in base 10. Its binary form, 0.000110011001100…, repeats forever. A double keeps only 53 bits, so it stores the nearest binary fraction. Call this the **stored value**: + +| Written | Stored value (first 34 digits) | +|---|---| +| 0.1 | 0.1000000000000000055511151231257827 | +| 0.2 | 0.2000000000000000111022302462515654 | +| 0.3 | 0.2999999999999999888977697537484345 | + +The stored 0.1 and 0.2 are slightly too high. The stored 0.3 is slightly too low. + +## What happens in the addition + + +1. The computer adds the two stored values. The exact sum is 0.3000000000000000166533453693773481. +2. That sum is not a double either, so the computer rounds it to the nearest double. The sum is exactly halfway between two doubles; the tie rule picks the upper one: 0.3000000000000000444089209850062616. +3. The stored 0.3 is the double just below that result. The two differ by 5.551115123125783e-17, the gap between neighbouring doubles near 0.3. +4. `==` compares the doubles bit for bit. They differ, so the answer is false. + +## When addition is exact + + +Before you read on, predict: is `0.5 + 0.25 == 0.75` true? + +It is true. 0.5 is 1/2 and 0.25 is 1/4, so both are short binary fractions. Their sum, 0.75, is also one. No step rounds, so no error appears. The problem comes only from numbers that have no short binary form, such as 0.1, 0.2, and 0.3. + +## "But print(0.1) shows 0.1" + + +Languages print the shortest decimal that maps back to the same double. For the stored 0.1, that decimal is "0.1". For the result of 0.1 + 0.2, the shortest such decimal is 0.30000000000000004, because "0.3" maps to a different double. The print hides the error; it does not remove it. + +## What to do + + +- Compare with a tolerance, not `==`. `abs(0.1 + 0.2 - 0.3) < 1e-9` is true. +- For money, count in integer cents: 10 + 20 == 30 is always true. +- If you need exact decimal arithmetic, use a decimal type, for example Python `Decimal` or Java `BigDecimal`. These are slower than doubles. + +This behaviour is not a bug and not random. Every language that uses IEEE 754 doubles (JavaScript, Python, Java, C, and most others) gives the same result every time. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/model.md new file mode 100644 index 0000000..892e7d2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/model.md @@ -0,0 +1,80 @@ +# Semantic model: why 0.1 + 0.2 != 0.3 + +> Source of truth. Every rendering derives from this file. + +## Central question + +Why does `0.1 + 0.2 == 0.3` evaluate to false in most programming languages? + +## Audience and prior knowledge + +A technical reader: school mathematics (fractions, powers, repeating decimals), basic programming (variables, `==`, printing). New to binary fractions and IEEE 754. Does not know that a float stores a binary fraction with a fixed number of bits. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| double | The default number type for decimals in most languages: a binary fraction with 53 significant bits (IEEE 754 binary64). | fact | +| stored value | The double nearest to the number written in the source code. | fact | +| rounding error | The difference between a written number and its stored value. | fact | +| spacing | The distance between two neighbouring doubles; 2^-54 ≈ 5.55e-17 for numbers between 0.25 and 0.5. | fact | + +## Causal chain + +```text +0.1 has an infinite binary expansion → the double keeps 53 bits → stored value ≠ 0.1 +→ the errors of 0.1 and 0.2 add → the exact sum is rounded to a double +→ that double is one spacing above the stored value of 0.3 → == is false +``` + +## Quantities + +- stored 0.1 = 0.1000000000000000055511151231257827… (error +5.55e-18) +- stored 0.2 = 0.2000000000000000111022302462515654… (error +1.11e-17) +- stored 0.3 = 0.2999999999999999888977697537484345… (error −1.11e-17) +- exact sum of stored 0.1 and 0.2 = 0.3000000000000000166533453693773481… +- result of 0.1 + 0.2 = 0.3000000000000000444089209850062616… (prints as 0.30000000000000004) +- difference result − stored 0.3 = 2^-54 ≈ 5.551115123125783e-17 = one spacing +- The exact sum lies exactly halfway between the two neighbouring doubles; round-half-to-even chooses the upper one. + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Store a binary fraction | Store decimal digits | Binary hardware is faster and simpler; decimal types exist (Python `Decimal`, Java `BigDecimal`) but are slower. | +| Fixed 53 bits | Unlimited bits | 0.1 in binary is 0.000110011001100… and never ends, so no finite memory holds it exactly. | +| Compare with a tolerance | `==` | `0.1 + 0.2 == 0.3` is false; `abs(0.1 + 0.2 - 0.3) < 1e-9` is true. | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| Sums of doubles are exact when every value is a sum of powers of 2 that fits in 53 bits | 0.5 + 0.25 == 0.75 is true | 0.1 + 0.2 == 0.3 is false | +| Decimal analogy: 3 decimal digits | 0.5 + 0.25 = 0.75 exact | 1/3 → 0.333; 0.333 + 0.333 = 0.666, but 2/3 → 0.667 | + +## Terms + +- **double**: as above. +- **stored value**: the double the computer actually holds. Never call it "the number 0.1". +- **rounding error**: written number minus stored value. +- **spacing**: gap between neighbouring doubles; small near 0, large for large numbers. + +## Epistemic status + +- **Established:** IEEE 754 binary64 values above; they follow mathematically from the standard. + +## Confusion points + +- "The computer adds badly." → The addition is correctly rounded. The inputs are already not 0.1 and 0.2. +- "But `print(0.1)` shows 0.1." → Languages print the shortest decimal that maps back to the same double. The stored value is still not 0.1. +- "The error is random." → It is deterministic: the same code gives the same result every time. + +## Scope + +- **Out of scope:** exponent field and bit layout — see IEEE 754; error accumulation in long sums (Kahan summation); NaN and infinities. + +## Representation decision + +- **Stage:** 1 +- **Reason:** A short linear causal chain with worked numbers; a table carries the numbers. No parameters to explore. +- **Levels:** L1 the answer / L2 the stored values / L3 why binary cannot hold 0.1 / L4 what to do. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/model.yaml new file mode 100644 index 0000000..66b3a57 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/model.yaml @@ -0,0 +1,60 @@ +# Machine-readable half of model.md. `explainer check float-sum` holds every rendering to it. + +values: + bits: 53 + stored_01: "0.1000000000000000055511151231257827" + stored_02: "0.2000000000000000111022302462515654" + stored_03: "0.2999999999999999888977697537484345" + exact_sum: "0.3000000000000000166533453693773481" + result: "0.3000000000000000444089209850062616" + result_printed: "0.30000000000000004" + gap: "5.551115123125783e-17" + +# Written inputs, the binary expansion, the standard's name, and small illustrative integers. +allow: [0.1, 0.2, 0.3, 0.5, 0.25, 0.75, 0.333, 0.666, 0.667, 1e-9, 2, 3, 4, 8, 10, 20, 30, 34, 754, + "0.000110011001100", + "0.1000000000000000055511151231257827", "0.2000000000000000111022302462515654", + "0.2999999999999999888977697537484345", "0.3000000000000000166533453693773481", + "0.3000000000000000444089209850062616", "0.30000000000000004", "5.551115123125783e-17"] + +require: + - path: result_printed + in: [prose] + - path: stored_01 + in: [prose] + - path: stored_03 + in: [prose] + +claims: + - id: why-binary-inexact + kind: why + statement: "0.1 has an infinite binary expansion, so a double with 53 bits stores only the nearest binary fraction." + alternative: "Store 0.1 exactly." + counterexample: "0.1 = 0.000110011001100… in binary and never ends; stored 0.1 = 0.1000000000000000055511…" + ask: "Why can a double not hold 0.1 exactly when it can hold 0.75 exactly?" + - id: mech-sum + kind: mechanism + statement: "The rounding errors of the stored 0.1 and 0.2 add, and the sum is rounded to the double one spacing above the stored 0.3." + example: "stored 0.1 + stored 0.2 = 0.30000000000000001665…; rounded to 0.30000000000000004441…; stored 0.3 = 0.29999999999999998889…" + - id: guarantee-exact + kind: guarantee + statement: "Addition gives the exact answer when the inputs and the answer are sums of powers of 2 that fit in 53 bits." + example: "0.5 + 0.25 == 0.75 is true." + counterexample: "0.1 + 0.2 == 0.3 is false, because 0.1, 0.2, 0.3 are not sums of powers of 2." + - id: misc-print + kind: misconception + correction: "The stored value of 0.1 is 0.1000000000000000055511…; printing chooses a short decimal for display only." + statement: "print(0.1) shows 0.1 because the language prints the shortest decimal that maps back to the same double, not because the double equals 0.1." + ask: "If print(0.1) shows 0.1, why is the stored value not 0.1?" + - id: why-tolerance + kind: why + statement: "Compare doubles with a tolerance, or use integers or a decimal type, not ==." + alternative: "Compare with ==" + counterexample: "0.1 + 0.2 == 0.3 is false; abs(0.1 + 0.2 - 0.3) < 1e-9 is true." + +accept_unjustified: {} + +terms: + - term: stored value + means: "the double the computer holds for a written number" + avoid: [imprecise number, fuzzy] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/numbers.py b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/numbers.py new file mode 100644 index 0000000..03f6bc5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/explainers/float-sum/numbers.py @@ -0,0 +1,10 @@ +from decimal import Decimal as D +import math + +for x in [0.1, 0.2, 0.3, 0.1 + 0.2]: + print(repr(x), D(x), x.hex()) +print("errors", D(0.1) - D("0.1"), D(0.2) - D("0.2"), D(0.3) - D("0.3")) +print("exact sum", D(0.1) + D(0.2)) +print("gap", D(0.1 + 0.2) - D(0.3)) +print("ulp(0.3)", D(math.ulp(0.3))) +print(0.5 + 0.25 == 0.75, abs(0.1 + 0.2 - 0.3) < 1e-9, 0.1 + 0.2 - 0.3) diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/read.json new file mode 100644 index 0000000..1f3d8a8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/read.json @@ -0,0 +1,40 @@ +{ + "answers": { + "1": { + "answer": "0.1 in binary form is a repeating fraction like 1/3 in decimal. Computers can only store 53 bits, so they store the nearest binary fraction instead of the infinite repeating form.", + "quote": "Its binary form, 0.000110011001100…, repeats forever. A double keeps only 53 bits, so it stores the nearest binary fraction." + }, + "2": { + "answer": "The stored 0.1 and 0.2 are slightly too high, and when added together, the result must be rounded again to the nearest double. This rounded sum differs from the stored 0.3, which is slightly too low.", + "quote": "The stored 0.1 and 0.2 are slightly too high. The stored 0.3 is slightly too low... so the computer rounds it to the nearest double... The stored 0.3 is the double just below that result." + }, + "3": { + "answer": "Compare with a tolerance instead of using ==. For example, test abs(0.1 + 0.2 - 0.3) < 1e-9. Alternatively, for money use integer cents, or use a decimal type like Python Decimal or Java BigDecimal.", + "quote": "Compare with a tolerance, not `==`. `abs(0.1 + 0.2 - 0.3) < 1e-9` is true." + }, + "4": { + "answer": "Yes, it is true. 0.5 and 0.25 are short binary fractions (1/2 and 1/4), so both can be stored exactly. Their sum 0.75 can also be stored exactly, so no rounding error occurs.", + "quote": "0.5 is 1/2 and 0.25 is 1/4, so both are short binary fractions. Their sum, 0.75, is also one. No step rounds, so no error appears." + } + }, + "stops": [ + "Line 15: 'IEEE 754 standard' is unexplained.", + "Line 32: 'tie rule' is mentioned but not defined.", + "Line 33: 'gap between neighbouring doubles' is a technical concept not fully explained.", + "Line 46: 'maps back to the same double' assumes understanding of the relationship between decimal and binary representation.", + "Line 18: 'binary fraction' is used before being clearly defined.", + "Line 9: 'no finite decimal form' assumes reader knows what that means." + ], + "excess": [ + "Line 3: HTML comment tags are not needed.", + "Line 55: 'Every language that uses IEEE 754 doubles...' provides context not needed to answer the questions.", + "Line 53: 'These are slower than doubles' is off-topic.", + "Line 9: 'The calculator adds correctly' is a philosophical point not needed." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/run.json new file mode 100644 index 0000000..b1120da --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/float-sum/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/float-sum/explanation.md` as prose. The cause is a short chain of steps with exact numbers, and text plus one table explains that better than a diagram or interactive page; it passes `explainer check`, and the files also include `model.md`/`model.yaml` and a leftover `numbers.py` I couldn't run or delete, so the decimal values come from the IEEE 754 standard and I didn't recompute them.", + "error": null, + "cost": 0.4691284, + "seconds": 83, + "turns": 24, + "files": [ + "explainers/float-sum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/explainers/gps-relativity/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/explainers/gps-relativity/explanation.md new file mode 100644 index 0000000..e2a4958 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/explainers/gps-relativity/explanation.md @@ -0,0 +1,150 @@ +# How GPS finds your position, and why it needs relativity + +The short version: GPS is a clock system. Each satellite says what time it is. Your receiver works out how long each message took to arrive, and those delays tell it where it is. Light covers about 30 cm in a nanosecond, so the clocks have to agree to within a few billionths of a second. At that level of precision, Einstein's two theories of relativity both change how fast the satellite clocks tick, and the change is large enough that it has to be designed out. + +The rest of this file explains that in four steps: + +1. Distance from time +2. Why you need four satellites, not three +3. How the receiver actually solves for position (with code) +4. Relativity: what changes the satellite clocks' rate, by how much, and what the system does about it + +--- + +## 1. Distance from time + +About 30 GPS satellites orbit roughly 20,200 km above the ground, and each goes round about twice a day. Each one carries atomic clocks and constantly broadcasts a radio signal containing, in effect: + +> "I am satellite #12. This part of the signal left me at exactly time *t*ₛ. At that moment I was at position (*x*, *y*, *z*)." + +The satellite knows its own position because ground stations track every satellite and regularly upload a precise description of its orbit, called the *ephemeris*. + +Your receiver notes the time *t*ᵣ when that part of the signal arrives. Radio waves travel at the speed of light, *c* ≈ 299,792,458 m/s, so + +``` +distance to satellite = c × (t_r − t_s) +``` + +The signal takes about 67–86 milliseconds to arrive. Getting a distance accurate to a metre means measuring that delay to about 3 nanoseconds. **Every nanosecond of timing error is about 30 cm of position error.** Keep that conversion in mind, because the rest of the story depends on it. + +One distance puts you somewhere on a sphere centred on that satellite. With a second satellite you are on the circle where two spheres meet. A third sphere cuts that circle at two points. One of those points is usually far out in space or deep underground, so in principle three satellites give you a position. + +In practice, three is not enough. + +## 2. Why four satellites: your receiver's clock is bad + +The satellites have atomic clocks that cost a lot and are steered against a master time scale on the ground. Your phone has a cheap quartz oscillator. It may easily be off by a millisecond, and a millisecond is 300 km. + +So *t*ᵣ in the formula above isn't the true arrival time. It is the true arrival time plus an unknown receiver clock error δ*t*. That error is the **same for every satellite**, because all the measurements are made with the same clock. Each measured distance is therefore wrong by the same amount, *b* = *c*·δ*t*. These measured distances are called **pseudoranges**: + +``` +ρᵢ = |sᵢ − p| + b +``` + +Here `sᵢ` is satellite *i*'s known position, `p` = (*x*, *y*, *z*) is your unknown position, and `b` is your unknown clock error expressed in metres. + +That makes **four unknowns**: *x*, *y*, *z* and *b*. Four unknowns need four equations, so you need four satellites. A useful side effect is that once the receiver has solved for *b*, it knows the time to within tens of nanoseconds. That is why GPS is also one of the world's main sources of precise time, used by phone networks, power grids and financial systems. + +Usually 8–12 satellites are visible. With more than four, the receiver does a least-squares fit, which averages out noise. + +## 3. Solving the equations + +The equations are non-linear because of the square root inside |sᵢ − p|. The standard method is to guess, linearise and repeat: + +1. Start with a rough guess, such as the centre of the Earth with *b* = 0. +2. For each satellite, predict the pseudorange from the current guess and compare it with the measured one. +3. Work out how much each prediction would change if you nudged *x*, *y*, *z* or *b*. For the position this is the unit vector pointing from the satellite to you. For *b* it is simply 1. +4. Solve the resulting *linear* least-squares problem for the correction, apply it, and go back to step 2. + +This usually converges in 4–6 iterations, even from the centre of the Earth. + +```python +import numpy as np + +def solve_position(sat_pos, pseudoranges, iters=10): + """sat_pos: (N,3) array of satellite positions in metres (N >= 4). + pseudoranges: (N,) measured pseudoranges in metres. + Returns (receiver position, receiver clock bias in metres).""" + x = np.zeros(4) # [x, y, z, b], starting at Earth's centre + for _ in range(iters): + diff = sat_pos - x[:3] + ranges = np.linalg.norm(diff, axis=1) + predicted = ranges + x[3] + residual = pseudoranges - predicted + # Jacobian: d(predicted)/d(x,y,z) = -unit vector to satellite; d/db = 1 + J = np.hstack([-diff / ranges[:, None], np.ones((len(ranges), 1))]) + dx, *_ = np.linalg.lstsq(J, residual, rcond=None) + x += dx + if np.linalg.norm(dx) < 1e-4: + break + return x[:3], x[3] +``` + +Real receivers also correct for several known sources of error: the signal slowing down in the ionosphere and the troposphere, reflections off buildings, small errors in the broadcast orbits, and the effects described below. All of this together gives a typical phone accuracy of a few metres. + +Note what the whole method assumes: **the satellites' clocks agree with each other and with GPS system time.** The receiver can absorb an error in its own clock, because that error is the same for every satellite. It cannot absorb errors that differ from satellite to satellite. The receiver has no way to tell one satellite's clock being 10 ns fast from that satellite being 3 m farther away. + +## 4. Where relativity comes in + +Relativity says clocks don't all run at the same rate. Two effects matter for GPS, and they push in opposite directions. + +### Effect 1: moving clocks run slow (special relativity) + +A clock moving at speed *v* relative to you ticks slower by a fraction of roughly + +``` +v² / (2c²) +``` + +GPS satellites move at about 3.87 km/s: + +``` +(3874 m/s)² / (2 × (3.0×10⁸ m/s)²) ≈ 8.3 × 10⁻¹¹ +``` + +Over one day (86,400 s) that adds up to about **7 microseconds slow**. + +### Effect 2: clocks higher up run fast (general relativity) + +Clocks deeper in a gravitational field run slower. A clock at the satellite's altitude feels weaker gravity than one on the ground, so it runs faster. The fractional difference is + +``` +(GM/c²) × (1/R_earth − 1/r_orbit) +``` + +Here *GM* = 3.986×10¹⁴ m³/s² is Earth's gravitational parameter, *R*_earth ≈ 6,371 km and *r*_orbit ≈ 26,560 km, measured from Earth's centre: + +``` +3.986×10¹⁴ × (1/6.371×10⁶ − 1/2.656×10⁷) / (3.0×10⁸)² ≈ 5.3 × 10⁻¹⁰ +``` + +Over one day that is about **45 microseconds fast**. + +### The net effect + +``` ++45 μs/day (higher up → faster) + −7 μs/day (moving → slower) +─────────── ++38 μs/day satellite clocks gain on ground clocks +``` + +38 microseconds is 38,000 nanoseconds. At 30 cm per nanosecond, that much timing error is about **11 km of light travel per day**. The system needs nanosecond agreement, and relativity puts tens of thousands of nanoseconds of disagreement into it every day. + +A note on the popular claim that "without relativity GPS would drift 11 km a day". That number shows the *scale* of the effect, but it isn't literally what would happen. If every satellite clock gained exactly the same amount, part of the error would end up in the receiver's *b*. The ground stations also already estimate each satellite's clock drift and broadcast corrections. A more accurate statement is this: if nobody had accounted for relativity, the satellite clocks would steadily drift away from ground time, and the parts of the effect that differ from satellite to satellite and moment to moment (next section) would put metre-level errors directly into your position. Relativity isn't an optional refinement. You have to account for it for the system to work. + +### What GPS actually does about it + +**1. Detune the clocks at the factory.** Each satellite's reference oscillator is meant to run at 10.23 MHz. It is deliberately set slightly slow, to 10.22999999543 MHz, a fractional offset of about 4.465×10⁻¹⁰. Once the satellite is in orbit, the relativistic speed-up brings it to exactly 10.23 MHz as seen from the ground. This one adjustment removes almost all of the 38 μs/day. + +**2. Correct for the slightly elliptical orbit.** The orbits are not perfect circles. When a satellite is nearest Earth it moves faster and sits deeper in gravity, so its clock runs slower. When it is farthest away, its clock runs faster. The resulting periodic clock error reaches a few tens of nanoseconds, which means up to about 10 m of range error. A fixed factory offset can't remove a varying error, so the receiver calculates it from the broadcast orbit and subtracts it. In the GPS specification it appears as `Δt_r = F · e · √a · sin(E)`, where *e* is how elliptical the orbit is, *a* is the size of the orbit, *E* is where the satellite is in its orbit, and *F* = −4.4428×10⁻¹⁰ s/√m is a constant built from *G*, Earth's mass and *c*. + +**3. Account for Earth turning while the signal travels (the Sagnac effect).** Positions are computed in a frame that rotates with Earth. During the roughly 70 ms the signal is in flight, the ground under you moves by up to about 30 m. The receiver handles this by rotating each satellite's position back by the angle Earth turns during the signal's flight. Leave this out and you get errors of tens of metres. + +## Summary + +- GPS measures **time**, not distance. A delay multiplied by the speed of light gives a range, and 1 ns of error gives 30 cm of range error. +- Your receiver's clock is unknown, so it solves for **four unknowns** (*x*, *y*, *z*, clock error) using **at least four satellites**, by repeatedly linearising and fitting. +- That only works if the **satellite clocks keep nanosecond agreement** with ground time. +- Relativity breaks that agreement by a lot. Orbital speed makes satellite clocks lose about 7 μs/day, and weaker gravity makes them gain about 45 μs/day, for a net gain of about **38 μs/day**, thousands of times the allowed error. +- GPS deals with this by **detuning the satellite clocks before launch**, having **receivers correct for elliptical orbits**, and **accounting for Earth's rotation** during signal travel. Each of these is engineering that relativity requires. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/read.json new file mode 100644 index 0000000..0bc84de --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/read.json @@ -0,0 +1,45 @@ +{ + "answers": { + "1": { + "answer": "The receiver notes the arrival time of each signal and subtracts the broadcast transmission time to find the delay. It multiplies this delay by the speed of light to calculate distance.", + "quote": "Your receiver notes the time *t*ᵣ when that part of the signal arrives. Radio waves travel at the speed of light, *c* ≈ 299,792,458 m/s, so distance to satellite = c × (t_r − t_s)" + }, + "2": { + "answer": "The receiver's clock is unknown and may be milliseconds off, causing the same error in all distance measurements. This unknown clock error is a fourth unknown alongside position coordinates x, y, z, requiring four equations and thus four satellites.", + "quote": "That makes **four unknowns**: *x*, *y*, *z* and *b*. Four unknowns need four equations, so you need four satellites." + }, + "3": { + "answer": "Relativistic effects make satellite clocks gain about 38 microseconds daily (45 µs from weaker gravity minus 7 µs from orbital speed). This produces about 11 kilometers of timing error daily, vastly exceeding the nanosecond-level precision required.", + "quote": "38 microseconds is 38,000 nanoseconds. At 30 cm per nanosecond, that much timing error is about **11 km of light travel per day**." + }, + "4": { + "answer": "About 300 meters. Since 1 nanosecond of timing error produces 30 cm of error, 1 microsecond (1000 nanoseconds) produces 300 meters of distance error.", + "quote": "Every nanosecond of timing error is about 30 cm of position error." + } + }, + "stops": [ + "\"Jacobian\"", + "\"unit vector pointing from the satellite to you\"", + "\"v²/(2c²)\" - formula without derivation", + "\"Clocks deeper in a gravitational field run slower\" - principle not explained", + "\"ρᵢ = |sᵢ − p| + b\" - notation introduced", + "\"pseudoranges\" - why called 'pseudo' unclear", + "\"least-squares fit\" - method not defined", + "\"(GM/c²) × (1/R_earth − 1/r_orbit)\" - complex formula" + ], + "excess": [ + "Python code implementation with numpy", + "\"Usually 8–12 satellites are visible. With more than four, the receiver does a least-squares fit\"", + "\"Real receivers also correct for several known sources of error: the signal slowing down in the ionosphere and the troposphere\"", + "Detailed gravitational parameter values: \"GM = 3.986×10¹⁴ m³/s²\" and orbital distances", + "\"In the GPS specification it appears as Δt_r = F · e · √a · sin(E)\"", + "Sagnac effect: \"The receiver handles this by rotating each satellite's position back\"", + "Discussion of popular misconception: \"A note on the popular claim that 'without relativity GPS would drift 11 km a day'\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/run.json new file mode 100644 index 0000000..256ef70 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/gps-relativity/explanation.md`, a prose explanation, because this topic is a chain of reasoning with worked numbers and a short Python position solver, and prose with inline calculations and code follows that chain best for a technical reader.", + "error": null, + "cost": 0.17838859999999995, + "seconds": 57, + "turns": 2, + "files": [ + "explainers/gps-relativity/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/explanation.md new file mode 100644 index 0000000..a95d00a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/explanation.md @@ -0,0 +1,104 @@ +# How GPS finds your position, and why it needs relativity + +**The short answer.** A GPS receiver does not send anything. It only listens. Each satellite broadcasts its position and the exact time it sent its signal. The receiver measures how long each signal took to arrive. Then it multiplies that time by the speed of light to get a distance. With distances to four satellites, it can solve for where it is. + +This works only if the satellite clocks are correct to a few billionths of a second (nanoseconds). Relativity says a clock in orbit ticks at a different rate from a clock on the ground: about 38 microseconds per day faster. GPS builds that correction into every satellite and every receiver. + +Below, each step follows from the one before it. + +--- + +## 1. Time becomes distance + +Radio signals travel at the speed of light, *c* ≈ 300,000 km/s. The distance to a satellite is: + +``` +distance = c × (time received − time sent) +``` + +**Example.** GPS satellites orbit about 20,200 km above the ground. A signal from a satellite directly overhead takes about 0.067 s (67 ms) to arrive. A satellite near the horizon is farther away, so its signal takes up to about 86 ms. + +This conversion is unforgiving. Light travels **30 cm in one nanosecond**. A timing error of 10 ns gives a distance error of 3 m. A timing error of 1 microsecond (1,000 ns) gives 300 m. So GPS is really a system for distributing very precise time. + +## 2. Distances become a position + +One distance tells you that you are somewhere on a sphere around that satellite. + +- Two spheres intersect in a circle. +- A third sphere cuts that circle at two points. One point is near the Earth. The other is far out in space, so the receiver discards it. + +So three exact distances give a position. A flat version of the same idea: if you are 3 km from tower A, 4 km from tower B, and 5 km from tower C, only one spot on the map fits all three. + +## 3. The receiver's clock is the weak link, so use a fourth satellite + +Each satellite carries atomic clocks. Your phone has a cheap quartz clock. Suppose the quartz clock is wrong by 1 millisecond. Then every distance is wrong by 300 km, and the three spheres do not meet at a sensible point. + +The simplest fix is an atomic clock in every receiver. That fails on cost, size, and power. GPS uses a better trick: treat the receiver clock error *b* as a fourth unknown. The receiver's error adds the same amount, *c·b*, to every measured distance. For each satellite *i* at known position (*xᵢ, yᵢ, zᵢ*): + +``` +measured_distanceᵢ = √((x − xᵢ)² + (y − yᵢ)² + (z − zᵢ)²) + c·b +``` + +There are four unknowns (*x, y, z, b*), so the receiver needs four satellites. It solves the equations numerically, usually with more than four satellites to average out noise. A bonus: once it knows *b*, the receiver knows the true time to within tens of nanoseconds. + +**What this trick cannot fix.** The term *c·b* removes only an error that is **the same for every signal**. If one satellite's clock is off by 30 ns and the others are not, that satellite's distance is off by 9 m, and the solve cannot detect it. So every satellite clock must agree with one shared time scale, called **GPS time**, to within a few nanoseconds. Relativity enters here. + +## 4. Clocks in orbit do not tick at ground rate + +*Before you read on, predict: does a satellite clock run fast or slow compared with a clock on the ground?* + +Two established effects act on the satellite clock. Both are measured in experiments, not only predicted. They act in opposite directions. + +**Effect 1: speed (special relativity).** A moving clock ticks slower than a clock at rest, as seen by the clock at rest. For speeds much less than *c*, the clock loses a fraction *v²/2c²* of its time. A GPS satellite moves at about 3.87 km/s: + +``` +v² / 2c² = (3,870 m/s)² / (2 × (3.0×10⁸ m/s)²) ≈ 8.3 × 10⁻¹¹ +× 86,400 s per day ≈ 7.2 µs per day slower +``` + +**Effect 2: gravity (general relativity).** A clock deeper in a gravitational field ticks slower. The ground clock sits closer to the Earth's centre (6,371 km) than the satellite (26,560 km). So the satellite clock ticks *faster*. The fraction is the difference in gravitational potential divided by *c²*: + +``` +(GM/c²) × (1/R_earth − 1/r_orbit) += 4.43 mm × (1/6,371 km − 1/26,560 km) ≈ 5.29 × 10⁻¹⁰ +× 86,400 s per day ≈ 45.7 µs per day faster +``` + +(*GM/c²* for the Earth is a length, about 4.43 mm. It sets how strong the effect is.) + +**Net result.** Gravity wins: + +``` ++45.7 µs/day − 7.2 µs/day ≈ +38.5 µs/day +``` + +**What that number means.** 38.5 µs × 30 cm/ns ≈ **11.5 km**. That is the distance error that one day of uncorrected drift represents. Compare it with the few-nanosecond budget from section 3. The drift is about 0.45 ns per second, so it uses up that budget in about 10 seconds. + +## 5. How GPS corrects it + +**Before launch.** Engineers set each satellite clock slightly slow on the ground. The nominal clock frequency is 10.23 MHz. The factory setting is **10.22999999543 MHz**, which is lower by the net fraction 4.46 × 10⁻¹⁰. In orbit, the two relativistic effects speed the clock up by exactly that fraction. Seen from the ground, it then ticks at 10.23 MHz. + +**In the receiver.** The fixed offset assumes a perfectly circular orbit. Real orbits are slightly elliptical. As a satellite moves closer to the Earth and then farther away, its speed and gravity change, so its clock rate changes too. For an orbit with eccentricity 0.02, this swings the clock by up to about ±46 ns (±14 m). The amount differs for each satellite, so the fourth-satellite trick cannot absorb it. The receiver calculates this term from the broadcast orbit data and subtracts it. + +**From the ground.** Control stations track every satellite clock and upload small corrections for the remaining drift. Relativity is built into the model that these corrections assume. + +## 6. What if relativity were ignored? + +You often read that GPS would "drift 10 km per day" without relativity. That figure assumes the receiver trusts the satellite clocks. It is a fair measure of the size of the effect. It is not an exact forecast: + +- **The part common to all satellites** would partly cancel. It shifts every distance equally, so the receiver would absorb it into *b*, like its own clock error. The *time* the receiver reports would still drift by 38.5 µs per day. Power grids, phone networks, and financial exchanges use GPS time and would fail. +- **The parts that differ between satellites** would not cancel. Each satellite's eccentricity term alone produces errors of up to about 14 m. Satellite clocks would also drift away from GPS time, and from the ground stations that track them. + +Either way, GPS cannot meet its metre-level accuracy without relativity. + +--- + +## Summary + +1. The receiver measures signal travel time; time × *c* = distance. 1 ns = 30 cm. +2. Three distances fix a point. A fourth satellite removes the receiver's own clock error. +3. That trick removes only a shared error, so satellite clocks must agree to nanoseconds. +4. In orbit, speed slows the clock by 7.2 µs/day and weaker gravity speeds it up by 45.7 µs/day: net +38.5 µs/day, or 11.5 km of light travel. +5. GPS sets the clocks slow before launch and corrects the per-satellite orbit effect in the receiver. + +**Not covered here:** signal delays in the atmosphere, the Sagnac effect (the Earth rotates while the signal travels), how orbits are predicted, and centimetre-level methods such as RTK. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/model.md new file mode 100644 index 0000000..bbc70a0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/model.md @@ -0,0 +1,21 @@ +# Model: GPS and relativity + +**Central question.** How does a receiver find its position from satellite signals, and why must relativity be included? + +**Reader.** Technical, school physics and maths, basic programming. New to GPS and to relativity. + +**Entities.** Satellite (atomic clock, known position), signal (carries send time and orbit data), receiver (cheap quartz clock), GPS time (the shared time scale). + +**Causal chain.** +1. Distance = c × travel time. 1 ns of timing error = 30 cm of distance error. +2. One distance → sphere. Three spheres → a point (plus one point far in space). +3. Receiver clock is unknown → add unknown b → need four satellites. +4. Step 3 removes only an error common to all signals. Each satellite clock must agree with GPS time to ~ns. +5. Satellite clocks run at a different rate from ground clocks: −7.2 µs/day (speed, special relativity) + 45.7 µs/day (weaker gravity, general relativity) = +38.5 µs/day. +6. Fix: set the clock to 10.22999999543 MHz before launch; receiver adds a per-satellite eccentricity term (up to ~46 ns ≈ 14 m for e = 0.02). + +**Epistemic notes.** Both effects are established and measured. The popular "10 km/day of position error" assumes the receiver trusts the satellite clocks; a drift common to all satellites partly cancels in the four-unknown solve. Per-satellite differences and the time output do not cancel. + +**Confusion points.** "Doesn't the 4th satellite fix all clock errors?" — only the shared one. "Fast or slow?" — two effects with opposite sign; gravity wins. + +**Scope (out).** Atmospheric delay, Sagnac effect, ephemeris fitting, carrier-phase / RTK, other GNSS. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/model.yaml new file mode 100644 index 0000000..84b921f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/explainers/gps-relativity/model.yaml @@ -0,0 +1,22 @@ +c_m_per_s: 299792458 +light_per_ns_cm: 30 +earth_radius_km: 6371 +orbit_radius_km: 26560 +orbit_altitude_km: 20200 +orbit_speed_km_s: 3.87 +orbit_period_h: 11.97 +signal_travel_ms: [67, 86] +receiver_clock_error_example_ms: 1 +receiver_clock_error_example_km: 300 +special_relativity_fraction: 8.3e-11 +special_relativity_us_per_day: -7.2 +general_relativity_fraction: 5.29e-10 +general_relativity_us_per_day: 45.7 +net_fraction: 4.46e-10 +net_us_per_day: 38.5 +net_range_km_per_day: 11.5 +nominal_clock_mhz: 10.23 +factory_clock_mhz: 10.22999999543 +eccentricity_example: 0.02 +eccentricity_term_ns: 46 +eccentricity_term_m: 14 diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/read.json new file mode 100644 index 0000000..95a8366 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "The receiver measures the time radio signals take to arrive from each satellite. It multiplies this time by the speed of light (about 300,000 km/s) to calculate the distance to each satellite.", + "quote": "The receiver measures how long each signal took to arrive. Then it multiplies that time by the speed of light to get a distance." + }, + "2": { + "answer": "Three distances alone can locate a position (spheres intersect at a point near Earth), but the receiver's cheap quartz clock has errors that shift all measured distances equally, preventing a valid solution. The fourth satellite lets the receiver solve for the clock error as an additional unknown variable.", + "quote": "GPS uses a better trick: treat the receiver clock error *b* as a fourth unknown. There are four unknowns (*x, y, z, b*), so the receiver needs four satellites." + }, + "3": { + "answer": "Without relativity corrections, the satellite clocks would run about 38.5 microseconds per day faster than ground clocks. This corresponds to about 11.5 kilometers of distance error per day, far exceeding the nanosecond-level accuracy budget required for GPS.", + "quote": "38.5 µs × 30 cm/ns ≈ **11.5 km**. That is the distance error that one day of uncorrected drift represents." + }, + "4": { + "answer": "A 1 microsecond clock error produces a distance error of approximately 300 meters. Since the error is the same for all satellites, all three measured distances would be wrong by about 300 m.", + "quote": "A timing error of 1 microsecond (1,000 ns) gives 300 m." + } + }, + "stops": [ + "For an orbit with eccentricity 0.02, this swings the clock by up to about ±46 ns", + "the difference in gravitational potential divided by *c²*", + "the clock loses a fraction *v²/2c²* of its time", + "The drift is about 0.45 ns per second, so it uses up that budget in about 10 seconds", + "(*GM/c²* for the Earth is a length, about 4.43 mm", + "The term *c·b* removes only an error that is the same for every signal", + "The receiver calculates this term from the broadcast orbit data and subtracts it", + "There are four unknowns (*x, y, z, b*), so the receiver needs four satellites", + "called **GPS time**, to within a few nanoseconds", + "A third sphere cuts that circle at two points. One point is near the Earth. The other is far out in space" + ], + "excess": [ + "A satellite near the horizon is farther away, so its signal takes up to about 86 ms", + "if you are 3 km from tower A, 4 km from tower B, and 5 km from tower C, only one spot on the map fits all three", + "Both are measured in experiments, not only predicted", + "GPS satellites orbit about 20,200 km above the ground", + "(*GM/c²* for the Earth is a length, about 4.43 mm", + "Control stations track every satellite clock and upload small corrections for the remaining drift", + "Power grids, phone networks, and financial exchanges use GPS time and would fail", + "signal delays in the atmosphere, the Sagnac effect (the Earth rotates while the signal travels), how orbits are predicted, and centimetre-level methods such as RTK", + "The nominal clock frequency is 10.23 MHz", + "Real orbits are slightly elliptical. As a satellite moves closer to the Earth and then farther away, its speed and gravity change, so its clock rate changes too" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/run.json new file mode 100644 index 0000000..a891a46 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I wrote `explainers/gps-relativity/explanation.md` as prose with worked numbers, because the subject is one causal chain (time → distance → position → clock agreement → relativity) and needs no diagram or interactive page; I also wrote `model.md` and `model.yaml` with the values, but I had no shell, so `explainer check` was not run and the numbers are hand-checked only.", + "error": null, + "cost": 0.28801780000000005, + "seconds": 89, + "turns": 6, + "files": [ + "explainers/gps-relativity/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/explanation.md new file mode 100644 index 0000000..ff7aaeb --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/explanation.md @@ -0,0 +1,75 @@ +# How does GPS find your position, and why does it need relativity? + +**A GPS receiver times radio signals from four satellites and solves for its position and its own clock error. This works only if the satellite clocks agree to a few billionths of a second. Relativity makes those clocks gain 38 µs a day on ground clocks. Light covers 11.4 km in 38 µs, so engineers build the clocks to tick slow.** + +## 1. Time gives distance + +About 30 GPS satellites orbit 20,200 km above Earth. Each one carries atomic clocks and broadcasts a signal that says: "I am satellite k, I am at position P, and I sent this at time t." Your receiver (the GPS chip in a phone) only listens. + +The receiver notes when each signal arrives. Then: + +- travel time = receive time − send time +- range = travel time × c, where c is light speed and the range is the true distance to the satellite. + +From a satellite directly overhead, the travel time is about 67 ms. Light covers 300 m in one microsecond (1 µs = 10⁻⁶ s) and 0.3 m in one nanosecond (1 ns = 10⁻⁹ s). So every timing error becomes a range error at that rate. + +One range puts you somewhere on a sphere around the satellite: every point at that distance. Three spheres meet in two points. The receiver rejects the one far out in space. Three satellites would be enough, if every clock were perfect. + +## 2. The receiver's clock is wrong, so it uses a fourth satellite + + +Your phone's clock is quartz, not atomic. It can easily be off by 1 ms, and 1 ms × c = 300 km. + + +So the receiver does not compute a range. It computes a **pseudorange**: light speed × (receive time on the receiver clock − send time on the satellite clock). A pseudorange equals the range plus **b**, the receiver clock error expressed in metres. One clock makes every measurement, so b is the same in every pseudorange. That makes b one more unknown to solve for, next to the position. + + +**Worked example, on a line.** Put beacon A at 0 km and beacon B at 30 km. The receiver is at 12 km, so the true travel times are 40 µs and 60 µs. The receiver clock is 5 µs fast, so it measures 45 µs and 65 µs. Those give pseudoranges of 13.5 km and 19.5 km. + +The receiver lies between the beacons, so the two ranges must add up to 30 km. The pseudoranges add up to 33 km. The excess, 3 km, is b counted twice, so b = 1.5 km (5 µs × c). Subtract b from each: 12 km and 18 km. The receiver is at 12 km. With only beacon A, the receiver could not detect the 1.5 km error. + +In three dimensions the unknowns are x, y, z, and b. Four unknowns need four equations, so the receiver needs four satellites. +## 3. The satellite clocks must be almost perfect + + +The receiver can solve for its own clock error because every pseudorange shares it. A satellite clock error sits in only one pseudorange, so no unknown absorbs it. Each satellite clock must therefore match **GPS time**, the time kept by atomic clocks at ground stations. + +How close? Each nanosecond of satellite clock error is 0.3 m of range error. For a fix within about 5 m, the satellite clocks must agree with GPS time to about 17 ns. + +## 4. Relativity moves the satellite clocks by microseconds a day + + +**Time dilation** means that clocks in different motion, or at different heights in gravity, tick at different rates. It is measured, not only predicted. Two effects act on a satellite clock, compared with a clock on the ground: + +| Cause | Theory | Effect on the satellite clock | +|---|---|---| +| Speed: 3.9 km/s in orbit | special relativity: a moving clock ticks slower | loses 7 µs per day | +| Height: gravity is weaker 20,200 km up | general relativity: a clock higher above Earth ticks faster | gains 45 µs per day | +| **Net** | | **gains 38 µs per day** | + +The gravity effect is larger, so the clock runs fast. 38 µs is only 4.5 × 10⁻¹⁰ of a day, but light covers 11.4 km in 38 µs. Each uncorrected satellite clock would move 11.4 km of range further from GPS time every day. + +## 5. Why the receiver's b does not cancel the drift + + +Every satellite clock drifts at the same rate. Why does b not absorb the drift? + +Only if every clock had the same error at the same moment. If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right. + +But each satellite clock is set to GPS time at a different moment. Suppose satellite A is set on day 0 and satellite B on day 3. On day 5, A is 190 µs fast (57 km) and B is 76 µs fast (22.8 km). The 34 km difference is not shared, so no single b can absorb it, and the position fix fails. Every clock set on a different day carries a different error. + +## 6. The remedy: make the clocks tick slow before launch + + +Each satellite clock counts the ticks of a 10.23 MHz signal. Before launch, engineers set it to 10.22999999543 MHz: slower by 4.5 × 10⁻¹⁰, the fraction the clock gains in orbit. On the ground the clock runs slow. In orbit, relativity speeds it up to the rate of GPS time. Ground stations upload small corrections for what remains; the built-in offset keeps those corrections small instead of growing by 38 µs a day. + +## Summary + +Four pseudoranges give x, y, z, and the receiver's clock error, but only if satellite clocks stay within about 17 ns of GPS time. Relativity makes them gain 38 µs a day, so each clock is built to tick slow. + +## Not covered here + +- **Atmosphere:** charged and humid layers of the atmosphere slow the signal. These delays are the largest remaining errors. +- **Earth's rotation during the 67 ms flight (Sagnac effect)** and the **orbit-shape correction** (orbits are slightly elliptical, so each satellite clock error swings by up to about 23 ns): receivers correct both. See N. Ashby, "Relativity in the Global Positioning System", *Living Reviews in Relativity* (2003). +- **Why a moving clock ticks slow:** see the "light clock" argument in special relativity. +- **How the receiver solves the equations:** iteration from a guess, with a best fit when it hears more than four satellites. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/model.md new file mode 100644 index 0000000..98a06cc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/model.md @@ -0,0 +1,141 @@ +# Semantic model: GPS positioning and relativity + +> Source of truth. Every rendering derives from this file. + +## Central question + +How does a GPS receiver find its position from satellite signals, and why must the system correct for relativity? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (speed = distance / time, the speed of light, gravity, solving simultaneous equations), basic programming. New to GPS and to relativity. Knows that light speed is about 300,000 km/s. Does not know what a pseudorange is, why four satellites are needed, or what time dilation is in numbers. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Satellite | One of about 30 GPS satellites, orbiting at about 20,200 km altitude, each carrying atomic clocks. | fact | +| Satellite clock | An atomic clock on a satellite; it stamps each signal with its send time. | fact | +| Signal | A radio message from a satellite that says "I am satellite k, at position P, and I sent this at time t". | fact (simplified) | +| Receiver | The GPS chip in a phone or car; it only listens, it never transmits. | fact | +| Receiver clock | A cheap quartz clock in the receiver; its error is not known in advance. | fact | +| Range | The true distance from a satellite to the receiver. | definition | +| Pseudorange | Light speed × (receive time on the receiver clock − send time on the satellite clock): a range plus an unknown error from the receiver clock. | definition | +| Clock bias | b: how far the receiver clock is ahead of GPS time, times light speed (in metres). | definition | +| GPS time | The reference time kept by atomic clocks at ground stations; satellite clocks are steered to it. | definition | +| Time dilation | Clocks in different motion or at different heights in gravity tick at different rates. A moving clock ticks slower than a clock at rest, by a fraction ≈ v²/2c². A clock higher above Earth ticks faster than one on the ground. | established fact (measured: Pound–Rebka 1959, Hafele–Keating 1971, GPS clocks) | +| Rate offset | The built-in setting that makes each satellite clock tick slow before launch: 10.23 MHz → 10.22999999543 MHz. 10.23 MHz is the nominal tick frequency of the satellite clock; the signal is derived from it. | fact | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| Signal travel time | times light speed gives | range | 1 µs ↔ 300 m | +| Receiver clock bias | adds the same error b to | every pseudorange | same instant, same clock | +| Four pseudoranges | determine | x, y, z, b | four equations, four unknowns | +| Orbit speed | slows | satellite clock | special relativity, −7 µs/day | +| Weaker gravity at altitude | speeds up | satellite clock | general relativity, +45 µs/day | +| Satellite clock rate error | grows into | range error | 38 µs/day ↔ 11 km/day | + +## Dependencies (learning order) + +1. Time of flight gives distance (range). +2. Several ranges fix a point (intersection of spheres). +3. The receiver clock is wrong, so each range is a pseudorange; add b as a fourth unknown, use four satellites. +4. The method needs satellite clocks that agree with ground time to nanoseconds. +5. Relativity makes satellite clocks run 38 µs/day fast; the system corrects it. + +## Causal chain + +```text +satellite sends time stamp → receiver measures travel time → × c → pseudorange +4 pseudoranges → solve x, y, z, b → position (and exact time) +satellite clock error of 1 ns → range error of 0.3 m +orbit: speed (−7 µs/day) + weaker gravity (+45 µs/day) → +38 µs/day → 11 km/day of range error if not corrected +→ rate offset: clocks set to tick slow before launch (10.23 MHz → 10.22999999543 MHz) +``` + +Why the drift would not cancel: b absorbs only an error that is the same in every pseudorange. All satellite clocks drift at the same rate, but each one was last set to GPS time on a different day. Satellite A set on day 0 and satellite B set on day 3: on day 5, A is 190 µs (57 km) off and B is 76 µs (22.8 km) off. The 34 km difference cannot go into one b. If all clocks were off by exactly the same 38 µs, b would absorb it: the position would be right and only the reported time would be wrong. + +## Temporal sequence + +1. Before launch: engineers set each satellite clock slightly slow. +2. In orbit: each satellite broadcasts its position and send time continuously. +3. The receiver records the receive time of signals from 4+ satellites. +4. The receiver computes pseudoranges and solves for x, y, z, b. + +## Quantities + +- Light speed c = 299,792,458 m/s ≈ 0.3 m per nanosecond ≈ 300 m per microsecond. +- Orbit radius ≈ 26,560 km (altitude ≈ 20,200 km); orbit speed ≈ 3.9 km/s. +- Accuracy target: a position error of about 5 m needs satellite clocks within about 17 ns of GPS time (5 m / 0.3 m per ns). 1 µs gives 300 m. +- Signal travel time ≈ 67 ms from directly overhead. +- Special relativity (speed): satellite clock loses ≈ 7 µs/day (v²/2c² ≈ 8.3 × 10⁻¹¹). +- General relativity (gravity): satellite clock gains ≈ 45 µs/day (GM(1/R_earth − 1/r)/c² ≈ 5.3 × 10⁻¹⁰). +- Net: gains ≈ 38 µs/day (fractional rate ≈ 4.465 × 10⁻¹⁰). 38 µs × c ≈ 11.4 km. +- Factory offset: 10.23 MHz × (1 − 4.465 × 10⁻¹⁰) = 10.22999999543 MHz. +- Receiver quartz clock: easily off by 1 ms → 300 km. + +## Alternative states + +| State | What changes | Why | +|---|---|---| +| Receiver clock perfect | 3 satellites suffice | no b to solve | +| Receiver clock off by 5 µs, 3D, only 3 satellites | position wrong by km, no way to detect | 4 unknowns, 3 equations | +| No relativity correction | ranges drift ≈ 11 km per day | clocks gain 38 µs/day | +| Only the speed effect corrected | clocks gain 45 µs/day → 13.5 km/day | the speed correction cancels −7 µs/day; the +45 µs/day gravity gain remains | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Solve for receiver clock bias b as a fourth unknown | Trust the receiver clock | Quartz clock off by 1 ms → every range off by 300 km. Even 1 µs → 300 m. | +| Use 4 satellites | Use 3 (three spheres meet at two points; the one far from Earth is rejected) | With b unknown, 3 equations, 4 unknowns: any b gives a different point. | +| Pre-set satellite clocks slow | Let them run at the nominal 10.23 MHz | They gain 38 µs/day; ranges drift 11.4 km per day. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here (numbers) | +|---|---|---| +| Inconsistent pseudoranges reveal the clock bias (1D worked example) | Beacons at 0 and 30 km, receiver at 12 km. True travel times 40 µs and 60 µs. Receiver clock 5 µs fast: measured 45 and 65 µs → pseudoranges 13.5 and 19.5 km. Sum 33 km ≠ 30 km → excess 3 km = 2b → b = 1.5 km = 5 µs. Subtract: 12 and 18 km → x = 12 km. (Assumes the receiver lies between the beacons, as satellites surround a receiver.) | With one beacon only: 13.5 km, no check → position off by 1.5 km, undetected. | +| Satellite clock error becomes range error | 1 ns → 0.3 m | 38 µs → 11.4 km | +| b absorbs a shared error only | all clocks 38 µs off → b absorbs 11.4 km, position right | clocks set on day 0 and day 3, read on day 5: 57 km vs 22.8 km, 34 km difference survives | +| The two relativity effects oppose | at GPS altitude: −7 + 45 = +38 µs/day | — | + +## Terms + +- **Range:** true distance satellite–receiver. +- **Pseudorange:** measured travel time × c, using the receiver clock; equals range + b. Never call it "distance". +- **Clock bias (b):** receiver clock error expressed in metres (error in seconds × c). 5 µs ↔ 1.5 km. Low b (1 ns) → 0.3 m; high (1 ms) → 300 km. +- **GPS time:** the reference time kept by ground stations; satellite clocks are steered to it. +- **Time dilation:** the rate difference between clocks. 38 µs/day is a fraction 4.5 × 10⁻¹⁰: tiny as a fraction, large as distance. + +## Epistemic status + +- **Established:** both relativistic effects are measured on GPS clocks and match theory; the clock offset has been built in since the first satellites. +- **Mathematical consequence:** 11.4 km/day = 38 µs/day × c. +- **Simplification:** the "11 km/day position error" is the growth of the range error if satellite clocks were left uncorrected against GPS time. b absorbs the part shared equally by all satellites, so the position error comes from how the clock errors differ between satellites; that difference grows by tens of km within days. The ground stations also upload small clock corrections. + +## Confusion points + +- "Relativity is about the satellite's speed, so the clock runs slow" → gravity effect is larger and opposite; net is fast. +- "GPS needs 3 satellites" → 3 is enough only with a perfect receiver clock; real receivers need 4. +- "The phone sends a signal to the satellite" → receiver only listens. +- "The receiver clock bias would absorb the relativity drift" → only the part shared equally by all satellites; clocks set on different days carry different errors. +- "38 µs is negligible" → light goes 11.4 km in 38 µs. + +## Scope + +- **Out of scope:** atmospheric delays (ionosphere, troposphere) — the largest remaining error sources; dual-frequency receivers remove most of the ionospheric part. +- **Out of scope:** Sagnac effect (Earth rotates during the 67 ms flight) — receivers correct it; see Ashby, "Relativity in the Global Positioning System", Living Reviews in Relativity (2003). +- **Out of scope:** the eccentricity correction (the receiver removes a periodic clock error up to about 23 ns, 7 m, from slightly elliptical orbits) and the forms of the time-dilation formulas — Ashby (2003). +- **Out of scope:** more than four satellites (a least-squares fit) and satellite geometry (bunched satellites amplify errors). +- **Out of scope:** why one-way ranging (unlimited users, receivers never transmit) and why atomic clocks on satellites. +- **Out of scope:** how the receiver solves the nonlinear equations (iterative least squares from a guess). +- **Out of scope:** satellite orbit data (ephemeris) and how the ground segment computes it. + +## Representation decision + +- **Stage:** 1 +- **Reason:** the subject is one sequential causal argument with worked numbers; there is no parameter space a reader must explore, and the 1D worked example fits in text. +- **Levels:** L1 answer / L2 positioning with pseudoranges / L3 why clocks must agree / L4 relativity numbers and the fix. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/model.yaml new file mode 100644 index 0000000..a86253f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/model.yaml @@ -0,0 +1,117 @@ +# Machine-readable half of model.md. `explainer check gps-relativity` holds every rendering to it. + +values: + light: + m_per_ns: 0.3 + m_per_us: 300 + km_per_us: 0.3 + orbit: + altitude_km: 20200 + speed_km_s: 3.9 + travel_ms: 67 + satellites: 30 + accuracy: + position_m: 5 + clock_ns: 17 + quartz: + error_ms: 1 + error_km: 300 + example1d: + beacon_b_km: 30 + receiver_km: 12 + other_km: 18 + t_a_us: 40 + t_b_us: 60 + bias_us: 5 + meas_a_us: 45 + meas_b_us: 65 + pr_a_km: 13.5 + pr_b_km: 19.5 + sum_km: 33 + excess_km: 3 + bias_km: 1.5 + relativity: + speed_us_day: 7 + gravity_us_day: 45 + net_us_day: 38 + net_km_day: 11.4 + speed_only_km_day: 13.5 + fraction: 4.5e-10 + ecc_ns: 23 + nominal_mhz: 10.23 + offset_mhz: 10.22999999543 + reset: + day_a: 0 + day_b: 3 + day_read: 5 + err_a_us: 190 + err_a_km: 57 + err_b_us: 76 + err_b_km: 22.8 + diff_km: 34 + +allow: [1, 2, 3, 4, 10] + +require: + - path: relativity/net_us_day + in: [prose] + - path: relativity/net_km_day + in: [prose] + - path: relativity/speed_us_day + in: [prose] + - path: relativity/gravity_us_day + in: [prose] + - path: example1d/bias_km + in: [prose] + +claims: + - id: def-pseudorange + kind: definition + statement: "A pseudorange is light speed times (receive time on the receiver clock minus send time on the satellite clock): the range plus the receiver clock bias b." + - id: why-fourth-unknown + kind: why + statement: "The receiver treats its own clock error b as a fourth unknown, so it needs four satellites." + alternative: "Trust the receiver's quartz clock and use three satellites." + counterexample: "A quartz clock off by 1 ms puts every range off by 300 km; even 1 µs gives 300 m." + ask: "Why does a GPS receiver need signals from four satellites and not three?" + - id: mech-bias-1d + kind: mechanism + statement: "Pseudoranges that do not fit together reveal the clock bias." + example: "Beacons 30 km apart, receiver at 12 km, clock 5 µs fast: pseudoranges 13.5 and 19.5 km sum to 33 km, 3 km too much, so b = 1.5 km and x = 12 km." + - id: mech-clock-to-range + kind: mechanism + statement: "Every nanosecond of satellite clock error becomes 0.3 m of range error." + example: "5 m accuracy needs satellite clocks within about 17 ns; 38 µs gives 11.4 km." + - id: mech-two-effects + kind: mechanism + statement: "The orbit speed slows the satellite clock by 7 µs/day; the weaker gravity at altitude speeds it up by 45 µs/day; net, the clock gains 38 µs/day." + example: "−7 + 45 = +38 µs/day; 38 µs × 300 m/µs = 11.4 km per day." + ask: "Do GPS satellite clocks run fast or slow compared with clocks on the ground, and why?" + - id: guar-bias-absorbs-shared + kind: guarantee + statement: "b absorbs only an error shared equally by all satellites, so uncorrected relativistic drift still ruins positions." + example: "If every clock were off by exactly 38 µs, b would absorb 11.4 km and the position would be right." + counterexample: "Clocks set on day 0 and day 3, read on day 5: 57 km and 22.8 km off; the 34 km difference cannot go into one b." + ask: "The receiver already solves for a clock error. Why can that not soak up the relativity drift of the satellite clocks?" + - id: fix-rate-offset + kind: mechanism + statement: "Engineers set each satellite clock to tick slow before launch, so that in orbit it matches GPS time." + example: "10.23 MHz is set to 10.22999999543 MHz, a fraction 4.5 × 10⁻¹⁰ slower." + +terms: + - term: pseudorange + means: "measured travel time × c using the receiver clock; 'range' means only the true distance" + avoid: [measured distance, true pseudorange] + +budget: {prose: 1000, reason: "two mechanisms (positioning with clock bias, relativity) and one guarantee whose failing case needs its own numbers"} + +quiz: + - q: "A receiver's quartz clock is 2 µs fast. Roughly how wrong is each pseudorange, and does that make the final position wrong by that much?" + expect: "Each pseudorange is about 600 m too long, but the same error is in every pseudorange, so the receiver solves for it as b and the position comes out right." + misconception: "The position is 600 m wrong." + - q: "If engineers had corrected only for the satellites' speed and not for gravity, would the clocks run fast or slow, and by how much per day?" + expect: "Fast, by about 45 µs per day (about 13.5 km of range per day), because the gravity effect remains." + misconception: "Slow, because moving clocks run slow." + - q: "Why does a 38 µs per day clock drift matter so much for positioning?" + expect: "Signals travel at light speed, so 38 µs of clock error is about 11.4 km of range error, growing every day." + misconception: "38 µs is too small to matter." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/narrative.md new file mode 100644 index 0000000..eb5db07 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/narrative.md @@ -0,0 +1,52 @@ +# Narrative: gps-relativity + +## 1. Reader before and after + +- **Before:** a technical reader: school mathematics and science (distance = speed × time, light speed ≈ 300,000 km/s, gravity, simultaneous equations), basic programming. New to GPS and to relativity. +- **After:** "A GPS receiver times signals from four satellites, turns each time into a distance, and solves for its position and its own clock error. This works only if the satellite clocks agree to nanoseconds, and relativity makes them gain 38 µs a day — 11 km of range — so they are set to tick slow before launch." + +## 2. Question and motive + +- **Question:** How does GPS find your position, and why does it need relativity? Result in words: the receiver measures how long signals take to arrive from four satellites; relativity makes the satellite clocks run fast by 38 µs a day, which would add about 11 km of range error per day, so the clocks are built to run slow. +- **Why care:** a phone with a cheap clock finds itself to a few metres, and the everyday device depends on Einstein's theory. +- **Why this approach:** distance = speed × time; the signal speed is known exactly (light speed), so measure time. + +## 3. Introduction ledger + +| Reference | Means | Grounded by | Beat | +|---|---|---|---| +| range | true distance from a satellite to the receiver | 67 ms of travel × light speed ≈ 20,200 km | 2 | +| pseudorange | travel time measured with the receiver clock × light speed; range + b | 1D example: 13.5 km where the range is 12 km | 4 | +| clock bias b | receiver clock error, in metres | 5 µs → 1.5 km | 4 | +| GPS time | reference time from atomic clocks at ground stations | satellites steered to it | 5 | +| time dilation | clocks in different motion or height tick at different rates | −7 and +45 µs/day | 6 | +| rate offset | satellite clock set to tick slow before launch | 10.23 → 10.22999999543 MHz | 8 | +| µs, ns | microsecond (10⁻⁶ s), nanosecond (10⁻⁹ s) | 1 µs ↔ 300 m, 1 ns ↔ 0.3 m | 2 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How does GPS work, and where does relativity come in? | — | answer paragraph (the result of §2 in words) | "The receiver times signals from four satellites and solves for its position and its own clock error. Relativity makes satellite clocks gain 38 µs a day, about 11 km of range, so they are built to tick slow." | the whole answer in outline | +| 2 | mechanism | How can a satellite tell you how far away it is? | so | the signal carries the satellite's position and its send time (the time stamp) | "Travel time = receive time − send time. Range = travel time × light speed c. From overhead, 67 ms × c ≈ 20,200 km. Light goes 300 m in 1 µs and 0.3 m in 1 ns, so a timing error becomes a range error at that rate." | timing gives range; timing must be precise | +| 3 | mechanism | How many ranges fix a point? | therefore | spheres | "One range puts the receiver somewhere on a sphere around the satellite (every point at that distance). Three spheres meet in two points; one is far from Earth and is rejected." | 3 satellites if all clocks were perfect | +| 4 | objection + instance | But my phone's clock is quartz, not atomic. | but | 1D worked example: beacons 0 and 30 km, receiver at 12 km, clock 5 µs fast; 13.5 + 19.5 = 33 km ≠ 30 km | "The receiver measures travel time with its own clock, so each result is a pseudorange: range + b, where b is the receiver clock error in metres (5 µs → 1.5 km). One clock makes every measurement, so b is the same in all of them. The excess 3 km = 2b gives b = 1.5 km. In 3D, b is a fourth unknown next to x, y, z, so the receiver needs a fourth satellite: four equations, four unknowns." | pseudorange, b, why four satellites | +| 5 | mechanism | So what accuracy do the satellite clocks need? | but | 17 ns ↔ 5 m | "The receiver can solve for its own clock error, because it is shared. Each satellite clock error is different and goes into only one pseudorange, so the receiver cannot solve for it. Satellite clocks must match GPS time — the time of atomic clocks at ground stations. For a 5 m fix (a typical phone), the error must be under about 17 ns." | satellite clocks must agree to ~10 ns | +| 6 | mechanism | What does relativity do to those clocks? | so | −7, +45, +38 µs/day vs ground clocks | "Time dilation: clocks in different motion or height tick at different rates (measured, not only predicted). The speed of 3.9 km/s makes a satellite clock lose 7 µs a day. The weaker gravity 20,200 km up makes it gain 45 µs a day. Net: it gains 38 µs a day, which is 11.4 km of range per day." | net fast 38 µs/day = 11.4 km/day | +| 7 | objection | Doesn't b absorb it, if every satellite drifts the same? | but | satellite A set to GPS time on day 0, B on day 3, read on day 5: 190 µs (57 km) vs 76 µs (22.8 km) | "b absorbs only an error that is equal in every pseudorange. The drift rate is equal, but each clock was set on a different day, so the accumulated errors differ: 34 km that no single b can absorb." | why the drift still ruins fixes | +| 8 | close | So what does GPS do? | therefore | 10.23 MHz → 10.22999999543 MHz | "Each satellite clock counts ticks of a 10.23 MHz signal. Before launch, engineers set it 4.5 × 10⁻¹⁰ slower — exactly the 38 µs/day gain as a fraction — so in orbit it ticks at the ground rate. Ground stations upload small remaining corrections." | the fix | +| 9 | recap | — | — | — | one-sentence answer; Scope list: atmosphere, Earth's rotation during flight, orbit-shape correction, more than four satellites | after-sentence | + +## 5. Concrete to symbol + +Pseudorange = range + b shown in the 1D example (13.5 = 12 + 1.5) before the general equation. + +## 6. Links between representations + +Times and distances always paired: µs with km via 0.3 km per µs. + +## 7. Close the loop + +- **Answer:** four time measurements give position and receiver clock error; relativity's 38 µs/day would add 11 km/day of range error, so satellite clocks tick slow by design. +- **Payoff:** the reader can turn any timing error into a range error, and predict which effect wins. +- **Scope:** atmosphere, Sagnac, eccentricity correction, least squares with more satellites. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/decisions.yaml new file mode 100644 index 0000000..3c9df8d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/decisions.yaml @@ -0,0 +1,185 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T145628-probe#1: + decision: adopted +2026-10-04T145628-probe#10: + decision: declined +2026-10-04T145628-probe#11: + decision: declined +2026-10-04T145628-probe#12: + decision: declined +2026-10-04T145628-probe#13: + decision: adopted +2026-10-04T145628-probe#14: + decision: adopted +2026-10-04T145628-probe#15: + decision: adopted +2026-10-04T145628-probe#16: + decision: adopted +2026-10-04T145628-probe#2: + decision: adopted +2026-10-04T145628-probe#3: + decision: adopted +2026-10-04T145628-probe#4: + decision: adopted +2026-10-04T145628-probe#5: + decision: declined +2026-10-04T145628-probe#6: + decision: adopted +2026-10-04T145628-probe#7: + decision: adopted +2026-10-04T145628-probe#8: + decision: adopted +2026-10-04T145628-probe#9: + decision: adopted +2026-10-04T145834-coldread-narrative#1: + decision: adopted +2026-10-04T145834-coldread-narrative#10: + decision: adopted +2026-10-04T145834-coldread-narrative#11: + decision: adopted +2026-10-04T145834-coldread-narrative#12: + decision: adopted +2026-10-04T145834-coldread-narrative#13: + decision: adopted +2026-10-04T145834-coldread-narrative#14: + decision: adopted +2026-10-04T145834-coldread-narrative#15: + decision: adopted +2026-10-04T145834-coldread-narrative#16: + decision: adopted +2026-10-04T145834-coldread-narrative#17: + decision: adopted +2026-10-04T145834-coldread-narrative#18: + decision: adopted +2026-10-04T145834-coldread-narrative#19: + decision: adopted +2026-10-04T145834-coldread-narrative#2: + decision: adopted +2026-10-04T145834-coldread-narrative#20: + decision: adopted +2026-10-04T145834-coldread-narrative#21: + decision: adopted +2026-10-04T145834-coldread-narrative#22: + decision: adopted +2026-10-04T145834-coldread-narrative#23: + decision: adopted +2026-10-04T145834-coldread-narrative#24: + decision: adopted +2026-10-04T145834-coldread-narrative#25: + decision: adopted +2026-10-04T145834-coldread-narrative#26: + decision: adopted +2026-10-04T145834-coldread-narrative#27: + decision: adopted +2026-10-04T145834-coldread-narrative#28: + decision: declined +2026-10-04T145834-coldread-narrative#29: + decision: adopted +2026-10-04T145834-coldread-narrative#3: + decision: adopted +2026-10-04T145834-coldread-narrative#30: + decision: adopted +2026-10-04T145834-coldread-narrative#31: + decision: adopted +2026-10-04T145834-coldread-narrative#32: + decision: adopted +2026-10-04T145834-coldread-narrative#33: + decision: declined +2026-10-04T145834-coldread-narrative#34: + decision: adopted +2026-10-04T145834-coldread-narrative#35: + decision: adopted +2026-10-04T145834-coldread-narrative#36: + decision: adopted +2026-10-04T145834-coldread-narrative#37: + decision: adopted +2026-10-04T145834-coldread-narrative#4: + decision: adopted +2026-10-04T145834-coldread-narrative#5: + decision: declined +2026-10-04T145834-coldread-narrative#6: + decision: adopted +2026-10-04T145834-coldread-narrative#7: + decision: adopted +2026-10-04T145834-coldread-narrative#8: + decision: adopted +2026-10-04T145834-coldread-narrative#9: + decision: adopted +2026-10-04T150100-coldread-prose#1: + decision: adopted +2026-10-04T150100-coldread-prose#10: + decision: declined +2026-10-04T150100-coldread-prose#11: + decision: adopted +2026-10-04T150100-coldread-prose#12: + decision: adopted +2026-10-04T150100-coldread-prose#13: + decision: adopted +2026-10-04T150100-coldread-prose#14: + decision: adopted +2026-10-04T150100-coldread-prose#15: + decision: declined +2026-10-04T150100-coldread-prose#16: + decision: adopted +2026-10-04T150100-coldread-prose#2: + decision: adopted +2026-10-04T150100-coldread-prose#3: + decision: declined +2026-10-04T150100-coldread-prose#4: + decision: declined +2026-10-04T150100-coldread-prose#5: + decision: adopted +2026-10-04T150100-coldread-prose#6: + decision: adopted +2026-10-04T150100-coldread-prose#7: + decision: declined +2026-10-04T150100-coldread-prose#8: + decision: declined +2026-10-04T150100-coldread-prose#9: + decision: adopted +2026-10-04T150300-quiz-prose#1: + decision: declined +2026-10-04T150300-quiz-prose#10: + decision: declined +2026-10-04T150300-quiz-prose#11: + decision: declined +2026-10-04T150300-quiz-prose#12: + decision: declined +2026-10-04T150300-quiz-prose#13: + decision: declined +2026-10-04T150300-quiz-prose#14: + decision: declined +2026-10-04T150300-quiz-prose#15: + decision: declined +2026-10-04T150300-quiz-prose#16: + decision: declined +2026-10-04T150300-quiz-prose#17: + decision: declined +2026-10-04T150300-quiz-prose#18: + decision: declined +2026-10-04T150300-quiz-prose#19: + decision: adopted +2026-10-04T150300-quiz-prose#2: + decision: declined +2026-10-04T150300-quiz-prose#20: + decision: declined +2026-10-04T150300-quiz-prose#21: + decision: declined +2026-10-04T150300-quiz-prose#22: + decision: declined +2026-10-04T150300-quiz-prose#23: + decision: declined +2026-10-04T150300-quiz-prose#3: + decision: adopted +2026-10-04T150300-quiz-prose#4: + decision: adopted +2026-10-04T150300-quiz-prose#5: + decision: declined +2026-10-04T150300-quiz-prose#6: + decision: adopted +2026-10-04T150300-quiz-prose#7: + decision: declined +2026-10-04T150300-quiz-prose#8: + decision: declined +2026-10-04T150300-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145628-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145628-probe.json new file mode 100644 index 0000000..00da2a5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145628-probe.json @@ -0,0 +1,231 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "gps-relativity", + "at": "2026-10-04T14:56:28", + "model": "default", + "prompt_sha": "4e9abe4986eb", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "Why an uncorrected 38 µs/day drift would actually ruin positions, given that b absorbs any error shared by all satellites", + "why_it_matters": "The central 'why must it correct' answer depends on this. The file grants that b absorbs common error, then says the position error 'would still grow without limit' without saying why. All GPS satellites share roughly the same orbit radius, so the drift rate is the same for each, and a sharp reader will conclude that b cancels it. The step that makes per-satellite errors differ is missing.", + "suggested_addition": "Add a concrete instance where the drift is NOT common. Satellite clocks are synchronized and adjusted at different times. If satellite A was last set to GPS time on day 0 and satellite B on day 3, then on day 5 A is off by 5×38 = 190 µs (57 km) and B by 2×38 = 76 µs (22.8 km). The 34 km difference cannot be absorbed by one b. Also give the instance where it holds: if all clocks were off by exactly 38 µs, b would absorb 11.4 km, the position would come out right, and the time the receiver reports would be wrong by 38 µs." + }, + { + "rule": 4, + "severity": "main", + "about": "Dependency step 2 ('several ranges fix a point / intersection of spheres') has no worked example", + "why_it_matters": "The only worked example is 1D. The reader never sees ranges intersect in 2D or 3D, or why the count of unknowns sets the number of satellites.", + "suggested_addition": "2D example: beacons at (0,0) and (10,0) km with true ranges 5 and √65 ≈ 8.06 km. The two circles meet at (3,4) and (3,−4): two points. A third beacon at (0,10) with range √45 ≈ 6.71 km picks (3,4). State the 3D analogue: three spheres meet at two points, and one of them is usually far from Earth's surface and gets rejected." + }, + { + "rule": 2, + "severity": "main", + "about": "The claim 'four pseudoranges determine x, y, z, b' has no numeric instance and no breaking instance", + "why_it_matters": "This is the core positioning guarantee. Its assumptions are never stated: the geometry must not be degenerate, and the solution must be unique. The text also says 'three spheres meet at a point', but they meet at two.", + "suggested_addition": "Breaking cases: four satellites lying in one plane, or bunched close together, give a nearly singular system. For example, a timing error of 3 m that turns into 30 m of position error (geometry amplification ×10). Also fix 'three spheres meet at a point' to 'at two points'." + }, + { + "rule": 2, + "severity": "edge", + "about": "The 1D worked example silently assumes the receiver lies between the two beacons", + "why_it_matters": "If the receiver is outside the segment, e.g. at x = 40 km, the pseudoranges are 40+b and 10+b. Their difference is always 30 km, whatever x is, so x and b can't be separated. The claim 'inconsistent pseudoranges reveal the bias' breaks under an assumption the file never states.", + "suggested_addition": "State the assumption, and give the breaking case: receiver at 40 km with a 5 µs fast clock gives 41.5 and 11.5 km. Every (x, b) with x − b = 38.5 fits, so the position is undetermined. This links to why satellite geometry matters." + }, + { + "rule": 1, + "severity": "main", + "about": "Why the time-dilation formulas have these forms: v²/2c² and GM(1/R_earth − 1/r)/c²", + "why_it_matters": "The audience is new to relativity. Without the why, the formulas are unexplained magic. The natural school-physics alternative for the gravity term is g·h/c², and that gives the wrong answer.", + "suggested_addition": "Say that v²/2c² is the small-speed approximation of 1 − √(1 − v²/c²). At 3.87 km/s the two agree to about 10⁻²⁰. The natural alternative g·h/c² = 9.8 × 2.02×10⁷ / 8.99×10¹⁶ ≈ 2.2×10⁻⁹ (≈190 µs/day) overestimates about 4× because gravity weakens with height. The 1/r form accounts for that. Note too that the reference is a clock on the ground at R_earth, which is why R_earth appears." + }, + { + "rule": 3, + "severity": "main", + "about": "'Time dilation' is used with no intuition for why motion slows a clock and weaker gravity speeds it up", + "why_it_matters": "The audience 'does not know what time dilation is'. The file gives numbers but no reason for the signs. That is the very confusion point it lists, and readers may take the result on faith.", + "suggested_addition": "Give one sentence per effect. A moving clock ticks slower as seen from the ground, by a fraction of about v²/2c² (8.3×10⁻¹¹ at 3.9 km/s). A clock higher in a gravity well ticks faster than one lower down, by a fraction of about the potential difference/c² (5.3×10⁻¹⁰). Add one sentence noting that both effects have been confirmed independently, e.g. Hafele–Keating (1971) and Pound–Rebka (1959)." + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts are given two names, and some terms are used before they are defined", + "why_it_matters": "Readers may think these are different things.", + "suggested_addition": "Pick one name each: 'orbit-shape correction' vs 'eccentricity correction'; 'factory offset' vs 'pre-set slow' vs 'clock offset built in'; 'ground time' vs 'GPS time' (GPS time is used in the Clock bias entity before Terms defines it, and it is not in Entities); 'range error' vs 'position error' vs 'ranges drift' for the 11 km figure; 'time of flight' vs 'signal travel time'; 'time dilation' as both the phenomenon (Entities) and the rate difference (Terms). Also define '10.23 MHz' as the satellite clock's nominal tick frequency, from which the signal is derived." + }, + { + "rule": 5, + "severity": "edge", + "about": "'Nanoseconds' clock agreement (dependency 4) is not tied to an accuracy target", + "why_it_matters": "The reader can't tell why nanoseconds matter rather than microseconds.", + "suggested_addition": "State that a typical position error of about 5 m needs satellite clock errors under about 17 ns (5 m / 0.3 m per ns). Contrast with 1 µs, which gives 300 m: useless for driving directions." + }, + { + "rule": 2, + "severity": "edge", + "about": "The 'Only special relativity applied' row explains itself as 'wrong sign of the bigger effect', which is confusing", + "why_it_matters": "The reasoning step is implicit. If the SR correction makes the clock run faster to cancel −7 µs/day, the remaining error is +45 µs/day because GR goes uncorrected. 'Wrong sign' does not explain that.", + "suggested_addition": "Rewrite as: the SR correction speeds the clock by 7 µs/day, which cancels the −7. The +45 µs/day GR gain remains uncorrected, which gives 13.5 km/day of range drift." + }, + { + "rule": 1, + "severity": "edge", + "about": "Why one-way listening ranging rather than two-way (radar-like) ranging", + "why_it_matters": "Two-way ranging is the natural alternative, and it would remove the receiver clock bias altogether. The file says the receiver only listens but never says why that design was chosen.", + "suggested_addition": "With two-way ranging, every user transmits, so the number of users is limited and receivers become costly and detectable. One-way ranging serves unlimited users, and the price is the unknown b plus a fourth satellite." + }, + { + "rule": 1, + "severity": "edge", + "about": "Why atomic clocks on the satellites rather than quartz", + "why_it_matters": "The receiver gets away with quartz because b is solved for; the reader may ask why satellites can't do the same.", + "suggested_addition": "A good quartz clock drifts about 10⁻⁹ per day, roughly 86 µs/day, or 26 km. Satellite clocks can't be solved for by the user, so they need atomic stability of about 10⁻¹³, roughly 9 ns/day, or 2.6 m." + }, + { + "rule": 6, + "severity": "edge", + "about": "What happens with more than four satellites", + "why_it_matters": "Real receivers track 8–12 satellites. Having 5 or more equations for 4 unknowns turns the problem into a best fit and gives an error check. The file never mentions this.", + "suggested_addition": "With 8 satellites there are 8 equations and 4 unknowns. A least-squares fit averages down noise, and the leftover mismatch (e.g. residuals of a few metres) shows how much to trust the fix." + } + ], + "excess": [ + "Orbit period ≈ 12 h: not used in any argument", + "Eccentricity correction (23 ns ≈ 7 m): a secondary detail, not needed to answer why relativity must be corrected", + "'GPS time itself must stay right for the system's timing users': a side point unless it is used to justify the correction", + "ISS low-orbit counterexample: useful for rule 2 but marked as not rendered" + ], + "next_questions": [ + { + "q": "If every satellite clock gains the same 38 µs/day, why doesn't the receiver's b just absorb it?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why does moving fast slow a clock, and why does weaker gravity speed it up?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How accurate is GPS in practice, and what limits it once relativity is corrected?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "How does the receiver know where each satellite is at the moment it sent its signal?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "What does the receiver do with more than four satellites, and how does it actually solve the nonlinear equations?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.18053100000000002, + "seconds": 70 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "Why an uncorrected 38 µs/day drift would actually ruin positions, given that b absorbs any error shared by all satellites → Add a concrete instance where the drift is NOT common. Satellite clocks are synchronized and adjusted at different times. If satellite A was last set to GPS time on day 0 and satellite B on day 3, then on day 5 A is off by 5×38 = 190 µs (57 km) and B by 2×38 = 76 µs (22.8 km). The 34 km difference cannot be absorbed by one b. Also give the instance where it holds: if all clocks were off by exactly 38 µs, b would absorb 11.4 km, the position would come out right, and the time the receiver reports would be wrong by 38 µs." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 4", + "text": "Dependency step 2 ('several ranges fix a point / intersection of spheres') has no worked example → 2D example: beacons at (0,0) and (10,0) km with true ranges 5 and √65 ≈ 8.06 km. The two circles meet at (3,4) and (3,−4): two points. A third beacon at (0,10) with range √45 ≈ 6.71 km picks (3,4). State the 3D analogue: three spheres meet at two points, and one of them is usually far from Earth's surface and gets rejected." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 2", + "text": "The claim 'four pseudoranges determine x, y, z, b' has no numeric instance and no breaking instance → Breaking cases: four satellites lying in one plane, or bunched close together, give a nearly singular system. For example, a timing error of 3 m that turns into 30 m of position error (geometry amplification ×10). Also fix 'three spheres meet at a point' to 'at two points'." + }, + { + "n": 4, + "severity": "edge", + "kind": "rule 2", + "text": "The 1D worked example silently assumes the receiver lies between the two beacons → State the assumption, and give the breaking case: receiver at 40 km with a 5 µs fast clock gives 41.5 and 11.5 km. Every (x, b) with x − b = 38.5 fits, so the position is undetermined. This links to why satellite geometry matters." + }, + { + "n": 5, + "severity": "main", + "kind": "rule 1", + "text": "Why the time-dilation formulas have these forms: v²/2c² and GM(1/R_earth − 1/r)/c² → Say that v²/2c² is the small-speed approximation of 1 − √(1 − v²/c²). At 3.87 km/s the two agree to about 10⁻²⁰. The natural alternative g·h/c² = 9.8 × 2.02×10⁷ / 8.99×10¹⁶ ≈ 2.2×10⁻⁹ (≈190 µs/day) overestimates about 4× because gravity weakens with height. The 1/r form accounts for that. Note too that the reference is a clock on the ground at R_earth, which is why R_earth appears." + }, + { + "n": 6, + "severity": "main", + "kind": "rule 3", + "text": "'Time dilation' is used with no intuition for why motion slows a clock and weaker gravity speeds it up → Give one sentence per effect. A moving clock ticks slower as seen from the ground, by a fraction of about v²/2c² (8.3×10⁻¹¹ at 3.9 km/s). A clock higher in a gravity well ticks faster than one lower down, by a fraction of about the potential difference/c² (5.3×10⁻¹⁰). Add one sentence noting that both effects have been confirmed independently, e.g. Hafele–Keating (1971) and Pound–Rebka (1959)." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts are given two names, and some terms are used before they are defined → Pick one name each: 'orbit-shape correction' vs 'eccentricity correction'; 'factory offset' vs 'pre-set slow' vs 'clock offset built in'; 'ground time' vs 'GPS time' (GPS time is used in the Clock bias entity before Terms defines it, and it is not in Entities); 'range error' vs 'position error' vs 'ranges drift' for the 11 km figure; 'time of flight' vs 'signal travel time'; 'time dilation' as both the phenomenon (Entities) and the rate difference (Terms). Also define '10.23 MHz' as the satellite clock's nominal tick frequency, from which the signal is derived." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 5", + "text": "'Nanoseconds' clock agreement (dependency 4) is not tied to an accuracy target → State that a typical position error of about 5 m needs satellite clock errors under about 17 ns (5 m / 0.3 m per ns). Contrast with 1 µs, which gives 300 m: useless for driving directions." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 2", + "text": "The 'Only special relativity applied' row explains itself as 'wrong sign of the bigger effect', which is confusing → Rewrite as: the SR correction speeds the clock by 7 µs/day, which cancels the −7. The +45 µs/day GR gain remains uncorrected, which gives 13.5 km/day of range drift." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 1", + "text": "Why one-way listening ranging rather than two-way (radar-like) ranging → With two-way ranging, every user transmits, so the number of users is limited and receivers become costly and detectable. One-way ranging serves unlimited users, and the price is the unknown b plus a fourth satellite." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 1", + "text": "Why atomic clocks on the satellites rather than quartz → A good quartz clock drifts about 10⁻⁹ per day, roughly 86 µs/day, or 26 km. Satellite clocks can't be solved for by the user, so they need atomic stability of about 10⁻¹³, roughly 9 ns/day, or 2.6 m." + }, + { + "n": 12, + "severity": "edge", + "kind": "rule 6", + "text": "What happens with more than four satellites → With 8 satellites there are 8 equations and 4 unknowns. A least-squares fit averages down noise, and the leftover mismatch (e.g. residuals of a few metres) shows how much to trust the fix." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Orbit period ≈ 12 h: not used in any argument" + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Eccentricity correction (23 ns ≈ 7 m): a secondary detail, not needed to answer why relativity must be corrected" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "'GPS time itself must stay right for the system's timing users': a side point unless it is used to justify the correction" + }, + { + "n": 16, + "severity": "excess", + "kind": "model entry", + "text": "ISS low-orbit counterexample: useful for rule 2 but marked as not rendered" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145834-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145834-coldread-narrative.json new file mode 100644 index 0000000..c02ef3a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145834-coldread-narrative.json @@ -0,0 +1,401 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "gps-relativity", + "at": "2026-10-04T14:58:34", + "model": "default", + "prompt_sha": "c53fdd66ab92", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: answer paragraph. Said: the result in two sentences.", + "unresolved": [], + "unsaid": [ + "The table never gives the 'answer paragraph' or the 'two sentences', so the reader can't check what the result actually is." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (mechanism)", + "text": "Shown: signal content; 67 ms; 1 µs ↔ 300 m. Said: time stamp × c = range.", + "unresolved": [ + "time stamp", + "c" + ], + "unsaid": [ + "'signal content': the words never say what the signal carries.", + "'67 ms': no one says this is the travel time, or that it comes to about 20,200 km.", + "'1 µs ↔ 300 m': the words never say what this conversion is for." + ], + "leap": [ + "'time stamp × c = range': a time stamp is a clock reading, not a travel time. The step 'travel time = arrival time − time stamp' is never said, so the formula as written doesn't follow." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Shown: spheres. Said: 3 spheres meet in 2 points; one is far from Earth.", + "unresolved": [], + "unsaid": [], + "leap": [ + "Nobody says why a single range is a sphere (every point at that distance from the satellite)." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (objection + instance)", + "text": "Shown: 1D worked example. Said: bias b is the same for all; extra measurement reveals it.", + "unresolved": [ + "bias b", + "pseudorange (listed as something the reader now knows, but it is never said or shown)" + ], + "unsaid": [ + "The numbers in the 1D example (13.5 = 12 + 1.5) and the step 5 µs → 1.5 km never appear in Shown or Said.", + "'four satellites' is never said; the reader has to work out 3 + 1 alone." + ], + "leap": [ + "'the same for all': the reason (one receiver clock is used for every measurement) is not given.", + "'extra measurement reveals it': it isn't said that this works by adding a fourth unknown and a fourth equation (simultaneous equations)." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: 17 ns ↔ 5 m. Said: satellite clock errors are not solved; they must match GPS time.", + "unresolved": [ + "GPS time", + "ns" + ], + "unsaid": [ + "Nothing says why 5 m is the target, or where the ~10 ns requirement comes from." + ], + "leap": [ + "'satellite clock errors are not solved': the reader has just learned that a clock error can be solved for, and the reason the satellites' errors can't be (each satellite has its own error, not a shared one) only comes in beat 7." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Shown: −7, +45, +38 µs/day. Said: two effects, opposite signs.", + "unresolved": [ + "which 'two effects' (speed and gravity are never named)", + "time dilation (never said)" + ], + "unsaid": [ + "Nothing says which number belongs to which cause, or what the signs mean (slow or fast compared with ground clocks).", + "The 11.4 km/day figure (listed as something the reader now knows) is neither shown nor said." + ], + "leap": [ + "There is no reason at all for why a moving clock runs slow or a higher clock runs fast. Relativity, the topic named in the title, arrives as bare numbers." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (objection)", + "text": "Shown: day 0/3/5 example. Said: only shared error is absorbed.", + "unresolved": [], + "unsaid": [ + "The day 0/3/5 example isn't described, so the reader can't tell what it shows." + ], + "leap": [ + "If every satellite gains the same 38 µs/day, that looks like a shared error that b should absorb. The beat doesn't say why the drift is not shared." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (close)", + "text": "Shown: 10.22999999543 MHz. Said: rate offset; ground corrections.", + "unresolved": [ + "MHz", + "rate offset", + "ground corrections", + "what 10.23 MHz is (the clock's nominal frequency is never given)" + ], + "unsaid": [ + "The shown frequency is never linked to 38 µs/day. The fractional offset (about 4.5 × 10⁻¹⁰) is not explained." + ], + "leap": [ + "The table doesn't say how a slower tick rate cancels the relativistic gain." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (recap)", + "text": "Said: answer in the opening words; scope.", + "unresolved": [ + "'scope': the items left out are not listed in the beat" + ], + "unsaid": [], + "leap": [], + "excess": [ + "'answer in the opening words': this repeats beat 1 and gives no sign of saying anything new." + ], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 2: 'time stamp × c = range'. Travel time is never defined as arrival time minus the stamp, and 'c' is not introduced.", + "Beat 4: 'bias b is the same for all; extra measurement reveals it'. The bias b and 'pseudorange' are never defined, and the move to four satellites and four unknowns is not said.", + "Beat 6: 'two effects, opposite signs'. The effects are not named or explained. The relativity step, which is the core of the explanation, is only numbers.", + "Beat 7: 'only shared error is absorbed'. It doesn't say why the satellites' relativistic drift counts as unshared." + ], + "cuts": [ + "Beat 9: 'answer in the opening words' repeats beat 1. Merge it into beat 8's close or cut it to one sentence." + ], + "question": { + "text": "How does a GPS receiver find its position from satellite signals, and why does relativity have to be corrected for?", + "known_at": "Beat 1 (reader's question: 'How does GPS work, and where does relativity come in?')", + "result_stated": "Only as a reference. Beat 1 says 'the result in two sentences', but the beats never show the sentences themselves (they appear only in the author's plan, section 2)." + }, + "motive": "Missing from the beats. The plan's reasons to care ('a phone with a cheap clock finds itself to a few metres', and that an everyday device depends on Einstein's theory) never appear in Said or Shown. The reason for the approach is only implied in beat 2 ('time stamp × c = range'); the line 'signal speed is known exactly, so measure time' appears only in the plan.", + "close": "Partial. Beat 8 says 'rate offset; ground corrections' and beat 9 says 'answer in the opening words'. Neither states the general argument in words: four timings give position plus clock error, a 38 µs/day drift means about 11 km/day, so the clocks are set slow.", + "recap": "GPS finds you by timing signals from four satellites (the fourth one cancels your phone's clock error), and because relativity makes satellite clocks gain about 38 µs a day, they are built to tick slightly slow.", + "strongest": "Beat 4: answering 'my phone clock is not atomic' with a worked 1D example that shows the extra unknown being solved for.", + "weakest": "Beat 6: relativity, the topic named in the title, arrives as '−7, +45, +38 µs/day' and 'two effects, opposite signs', with no names and no reasons.", + "suggestions": [ + "In beat 2, say 'travel time = arrival time − time stamp' before multiplying by c, and define c as light speed.", + "In beat 6, name the effects: speed (special relativity, −7 µs/day) and weaker gravity at altitude (general relativity, +45 µs/day). Say the 11.4 km/day figure aloud.", + "In beat 7, explain why b can't absorb the drift. The satellite clocks drift away from GPS time, so the time stamps and the satellite positions computed from them become inconsistent, and the satellites' clock states differ.", + "In beat 8, link 10.23 → 10.22999999543 MHz to a fractional change of about 4.5 × 10⁻¹⁰, which is 38 µs per 86,400 s.", + "Put the 'why care' line (cheap phone clock, metre-level accuracy) into beat 1." + ] + }, + "usage": { + "cost": 0.11163419999999998, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'time stamp × c = range'. Travel time is never defined as arrival time minus the stamp, and 'c' is not introduced." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'bias b is the same for all; extra measurement reveals it'. The bias b and 'pseudorange' are never defined, and the move to four satellites and four unknowns is not said." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 6: 'two effects, opposite signs'. The effects are not named or explained. The relativity step, which is the core of the explanation, is only numbers." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 7: 'only shared error is absorbed'. It doesn't say why the satellites' relativistic drift counts as unshared." + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 9: 'answer in the opening words' repeats beat 1. Merge it into beat 8's close or cut it to one sentence." + }, + { + "n": 6, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): The table never gives the 'answer paragraph' or the 'two sentences', so the reader can't check what the result actually is." + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): time stamp" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): c" + }, + { + "n": 9, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): 'signal content': the words never say what the signal carries." + }, + { + "n": 10, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): '67 ms': no one says this is the travel time, or that it comes to about 20,200 km." + }, + { + "n": 11, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): '1 µs ↔ 300 m': the words never say what this conversion is for." + }, + { + "n": 12, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (mechanism): 'time stamp × c = range': a time stamp is a clock reading, not a travel time. The step 'travel time = arrival time − time stamp' is never said, so the formula as written doesn't follow." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): Nobody says why a single range is a sphere (every point at that distance from the satellite)." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (objection + instance): bias b" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (objection + instance): pseudorange (listed as something the reader now knows, but it is never said or shown)" + }, + { + "n": 16, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (objection + instance): The numbers in the 1D example (13.5 = 12 + 1.5) and the step 5 µs → 1.5 km never appear in Shown or Said." + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (objection + instance): 'four satellites' is never said; the reader has to work out 3 + 1 alone." + }, + { + "n": 18, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (objection + instance): 'the same for all': the reason (one receiver clock is used for every measurement) is not given." + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (objection + instance): 'extra measurement reveals it': it isn't said that this works by adding a fourth unknown and a fourth equation (simultaneous equations)." + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): GPS time" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): ns" + }, + { + "n": 22, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): Nothing says why 5 m is the target, or where the ~10 ns requirement comes from." + }, + { + "n": 23, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 'satellite clock errors are not solved': the reader has just learned that a clock error can be solved for, and the reason the satellites' errors can't be (each satellite has its own error, not a shared one) only comes in beat 7." + }, + { + "n": 24, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): which 'two effects' (speed and gravity are never named)" + }, + { + "n": 25, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): time dilation (never said)" + }, + { + "n": 26, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): Nothing says which number belongs to which cause, or what the signs mean (slow or fast compared with ground clocks)." + }, + { + "n": 27, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): The 11.4 km/day figure (listed as something the reader now knows) is neither shown nor said." + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): There is no reason at all for why a moving clock runs slow or a higher clock runs fast. Relativity, the topic named in the title, arrives as bare numbers." + }, + { + "n": 29, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (objection): The day 0/3/5 example isn't described, so the reader can't tell what it shows." + }, + { + "n": 30, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (objection): If every satellite gains the same 38 µs/day, that looks like a shared error that b should absorb. The beat doesn't say why the drift is not shared." + }, + { + "n": 31, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): MHz" + }, + { + "n": 32, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): rate offset" + }, + { + "n": 33, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): ground corrections" + }, + { + "n": 34, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): what 10.23 MHz is (the clock's nominal frequency is never given)" + }, + { + "n": 35, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (close): The shown frequency is never linked to 38 µs/day. The fractional offset (about 4.5 × 10⁻¹⁰) is not explained." + }, + { + "n": 36, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (close): The table doesn't say how a slower tick rate cancels the relativistic gain." + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (recap): 'scope': the items left out are not listed in the beat" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150100-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150100-coldread-prose.json new file mode 100644 index 0000000..9d33cd5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150100-coldread-prose.json @@ -0,0 +1,375 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "gps-relativity", + "at": "2026-10-04T15:01:00", + "model": "default", + "prompt_sha": "05a1f51134c7", + "reply": { + "paragraphs": [ + { + "at": "Title", + "text": "How does GPS find your position, and why does it need relativity?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Lead (bold paragraph)", + "text": "A GPS receiver times radio signals from four satellites and solves for its position and its own clock error. ... engineers build them to tick slow.", + "unresolved": [], + "leap": [ + "\"Relativity makes those clocks gain 38 µs a day\": the reader is told this here and has to wait until §4 for the reason. That is fine in a summary.", + "\"which would add 11.4 km of range error per day\": this reads as if the error is the clock drift itself. §5 later says a drift shared by all clocks would be absorbed, so this framing needs revising once the reader gets there." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 para 1", + "text": "About 30 GPS satellites orbit 20,200 km above Earth. Each one carries atomic clocks and broadcasts ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 para 2 + list", + "text": "The receiver notes when each signal arrives. Then: travel time = receive time − send time; range = travel time × c ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 para 3", + "text": "From a satellite directly overhead, the travel time is about 67 ms. Light covers 300 m in one microsecond ...", + "unresolved": [], + "leap": [], + "excess": [ + "\"(1 µs = 10⁻⁶ s)\": this definition comes after µs was already used in the lead, and this reader probably knows the unit anyway. It costs very little." + ], + "purpose_unclear": false + }, + { + "at": "§1 para 4", + "text": "One range puts you somewhere on a sphere around the satellite ... Three satellites would be enough, if every clock were perfect.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 para 1", + "text": "Your phone's clock is quartz, not atomic. It can easily be off by 1 ms, and 1 ms × c = 300 km.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 para 2", + "text": "So the receiver does not compute a range. It computes a pseudorange ... That makes b one more unknown to solve for, next to the position.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 worked example", + "text": "Worked example, on a line. Put beacon A at 0 km and beacon B at 30 km ... Those give pseudoranges of 13.5 km and 19.5 km.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 para 4", + "text": "The two pseudoranges add up to 33 km, but the beacons are only 30 km apart. The excess, 3 km, is b counted twice ...", + "unresolved": [], + "leap": [ + "\"The two pseudoranges add up to 33 km, but the beacons are only 30 km apart\": the shortcut of adding the ranges only works because the receiver lies between the beacons. The text does not say so, but it is a small point." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 para 5", + "text": "In three dimensions the unknowns are x, y, z, and b. Four unknowns need four equations ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 para 1", + "text": "The receiver can solve for its own clock error because that error is shared by all the pseudoranges. A satellite clock error is different ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 para 2", + "text": "How close? Each nanosecond of satellite clock error is 0.3 m of range error. For a fix within about 5 m ... about 17 ns.", + "unresolved": [], + "leap": [ + "\"For a fix within about 5 m\": the 5 m target is given without a reason. It works as an example." + ], + "excess": [ + "\"Each nanosecond of satellite clock error is 0.3 m of range error\": this repeats the conversion from §1. The repeat is short and useful here." + ], + "purpose_unclear": false + }, + { + "at": "§4 para 1", + "text": "Time dilation means that clocks in different motion, or at different heights in gravity, tick at different rates ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 table", + "text": "Speed: 3.9 km/s ... loses 7 µs per day | Height ... gains 45 µs per day | Net gains 38 µs per day", + "unresolved": [], + "leap": [ + "\"a clock higher above Earth ticks faster\": the text gives no reason, only a pointer to experiment. The 'Not covered' section later admits this, and the main line does not need the reason." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 para 2", + "text": "The gravity effect is larger, so the clock runs fast. A correction for speed alone would leave the full 45 µs a day, about 13.5 km of range per day.", + "unresolved": [], + "leap": [], + "excess": [ + "\"A correction for speed alone would leave the full 45 µs a day, about 13.5 km of range per day.\": nobody has suggested correcting for speed alone, so the reader stops to work out why this hypothetical is here. Nothing later uses it." + ], + "purpose_unclear": false + }, + { + "at": "§4 para 3", + "text": "38 µs is only 4.5 × 10⁻¹⁰ of a day, but light covers 11.4 km in 38 µs.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 para 1", + "text": "Every satellite clock drifts at the same rate. So does b not absorb the drift, as it absorbs the receiver clock error?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 para 2", + "text": "Only if every clock had the same error at the same moment. If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right.", + "unresolved": [], + "leap": [ + "\"b would absorb 11.4 km, and the position would come out right\": this contradicts the lead's \"would add 11.4 km of range error per day\". The reader has to go back and work out that the real damage comes from differences between clocks, not from the 11.4 km itself." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 para 3", + "text": "But each satellite clock is set to GPS time at a different moment. Suppose satellite A is set on day 0 and satellite B on day 3 ...", + "unresolved": [], + "leap": [ + "\"each satellite clock is set to GPS time at a different moment\": the obvious question is why not reset them often, or all at once. The text never answers it." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§6", + "text": "Each satellite clock counts the ticks of a signal at a nominal 10.23 MHz ... Ground stations upload small corrections for what remains.", + "unresolved": [ + "\"nominal\"" + ], + "leap": [ + "\"Ground stations upload small corrections for what remains\": if uploads can correct the clocks at all, why not upload corrections for the whole 38 µs a day? The text does not say why a rate offset built in before launch is better." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Summary", + "text": "Four pseudoranges give x, y, z, and the receiver's clock error, but only if satellite clocks stay within about 17 ns of GPS time; relativity's net 38 µs a day would break that by 11.4 km a day, so each clock is built to tick slow.", + "unresolved": [], + "leap": [ + "\"would break that by 11.4 km a day\": this repeats the framing that §5 qualified. The damage is the unshared part of the drift, not 11.4 km as such." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered: Atmosphere", + "text": "the ionosphere and troposphere delay the signal ...", + "unresolved": [ + "\"ionosphere\"", + "\"troposphere\"" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered: Sagnac / orbit-shape", + "text": "Earth's rotation during the 67 ms flight (Sagnac effect) and the orbit-shape correction ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered: moving clock", + "text": "any introduction to special relativity (the \"light clock\" argument).", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered: solving", + "text": "iteration from a guess, with least squares when it hears more than four satellites.", + "unresolved": [ + "\"least squares\"" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Lead (\"would add 11.4 km of range error per day\") and Summary (\"would break that by 11.4 km a day\") vs §5 para 2 (\"If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right\"): the headline cost of relativity is stated in a way that the argument itself then says is not how the failure happens. A careful reader stops at §5 and has to rebuild the claim as 'the clocks drift apart by up to 11.4 km a day'." + ], + "cuts": [ + "§4 para 2: cut \"A correction for speed alone would leave the full 45 µs a day, about 13.5 km of range per day.\" It answers a question nobody asked.", + "§1 para 3: the \"(1 µs = 10⁻⁶ s)\" and \"(1 ns = 10⁻⁹ s)\" definitions are probably unneeded for this reader. They are very short, so cutting them saves little." + ], + "question": { + "text": "How does a GPS receiver work out its position from satellite signals, and why does that depend on correcting for relativity?", + "known_at": "Title, with the answer sketched in the bold lead.", + "result_stated": "Yes. The lead says: \"Relativity makes those clocks gain 38 µs a day, which would add 11.4 km of range error per day, so engineers build them to tick slow.\" The Summary restates it." + }, + "motive": "Reason to care: this is weak and only implied, through \"Your receiver (the GPS chip in a phone or car)\". It is an everyday device with a surprising dependence on relativity. Reason for the approach: present. §3 says \"A satellite clock error is different: it sits in only one pseudorange, so no unknown can absorb it\", and §5 explains why b cannot cancel the drift. Together these justify building in the rate offset. Missing: why a fixed slow rate is better than ground uploads alone.", + "close": "Yes. The Summary says: \"Four pseudoranges give x, y, z, and the receiver's clock error, but only if satellite clocks stay within about 17 ns of GPS time; relativity's net 38 µs a day would break that by 11.4 km a day, so each clock is built to tick slow.\" It answers the opening question in words, though the 11.4 km framing ignores the shared-versus-unshared point from §5.", + "recap": "GPS solves for your position plus your phone's clock error from four satellites, which only works if the satellite clocks agree almost perfectly, and since relativity makes orbiting clocks run 38 µs a day fast, they are built to tick slightly slow.", + "strongest": "§2 worked example on a line (the 12 km receiver with a clock 5 µs fast): it makes the clock-bias unknown b concrete and easy to check.", + "weakest": "§5 para 2 together with the Lead and Summary: they disagree about whether 11.4 km a day is the actual error. Second weakest is the unexplained \"Ground stations upload small corrections\" in §6.", + "suggestions": [ + "Rephrase the lead and summary as something like 'clocks set at different times would drift apart, reaching tens of km of disagreement within days', so they agree with §5.", + "Add one sentence to §6 on why a built-in rate offset is preferred over large, frequent uploaded corrections.", + "Drop the speed-only hypothetical in §4, or say why it matters.", + "Optionally note that adding the two pseudoranges in the 1-D example works because the receiver is between the beacons." + ] + }, + "usage": { + "cost": 0.15960499999999997, + "seconds": 58 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Lead (\"would add 11.4 km of range error per day\") and Summary (\"would break that by 11.4 km a day\") vs §5 para 2 (\"If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right\"): the headline cost of relativity is stated in a way that the argument itself then says is not how the failure happens. A careful reader stops at §5 and has to rebuild the claim as 'the clocks drift apart by up to 11.4 km a day'." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "§4 para 2: cut \"A correction for speed alone would leave the full 45 µs a day, about 13.5 km of range per day.\" It answers a question nobody asked." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "§1 para 3: the \"(1 µs = 10⁻⁶ s)\" and \"(1 ns = 10⁻⁹ s)\" definitions are probably unneeded for this reader. They are very short, so cutting them saves little." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Lead (bold paragraph): \"Relativity makes those clocks gain 38 µs a day\": the reader is told this here and has to wait until §4 for the reason. That is fine in a summary." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Lead (bold paragraph): \"which would add 11.4 km of range error per day\": this reads as if the error is the clock drift itself. §5 later says a drift shared by all clocks would be absorbed, so this framing needs revising once the reader gets there." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "§2 para 4: \"The two pseudoranges add up to 33 km, but the beacons are only 30 km apart\": the shortcut of adding the ranges only works because the receiver lies between the beacons. The text does not say so, but it is a small point." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "§3 para 2: \"For a fix within about 5 m\": the 5 m target is given without a reason. It works as an example." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "§4 table: \"a clock higher above Earth ticks faster\": the text gives no reason, only a pointer to experiment. The 'Not covered' section later admits this, and the main line does not need the reason." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "§5 para 2: \"b would absorb 11.4 km, and the position would come out right\": this contradicts the lead's \"would add 11.4 km of range error per day\". The reader has to go back and work out that the real damage comes from differences between clocks, not from the 11.4 km itself." + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "§5 para 3: \"each satellite clock is set to GPS time at a different moment\": the obvious question is why not reset them often, or all at once. The text never answers it." + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "§6: \"nominal\"" + }, + { + "n": 12, + "severity": "edge", + "kind": "leap", + "text": "§6: \"Ground stations upload small corrections for what remains\": if uploads can correct the clocks at all, why not upload corrections for the whole 38 µs a day? The text does not say why a rate offset built in before launch is better." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Summary: \"would break that by 11.4 km a day\": this repeats the framing that §5 qualified. The damage is the unshared part of the drift, not 11.4 km as such." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: Atmosphere: \"ionosphere\"" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: Atmosphere: \"troposphere\"" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: solving: \"least squares\"" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150300-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150300-quiz-prose.json new file mode 100644 index 0000000..8ae8363 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150300-quiz-prose.json @@ -0,0 +1,237 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "gps-relativity", + "at": "2026-10-04T15:02:47", + "model": "default", + "prompt_sha": "8989a402f1c2", + "reply": { + "answers": { + "1": "The GPS satellites (about 30, 20,200 km up) and their atomic clocks; the receiver (for example a phone chip) and its quartz clock; the broadcast signal (satellite ID, position P, send time t); range and pseudorange; b, the receiver's clock error in metres; GPS time, kept by atomic clocks at ground stations; the two relativity effects (special relativity from speed, general relativity from gravity/height); and the 10.23 MHz clock frequency offset set before launch.", + "2": "Each satellite broadcasts its position and send time. The receiver times when each signal arrives. Travel time × c gives a range. Because the receiver clock is wrong, each measurement is a pseudorange = range + b, and b is the same in every pseudorange. Four pseudoranges let it solve for x, y, z and b. Satellite clocks must match GPS time because an error in one satellite clock sits in only one pseudorange, so nothing absorbs it. Relativity makes satellite clocks run fast compared with GPS time, so engineers set their frequency slightly low before launch. Ground stations upload small corrections for what is left.", + "3": "Position comes from signal travel times × light speed, so every nanosecond of satellite clock error is 0.3 m of range error. Relativity makes satellite clocks gain 38 µs/day (11.4 km of range per day), so the clocks are built to tick slow by 4.5×10⁻¹⁰ to cancel this.", + "4": "Satellite clock error is the key variable. Each extra ns adds 0.3 m of range error to that satellite's pseudorange only, so the fix gets worse, and past about 17 ns it is worse than about 5 m. If the relativistic drift were left uncorrected, each clock would move away from GPS time by 11.4 km of range per day, and the fix would fail. A change in the receiver clock error instead does not hurt accuracy, because b absorbs it.", + "5": "Assumption/simplification: the 1D worked example with two beacons on a line (and leaving out atmosphere, Sagnac and orbit shape). Established fact: time dilation 'is measured, not only predicted'. Net, satellite clocks gain 38 µs/day (lose 7 from speed, gain 45 from gravity).", + "6": "A GPS receiver turns signal travel times into distances by multiplying by light speed. Its cheap clock adds the same unknown offset to every distance, so it uses four satellites to solve for x, y, z and that offset. A satellite clock error affects only one distance, so satellite clocks must stay within about 17 ns of GPS time. Relativity makes them gain a net 38 µs per day (gravity +45, speed −7), which is 11.4 km of range per day. Engineers therefore set the clocks slightly slow before launch so that they run at the right rate in orbit.", + "7": "Section 5 was hardest. It does not say why satellites are set to GPS time at different moments. It also says the difference 'grows every day', but in its own example both clocks gain 38 µs/day, so the A−B gap stays at 114 µs (34.2 km). An explicit day-by-day table, and a reason why per-satellite drift becomes uneven (different set times, different clocks), would help. Also the text says 34 km, but 57 − 22.8 = 34.2.", + "8": "2 µs × 300 m/µs ≈ 600 m added to every pseudorange. It does not make the position wrong by 600 m. The error is the same in all pseudoranges, so the receiver solves for it as b (about 600 m) and subtracts it.", + "9": "They would run fast by about 45 µs per day, the gravity gain with nothing to cancel it. That is about 13.5 km of range per day.", + "10": "Range = time × c, and light covers 11.4 km in 38 µs. A 5 m fix needs satellite clocks within about 17 ns, so 38 µs/day is thousands of times over that budget after one day. Each satellite's error is its own and is not absorbed.", + "11": "The receiver's quartz clock error b is a fourth unknown alongside x, y, z. Four unknowns need four equations (pseudoranges). With perfect clocks, three would do: three spheres meet in two points and one is rejected as being in space.", + "12": "Fast, by a net 38 µs/day. Weaker gravity at 20,200 km makes them gain 45 µs/day (general relativity). Their 3.9 km/s speed makes them lose 7 µs/day (special relativity). Gravity wins. (Once built slow, they match GPS time in orbit.)", + "13": "b can only absorb an error that is the same in every pseudorange at the same moment. Satellites are set to GPS time at different moments, so their accumulated errors differ (for example 57 km vs 22.8 km on day 5). The difference is not shared, so no single b removes it. The explanation does not say why setting times differ." + }, + "audit": { + "terms": [ + "The ground reference has three names: 'ground clocks' (lead), 'GPS time' (§3, 'atomic clocks at ground stations'), and 'a clock on the ground' (§4). It is unclear whether these are the same thing.", + "Satellite clock error is called 'drift' (§5 title, §5), 'gain' (§4 table, lead), and 'error' (§3). For example: 'Relativity makes those clocks gain 38 µs a day' (lead) vs 'relativity drift' / 'cancel the drift' (§5).", + "'fix' means two things: 'For a fix within about 5 m' (§3) and 'position fix fails' (§5) mean a computed position, but '## 6. The fix: make the clocks tick slow' means the engineering correction.", + "b is given in two units without the link being stated: 'the receiver clock error expressed in metres' (§2) vs 'b = 1.5 km (5 µs)' (§2 example).", + "'range' / 'true distance' / 'distance' (§1) all mean the same thing, and pseudorange is introduced right after. This is minor but should stay consistent." + ], + "unexplained": [ + "'This difference grows every day' (§5): in the example both clocks gain 38 µs/day, so the 114 µs gap between A and B stays constant. The text does not say what makes the difference grow.", + "'10.23 MHz' and '10.22999999543 MHz' (§6): no reason is given for this base frequency, or for what the many decimal places mean for the reader.", + "'For a fix within about 5 m' (§3): the 5 m target is given without saying why 5 m, or what a looser or tighter target would do to the 17 ns budget.", + "'swings by up to about 23 ns' (Not covered): there is no comparison with the 17 ns budget, although 23 ns exceeds it. That is the key implication.", + "'3.9 km/s', 'loses 7 µs', 'gains 45 µs' (§4 table) are given with no way to see how they scale (for example, how the loss would change at higher speed or lower orbit)." + ], + "missing_why": [ + "The §4 table gives −7 and +45 µs/day with no formula or reasoning for their size. The text does not say why higher altitude makes a clock tick faster, only that it does.", + "§5: 'each satellite clock is set to GPS time at a different moment' is given without saying why the satellites are not all set together, or why a common rate still leaves unequal errors.", + "§6: why a rate (frequency) change is the right fix rather than periodic resets. Ground corrections exist, so why not just reset? This is only partly answered by 'keeps those corrections small'.", + "§1: 'Three spheres meet in two points': no reason is given why it is exactly two points.", + "§2: why the receiver clock error adds (pseudorange = range + b) rather than scaling the range. The 1D example shows it but does not state why." + ], + "excess": [ + "The Summary (lines 66–68) repeats the lead paragraph almost word for word.", + "'Light covers 300 m in one microsecond … 0.3 m in one nanosecond' with unit definitions (§1): this reader knows c ≈ 300,000 km/s, though the conversion rates are mildly useful.", + "The 'Not covered here' bullets on atmosphere and solving by iteration are detours the questions do not use." + ] + }, + "gaps": [ + "Section 5 says the satellite-to-satellite difference 'grows every day', but its own example (same drift rate, different set days) gives a constant 114 µs gap. The explanation never says what makes satellite errors diverge over time.", + "No derivation or scaling of the 7 µs and 45 µs figures, and no explanation of why higher gravitational potential means faster ticking.", + "There is no explanation of why satellites are synchronised at different moments.", + "The orbit-shape swing of 23 ns exceeds the stated 17 ns budget, but the effect is only noted under 'Not covered'.", + "Small arithmetic rounding: 57 − 22.8 = 34.2 km, stated as 34 km." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "1": "Full marks: lists every entity in model.md with correct details (30 satellites at 20,200 km, atomic vs quartz clocks, signal contents, range/pseudorange, b in metres, GPS time, both relativity effects, 10.23 MHz offset).", + "2": "Full marks: the causal chain is complete and in order. It explains why a satellite clock error is not absorbed by b and mentions the ground corrections.", + "3": "Full marks: gives the central answer with 0.3 m/ns, +38 µs/day, 11.4 km/day and the 4.5×10⁻¹⁰ slow-tick fix.", + "4": "Full marks: identifies satellite clock error as the key variable, with the 17 ns to 5 m budget, and contrasts it with receiver clock error, which b absorbs.", + "5": "Full marks: correctly separates simplifications (1D example; atmosphere, Sagnac and eccentricity out of scope) from measured fact (time dilation).", + "6": "Full marks: accurate summary with correct numbers (+45, −7, net 38 µs/day, 11.4 km/day, 17 ns).", + "7": "Full marks: specific and valid critique. If both clocks drift at the same rate, the A−B gap stays constant at 114 µs (34.2 km), which conflicts with 'grows every day'. It also notes the rounding (34 vs 34.2) and that the rendering never says why setting times differ. This is a candidate audit finding for the explainer to decide.", + "8": "Full marks: 600 m per pseudorange, absorbed as b, so the position is not off by 600 m.", + "9": "Full marks: fast by 45 µs/day, about 13.5 km/day, because the gravity gain remains.", + "10": "Full marks: 38 µs × c ≈ 11.4 km per day, set against the 17 ns budget.", + "11": "Full marks: b is a fourth unknown, so four equations are needed. Also covers the perfect-clock three-satellite case.", + "12": "Full marks: net fast by 38 µs/day; gravity +45 (GR) and speed −7 (SR).", + "13": "Full marks: b absorbs only a shared error. Gives the day-5 example (57 vs 22.8 km) showing the difference cannot go into one b." + }, + "pass": true + }, + "usage": { + "cost": 0.1966034, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The ground reference has three names: 'ground clocks' (lead), 'GPS time' (§3, 'atomic clocks at ground stations'), and 'a clock on the ground' (§4). It is unclear whether these are the same thing." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Satellite clock error is called 'drift' (§5 title, §5), 'gain' (§4 table, lead), and 'error' (§3). For example: 'Relativity makes those clocks gain 38 µs a day' (lead) vs 'relativity drift' / 'cancel the drift' (§5)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'fix' means two things: 'For a fix within about 5 m' (§3) and 'position fix fails' (§5) mean a computed position, but '## 6. The fix: make the clocks tick slow' means the engineering correction." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "b is given in two units without the link being stated: 'the receiver clock error expressed in metres' (§2) vs 'b = 1.5 km (5 µs)' (§2 example)." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "'range' / 'true distance' / 'distance' (§1) all mean the same thing, and pseudorange is introduced right after. This is minor but should stay consistent." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'This difference grows every day' (§5): in the example both clocks gain 38 µs/day, so the 114 µs gap between A and B stays constant. The text does not say what makes the difference grow." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'10.23 MHz' and '10.22999999543 MHz' (§6): no reason is given for this base frequency, or for what the many decimal places mean for the reader." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'For a fix within about 5 m' (§3): the 5 m target is given without saying why 5 m, or what a looser or tighter target would do to the 17 ns budget." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'swings by up to about 23 ns' (Not covered): there is no comparison with the 17 ns budget, although 23 ns exceeds it. That is the key implication." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "'3.9 km/s', 'loses 7 µs', 'gains 45 µs' (§4 table) are given with no way to see how they scale (for example, how the loss would change at higher speed or lower orbit)." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "The §4 table gives −7 and +45 µs/day with no formula or reasoning for their size. The text does not say why higher altitude makes a clock tick faster, only that it does." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "§5: 'each satellite clock is set to GPS time at a different moment' is given without saying why the satellites are not all set together, or why a common rate still leaves unequal errors." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "§6: why a rate (frequency) change is the right fix rather than periodic resets. Ground corrections exist, so why not just reset? This is only partly answered by 'keeps those corrections small'." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "§1: 'Three spheres meet in two points': no reason is given why it is exactly two points." + }, + { + "n": 15, + "severity": "gap", + "kind": "missing_why", + "text": "§2: why the receiver clock error adds (pseudorange = range + b) rather than scaling the range. The 1D example shows it but does not state why." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "The Summary (lines 66–68) repeats the lead paragraph almost word for word." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "'Light covers 300 m in one microsecond … 0.3 m in one nanosecond' with unit definitions (§1): this reader knows c ≈ 300,000 km/s, though the conversion rates are mildly useful." + }, + { + "n": 18, + "severity": "excess", + "kind": "excess", + "text": "The 'Not covered here' bullets on atmosphere and solving by iteration are detours the questions do not use." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "Section 5 says the satellite-to-satellite difference 'grows every day', but its own example (same drift rate, different set days) gives a constant 114 µs gap. The explanation never says what makes satellite errors diverge over time." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "No derivation or scaling of the 7 µs and 45 µs figures, and no explanation of why higher gravitational potential means faster ticking." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "There is no explanation of why satellites are synchronised at different moments." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "The orbit-shape swing of 23 ns exceeds the stated 17 ns budget, but the effect is only noted under 'Not covered'." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "Small arithmetic rounding: 57 − 22.8 = 34.2 km, stated as 34 km." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/read.json new file mode 100644 index 0000000..792a192 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "The receiver measures the arrival time of radio signals. It subtracts the signal's timestamp (send time) from the arrival time to get travel time, then multiplies by light speed to convert to distance.", + "quote": "The receiver notes when each signal arrives. Then: travel time = receive time − send time; range = travel time × c, where c is light speed and the range is the true distance to the satellite." + }, + "2": { + "answer": "The receiver's quartz clock is inaccurate (off by milliseconds, which is hundreds of kilometers). This clock error becomes an unknown to solve for, requiring a fourth satellite to provide the fourth equation.", + "quote": "In three dimensions the unknowns are x, y, z, and b. Four unknowns need four equations, so the receiver needs four satellites." + }, + "3": { + "answer": "Relativity makes satellite clocks gain 38 microseconds per day. Light travels 11.4 km in 38 microseconds, so each uncorrected clock would drift 11.4 km further from correct GPS time daily.", + "quote": "38 µs is only 4.5 × 10⁻¹⁰ of a day, but light covers 11.4 km in 38 µs. Each uncorrected satellite clock would move 11.4 km of range further from GPS time every day." + }, + "4": { + "answer": "A 1 microsecond clock error produces a 300 meter range error. Using three satellites while ignoring the 1 µs clock error would give roughly 300 m error in the distance measurements.", + "quote": "Light covers 300 m in one microsecond (1 µs = 10⁻⁶ s)" + } + }, + "stops": [ + "I am satellite k, I am at position P, and I sent this at time t", + "One range puts you somewhere on a sphere around the satellite: every point at that distance", + "The receiver rejects the one far out in space", + "GPS time, the time kept by atomic clocks at ground stations", + "The excess, 3 km, is b counted twice, so b = 1.5 km", + "It is measured, not only predicted", + "4.5 × 10⁻¹⁰, the fraction the clock gains in orbit" + ], + "excess": [ + "About 30 GPS satellites orbit 20,200 km above Earth", + "Each one carries atomic clocks and broadcasts a signal", + "From a satellite directly overhead, the travel time is about 67 ms", + "See N. Ashby, 'Relativity in the Global Positioning System', Living Reviews in Relativity (2003)", + "Why a moving clock ticks slow: see the 'light clock' argument in special relativity", + "Each satellite clock counts the ticks of a 10.23 MHz signal", + "charged and humid layers of the atmosphere slow the signal" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/run.json new file mode 100644 index 0000000..9bea378 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/gps-relativity/explanation.md` as prose only, because the subject is one chain of reasoning with worked numbers at each step, which needs no diagram or interactive page. The probe, the cold read and the blind test have all run, I fixed what they found, and `explainer check` passes.", + "error": null, + "cost": 1.5428533999999998, + "seconds": 567, + "turns": 79, + "files": [ + "explainers/gps-relativity/explanation.md" + ], + "review_cost": 0.6484, + "reviews": 4 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/explanation.md new file mode 100644 index 0000000..6705031 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/explanation.md @@ -0,0 +1,51 @@ +# How does GPS find your position, and why does it need relativity? + + + +**GPS finds your position by timing radio signals. Each satellite sends the time it transmitted. Your receiver measures how long each signal took to arrive, and multiplies by the speed of light to get a distance. Four distances fix your position. Relativity matters because the satellite clocks tick faster than ground clocks by 38 microseconds per day. At the speed of light, that error is about 11 km of distance per day.** + +## Step 1: time becomes distance + +About 30 GPS satellites orbit at 20,200 km altitude. Each satellite carries an atomic clock. Each satellite continuously broadcasts a **signal**: a radio message that says "I sent this at time t, and I was at position (x, y, z)". + +Your receiver notes when the signal arrives. The **travel time** is receive time minus send time. For a satellite straight overhead, the travel time is about 67 ms (20,200 km ÷ 299,792,458 m/s). + +Distance = travel time × c. Light travels about 30 cm per nanosecond, so the clocks must agree to within a few nanoseconds for metre-level accuracy. + +## Step 2: why four satellites, not three + +One distance puts you somewhere on a sphere around one satellite. Three spheres meet at a point. So three satellites should be enough. + +But your phone has a cheap quartz clock, not an atomic clock. Suppose the phone clock is 1 ms wrong. Then every distance is wrong by 1 ms × c = 300 km. The three spheres no longer meet at your true position. + +GPS solves this by treating the clock error as a fourth unknown. The **receiver clock bias** b is the offset of your clock from GPS time. Each measured distance is a **pseudorange** ρ: the true distance plus the same error c·b. For satellite i at (xᵢ, yᵢ, zᵢ): + +```text +ρᵢ = √((x − xᵢ)² + (y − yᵢ)² + (z − zᵢ)²) + c·b +``` + +There are four unknowns (x, y, z, b), so the receiver needs four equations: four satellites. It solves them by iteration: guess a position, compute the predicted pseudoranges, and correct the guess until the predicted values match the measured ones. A side effect: your phone gets the time accurate to nanoseconds. + +This method works only if the satellite clocks all keep GPS time. Relativity says that they do not, if nobody corrects them. + +## Step 3: two effects change the satellite clock rate + +**Predict first:** a satellite moves fast and sits high above Earth. Does its clock run fast or slow compared with a clock on the ground? + +Two effects pull in opposite directions. + +1. **Speed slows the clock (special relativity).** A moving clock ticks slower by a fraction of about v²/(2c²). The orbit speed is 3.87 km/s, which gives 8.3 × 10⁻¹¹. Over one day (86,400 s), the satellite clock loses about 7 µs. +2. **Weaker gravity speeds the clock up (general relativity).** A clock deeper in a gravity well ticks slower. A ground clock sits at 6,371 km from Earth's center. A satellite sits at 26,560 km, where gravity is weaker. The fraction is about (GM/c²)(1/R_Earth − 1/r) = 5.3 × 10⁻¹⁰. Over one day, the satellite clock gains about 45 µs. + +The gravity effect wins. Net: the satellite clock gains about 38 µs per day. + +## Step 4: why 38 µs matters + +38 µs looks small. Multiply by c: 38 µs × 300 m per µs ≈ 11 km. The system needs agreement to a few nanoseconds. The drift is 0.44 ns per second, so after one minute it is 26 ns, or 8 m. After one day, the satellite time stamps would be off by the light-travel time of 11 km. + +The fix is simple. Engineers set each satellite clock to tick slow before launch. The nominal frequency is 10.23 MHz; the clock is set to 10.22999999543 MHz. In orbit, relativity speeds the clock up to exactly 10.23 MHz as seen from the ground. + +## Limits + +- **Estimate:** "11 km per day" is the light-travel distance of 38 µs. The actual position error would grow in a more complex way, because the bias b absorbs part of an error that all satellites share. The drift would still break the system within hours. +- **Out of scope:** Earth's rotation during signal travel (the Sagnac effect), a small extra correction for slightly elliptical orbits (tens of ns, computed by the receiver), and signal delay in the ionosphere. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/model.md new file mode 100644 index 0000000..c2bef06 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/model.md @@ -0,0 +1,99 @@ +# Semantic model: GPS positioning and relativity + +> Source of truth. Every rendering derives from this file. + +## Central question + +How does a GPS receiver find its position from satellite signals, and why must the system correct for relativity? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (speed = distance / time, Pythagoras, simultaneous equations, gravity), basic programming. New to GPS and to relativity. Does not know: pseudorange, receiver clock bias, why four satellites, that clocks tick at different rates with speed and gravity. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Satellite | One of about 30 GPS satellites at 20,200 km altitude, each with an atomic clock. | fact | +| Signal | A radio message that carries the satellite's send time and its position. | fact | +| Receiver | The device on the ground (phone) with a cheap quartz clock. | fact | +| Travel time | Receive time minus send time; about 67 ms for a satellite overhead. | fact | +| Pseudorange | Travel time × c; a distance that still contains the receiver clock error. | definition | +| Receiver clock bias | The unknown offset of the receiver's clock from GPS time. | definition | +| Clock rate shift | The change in how fast a satellite clock ticks compared with a ground clock. | fact (relativity) | + +## Relationships + +| From | Relationship | To | +|---|---|---| +| Satellite clock | stamps | Signal (send time) | +| Travel time × c | gives | Pseudorange | +| 4 pseudoranges | solve for | x, y, z, receiver clock bias | +| Orbit speed | slows | Satellite clock (special relativity) | +| Weaker gravity at altitude | speeds up | Satellite clock (general relativity) | +| Clock rate shift | becomes | Range error (× c) | + +## Causal chain + +```text +time stamp → travel time → × c → pseudorange → 4 equations → position + clock bias +satellite speed (−7 µs/day) + weaker gravity (+45 µs/day) → net +38 µs/day → ×c → ≈11 km/day range drift → so: tune satellite clocks slow before launch +``` + +## Quantities + +- c = 299,792,458 m/s; light travels about 30 cm per ns, 300 m per µs. +- Altitude 20,200 km; orbit radius 26,560 km; Earth radius 6,371 km; orbit speed 3.87 km/s. +- Travel time overhead ≈ 20,200 km / c ≈ 67 ms. +- Special relativity: fractional rate ≈ −v²/(2c²) = −8.3 × 10⁻¹¹ → × 86,400 s ≈ −7 µs/day. +- General relativity: fractional rate ≈ (GM/c²)(1/R_E − 1/r) = +5.3 × 10⁻¹⁰ → ≈ +45 µs/day. +- Net ≈ +38 µs/day → 38 µs × 300 m/µs ≈ 11 km of light travel per day. +- Fix: nominal 10.23 MHz clock set to 10.22999999543 MHz before launch. +- Receiver clock 1 ms wrong → 300 km range error. + +## Alternative states + +| Scenario | What changes | Why | +|---|---|---| +| No relativity correction | Satellite clocks gain 38 µs/day on ground clocks | ranges drift by about 11 km per day | +| Only 3 satellites | cannot solve for clock bias | 4 unknowns, 3 equations | +| Perfect receiver clock | 3 satellites would suffice | bias is no longer unknown | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Use 4 satellites | 3 spheres meet at a point; use 3 | receiver quartz clock may be 1 ms off → every range wrong by 300 km | +| Measure time, then multiply by c | measure distance directly | no way to measure 20,200 km directly; time stamps are what a radio can carry | +| Correct the satellite clock rate | ignore relativity; effects are tiny (5 parts in 10¹⁰) | ×c turns 38 µs/day into ≈11 km/day | + +## Concrete cases + +| Claim | Holds | Breaks | +|---|---|---| +| Position from 4 pseudoranges | all clocks agree with GPS time to ns → metres | satellite clocks drift 38 µs/day → ≈11 km/day | + +## Terms + +- travel time, pseudorange, receiver clock bias, clock rate shift. "GPS time" = the common time the satellite clocks are kept to. + +## Epistemic status + +- **Established:** both rate shifts are measured on GPS clocks and match relativity. +- **Estimate:** "11 km per day" is the light-travel equivalent of 38 µs. Real position error would grow in a more complex way, because the receiver's bias term absorbs part of a shift common to all satellites. The drift would still break the system. + +## Confusion points + +- "Relativity makes satellite clocks slow" → speed slows them (7 µs/day), but gravity speeds them up more (45 µs/day). Net: fast. +- "GPS measures distance" → it measures time; distance is time × c. +- "Third satellite is enough in 3D" → only if the receiver clock is perfect. + +## Scope + +- **Out of scope:** Sagnac effect from Earth's rotation; orbit eccentricity correction (tens of ns, computed by the receiver); ionosphere delay; least-squares with more than 4 satellites. + +## Representation decision + +- **Stage:** 1 +- **Reason:** a sequential argument with worked numbers; no state change or parameter space a reader must explore. +- **Levels:** L1 answer / L2 time → distance → 4 equations / L3 two clock effects with numbers / L4 the factory frequency offset. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/model.yaml new file mode 100644 index 0000000..1033f73 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/explainers/gps-relativity/model.yaml @@ -0,0 +1,77 @@ +# Machine-readable half of model.md. `explainer check gps-relativity` holds every rendering to it. + +values: + satellites_in_orbit: 30 + altitude_km: 20200 + c_m_per_s: 299792458 + travel_time_ms: 67 + light_cm_per_ns: 30 + light_m_per_us: 300 + receiver_clock_error_ms: 1 + receiver_range_error_km: 300 + unknowns: 4 + orbit_speed_km_s: 3.87 + sr_fraction: 8.3e-11 + seconds_per_day: 86400 + sr_us_per_day: 7 + earth_radius_km: 6371 + orbit_radius_km: 26560 + gr_fraction: 5.3e-10 + gr_us_per_day: 45 + net_us_per_day: 38 + drift_km_per_day: 11 + drift_ns_per_s: 0.44 + drift_ns_per_min: 26 + drift_m_per_min: 8 + nominal_mhz: 10.23 + set_mhz: 10.22999999543 + +allow: [1, 2, 3, 4, 10] + +require: + - path: net_us_per_day + in: [prose] + - path: drift_km_per_day + in: [prose] + - path: set_mhz + in: [prose] + +claims: + - id: time-to-distance + kind: mechanism + statement: "The receiver turns travel time into distance by multiplying by c." + example: "Satellite overhead at 20,200 km: travel time about 67 ms. 1 ns of clock error is 30 cm." + - id: why-four + kind: why + statement: "Four satellites are needed because the receiver clock bias is a fourth unknown." + alternative: "Use three satellites; three spheres meet at a point." + counterexample: "A phone clock 1 ms wrong makes every distance wrong by 300 km." + ask: "Why does a receiver need four satellites, not three?" + - id: two-effects + kind: mechanism + statement: "Speed slows the satellite clock; weaker gravity speeds it up more." + example: "-7 µs/day from speed, +45 µs/day from gravity, net +38 µs/day." + ask: "Does a GPS satellite clock run fast or slow compared with a ground clock, and why?" + - id: drift-size + kind: why + statement: "A tiny clock drift matters because c converts it into a large distance." + alternative: "Ignore relativity; the rate shift is only a few parts in 10^10." + counterexample: "38 µs/day × 300 m/µs ≈ 11 km/day; 8 m after one minute." + - id: the-fix + kind: mechanism + statement: "Satellite clocks are set slow before launch so they tick at the nominal rate in orbit." + example: "10.23 MHz nominal, set to 10.22999999543 MHz." + +accept_unjustified: {} + +budget: {prose: 750, reason: "two mechanisms (positioning, then two opposing clock effects), each with worked numbers"} + +terms: + - term: pseudorange + means: "travel time × c, which still contains the receiver clock bias" + avoid: [true range] + +quiz: + - q: "Does a GPS satellite clock run fast or slow compared with a ground clock?" + expect: "Fast, by about 38 µs/day: gravity's +45 beats speed's -7." + misconception: "Slow, because moving clocks run slow." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/read.json new file mode 100644 index 0000000..935ee45 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "The receiver measures the travel time of the radio signal (time received minus time sent), then multiplies by the speed of light to calculate distance.", + "quote": "Each satellite sends the time it transmitted. Your receiver measures how long each signal took to arrive, and multiplies by the speed of light to get a distance." + }, + "2": { + "answer": "Because the receiver's clock may be inaccurate. Any clock error affects all measured distances equally. Treating the clock error as an unknown quantity requires four equations to solve four unknowns: x, y, z position and clock bias.", + "quote": "There are four unknowns (x, y, z, b), so the receiver needs four equations: four satellites." + }, + "3": { + "answer": "Relativity causes satellite clocks to run 38 microseconds per day faster than ground clocks. At light speed, this error accumulates to about 11 km of distance error per day, which overwhelms meter-level accuracy requirements.", + "quote": "Relativity matters because the satellite clocks tick faster than ground clocks by 38 microseconds per day. At the speed of light, that error is about 11 km of distance per day." + }, + "4": { + "answer": "Since 1 ms of clock error causes 300 km distance error, a 1 µs error (1000 times smaller) causes 300 m error, proportionally distributed across all distance measurements.", + "quote": "Suppose the phone clock is 1 ms wrong. Then every distance is wrong by 1 ms × c = 300 km." + } + }, + "stops": [ + "pseudorange ρ: the true distance plus the same error c·b", + "A moving clock ticks slower by a fraction of about v²/(2c²)", + "solves them by iteration: guess a position, compute the predicted pseudoranges, and correct", + "The fraction is about (GM/c²)(1/R_Earth − 1/r) = 5.3 × 10⁻¹⁰", + "Three spheres meet at a point. So three satellites should be enough.", + "The Sagnac effect", + "The receiver clock bias b is the offset of your clock from GPS time", + "The bias b absorbs part of an error that all satellites share" + ], + "excess": [ + "About 30 GPS satellites orbit at 20,200 km altitude", + "For a satellite straight overhead, the travel time is about 67 ms", + "A side effect: your phone gets the time accurate to nanoseconds", + "Light travels about 30 cm per nanosecond", + "Each satellite continuously broadcasts a signal: a radio message", + "A ground clock sits at 6,371 km from Earth's center. A satellite sits at 26,560 km" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/run.json new file mode 100644 index 0000000..78ebe80 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/gps-relativity/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/gps-relativity/explanation.md` (prose, with `model.md` and `model.yaml` behind it; `explainer check` passes) because the subject is one chain of reasoning with worked numbers: timing gives distance, four satellites solve the clock error, and relativity makes the satellite clocks gain 38 µs/day (about 11 km/day) — there is nothing to explore or animate.", + "error": null, + "cost": 0.44009420000000005, + "seconds": 91, + "turns": 18, + "files": [ + "explainers/gps-relativity/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/explainers/greenhouse/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/explainers/greenhouse/explanation.md new file mode 100644 index 0000000..2096434 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/explainers/greenhouse/explanation.md @@ -0,0 +1,167 @@ +# How greenhouse gases warm the Earth + +**The short answer:** The Earth gets energy from sunlight and loses it as infrared radiation to space. A few gases in the air, mainly water vapour and carbon dioxide, absorb that infrared. Because of them, the radiation that reaches space comes mostly from high, cold layers of the atmosphere rather than from the warm surface. Cold things radiate less. To get rid of all the energy it receives, the planet has to be warmer than it would be without those gases. Adding more of them pushes the radiating layer higher, where it is colder. The planet then loses less energy than it takes in, and it warms until the two match again. + +The rest of this document builds that answer one step at a time. Each step is a small piece of physics you can check with a calculator. The companion page `index.html` lets you change the inputs of the two models below and see what happens. + +--- + +## 1. The planet is a budget: energy in = energy out + +Leave the atmosphere out at first. The Earth sits in sunlight and gains energy. It also radiates energy away into space. If it gains more than it loses, it warms up. A warmer body radiates more, so the warming continues until the losses equal the gains. That balanced state sets the planet's temperature. + +**Energy in.** At Earth's distance from the Sun, sunlight carries about **S = 1361 W/m²** through a surface facing it. The Earth catches this on a disc of area πR², but it is a sphere of area 4πR². Averaged over the whole surface, day and night, that gives S/4 ≈ 340 W/m². About 30% of it is reflected straight back by clouds, ice and bright surfaces (this fraction is the *albedo*, α ≈ 0.3). What is actually absorbed is + +``` +absorbed = S (1 − α) / 4 ≈ 1361 × 0.7 / 4 ≈ 238 W/m² +``` + +**Energy out.** Every object radiates according to its temperature. For an ideal emitter (a "black body") the **Stefan–Boltzmann law** gives + +``` +emitted = σ T⁴, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴ +``` + +Note the fourth power. A small rise in temperature makes a body radiate quite a lot more. + +**Balance.** Set the two equal and solve for T: + +``` +σ T⁴ = 238 → T = (238 / 5.67e-8)^(1/4) ≈ 255 K ≈ −18 °C +``` + +The measured average surface temperature is about **288 K (+15 °C)**. That leaves a gap of roughly **33 K**. This gap is the greenhouse effect, and the rest of the document explains where it comes from. + +## 2. Sunlight and earthlight are different colours + +The radiation a body gives off peaks at a wavelength set by its temperature (**Wien's law**: peak ≈ 2898 µm·K / T). + +| Emitter | Temperature | Peak wavelength | Kind of light | +|---|---|---|---| +| Sun's surface | ~5800 K | ~0.5 µm | visible | +| Earth's surface | ~288 K | ~10 µm | thermal infrared | + +The two spectra hardly overlap. This is what makes the greenhouse effect possible. Air can be almost transparent to incoming sunlight and still be largely opaque to outgoing infrared. The air acts on the energy going out and does little to the energy coming in. + +## 3. Why only some gases absorb infrared + +About 99% of dry air is nitrogen (N₂) and oxygen (O₂), and these barely absorb thermal infrared. The reason is molecular. + +A molecule absorbs or emits infrared by changing how it vibrates or rotates. It can only do this if the motion **changes the molecule's electric dipole**, because the dipole is what the light's oscillating electric field can push on. N₂ and O₂ are two identical atoms. However they stretch, the molecule stays symmetric and has no dipole, so infrared light has almost nothing to act on. + +Molecules with three or more atoms, or with two different atoms, behave differently: + +- **CO₂** (O=C=O) is straight and symmetric at rest. When it **bends**, or when it stretches asymmetrically, a dipole appears. The bending mode absorbs strongly near **15 µm**, close to the peak of Earth's emission. +- **H₂O** is bent to begin with and absorbs across wide ranges of the infrared. +- **CH₄** (methane), **N₂O** (nitrous oxide) and **O₃** (ozone) each have their own absorption bands. + +Together these absorb most outgoing infrared at most wavelengths. The main exception is a gap between about 8 and 13 µm, the **atmospheric window**, where clear air is fairly transparent. Through that window the surface radiates fairly directly to space. + +One detail matters for later. In the lower atmosphere, a CO₂ molecule that absorbs a photon nearly always collides with neighbouring molecules (mostly N₂ and O₂) before it can re-emit. Collisions happen roughly every nanosecond, and the excited state would take much longer than that to radiate. So the absorbed energy turns into ordinary heat in the surrounding air. Separately, CO₂ is constantly excited by collisions and **emits** at a rate set by the *local air temperature*. Remember this point: **what a layer of air emits depends on how warm that layer is.** + +## 4. The simplest model: one absorbing layer + +Model the atmosphere as a single layer. It is transparent to sunlight and absorbs a fraction ε of the surface's infrared, with 0 ≤ ε ≤ 1. A layer that absorbs a fraction ε also emits ε times the black-body amount (Kirchhoff's law). It emits from both faces, so the same amount goes up and down. + +Write E_s = σT_s⁴ for the surface emission and E_a = σT_a⁴ for the black-body emission at the layer's temperature. At equilibrium every part must balance: + +``` +Atmosphere layer: absorbs ε·E_s = emits ε·E_a up + ε·E_a down + → E_a = E_s / 2 + +Top of atmosphere: 238 (in) = (1 − ε)·E_s + ε·E_a (out) + = E_s (1 − ε/2) + +So: T_s = [ 238 / (σ (1 − ε/2)) ]^(1/4) +``` + +Some values: + +| ε | T_s | +|---|---| +| 0 (no greenhouse gases) | 255 K | +| 0.78 | ≈ 288 K, today's value | +| 1 (fully opaque) | 255 × 2^(1/4) ≈ 303 K | + +This shows the mechanism directly. The surface now gets sunlight **plus** downward infrared from the warm air, so it has to warm up until its own emission gets rid of both. The downward infrared is real and measurable: instruments on the ground record about 340 W/m² of it on average, which is more than the surface receives from the Sun. + +The model has a problem, though. In it, the effect stops growing once the layer is fully opaque (ε = 1). The real atmosphere is already opaque at the centre of the CO₂ band, yet adding CO₂ still causes warming. Explaining that needs a better picture. + +## 5. The better picture: the height at which Earth radiates to space + +Two facts about the real atmosphere matter here. + +1. **It gets colder with height.** In the lowest ~10–15 km (the troposphere), air rising and expanding cools by about **6.5 K per km** on average. This rate of cooling with height is called the *lapse rate*. Convection keeps this profile in place. +2. **It is thick in infrared.** At a wavelength the gases absorb strongly, a photon emitted low down is absorbed again almost immediately. Only photons emitted from fairly **high up**, where there is little absorbing gas above them, get out to space. + +So at each wavelength there is an **effective emission height**, roughly the level above which the air is too thin to stop the radiation. Space receives radiation at the *temperature of that height*, not the temperature of the surface. + +Averaged over all wavelengths, the Earth radiates to space like a black body at 255 K. In the real atmosphere, 255 K is found at about **5 km** up. Starting from the emission height and going down at the lapse rate: + +``` +T_surface ≈ T_emission + Γ × H ≈ 255 K + 6.5 K/km × 5 km ≈ 288 K +``` + +The same 33 K appears again, this time explained through altitude. + +**Now add CO₂.** With more absorber, the air becomes opaque higher up, so the emission height rises. A higher level is colder, and colder air radiates less (T⁴ again). For a while the planet sends out **less** than the 238 W/m² it absorbs. This shortfall is called **radiative forcing**. The planet keeps gaining energy and warms until the whole temperature profile has shifted up enough that the new, higher emission level is back at 255 K. Because the lapse rate holds the profile together, the surface ends up warmer by about Γ × (rise in emission height). + +This also deals with the "CO₂ is already saturated" objection. Being saturated at the band centre means that **surface** radiation at those wavelengths cannot escape. That was never the source of the effect. The emission level can always move higher. At the edges of the band, where absorption is weaker, extra CO₂ also moves those wavelengths from "escapes from low and warm" to "escapes from high and cold". As a result the effect grows roughly with the **logarithm** of concentration, not linearly, but it does not stop. + +## 6. Putting numbers on it + +A widely used fit from detailed radiation calculations gives the forcing from CO₂ as + +``` +ΔF ≈ 5.35 × ln(C / C₀) W/m² +``` + +Doubling CO₂ (C/C₀ = 2) gives ΔF ≈ **3.7 W/m²**. Before industrialisation C₀ was about 280 ppm. By the mid-2020s it was above 420 ppm. + +How much warming does that cause? Differentiate σT⁴ at 255 K: + +``` +d(σT⁴)/dT = 4σT³ ≈ 3.8 W/m² per K +``` + +That is how much more a black body at 255 K radiates for each degree of warming. Detailed models give about 3.2 W/m² per K for the real Earth (this is called the *Planck response*). On its own, then, doubling CO₂ warms the planet by about + +``` +ΔT ≈ 3.7 / 3.2 ≈ 1.2 K +``` + +Using the lapse rate, that corresponds to the emission height rising by only about 1.2 / 6.5 ≈ 0.18 km, under 200 metres. + +## 7. Feedbacks: the planet responds to its own warming + +The 1.2 K figure assumes nothing else changes. Several other things do change: + +- **Water vapour (strongly amplifies).** Warmer air can hold about 7% more water vapour per kelvin (the Clausius–Clapeyron relation). Water vapour is itself a greenhouse gas, so this adds to the warming. +- **Lapse rate (partly offsets).** In the tropics, warming is greater aloft than at the surface. That lets the planet radiate a little more easily, which damps the warming somewhat. +- **Ice and snow (amplifies).** Melting replaces bright surfaces with darker ones, so the albedo falls and more sunlight is absorbed. +- **Clouds (uncertain, probably amplifying overall).** Clouds both reflect sunlight, which cools, and absorb infrared, which warms. How they change is the largest single source of uncertainty. + +Feedbacks can be included by replacing the 3.2 with a smaller net "restoring" value λ. Then ΔT = ΔF / λ. Current assessments put the long-term warming from doubled CO₂ (the *equilibrium climate sensitivity*) **likely between about 2.5 and 4 K, with about 3 K as the best estimate**. That is roughly 2–3 times the no-feedback value. + +**Why CO₂ and not water vapour?** Water vapour contributes the largest share of today's greenhouse effect, around half. Clouds contribute about a quarter, and CO₂ about a fifth. But the amount of water vapour is set by temperature: if there is too much, it rains out within days. CO₂ does not condense, and it stays in the air for centuries. CO₂ therefore acts as the control setting, and water vapour amplifies whatever change CO₂ makes. + +## 8. Common misconceptions + +- **"It works like a glass greenhouse."** Not really. A real greenhouse stays warm mainly because the glass stops warm air from rising away by convection. The name stuck anyway. +- **"The gases reflect heat back down."** They absorb it and then emit according to their own temperature. The emission-height picture explains the effect better than "trapping" does. +- **"It's to do with the ozone hole."** The ozone hole is a separate problem. Ozone is a greenhouse gas, but the ozone hole does not drive global warming. +- **"Extra CO₂ can't matter if the band is saturated."** See Section 5: the emission height keeps rising. + +## 9. A fingerprint you can check + +If the Sun were causing the warming, the whole atmosphere would warm. Greenhouse warming predicts something different: the **troposphere warms while the stratosphere cools**. Above about 15 km, extra CO₂ mainly helps the air radiate its energy to space, so that layer cools. Observations since the 1960s show this pattern: a warming troposphere under a cooling stratosphere. + +--- + +### Summary + +1. In equilibrium, absorbed sunlight (~238 W/m²) equals emitted infrared (σT⁴), which gives 255 K for a planet with no greenhouse gases. +2. Sunlight is visible and earthlight is infrared, so a gas can affect one and not the other. +3. Only molecules whose vibrations change their dipole (CO₂, H₂O, CH₄, …) absorb infrared. N₂ and O₂ do not. +4. These gases make the planet radiate to space from about 5 km up, where it is 255 K. The lapse rate then puts the surface 33 K warmer. +5. More CO₂ raises the emission height, into colder air. Less energy leaves than arrives, and the planet warms until balance returns. Doubling CO₂ gives about 3.7 W/m² of forcing and 1.2 K of warming directly, and about 3 K once feedbacks are included. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/explainers/greenhouse/index.html b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/explainers/greenhouse/index.html new file mode 100644 index 0000000..9337922 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/explainers/greenhouse/index.html @@ -0,0 +1,142 @@ + + + + +Greenhouse effect: two models you can play with + + + + +

How greenhouse gases warm the Earth: interactive models

+

This page goes with explanation.md. Each model below is a few lines of arithmetic. Move the sliders and the numbers and diagrams update. All fluxes are global averages in W/m².

+ +

Model 1: one absorbing layer

+

The atmosphere is modelled as one layer. Sunlight passes straight through it. It absorbs a fraction ε of the infrared coming up from the surface, and it emits ε·σTa⁴ both up and down. Start at ε = 0, then increase it, and watch the surface temperature and the downward infrared.

+
+ + + +
+

Things to notice: (1) at ε = 0 the surface is 255 K, the temperature set by energy balance alone. (2) Around ε ≈ 0.78 it reaches the observed 288 K. (3) The model cannot go above ε = 1. The real atmosphere is already nearly opaque at the centre of the CO₂ band and still warms when CO₂ is added, which is why Model 2 is needed.

+
+ +

Model 2: emission height and lapse rate

+

Temperature falls by about Γ = 6.5 K/km with height. On average the Earth radiates to space from the level where the air is 255 K, about 5 km up. Adding CO₂ lifts that level into colder air, so less energy leaves the planet (the forcing, ΔF = 5.35·ln(C/280)). The planet warms until the outgoing radiation balances the incoming again.

+
+ + + + + +
+

Gray line: pre-industrial temperature profile. Red line: the profile after the planet has warmed by ΔT. Dots: the effective emission height. With no feedbacks (λ = 3.2), the emission level has risen by ΔT/Γ but is back at 255 K, so outgoing radiation balances again. A doubling raises it by under 200 m. The effect is small on this scale, but it accounts for all of the warming.

+
+ + + + diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/read.json new file mode 100644 index 0000000..873f2c3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "Sunlight and infrared radiation are different wavelengths. Gases can be nearly transparent to incoming sunlight while remaining opaque to outgoing infrared, because the air acts on the outgoing radiation but does little to the incoming radiation.", + "quote": "The two spectra hardly overlap. This is what makes the greenhouse effect possible. Air can be almost transparent to incoming sunlight and still be largely opaque to outgoing infrared." + }, + "2": { + "answer": "Adding CO₂ makes the atmosphere opaque at higher altitudes. Higher altitudes are colder, so they radiate less energy to space. This reduces outgoing radiation below the incoming solar energy, causing the planet to warm until the temperature profile shifts upward enough that the new emission height is back at the temperature needed for energy balance.", + "quote": "With more absorber, the air becomes opaque higher up, so the emission height rises. A higher level is colder, and colder air radiates less, so the planet sends out less energy than it absorbs." + }, + "3": { + "answer": "N₂ and O₂ molecules are symmetric, made of two identical atoms. Their symmetric structure means they develop no electric dipole when vibrating. Since infrared light acts on electric dipoles to excite molecules, these gases cannot absorb infrared.", + "quote": "N₂ and O₂ are two identical atoms. However they stretch, the molecule stays symmetric and has no dipole, so infrared light has almost nothing to act on." + }, + "4": { + "answer": "Without greenhouse gases, radiation would escape directly from the surface to space (nothing above absorbs it). The surface would have to be at 255 K to emit enough radiation to balance the 238 W/m² absorbed from the sun, following the Stefan-Boltzmann balance.", + "quote": "In equilibrium, absorbed sunlight (~238 W/m²) equals emitted infrared (σT⁴), which gives 255 K for a planet with no greenhouse gases." + } + }, + "stops": [ + "what is an 'electric dipole'? 'because the dipole is what the light's oscillating electric field can push on'", + "'Kirchhoff's law' invoked without definition: 'a layer that absorbs a fraction ε also emits ε times the black-body amount (Kirchhoff's law)'", + "'Planck response' undefined: 'this is called the Planck response'", + "why temperature decreases with height: 'air rising and expanding cools by about 6.5 K per km on average'", + "'Convection keeps this profile in place' but convection not explained", + "how does 'emission height' work? 'Only photons emitted from fairly high up...get out to space'", + "why logarithmic not linear: 'the effect grows roughly with the logarithm of concentration'", + "what is 'asymmetric stretching' of CO₂? 'When it bends, or when it stretches asymmetrically, a dipole appears'", + "CO₂ saturation paradox: 'the emission level can always move higher...the effect does not stop'", + "'Clausius–Clapeyron relation' unnamed: 'hold about 7% more water vapour per kelvin (the Clausius–Clapeyron relation)'" + ], + "excess": [ + "Wien's law formula not needed: 'Wien's law: peak ≈ 2898 µm·K / T'", + "Section 8 addresses misconceptions not raised: ''It works like a glass greenhouse.' Not really'", + "Section 9 observational fingerprint beyond mechanism: 'the troposphere warms while the stratosphere cools'", + "other greenhouse gases unnecessary for understanding: 'CH₄ (methane), N₂O (nitrous oxide) and O₃ (ozone)'", + "historical context not needed: 'Before industrialisation C₀ was about 280 ppm. By the mid-2020s it was above 420 ppm'", + "forcing formula as intermediate calculation: 'ΔF ≈ 5.35 × ln(C / C₀) W/m²'", + "feedback percentages not essential: 'Water vapour...around half. Clouds...quarter, and CO₂ about a fifth'", + "algebraic intermediate steps: 'absorbs ε·E_s = emits ε·E_a up + ε·E_a down'", + "specific model values: '| ε | T_s | 0 (no greenhouse gases) | 255 K |'", + "specific water vapor percentage: 'can hold about 7% more water vapour per kelvin'" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/run.json new file mode 100644 index 0000000..20eff7a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/none/run.json @@ -0,0 +1,13 @@ +{ + "result": "I made `explainers/greenhouse/explanation.md` and `explainers/greenhouse/index.html`. The prose builds the explanation step by step, using equations a technical reader can check with a calculator: energy balance, infrared absorption, one-layer and emission-height models, forcing and feedbacks. The page adds two models whose sliders change the inputs, so the reader can see how surface temperature responds to absorptivity, CO₂ and feedback strength; I haven't opened it in a browser yet.", + "error": null, + "cost": 0.3816954, + "seconds": 124, + "turns": 3, + "files": [ + "explainers/greenhouse/explanation.md", + "explainers/greenhouse/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/explanation.md new file mode 100644 index 0000000..0bb9f11 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/explanation.md @@ -0,0 +1,127 @@ +# How greenhouse gases warm the Earth + +**Short answer.** Earth loses heat to space only as infrared light. Greenhouse gases absorb infrared light, so most of the infrared that reaches space comes from high in the atmosphere, not from the surface. High air is cold, and cold air emits less infrared. To send out as much energy as it gets from the Sun, the whole lower atmosphere and the surface must become warmer. More CO2 lifts the emitting altitude higher, into colder air, so the surface must warm more. + +The sections below build this answer step by step, with numbers. + +## 1. The puzzle: Earth is 33 K too warm + +Energy that enters a planet must leave it, or the planet keeps warming. Start with the energy that enters. + +- Sunlight at Earth's distance delivers **1361 W/m²** to a surface that faces the Sun. +- Earth intercepts sunlight with a disc of area πR², but it has a surface of area 4πR². Spread over the whole sphere, the mean input is 1361 / 4 ≈ **340 W/m²**. +- Clouds, ice, and other bright surfaces reflect about 30% of it. Earth absorbs 340 × 0.70 ≈ **238 W/m²**. + +Now the energy that leaves. Every object emits thermal radiation. The **Stefan–Boltzmann law** gives the power per square metre from an object at absolute temperature T: + + P = σT⁴, with σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴. + +The T⁴ makes emission very sensitive to temperature: 10% warmer means about 46% more emission. + +Set emission equal to absorption and solve for T: + + T = (238 / 5.67 × 10⁻⁸)^(1/4) ≈ **255 K** (−18 °C). + +The measured mean surface temperature is **288 K** (15 °C). The surface is 33 K warmer than the simple balance predicts. A surface at 288 K emits σ × 288⁴ ≈ **390 W/m²**, but only about 238 W/m² leaves the top of the atmosphere. The atmosphere stops about **150 W/m²** of the surface emission from reaching space. The rest of this explanation shows how. + +## 2. Sunlight and Earth-light are different colours + +The wavelength at which an object emits most strongly falls as the object gets hotter (**Wien's law**: peak wavelength ≈ 2898 µm·K / T). + +| Emitter | Temperature | Peak wavelength | Type | +|---|---|---|---| +| Sun | 5800 K | 2898 / 5800 ≈ 0.5 µm | visible light | +| Earth | 288 K | 2898 / 288 ≈ 10 µm | infrared | + +The two bands hardly overlap. So the atmosphere can be nearly transparent to the incoming energy and nearly opaque to the outgoing energy. The gases that do this are the greenhouse gases. + +## 3. Why CO2 and water absorb infrared, but N2 and O2 do not + +A molecule absorbs light when the light's electric field can push its charges into a new motion. For infrared, that motion is a vibration of the atoms. The push works only if the vibration changes the molecule's **dipole**, the separation of its positive and negative charge. + +- **N2 and O2** (99% of dry air) have two identical atoms. When they stretch, the charge stays symmetric, so the dipole stays zero. They are almost transparent to infrared. +- **CO2** is O=C=O. When it bends, the carbon moves off the line of the oxygens and a dipole appears. This bend absorbs strongly near **15 µm**, close to the peak of Earth's emission. +- **H2O** is already bent and has many vibration and rotation modes. It absorbs across much of the infrared. CH4 and N2O act in the same way. + +A gas that absorbs at a wavelength also emits at that wavelength, at a rate set by its own temperature. Greenhouse gases do not reflect infrared. They absorb it, share the energy with the surrounding air through collisions, and emit new infrared in all directions. + +## 4. A first model: one absorbing layer + +Take the simplest version. Put one layer of air above the surface. It is transparent to sunlight and absorbs all infrared. + +*Predict before you read on: will the surface be warmer than 255 K, colder, or the same?* + +- Space sees only the layer. The layer must emit 238 W/m² upward, so the layer is at 255 K. +- The layer emits equally up and down. It sends 238 W/m² down to the surface as well. +- The surface receives 238 (Sun) + 238 (layer) = 476 W/m². It must emit that much: T = (476 / σ)^(1/4) ≈ **303 K**. + +So the absorbing layer warms the surface. But 303 K is 15 K too warm. The model fails for two reasons: + +1. Real air does not absorb all infrared. Between about 8 and 13 µm, few gases absorb, and some surface emission escapes directly through this **window**. +2. Warm surface air rises, so **convection** carries heat upward. Radiation is not the only path. + +The one-layer model shows *that* absorption warms the surface. It cannot show why *more* CO2 warms it further. If the layer already absorbs everything, extra CO2 would change nothing. The next model fixes this. + +## 5. The real mechanism: the emission level and the lapse rate + +Two facts replace the single layer. + +**Fact 1: infrared escapes from a height, not from the surface.** An infrared photon emitted low in the air is soon absorbed again. A photon reaches space only if little absorbing gas lies above it. So, at each wavelength, the infrared that reaches space comes mostly from one altitude, the **emission level**. Averaged over all wavelengths, Earth's emission level is about **5 km** up. + +**Fact 2: air gets colder with height.** Rising air expands as pressure falls, and expansion cools it. Convection keeps mixing the lower atmosphere (the **troposphere**), so its temperature falls at a nearly fixed rate, the **lapse rate**. The mean lapse rate is about **6.5 K per km**. + +Combine them: + +```text + altitude temperature + ~5 km ─── emission level ─── 255 K → emits 238 W/m² to space + │ air cools 6.5 K/km + │ as you go up + 0 km ─── surface ────────── 255 + 6.5 × 5 ≈ 288 K +``` + +The emission level must be at 255 K, because it must emit 238 W/m². The surface sits 5 km lower, on the same temperature slope, so it is about 33 K warmer. This matches the measured 288 K. + +## 6. Why adding CO2 warms the surface + +*Predict: the centre of the CO2 band already absorbs almost all 15 µm light within the lowest few hundred metres. Does adding more CO2 still matter?* + +It does. More CO2 means more absorbing gas above each altitude. So a photon must start higher before it can escape, and the emission level rises. The chain is: + +1. More CO2 → emission level rises. +2. Higher air is colder (lapse rate) → the emission level emits less (σT⁴). +3. Less energy leaves than arrives → the planet gains energy. +4. The troposphere and surface warm until the new, higher emission level reaches 255 K again. +5. The surface is now further below the emission level on the same slope → it is warmer. + +The imbalance in step 3 is the **radiative forcing**. A fit to detailed radiation calculations gives: + + ΔF ≈ 5.35 × ln(C / C₀) W/m², where C is the CO2 concentration and C₀ the starting value. + +The logarithm appears because the band centre is already opaque. Extra CO2 acts mostly at the edges of the band, where absorption is weaker, and each doubling widens the band by a similar amount. + +**Worked example: double CO2.** + +- Forcing: 5.35 × ln 2 ≈ **3.7 W/m²** less energy leaving. +- How much warming removes this? Differentiate P = σT⁴: dP/dT = 4σT³ = 4 × 5.67 × 10⁻⁸ × 255³ ≈ **3.8 W/m² per K**. +- Warming needed: 3.7 / 3.8 ≈ **1 K**, with nothing else changed. (Detailed calculations give about 1.2 K.) +- Equivalent rise of the emission level: 1 K / 6.5 K per km ≈ **150 m**. + +**Feedbacks amplify this.** Warmer air holds about 7% more water vapour per K, and water vapour is a greenhouse gas, so warming adds more absorption. Ice loss and clouds also respond. Water vapour cannot start a warming by itself, because extra water condenses and rains out within days; its amount follows temperature. CO2 stays in the air for centuries, so CO2 sets the level and water vapour amplifies it. Including feedbacks, the warming for doubled CO2 is *estimated* at **2.5 to 4 K** (IPCC AR6 likely range). + +## 7. When the mechanism does not warm: the stratosphere + +The argument in section 6 needs one assumption: the emission level must be colder than the air below it. Remove that assumption and the result breaks. + +- **Holds (troposphere):** temperature falls 6.5 K per km. Raising the emission level by 150 m makes it about 1 K colder, so it emits less and the surface must warm. +- **Breaks (isothermal air):** if temperature were the same at all heights, raising the emission level would not change its temperature. Emission would not change, and there would be no warming. +- **Reverses (stratosphere):** above about 12 km, ozone absorbs sunlight and temperature *rises* with height. At the very centre of the CO2 band, the emission level is already in the stratosphere. Here, more CO2 raises the emission level into warmer air, and the stratosphere cools. Satellites observe this stratospheric cooling at the same time as surface warming. A brighter Sun would warm both, so this pattern points to greenhouse gases as the cause. + +## 8. What the numbers mean + +- **238 W/m²** — the energy Earth must shed. It is fixed by sunlight and reflectivity, not by greenhouse gases. +- **255 K** — the temperature of the layer that sheds it. Greenhouse gases decide *where* that layer is, not its temperature. +- **5 km** — the emission height. Near 0 km, there would be almost no greenhouse effect (surface near 255 K). At 10 km, with the same lapse rate, the surface would be near 320 K. +- **3.7 W/m²** — about 1.5% of 238 W/m². A small change in the energy balance, held for decades, is enough to warm the surface by several kelvin. + +**Not covered here:** cloud effects in detail, how fast the oceans take up heat (the delay between forcing and warming), and the carbon cycle that sets CO2 levels. A glass greenhouse also works differently: it warms mostly because the glass stops convection, not because of infrared absorption. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/model.md new file mode 100644 index 0000000..eb2714a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/model.md @@ -0,0 +1,35 @@ +# Model: how greenhouse gases warm the Earth + +**Central question:** Why is the surface warmer than sunlight alone predicts, and why does more CO2 make it warmer? + +**Reader:** technical, school physics, new to climate. Knows energy conservation and what a molecule is. Does not know Stefan–Boltzmann, Wien, lapse rate, emission height, forcing. + +**Representation:** Stage 1, controlled prose with one text figure. The argument is a sequential causal chain; one vertical profile figure carries the only spatial idea. + +## Entities +- Sunlight (short-wave, peak ~0.5 µm). Infrared from Earth (long-wave, peak ~10 µm). +- Greenhouse gases: CO2, H2O, CH4 (3+ atoms, vibrations change dipole). N2, O2: transparent to IR. +- Emission level: the altitude from which infrared escapes to space. +- Lapse rate: temperature fall with height in the troposphere, set by convection. + +## Causal chain +1. Absorbed sunlight 238 W/m² must be balanced by 238 W/m² of IR out. +2. A body that emits 238 W/m² has T = 255 K (Stefan–Boltzmann). +3. Surface is 288 K and emits 390 W/m². The atmosphere stops ~150 W/m² of it. +4. Greenhouse gases absorb and emit IR; IR reaches space mainly from the emission level (~5 km). +5. Air cools with height (6.5 K/km), so emission level at 255 K implies surface ≈ 288 K. +6. More CO2 raises the emission level → colder emitter → less IR out → imbalance → column warms until balance returns. + +## Confusion points +- "Gases reflect heat" — they absorb and re-emit. +- "CO2 is saturated" — the emission level still rises; band wings widen. +- One-layer model overshoots (303 K). +- Mechanism needs a lapse rate; with an isothermal atmosphere raising the emission level changes nothing (stratosphere case). + +## Epistemic status +- Steps 1–5: established physics and observation. +- 5.35 ln(C/C0): fit to radiative-transfer calculations. +- Climate sensitivity 2.5–4 K: estimate (IPCC AR6 likely range). + +## Scope +Out: clouds in detail, ocean heat uptake and time lags, carbon cycle, history of measurements. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/model.yaml new file mode 100644 index 0000000..50d871d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/explainers/greenhouse/model.yaml @@ -0,0 +1,25 @@ +solar_constant_w_m2: 1361 +mean_incoming_w_m2: 340 +albedo: 0.30 +absorbed_solar_w_m2: 238 +sigma: 5.67e-8 +effective_temp_k: 255 +surface_temp_k: 288 +greenhouse_warming_k: 33 +surface_emission_w_m2: 390 +atmosphere_net_stop_w_m2: 150 +sun_temp_k: 5800 +sun_peak_um: 0.5 +earth_peak_um: 10 +wien_constant_um_k: 2898 +one_layer_surface_k: 303 +lapse_rate_k_per_km: 6.5 +emission_height_km: 5 +co2_band_um: 15 +forcing_coefficient: 5.35 +doubling_forcing_w_m2: 3.7 +planck_response_w_m2_k: 3.8 +no_feedback_warming_k: 1.0 +emission_rise_doubling_m: 150 +water_vapour_per_k_percent: 7 +sensitivity_likely_range_k: [2.5, 4] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/read.json new file mode 100644 index 0000000..6192841 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/read.json @@ -0,0 +1,48 @@ +{ + "answers": { + "1": { + "answer": "Greenhouse gases absorb infrared radiation (Earth's heat) but not visible light (sunlight), because these have very different wavelengths. The atmosphere can be transparent to the incoming visible energy while being opaque to the outgoing infrared energy.", + "quote": "The two bands hardly overlap. So the atmosphere can be nearly transparent to the incoming energy and nearly opaque to the outgoing energy." + }, + "2": { + "answer": "Adding CO2 raises the altitude from which infrared escapes to space (the emission level). Higher altitudes are colder, so they emit less infrared. To maintain energy balance and continue emitting 238 W/m² to space, the entire troposphere and surface must warm up.", + "quote": "More CO2 means more absorbing gas above each altitude. So a photon must start higher before it can escape, and the emission level rises." + }, + "3": { + "answer": "N2 and O2 have two identical atoms. When they vibrate or stretch, the charge distribution remains perfectly symmetric around the molecule's center, producing no dipole moment. Without a changing dipole, they cannot absorb infrared radiation.", + "quote": "N2 and O2 (99% of dry air) have two identical atoms. When they stretch, the charge stays symmetric, so the dipole stays zero." + }, + "4": { + "answer": "Heat would radiate directly from the surface (0 km altitude). The surface temperature would be approximately 255 K, with essentially no greenhouse effect to warm it further.", + "quote": "Near 0 km, there would be almost no greenhouse effect (surface near 255 K)." + } + }, + "stops": [ + "\"10% warmer means about 46% more emission\" — why does T⁴ scaling produce 46%?", + "\"peak wavelength ≈ 2898 µm·K / T\" — Wien's law stated without derivation or explanation", + "\"dipole, the separation of its positive and negative charge\" — abstract concept needs more grounding", + "\"window\" in \"few gases absorb, and some surface emission escapes directly through this window\" — technical term introduced without definition", + "\"Rising air expands as pressure falls, and expansion cools it\" — why does expansion cause cooling?", + "\"The mean lapse rate is about 6.5 K per km\" — why this specific value, why is it constant?", + "\"More CO2 means more absorbing gas above each altitude. So a photon must start higher before it can escape\" — causation between upper CO2 and lower photons unclear", + "\"The surface is now further below the emission level on the same slope → it is warmer\" — logical leap from position to temperature not justified", + "\"each doubling widens the band by a similar amount\" — why does logarithmic scaling follow from doubling?", + "\"the emission level must be colder than the air below it\" — why is this assumption necessary for the mechanism?" + ], + "excess": [ + "\"The sections below build this answer step by step, with numbers.\" — meta-commentary about structure", + "\"Predict before you read on: will the surface be warmer than 255 K, colder, or the same?\" — pedagogical prompt", + "\"Predict: the centre of the CO2 band already absorbs almost all 15 µm light within the lowest few hundred metres. Does adding more CO2 still matter?\" — prediction prompt", + "\"Feedbacks amplify this. Warmer air holds about 7% more water vapour per K...\" — explains real warming vs basic mechanism, beyond scope", + "\"When the mechanism does not warm: the stratosphere\" — validates mechanism but doesn't explain surface warming", + "\"What the numbers mean\" (section 8) — interpretive summary rather than new explanation", + "\"Not covered here: cloud effects in detail, how fast the oceans take up heat...\" — acknowledgment of gaps", + "\"If the layer already absorbs everything, extra CO2 would change nothing. The next model fixes this.\" — redundant with following explanation" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/run.json new file mode 100644 index 0000000..4fb140c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I wrote `explainers/greenhouse/explanation.md`, plus `model.md` and `model.yaml` as the source facts. I chose prose with one small text figure because the answer is one chain of causes with numbers, which prose handles best, and the only spatial idea, how air cools with height, fits in that figure. I couldn't run `explainer check` because I had no shell, so the numbers are hand-checked only.", + "error": null, + "cost": 0.29882359999999997, + "seconds": 89, + "turns": 4, + "files": [ + "explainers/greenhouse/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/explanation.md new file mode 100644 index 0000000..81b640b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/explanation.md @@ -0,0 +1,91 @@ +# How do greenhouse gases warm the Earth? + + + +**Greenhouse gases make the Earth radiate its energy to space from high, cold air instead of from the warm surface. Cold air emits less, so the whole lower atmosphere, and the surface with it, must be warmer before enough energy leaves.** + +The rest of this page puts numbers on each step. + +## 1. Without greenhouse gases: 255 K + + +Start from conservation of energy. Over years, the Earth must emit as much power to space as it absorbs from the Sun. This is the **energy balance**. + +- Sunlight delivers 1361 W/m² to a panel that faces the Sun. +- The Earth catches sunlight on a disc of area πR², but it emits from its whole sphere, of area 4πR². Per square metre of surface, the average input is 1361 / 4 = 340 W/m². +- The **albedo** is the fraction of sunlight that clouds, ice, and ground reflect straight back to space: 0.30. The Earth absorbs 340 × 0.70 ≈ 240 W/m². + +A body at temperature T emits σT⁴ watts per square metre (the Stefan–Boltzmann law, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴). Emission grows fast with temperature: 255 K emits 240 W/m², and 288 K emits 390 W/m². + +Set σT⁴ = 240 W/m² and solve: T = 255 K (−18 °C). This is the **emission temperature**: whatever emits the Earth's energy to space must be at 255 K. If the surface emitted directly to space, the surface would be at 255 K. + + +The division by 4 matters: with 1361 × 0.70 instead, T = 360 K (87 °C), as if every point on Earth faced the noon Sun all day. + +The measured average surface temperature is 288 K (15 °C). The surface is 33 K warmer than the emission temperature. The greenhouse gases cause that 33 K. + +## 2. What greenhouse gases do + + +Sunlight peaks near 0.5 µm, in the visible. The Earth is far cooler than the Sun, so it emits at longer wavelengths, near 10 µm. This is **infrared**: invisible radiation from the ground and the air. + +A **greenhouse gas** lets most sunlight pass but absorbs and emits infrared. Water vapour, CO₂, and methane are greenhouse gases. Nitrogen and oxygen, 99% of dry air, are not: they barely absorb infrared at all. + +So sunlight reaches the ground. But infrared from the ground is absorbed by the air above it and emitted again, in all directions, at the temperature of that air. + +## 3. Where the infrared escapes: 5 km up + + +Infrared reaches space only from air that has little greenhouse gas above it. Lower down, the air above absorbs it again. Infrared really escapes from a spread of heights, which differ by wavelength. Call their average the **emission height**. + +The air at the emission height must emit 240 W/m², so it must be at the emission temperature, 255 K. + +The air gets colder with height. Rising air expands and cools, and convection keeps mixing the lower atmosphere. The measured fall is about 6.5 K per km, the **lapse rate**. At this rate, the air reaches 255 K about 5 km up. That is the emission height. + +| Height | Temperature | +|---|---| +| 0 km (surface) | 288 K | +| 5 km (emission height) | 255 K | + +Now the 33 K has a cause: 255 K + 6.5 K/km × 5 km ≈ 288 K. The surface is warmer than the emitting air because it sits 5 km below it. + + +The surface at 288 K emits 390 W/m², more than the 240 W/m² that leaves the top. The surface does not lose that much, because the warm air just above it emits about 330 W/m² of infrared back down (a measured value). + +## 4. Adding CO₂ + +**Predict first:** CO₂ doubles, from its pre-industrial 280 ppm to 560 ppm. What happens to the emission height, and then to the surface? + + +1. More gas means the infrared must start higher before little gas remains above it. The emission height rises, on average by about 150 m (a result of radiation calculations). +2. The air 150 m higher is about 1 K colder (6.5 K/km × 0.15 km). +3. Colder air emits less: σ·254⁴ ≈ 236 W/m², against 240 W/m² at 255 K. The Earth now emits about 3.7 W/m² less than it absorbs. This imbalance, just after the change and before any warming, is the **radiative forcing**. It is small next to 240 W/m², but it persists every second until something changes. +4. The Earth gains energy and warms. Convection still sets the lapse rate, so the temperature at every height rises by the same amount. (This assumes the lapse rate stays at 6.5 K/km; in reality it changes a little, one of the feedbacks below.) +5. Warming stops when the air at the new emission height is back at 255 K. The surface is then about 1 K warmer, because it now sits 150 m further below the emitting air. + +This 1 K counts only this mechanism. Warmer air holds more water vapour, which is itself a greenhouse gas; with this and other feedbacks, the estimated warming per doubling is about 3 K, likely 2.5–4 K (assessment of the Intergovernmental Panel on Climate Change, IPCC, from models and observations). + +## 5. The condition: colder air above + + +The usual picture, "greenhouse gases trap heat like a blanket", says more gas always warms. A test case shows that it does not. + +Suppose the atmosphere had the same temperature at every height (a lapse rate of 0). Add CO₂. The emission height still rises, but the air there is still 255 K. It emits the same 240 W/m². There is no imbalance and no warming. + +So the warming depends on two things together: the gas lifts the emission height, and the air is colder higher up. + +## 6. "But CO₂ is already saturated" + + +Near the ground, CO₂ already absorbs almost all infrared at its main wavelength (15 µm). People conclude that more CO₂ can do nothing. But what reaches space is decided high up, not near the ground. High up, the air is thin, and infrared emitted there still escapes. Add CO₂, and that thin air absorbs more, so the escape moves up again, to colder air. Each doubling adds about the same forcing: 280 → 560 ppm gives 3.7 W/m², and 560 → 1120 ppm gives 3.7 W/m² again. Today's 420 ppm gives about 2.2 W/m². + +## Summary + +Energy balance requires the Earth to emit 240 W/m² to space, which holds the emitting air at 255 K. Greenhouse gases move that emitting air up to about 5 km. The lapse rate of 6.5 K/km makes the surface 33 K warmer, at 288 K. More greenhouse gas lifts the emission height and warms the surface. + +## Not covered here + +- Feedbacks in detail (water vapour, clouds, ice): IPCC AR6 Working Group I, Chapter 7. +- Why only some molecules absorb infrared: a molecular spectroscopy text. +- Why each doubling gives the same forcing: Pierrehumbert, *Principles of Planetary Climate*. +- Where added CO₂ goes (the carbon cycle), and how fast the ocean lets the surface warm. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/model.md new file mode 100644 index 0000000..7a827c1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/model.md @@ -0,0 +1,146 @@ +# Semantic model: the greenhouse effect + +> Source of truth. Every rendering derives from this file. + +## Central question + +How do greenhouse gases make the Earth's surface warmer than it would be without them? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (energy, power in watts, temperature in kelvin, molecules, light as a spectrum of wavelengths, gases cool when they expand), basic programming. New to this subject. Knows that hot objects glow and that hotter objects radiate more. Does not know the Stefan–Boltzmann law, albedo, emission height, the lapse rate, or radiative forcing. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Sunlight | Radiation from the Sun, mostly visible light, wavelengths near 0.5 µm. | established | +| Infrared | Radiation that the Earth (ground and air) emits, wavelengths near 10 µm, invisible to the eye. | established | +| Albedo | The fraction of sunlight the Earth reflects straight back to space: 0.30. | observed | +| Energy balance | In a steady state, the power the Earth absorbs from sunlight equals the power it emits to space as infrared. | established (conservation of energy) | +| Stefan–Boltzmann law | A surface at temperature T emits power σT⁴ per square metre (σ = 5.67×10⁻⁸ W m⁻² K⁻⁴). | established | +| Emission temperature | The temperature that emits 240 W/m² by σT⁴: 255 K. Energy balance fixes it. | mathematical consequence | +| Greenhouse gas | A gas that absorbs and emits infrared: water vapour, CO₂, methane. N₂ and O₂ (99% of dry air) do not. | established | +| Emission height | The average altitude from which infrared escapes to space; above it, too little greenhouse gas remains to absorb it again. ≈ 5 km today. | established (a simplification of a spread of heights across wavelengths) | +| Lapse rate | The fall of air temperature with height in the lower atmosphere: about 6.5 K per km. | observed average | +| Radiative forcing | The imbalance (W/m²) at the top of the atmosphere just after a change, such as added CO₂, before the Earth warms. | established; value is model output | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Sunlight | passes through | atmosphere | greenhouse gases absorb little of it | +| Surface | emits | infrared | σT⁴ | +| Greenhouse gas | absorbs and re-emits in all directions | infrared | part goes back down to the surface | +| Greenhouse gas amount | sets | emission height | more gas → higher emission height | +| Convection | holds | lapse rate at ≈ 6.5 K/km | rising air expands and cools | +| Energy balance | fixes | air at emission height at 255 K | | +| Lapse rate × emission height | adds to 255 K to give | surface temperature | 255 + 6.5 × 5.08 ≈ 288 K | + +## Dependencies (learning order) + +1. Energy balance and Stefan–Boltzmann → emission temperature 255 K; surface would be 255 K with no greenhouse gas. +2. Sunlight vs infrared; which gases absorb infrared. +3. Infrared escapes from the emission height, not the surface. +4. Lapse rate ties the surface to the air at the emission height. +5. Adding gas raises emission height → imbalance → warming. + +## Causal chain + +```text +more greenhouse gas → infrared escapes from higher up → the air there is colder → it emits less to space +→ Earth absorbs more than it emits (radiative forcing) → Earth warms +→ warming stops when the air at the new emission height is back at 255 K +→ the lapse rate holds, so the surface is warmer by (6.5 K/km × rise in emission height) +``` + +## Temporal sequence + +1. CO₂ is added. Emission height rises at once. +2. Outgoing infrared drops; the Earth gains energy (forcing ≈ 3.7 W/m² for doubled CO₂). +3. Surface and air warm over years to decades (the ocean slows it). The imbalance shrinks as they warm. +4. Balance returns when the air at the emission height is at 255 K again. + +## Quantities + +- Solar constant: 1361 W/m² on a surface facing the Sun. +- Average over the sphere: 1361 / 4 = 340 W/m². +- Albedo 0.30. Absorbed: 340 × 0.70 ≈ 240 W/m². +- Emission temperature: σT⁴ = 240 → T ≈ 255 K (−18 °C). +- Observed mean surface temperature: 288 K (15 °C). Difference: 33 K. +- Surface at 288 K emits σ·288⁴ ≈ 390 W/m²; only 240 W/m² leaves to space. +- Surface budget (rounded, observed estimates): gains 160 W/m² sunlight + 330 W/m² infrared from the air = 490; loses 390 W/m² infrared + 100 W/m² convection and evaporation = 490. +- Peak wavelengths: Sun (5800 K) ≈ 0.5 µm; Earth (288 K) ≈ 10 µm. +- Lapse rate 6.5 K/km (dry rising air cools 9.8 K/km; condensing water releases heat and reduces it). 33 K / 6.5 K/km ≈ 5 km: average emission height. +- Doubling CO₂: forcing ≈ 5.35 × ln 2 ≈ 3.7 W/m². 280 → 420 ppm: 5.35 × ln 1.5 ≈ 2.2 W/m². +- Air 1 K colder (254 K vs 255 K) emits about 3.8 W/m² less. 1 K at 6.5 K/km is about 150 m of rise. +- Warming that restores balance, no feedbacks: 3.7 / 3.76 ≈ 1 K, where 3.76 W/m² per K = 4σ·255³ is the extra emission per 1 K of warming. +- With feedbacks (mainly water vapour): about 3 K per doubling (IPCC AR6 best estimate; likely 2.5–4 K). +- CO₂: about 280 ppm in 1750, about 420 ppm in 2024. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| No greenhouse gas | surface 255 K | surface emits straight to space | +| Present atmosphere | surface 288 K | emission height ≈ 5 km, 33 K colder than surface | +| Doubled CO₂, no feedbacks | ≈ +1 K | emission height rises ≈ 150 m | +| Isothermal atmosphere (no lapse rate) | added gas causes no warming | emission height rises but the air there is not colder | +| Stratosphere (above ≈ 12 km, temperature rises with height) | added CO₂ cools the stratosphere | observed | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Divide solar constant by 4 | Use 1361 W/m² directly | σT⁴ = 1361 × 0.7 → T ≈ 360 K (87 °C). The Earth intercepts sunlight on a disc of area πR² but emits from a sphere of area 4πR². | +| Explain by emission height + lapse rate | "Gases trap heat like a blanket" | A blanket picture says more gas always warms and gives no number. Isothermal air is a counterexample: emission height rises, emitting air is still 255 K, outgoing stays 240 W/m², no warming. The trapping picture also suggests saturation ends the effect; it does not. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| More greenhouse gas warms the surface (assumes temperature falls with height) | Emission height 5.08 → 5.23 km; air there 1 K colder emits ≈ 3.7 W/m² less; Earth warms until that air is 255 K; surface 288 → 289 K | Isothermal air: emitting air is 255 K at any height → no imbalance, no warming. Stratosphere, where temperature rises with height: added CO₂ cools it. | +| Mechanism: energy balance sets 255 K | 340 × 0.7 = 238 ≈ 240 W/m²; σ·255⁴ ≈ 240 W/m² | — | +| Mechanism: more gas raises emission height | infrared escapes where little gas remains above; with twice the gas, that amount of gas above lies higher | at wavelengths no gas absorbs (8–13 µm "window", mostly) the surface emits straight to space and nothing changes; averaged over all wavelengths the rise is ≈ 150 m | +| Saturation does not end the effect | 280 → 560 ppm: 3.7 W/m²; 560 → 1120 ppm: again 3.7 W/m² | — | + +## Terms + +- **Sunlight:** short-wave radiation from the Sun (near 0.5 µm). +- **Infrared:** long-wave radiation the Earth emits (near 10 µm). Here, "infrared" always means the Earth's. +- **Albedo:** fraction of sunlight reflected. 0.30 → 240 W/m² absorbed; at 0, 340 W/m² would be absorbed. +- **Greenhouse gas:** a gas that absorbs and emits infrared. +- **Energy balance:** absorbed power = emitted power. +- **Emission temperature:** 255 K, the temperature that emits 240 W/m². One name only; not "balance temperature", not "no-atmosphere temperature". +- **Emission height:** the average altitude from which infrared escapes to space; ≈ 5 km today. 0 km: no greenhouse gas, surface 255 K. Each extra km: surface 6.5 K warmer. +- **Lapse rate:** 6.5 K drop per km. If 0, added gas does nothing; if larger, the same rise warms more. +- **Radiative forcing:** the imbalance just after a change, before the Earth warms. 3.7 W/m² for doubled CO₂ (compare 240 W/m² absorbed). It shrinks to 0 as the Earth warms. + +## Epistemic status + +- **Observed / established:** solar constant, albedo, surface 288 K, CO₂ concentrations, absorption spectra, lapse rate average, stratospheric cooling, surface budget (rounded). +- **Mathematical consequence:** 255 K from σT⁴ = 240; ≈ 1 K no-feedback warming. +- **Assumed:** a single emission height (really a spread across wavelengths); a fixed lapse rate. +- **Estimated / model output:** 3.7 W/m² forcing (radiation models); 3 K with feedbacks (IPCC assessment, likely 2.5–4 K). +- **Disputed:** that the effect exists is not disputed; the size of cloud feedback is the main uncertainty. + +## Confusion points + +- "CO₂ already absorbs all the infrared at its main wavelength, so more does nothing" → more gas still raises the emission height; each doubling adds about the same 3.7 W/m². +- "Greenhouse gases trap heat like a blanket" → the gases absorb and re-emit; warming comes from escape at a colder height. (A real glass greenhouse works mainly by stopping warm air from rising: a different mechanism.) +- "N₂ and O₂ are most of the air, so they matter most" → they do not absorb infrared. +- "390 W/m² leaves the surface but only 240 reaches space — energy is lost" → the air sends 330 W/m² back down; the surface budget balances. + +## Scope + +- **Out of scope:** feedbacks in detail (water vapour, clouds, ice) — IPCC AR6 WG1 Chapter 7. +- **Out of scope:** why a molecule must change its charge separation when it vibrates to absorb infrared — a molecular spectroscopy text. +- **Out of scope:** the carbon cycle and where added CO₂ goes. +- **Out of scope:** why forcing is logarithmic in CO₂ — Pierrehumbert, *Principles of Planetary Climate*. +- **Out of scope:** how fast warming happens (ocean heat uptake). + +## Representation decision + +- **Stage:** 1 +- **Reason:** The argument is a short causal chain with arithmetic. One table of numbers carries the comparison. No parameter needs exploration that a worked example with numbers does not give. +- **Levels:** L1 gases make the surface 33 K warmer / L2 emission from a colder height / L3 energy balance and lapse rate / L4 forcing numbers. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/model.yaml new file mode 100644 index 0000000..a77b8ef --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/model.yaml @@ -0,0 +1,119 @@ +# Machine-readable half of model.md. `explainer check greenhouse` holds every rendering to it. + +values: + solar_constant_wm2: 1361 + sunlight_avg_wm2: 340 + albedo: 0.30 + absorbed_wm2: 240 + emission_temp_k: 255 + emission_temp_c: -18 + surface_temp_k: 288 + surface_temp_c: 15 + greenhouse_warming_k: 33 + surface_emits_wm2: 390 + surface_sun_gain_wm2: 160 + back_radiation_wm2: 330 + convection_wm2: 100 + surface_budget_wm2: 490 + sunlight_um: 0.5 + infrared_um: 10 + lapse_rate_k_per_km: 6.5 + emission_height_km: 5 + forcing_2xco2_wm2: 3.7 + forcing_now_wm2: 2.2 + height_rise_m: 150 + warming_no_feedback_k: 1 + warming_with_feedback_k: 3 + warming_likely_range_k: [2.5, 4] + sigma: 5.67e-8 + absorbed_fraction: 0.70 + sun_temp_k: 5800 + temp_at_2km_k: 275 + height_rise_km: 0.15 + emission_at_254k_wm2: 236 + wrong_temp_k: 360 + wrong_temp_c: 87 + co2_1750_ppm: 280 + co2_2024_ppm: 420 + co2_double_ppm: 560 + n2_o2_share_pct: 99 + +allow: [4, 2, 1750, 2024, 254, 12, 1120] + +require: + - path: emission_temp_k + in: [prose] + - path: surface_temp_k + in: [prose] + - path: absorbed_wm2 + in: [prose] + - path: lapse_rate_k_per_km + in: [prose] + - path: emission_height_km + in: [prose] + - path: forcing_2xco2_wm2 + in: [prose] + +claims: + - id: balance-255 + kind: mechanism + statement: "Energy balance fixes the temperature of whatever emits to space at 255 K: absorbed 240 W/m² = σT⁴." + example: "1361/4 = 340 W/m²; ×0.70 = 240 W/m²; σ·255⁴ ≈ 240 W/m². Observed surface is 288 K, 33 K warmer." + - id: why-divide-4 + kind: why + about: [divide-by-4] + statement: "Sunlight is averaged over the sphere: the Earth intercepts it on a disc πR² but emits from a sphere 4πR²." + alternative: "Use 1361 W/m² directly." + counterexample: "σT⁴ = 1361 × 0.7 gives T ≈ 360 K (87 °C), as if every point faced the noon Sun all day." + - id: which-gases + kind: definition + statement: "Greenhouse gases absorb and emit infrared (~10 µm) but let sunlight (~0.5 µm) through; N₂ and O₂ do not absorb infrared." + - id: emission-height + kind: mechanism + statement: "Infrared escapes to space from the emission height, about 5 km up, where too little greenhouse gas remains above to absorb it again; the air there must be 255 K." + example: "Surface 288 K = 255 K + 6.5 K/km × 5.08 km." + ask: "Where does the infrared that reaches space mostly come from, what is the temperature there, and why that height and that temperature?" + - id: more-gas-warms + kind: guarantee + statement: "Adding greenhouse gas raises the emission height; because temperature falls with height, the air there is colder and emits less, so the Earth warms until that air is back at 255 K." + example: "Doubled CO₂: emission height rises ~150 m; that air is 1 K colder and emits ~3.7 W/m² less; the surface warms ~1 K before feedbacks." + counterexample: "In isothermal air the higher emitting air is still 255 K: outgoing stays 240 W/m², no warming. In the stratosphere, where temperature rises with height, added CO₂ cools." + ask: "Explain, step by step, how doubling CO₂ in today's atmosphere leads to a warmer surface, and when the warming stops." + - id: why-not-blanket + kind: why + about: [blanket] + statement: "The 'blanket that traps heat' picture is replaced by emission from a colder height." + alternative: "Gases trap heat like a blanket." + counterexample: "The blanket picture predicts warming in isothermal air (there is none) and predicts that saturated absorption ends the effect (each doubling still adds 3.7 W/m²)." + - id: surface-budget + kind: mechanism + statement: "The surface emits 390 W/m², more than the 240 W/m² that reaches space, because the air sends 330 W/m² of infrared back down." + example: "Gains 160 sunlight + 330 from the air = 490; loses 390 infrared + 100 convection and evaporation = 490." + - id: saturation + kind: misconception + statement: "Saturation does not stop the effect: more gas still raises the emission height; each doubling adds about 3.7 W/m²." + correction: "Escape happens high up where the gas is thin; more gas still raises the emission height. 280→560 ppm and 560→1120 ppm each give about 3.7 W/m²." + ask: "CO₂ already absorbs nearly all infrared at its main wavelength near the ground. Does adding more CO₂ still warm the Earth? Why?" + +accept_unjustified: {ln: "forcing formula 5.35 ln(C/C0) only computes numbers; why it is logarithmic is out of scope"} + +terms: + - term: emission temperature + means: "255 K, the temperature that emits 240 W/m²" + avoid: [balance temperature, no-atmosphere temperature, effective temperature] + - term: infrared + means: "the long-wave radiation the Earth emits" + avoid: [heat radiation, longwave radiation] + +budget: {prose: 1200, reason: "the mechanism needs four linked steps (balance, emission height, lapse rate, adding gas), each with numbers, plus the counterexample and the saturation misconception"} + +quiz: + - q: "Without greenhouse gases, what would the Earth's average surface temperature be, roughly, and what sets that number?" + expect: "About 255 K (−18 °C): absorbed sunlight (240 W/m²) must equal emitted infrared σT⁴." + misconception: "Absolute zero, or the same as today." + - q: "Adding CO₂ makes the Earth emit less infrared to space at first. Why?" + expect: "Infrared escapes from a higher altitude, where the air is colder, and colder air emits less (σT⁴)." + misconception: "CO₂ traps heat / reflects heat back like a mirror, with no role for altitude or temperature falling with height." + - q: "If the atmosphere had the same temperature at every height, would adding CO₂ warm the surface?" + expect: "No: the emission height rises but the air there is at the same temperature, so outgoing infrared does not change." + misconception: "Yes, because more CO₂ always traps more heat." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/narrative.md new file mode 100644 index 0000000..1988c44 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/narrative.md @@ -0,0 +1,57 @@ +# Narrative: greenhouse + +> Pass 3. Every rendering follows these beats. + +## 1. Reader before and after + +- **Before:** a technical reader (school maths and science, basic programming), new to climate physics. Knows watts, kelvin, molecules, wavelengths, that hotter things radiate more, that a gas cools when it expands. +- **After:** "The Earth must emit 240 W/m² to space, which fixes the temperature of the air that emits it at 255 K. Greenhouse gases move that emitting air up to about 5 km, and the surface is 6.5 K warmer per km below it. More gas raises the emitting air, so the surface warms." + +## 2. Question and motive + +- **Question:** How do greenhouse gases warm the Earth? Result in words: they make the Earth emit its infrared to space from high, cold air instead of from the warm surface; to emit enough from cold air, the whole column (and the surface) must be warmer. +- **Why care:** the familiar "trapping heat" picture cannot say how much, and it suggests the effect stops once CO₂ absorbs everything. The correct picture gives numbers and answers that objection. +- **Why this approach:** start from energy conservation, the one thing no reader doubts. It gives a temperature (255 K) that disagrees with the measured 288 K. The 33 K gap is the thing to explain. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| W/m² | power per square metre of the Earth's surface | 1361 W/m² of sunlight on a panel facing the Sun | 2 | +| albedo | fraction of sunlight reflected | 0.30 → 30% of 340 W/m² reflected | 2 | +| energy balance | absorbed = emitted | 240 in, 240 out | 2 | +| σT⁴ (Stefan–Boltzmann) | power per m² a body emits at temperature T | 255 K → 240 W/m²; 288 K → 390 W/m² | 2 | +| emission temperature | 255 K, the temperature that emits 240 W/m² | σ·255⁴ ≈ 240 | 2 | +| infrared | radiation the Earth emits, near 10 µm | sunlight peaks near 0.5 µm, 20× shorter | 3 | +| greenhouse gas | gas that absorbs and emits infrared | CO₂, water vapour, methane; not N₂, O₂ | 3 | +| emission height | average height from which infrared escapes to space | ≈ 5 km | 4 | +| lapse rate | temperature drop per km of height, 6.5 K/km | 288 − 6.5 × 5 ≈ 255 | 4 | +| radiative forcing | imbalance right after a change, before warming | 3.7 W/m² for doubled CO₂ | 5 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How do greenhouse gases warm the Earth? | — | answer in two sentences | gases move the emission of infrared up into cold air; the surface must be warmer so that cold air emits enough | the result in words | +| 2 | instance | What temperature would the Earth have without them? | start from energy | sunlight 1361 W/m² on a panel facing the Sun; ÷4 because a disc πR² catches it and a sphere 4πR² emits (without ÷4: 360 K); albedo 0.30 reflected → 240 W/m² absorbed; σT⁴ = power per m² a body emits at temperature T; σT⁴ = 240 → 255 K. Measured surface: 288 K | energy balance fixes 255 K for whatever emits to space; the measured surface is 288 K | 33 K to explain | +| 3 | name | What is different about greenhouse gases? | so the atmosphere must change where the emission comes from | sunlight 0.5 µm vs infrared 10 µm (a cooler body emits at longer wavelengths); CO₂, water vapour, methane absorb infrared; N₂, O₂ do not | the gases pass sunlight, absorb and re-emit infrared | the gases act on outgoing infrared only | +| 4 | mechanism | Then where does the 240 W/m² leave from? | but absorbed infrared is re-emitted in all directions; only infrared emitted where little gas remains above gets out | table: height vs temperature | the air that emits to space must emit 240 W/m², so it must be 255 K. Measured air reaches 255 K about 5 km up: the emission height. Air cools 6.5 K/km because rising air expands and cools, and convection keeps this profile. 255 + 6.5 × 5 ≈ 288 | the 33 K is the lapse rate times the emission height | +| 5 | objection | The surface emits σ·288⁴ ≈ 390 W/m², but only 240 leaves the top. Where does the rest go? | but | surface budget, each term named | the air emits 330 W/m² of infrared down. Surface gains 160 sunlight + 330 from the air; loses 390 infrared + 100 carried up by rising warm air and evaporation | the numbers add up | +| 6 | test (predict) | What happens when CO₂ doubles (280 → 560 ppm)? | therefore | prediction prompt, then: emission height rises ≈ 150 m; that air is 1 K colder; it emits 3.7 W/m² less, the radiative forcing; the Earth warms until the emitting air is 255 K: surface ≈ 1 K warmer. One line: water vapour and other feedbacks raise this to ≈ 3 K (out of scope) | more gas → higher → colder → less out → warm | the causal chain with numbers | +| 7 | test | Does more gas always warm? | but the mechanism needs colder air above | isothermal case | if air had the same temperature at every height, the higher emitting air would still be 255 K: outgoing unchanged, no warming | the assumption that matters | +| 8 | objection | Isn't CO₂ already saturated? | so | 280 → 560 ppm: 3.7 W/m²; 560 → 1120 ppm: 3.7 W/m² again | saturation near the ground does not matter, because escape happens high up, where the gas is thin; more gas still lifts the emission height. Each doubling adds about the same forcing | the saturation answer | +| 9 | close | — | so | recap | "Greenhouse gases move the Earth's emission to space up to about 5 km, where the air is 33 K colder than the surface. Energy balance holds that air at 255 K, so the surface sits at 288 K. More gas lifts the emission height and warms the surface." Scope: feedbacks, carbon cycle, timing | the "after" sentence | + +## 5. Concrete to symbol + +σT⁴: first 255 K → 240 W/m² and 288 K → 390 W/m², then the symbol with σ = 5.67×10⁻⁸. Surface = 255 + 6.5 × h: first with h = 5 km (288 K), then h = 5.15 km (289 K). + +## 6. Links between representations + +The height-temperature table and the sentence "255 K at 5 km" use the same numbers; the doubled-CO₂ row repeats the table with the emission height moved. + +## 7. Close the loop + +- **Answer:** Greenhouse gases make the Earth emit to space from high, cold air. Energy balance holds that air at 255 K, and the lapse rate makes the surface warmer by 6.5 K per km below it. +- **General argument:** any change that raises the emission height, while temperature falls with height, warms the surface. +- **Payoff:** the reader can predict the effect of more CO₂, of no lapse rate, and answer the saturation objection. Scope: feedbacks, carbon cycle, timing. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/decisions.yaml new file mode 100644 index 0000000..b8d38d7 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/decisions.yaml @@ -0,0 +1,532 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T152224-probe#1: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#10: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#11: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#12: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#13: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#14: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#15: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#2: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#3: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#4: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#5: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#6: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#7: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#8: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#9: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152459-coldread-narrative#1: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#10: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#11: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#12: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#13: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#14: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#15: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#16: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#17: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#18: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#19: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#2: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#20: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#21: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#22: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#23: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#24: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#25: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#26: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#27: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#28: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#29: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#3: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#30: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#31: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#32: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#33: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#34: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#35: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#36: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#37: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#38: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#39: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#4: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#40: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#41: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#42: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#43: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#44: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#45: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#46: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#47: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#48: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#49: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#5: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#50: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#51: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#52: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#53: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#54: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#6: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#7: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#8: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#9: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152723-coldread-prose#1: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#10: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#2: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#3: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#4: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#5: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#6: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#7: + decision: declined + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#8: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#9: + decision: declined + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152840-quiz-prose#1: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#10: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#11: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#12: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#13: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#14: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#15: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#16: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#17: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#18: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#19: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#2: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#20: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#21: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#22: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#23: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#3: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#4: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#5: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#6: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#7: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#8: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#9: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T153030-quiz-prose#1: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#10: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#11: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#12: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#13: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#14: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#15: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#16: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#17: + decision: adopted + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#18: + decision: adopted + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#19: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#2: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#20: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#21: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#22: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#3: + decision: adopted + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#4: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#5: + decision: adopted + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#6: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#7: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#8: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#9: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152224-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152224-probe.json new file mode 100644 index 0000000..959d08f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152224-probe.json @@ -0,0 +1,212 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "greenhouse", + "at": "2026-10-04T15:22:24", + "model": "default", + "prompt_sha": "5539eb6fbccb", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "Why the surface warms by exactly 'lapse rate × rise in emission height'. Nothing says why air gets colder with height, or what links the surface temperature to the temperature at the emission height.", + "why_it_matters": "The causal chain jumps from 'the emission layer is back at 255 K' to 'the surface is warmer'. That only follows if convection holds the temperature profile at a fixed 6.5 K/km, so the whole profile shifts together. The lapse rate is listed only as an 'observed average', and the reader is never told why higher air is colder. The step the central answer depends on is left unstated.", + "suggested_addition": "Add: rising air expands at lower pressure and cools as it does (dry air cools about 9.8 K/km; moist air about 6.5 K/km on average because condensing water releases heat). Convection keeps this profile in place, so surface temperature = emission temperature + 6.5 K/km × emission height. Today that is 255 + 6.5×5.08 ≈ 288 K. After doubling: 255 + 6.5×5.23 ≈ 289 K, so the surface is about 1 K warmer. State that this assumes the lapse rate stays fixed." + }, + { + "rule": 2, + "severity": "main", + "about": "Why more greenhouse gas raises the emission height, and why the rise is only about 150 m", + "why_it_matters": "The relationship 'more gas → higher emission height' is the key step of the mechanism, but the model only asserts it. It gives no reason and no worked example. The 150 m figure is worked backwards from the forcing, not from the gas itself.", + "suggested_addition": "Add: infrared escapes from roughly the level where the gas above it can absorb it only about once (optical depth ≈ 1). Air thins by a factor e every ~8 km (the scale height). So doubling a well-mixed gas moves that level up by about 8 km × ln 2 ≈ 5.5 km, but only at wavelengths where the gas absorbs and the air at that level is still colder than below. At most wavelengths nothing changes. Averaged over the whole spectrum, the effective rise is ≈ 150 m (3.7 W/m² ÷ 3.76 W/m² per K ÷ 6.5 K/km ≈ 0.15 km)." + }, + { + "rule": 4, + "severity": "main", + "about": "The surface emits 390 W/m² but only 240 W/m² leaves to space. The model never says where the other 150 W/m² goes.", + "why_it_matters": "A reader who sees these two numbers will ask where the difference went. Without back-radiation, the numbers look like a violation of the energy balance the model itself sets up.", + "suggested_addition": "Add a surface budget with numbers: the surface absorbs ≈160 W/m² of sunlight plus ≈340 W/m² of infrared emitted downward by the greenhouse gases. It loses ≈390 W/m² as infrared plus ≈100 W/m² carried upward by convection and evaporation. Only ≈240 W/m² leaves the top of the atmosphere, from ≈5 km up." + }, + { + "rule": 3, + "severity": "main", + "about": "One quantity, 255 K, has several names: 'no-atmosphere temperature', 'balance temperature', 'emission temperature', and 'the new emission height is back at 255 K'", + "why_it_matters": "The reader may take these for different quantities. 'Emission temperature' appears in Relationships, but it is defined only in the last Terms section and is missing from Entities.", + "suggested_addition": "Use one name, 'emission temperature (255 K)'. Define it in Entities before Relationships: the temperature that radiates 240 W/m² by σT⁴. Change 'emission height is back at 255 K' to 'the air at the emission height is back at 255 K'." + }, + { + "rule": 3, + "severity": "edge", + "about": "Two pairs of names, and terms used without definition", + "why_it_matters": "Two concepts each have two names: 'imbalance' vs 'radiative forcing', and 'emission height' vs 'emitting layer'/'layer'/'that air'. Albedo, Wien's law, feedbacks, 'saturated' and 'band' are used but never defined. The Audience section says the reader does not know Wien's law.", + "suggested_addition": "Say that forcing is the imbalance at the moment CO₂ is added (3.7 W/m²), and that the imbalance shrinks toward 0 as the Earth warms. Define albedo as the fraction of sunlight reflected (0.30 → 240 W/m² absorbed; 0.0 would give 340 W/m² and ≈278 K). Define feedback, band and saturation in one line each, or drop Wien's formula." + }, + { + "rule": 2, + "severity": "edge", + "about": "The guarantee 'energy balance sets 255 K' has no example where it breaks", + "why_it_matters": "Rule 2 asks for a case where removing the assumption breaks the guarantee. The table leaves that cell as '—'. The two assumptions are steady state and fixed albedo.", + "suggested_addition": "Breaks without steady state: today the Earth absorbs ≈1 W/m² more than it emits, so emission is about 239 W/m² and the effective temperature is ≈254.7 K, not 255 K. Breaks without fixed albedo: at albedo 0.25, absorbed is 255 W/m² and T ≈ 259 K." + }, + { + "rule": 2, + "severity": "edge", + "about": "The claim that 'each doubling adds about the same 3.7 W/m²' (logarithmic) has no worked example and no case where it fails", + "why_it_matters": "This is an always-type claim, used in the saturation confusion point.", + "suggested_addition": "280→560 ppm: 5.35 ln 2 ≈ 3.7 W/m². 560→1120 ppm: also 3.7 W/m². 280→420 ppm: 5.35 ln 1.5 ≈ 2.2 W/m². It breaks at very low concentrations, where absorption grows roughly in proportion to the amount, so from 0→280 ppm the log formula would give an infinite forcing." + }, + { + "rule": 1, + "severity": "edge", + "about": "The 'why T⁴' argument is circular", + "why_it_matters": "The 390 W/m² figure is itself computed with σT⁴, so it cannot show that T⁴ is needed rather than linear T.", + "suggested_addition": "Say that 390 W/m² is a measured surface emission (≈ 396 W/m² in observed budgets). Show that a linear law fitted to 240 W/m² at 255 K would predict 240 × 288/255 ≈ 271 W/m²." + }, + { + "rule": 5, + "severity": "edge", + "about": "The meaning of 3.76 W/m² per K (the Planck response) and of the 3 K climate sensitivity", + "why_it_matters": "These numbers appear in the Quantities section, but the model never says what a low or high value would mean for the reader.", + "suggested_addition": "3.76 W/m² per K is the extra emission for each 1 K of warming. A larger value means less warming per W/m² of forcing. Sensitivity of 2.5 K vs 4 K means 2.5 K vs 4 K of eventual warming per doubling of CO₂." + }, + { + "rule": 4, + "severity": "edge", + "about": "The temporal sequence step 'warm over years to decades (the ocean slows it)' has no numbers", + "why_it_matters": "It is a mechanism step with no worked example.", + "suggested_addition": "With 3.7 W/m² heating a 100 m ocean mixed layer (heat capacity ≈ 4.2×10⁸ J/m²/K), the first 1 K takes about 4.2×10⁸ / 3.7 ≈ 1.1×10⁸ s ≈ 3.6 years if no extra heat were emitted. In reality it takes longer, because emission grows as the Earth warms and heat mixes into the deep ocean." + } + ], + "excess": [ + "Wien's law constant 2898 µm·K (only the two peak wavelengths are needed)", + "CO₂ history 280 ppm (1750) / 420 ppm (2024)", + "IPCC 2.5–4 K likely range and feedback-inclusive 3 K (feedbacks are declared out of scope)", + "Ozone-hole confusion point", + "Stratospheric cooling as a fingerprint (useful but not needed for the central question)" + ], + "next_questions": [ + { + "q": "Why is the air colder higher up, and why does the surface stay tied to the emission-height temperature?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Where do the other 150 W/m² that the surface emits go? Does the atmosphere radiate back down?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why does adding gas raise the emission height, and why by only about 150 m?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If no-feedback warming is about 1 K, why is the expected warming 3 K, and how much warming should 280→420 ppm already have caused?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "If CO₂ absorption is saturated, why does each doubling still add about 3.7 W/m²?", + "answered_by_model": true, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.1644216, + "seconds": 59 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "Why the surface warms by exactly 'lapse rate × rise in emission height'. Nothing says why air gets colder with height, or what links the surface temperature to the temperature at the emission height. → Add: rising air expands at lower pressure and cools as it does (dry air cools about 9.8 K/km; moist air about 6.5 K/km on average because condensing water releases heat). Convection keeps this profile in place, so surface temperature = emission temperature + 6.5 K/km × emission height. Today that is 255 + 6.5×5.08 ≈ 288 K. After doubling: 255 + 6.5×5.23 ≈ 289 K, so the surface is about 1 K warmer. State that this assumes the lapse rate stays fixed." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "Why more greenhouse gas raises the emission height, and why the rise is only about 150 m → Add: infrared escapes from roughly the level where the gas above it can absorb it only about once (optical depth ≈ 1). Air thins by a factor e every ~8 km (the scale height). So doubling a well-mixed gas moves that level up by about 8 km × ln 2 ≈ 5.5 km, but only at wavelengths where the gas absorbs and the air at that level is still colder than below. At most wavelengths nothing changes. Averaged over the whole spectrum, the effective rise is ≈ 150 m (3.7 W/m² ÷ 3.76 W/m² per K ÷ 6.5 K/km ≈ 0.15 km)." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 4", + "text": "The surface emits 390 W/m² but only 240 W/m² leaves to space. The model never says where the other 150 W/m² goes. → Add a surface budget with numbers: the surface absorbs ≈160 W/m² of sunlight plus ≈340 W/m² of infrared emitted downward by the greenhouse gases. It loses ≈390 W/m² as infrared plus ≈100 W/m² carried upward by convection and evaporation. Only ≈240 W/m² leaves the top of the atmosphere, from ≈5 km up." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 3", + "text": "One quantity, 255 K, has several names: 'no-atmosphere temperature', 'balance temperature', 'emission temperature', and 'the new emission height is back at 255 K' → Use one name, 'emission temperature (255 K)'. Define it in Entities before Relationships: the temperature that radiates 240 W/m² by σT⁴. Change 'emission height is back at 255 K' to 'the air at the emission height is back at 255 K'." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 3", + "text": "Two pairs of names, and terms used without definition → Say that forcing is the imbalance at the moment CO₂ is added (3.7 W/m²), and that the imbalance shrinks toward 0 as the Earth warms. Define albedo as the fraction of sunlight reflected (0.30 → 240 W/m² absorbed; 0.0 would give 340 W/m² and ≈278 K). Define feedback, band and saturation in one line each, or drop Wien's formula." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 2", + "text": "The guarantee 'energy balance sets 255 K' has no example where it breaks → Breaks without steady state: today the Earth absorbs ≈1 W/m² more than it emits, so emission is about 239 W/m² and the effective temperature is ≈254.7 K, not 255 K. Breaks without fixed albedo: at albedo 0.25, absorbed is 255 W/m² and T ≈ 259 K." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 2", + "text": "The claim that 'each doubling adds about the same 3.7 W/m²' (logarithmic) has no worked example and no case where it fails → 280→560 ppm: 5.35 ln 2 ≈ 3.7 W/m². 560→1120 ppm: also 3.7 W/m². 280→420 ppm: 5.35 ln 1.5 ≈ 2.2 W/m². It breaks at very low concentrations, where absorption grows roughly in proportion to the amount, so from 0→280 ppm the log formula would give an infinite forcing." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 1", + "text": "The 'why T⁴' argument is circular → Say that 390 W/m² is a measured surface emission (≈ 396 W/m² in observed budgets). Show that a linear law fitted to 240 W/m² at 255 K would predict 240 × 288/255 ≈ 271 W/m²." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 5", + "text": "The meaning of 3.76 W/m² per K (the Planck response) and of the 3 K climate sensitivity → 3.76 W/m² per K is the extra emission for each 1 K of warming. A larger value means less warming per W/m² of forcing. Sensitivity of 2.5 K vs 4 K means 2.5 K vs 4 K of eventual warming per doubling of CO₂." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 4", + "text": "The temporal sequence step 'warm over years to decades (the ocean slows it)' has no numbers → With 3.7 W/m² heating a 100 m ocean mixed layer (heat capacity ≈ 4.2×10⁸ J/m²/K), the first 1 K takes about 4.2×10⁸ / 3.7 ≈ 1.1×10⁸ s ≈ 3.6 years if no extra heat were emitted. In reality it takes longer, because emission grows as the Earth warms and heat mixes into the deep ocean." + }, + { + "n": 11, + "severity": "excess", + "kind": "model entry", + "text": "Wien's law constant 2898 µm·K (only the two peak wavelengths are needed)" + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "CO₂ history 280 ppm (1750) / 420 ppm (2024)" + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "IPCC 2.5–4 K likely range and feedback-inclusive 3 K (feedbacks are declared out of scope)" + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Ozone-hole confusion point" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Stratospheric cooling as a fingerprint (useful but not needed for the central question)" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152459-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152459-coldread-narrative.json new file mode 100644 index 0000000..4ec6281 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152459-coldread-narrative.json @@ -0,0 +1,526 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "greenhouse", + "at": "2026-10-04T15:24:59", + "model": "default", + "prompt_sha": "46e8105a72c2", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: answer in two sentences. Said: gases move the emission of infrared up into cold air; the surface must be warmer so that cold air emits enough", + "unresolved": [ + "emits enough" + ], + "unsaid": [], + "leap": [ + "\"the surface must be warmer so that cold air emits enough\": nothing yet says why the surface has to warm when the air higher up emits. The link between them (temperature falling with height) only comes in beat 4. That's fine for a hook, but the reader holds it as a puzzle." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (instance)", + "text": "Shown: 1361/4 = 340; ×0.70 = 240; σT⁴ = 240 → 255 K; why ÷4 (360 K otherwise). Said: energy balance fixes 255 K", + "unresolved": [ + "1361", + "×0.70", + "σT⁴", + "σ", + "energy balance", + "360 K otherwise" + ], + "unsaid": [ + "1361 is never said to be the sunlight arriving per m², and its units are not given", + "0.70 (the 30% reflected, i.e. albedo) is shown but never named or explained", + "the reason for ÷4 is announced ('why ÷4') but the words don't give it", + "σT⁴ appears as a formula, but the words never say it is the power a body emits per m² at temperature T", + "288 K observed and the 33 K gap: these are listed as what the reader now knows, but they appear in neither Shown nor Said" + ], + "leap": [ + "Why is 255 K a puzzle? Without the measured 288 K, the reader has no gap to explain." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (name)", + "text": "Shown: sunlight 0.5 µm vs infrared 10 µm; which gases absorb. Said: the gases pass sunlight, absorb and re-emit infrared; N₂, O₂ do not", + "unresolved": [], + "unsaid": [ + "the 0.5 µm vs 10 µm contrast is shown, but the words never say that the Earth emits at 10 µm because it is cold, or why the difference matters", + "'which gases absorb': the gases that do absorb (CO₂, water vapour) are never named in the words. Only the ones that don't (N₂, O₂) are." + ], + "leap": [ + "Why does the Earth's own emission sit in the infrared? This reader knows hotter things radiate more, but not that a cooler body emits at longer wavelengths." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Shown: table: height vs temperature; emission height ≈ 5 km at 255 K. Said: infrared escapes where little gas remains above; that air must be 255 K; air cools 6.5 K/km (convection: rising air expands and cools); 255 + 6.5 × 5 ≈ 288", + "unresolved": [], + "unsaid": [ + "'emission height ≈ 5 km' is shown, but the words never say where 5 km comes from" + ], + "leap": [ + "\"that air must be 255 K\": beat 2 presented 255 K as the Earth's temperature without the gases. Reusing it as the temperature of the air that emits to space needs a sentence like 'whatever emits the 240 W/m² must be at 255 K'.", + "Why does re-emission move the escape point upward? Re-emitted infrared goes in every direction, and the step from that to 'escape happens where little gas remains above' is compressed.", + "288 is treated as the target here, but the reader was never given it as the measured value" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (objection)", + "text": "Shown: surface budget. Said: the air sends 330 W/m² back down; 160 + 330 = 390 + 100", + "unresolved": [ + "160", + "100", + "390" + ], + "unsaid": [ + "the objection itself ('the surface emits 390 W/m² but only 240 leaves') appears in neither Said nor Shown", + "160 (sunlight absorbed at the surface) and 100 (heat carried up by convection and evaporation) are never identified", + "390 W/m² for the surface is never derived in the beats (σ·288⁴)" + ], + "leap": [ + "The equation balances, but the reader can't tell what each term is, so it can't be checked" + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 6 (test, predict)", + "text": "Shown: prediction prompt, then: rise ≈ 150 m, 1 K colder, −3.7 W/m², warm until balance, ≈ 1 K (≈ 3 K with feedbacks). Said: more gas → higher → colder → less out → warm", + "unresolved": [ + "feedbacks", + "−3.7 W/m²" + ], + "unsaid": [ + "how much CO₂ is added (a doubling) is never said", + "the 150 m rise is shown without saying where it comes from", + "−3.7 W/m² is shown but never named as the imbalance before warming" + ], + "leap": [ + "≈ 1 K becoming ≈ 3 K 'with feedbacks' is given without explanation" + ], + "excess": [ + "\"(≈ 3 K with feedbacks)\": the argument doesn't need it, and it brings in an idea it never explains" + ], + "purpose_unclear": false + }, + { + "at": "Beat 7 (test)", + "text": "Shown: isothermal case; stratosphere cools. Said: no lapse rate, no warming", + "unresolved": [ + "lapse rate", + "stratosphere", + "isothermal" + ], + "unsaid": [ + "'stratosphere cools' is shown and never mentioned in the words" + ], + "leap": [ + "Why would the stratosphere cool? Nothing says that temperature rises with height there." + ], + "excess": [ + "\"stratosphere cools\": not needed for the main line, and unexplained" + ], + "purpose_unclear": false + }, + { + "at": "Beat 8 (objection)", + "text": "Shown: 280→560: 3.7; 560→1120: 3.7. Said: saturation near the ground does not stop the rise of the emission height", + "unresolved": [ + "saturation", + "280→560", + "3.7" + ], + "unsaid": [ + "the units (ppm of CO₂, W/m²) are not given", + "that the figures show the same forcing for each doubling is not said in words" + ], + "leap": [ + "Why does each doubling give the same 3.7? Not explained." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (close)", + "text": "Shown: recap sentence. Said: answer in the opening's words; scope pointers", + "unresolved": [ + "scope pointers" + ], + "unsaid": [ + "neither the recap sentence nor the general argument is actually written in the beat. It only describes them." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 2: 288 K observed / 33 K gap is in neither Shown nor Said. Without it there is no puzzle, and beat 4's '≈ 288' has nothing to match.", + "Beat 2: 'σT⁴' and '×0.70' are never explained, so the 255 K that everything after depends on rests on symbols the reader hasn't been given.", + "Beat 4: 'that air must be 255 K' goes unexplained (it reuses beat 2's result for the emitting layer). This is the key step of the argument.", + "Beat 4: 'emission height ≈ 5 km' is shown with no source, so 255 + 6.5 × 5 ≈ 288 looks tuned to fit." + ], + "cuts": [ + "Beat 7: 'stratosphere cools' (unexplained and off the main line)", + "Beat 6: '(≈ 3 K with feedbacks)'", + "Beat 5: cut the whole surface budget, or state the objection and name the terms; as written it costs a reread and gives nothing back" + ], + "question": { + "text": "How do greenhouse gases warm the Earth's surface? The reader knows this at beat 1.", + "known_at": "Beat 1", + "result_stated": "Yes, at beat 1: \"gases move the emission of infrared up into cold air; the surface must be warmer so that cold air emits enough\"" + }, + "motive": "Missing in the beats. The reason to care (the 'trapping heat' picture can't give amounts and suggests saturation) appears only in the author's plan. The reason for the approach (start from energy conservation and show the 33 K gap) is only partly there: beat 2's bridge says 'start from energy', but the gap is never shown.", + "close": "Can't be confirmed. Beat 9 says only 'answer in the opening's words; scope pointers' and 'recap sentence', with no actual wording. The general argument (any rise in emission height warms the surface, as long as temperature falls with height) is not said in the beats. Beat 7's 'no lapse rate, no warming' comes closest.", + "recap": "Greenhouse gases make the Earth radiate to space from high, cold air that has to sit at 255 K, and because air warms by about 6.5 K per km going down, the surface 5 km below ends up about 33 K warmer.", + "strongest": "Beat 6: the said causal chain 'more gas → higher → colder → less out → warm', with a prediction prompt and numbers.", + "weakest": "Beat 5: an objection that is never stated, with unnamed numbers '160 + 330 = 390 + 100'. Runner-up: beat 2 leaves out 288 K.", + "suggestions": [ + "In beat 2, say '1361 W/m² of sunlight arrives; 30% is reflected; σT⁴ is the power per m² emitted at temperature T', and state 'measured: 288 K, so 33 K is missing'.", + "In beat 4, add 'whatever emits the 240 W/m² to space must be at 255 K, and now that is air at about 5 km, not the surface'.", + "In beat 1, voice the motive: 'trapping heat' can't say how much, and suggests the effect stops once CO₂ absorbs everything.", + "In beat 5, state the objection and name the 160 (absorbed sunlight) and 100 (convection and evaporation) terms.", + "In beat 8, give units (ppm, W/m²) and say 'each doubling adds the same 3.7 W/m²'.", + "Write the closing sentence and the general argument into beat 9 itself." + ] + }, + "usage": { + "cost": 0.15388179999999999, + "seconds": 57 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 288 K observed / 33 K gap is in neither Shown nor Said. Without it there is no puzzle, and beat 4's '≈ 288' has nothing to match." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'σT⁴' and '×0.70' are never explained, so the 255 K that everything after depends on rests on symbols the reader hasn't been given." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'that air must be 255 K' goes unexplained (it reuses beat 2's result for the emitting layer). This is the key step of the argument." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'emission height ≈ 5 km' is shown with no source, so 255 + 6.5 × 5 ≈ 288 looks tuned to fit." + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 7: 'stratosphere cools' (unexplained and off the main line)" + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 6: '(≈ 3 K with feedbacks)'" + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 5: cut the whole surface budget, or state the objection and name the terms; as written it costs a reread and gives nothing back" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): emits enough" + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): \"the surface must be warmer so that cold air emits enough\": nothing yet says why the surface has to warm when the air higher up emits. The link between them (temperature falling with height) only comes in beat 4. That's fine for a hook, but the reader holds it as a puzzle." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): 1361" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): ×0.70" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): σT⁴" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): σ" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): energy balance" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): 360 K otherwise" + }, + { + "n": 16, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): 1361 is never said to be the sunlight arriving per m², and its units are not given" + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): 0.70 (the 30% reflected, i.e. albedo) is shown but never named or explained" + }, + { + "n": 18, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): the reason for ÷4 is announced ('why ÷4') but the words don't give it" + }, + { + "n": 19, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): σT⁴ appears as a formula, but the words never say it is the power a body emits per m² at temperature T" + }, + { + "n": 20, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): 288 K observed and the 33 K gap: these are listed as what the reader now knows, but they appear in neither Shown nor Said" + }, + { + "n": 21, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (instance): Why is 255 K a puzzle? Without the measured 288 K, the reader has no gap to explain." + }, + { + "n": 22, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (name): the 0.5 µm vs 10 µm contrast is shown, but the words never say that the Earth emits at 10 µm because it is cold, or why the difference matters" + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (name): 'which gases absorb': the gases that do absorb (CO₂, water vapour) are never named in the words. Only the ones that don't (N₂, O₂) are." + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (name): Why does the Earth's own emission sit in the infrared? This reader knows hotter things radiate more, but not that a cooler body emits at longer wavelengths." + }, + { + "n": 25, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): 'emission height ≈ 5 km' is shown, but the words never say where 5 km comes from" + }, + { + "n": 26, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): \"that air must be 255 K\": beat 2 presented 255 K as the Earth's temperature without the gases. Reusing it as the temperature of the air that emits to space needs a sentence like 'whatever emits the 240 W/m² must be at 255 K'." + }, + { + "n": 27, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): Why does re-emission move the escape point upward? Re-emitted infrared goes in every direction, and the step from that to 'escape happens where little gas remains above' is compressed." + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): 288 is treated as the target here, but the reader was never given it as the measured value" + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (objection): 160" + }, + { + "n": 30, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (objection): 100" + }, + { + "n": 31, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (objection): 390" + }, + { + "n": 32, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (objection): the objection itself ('the surface emits 390 W/m² but only 240 leaves') appears in neither Said nor Shown" + }, + { + "n": 33, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (objection): 160 (sunlight absorbed at the surface) and 100 (heat carried up by convection and evaporation) are never identified" + }, + { + "n": 34, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (objection): 390 W/m² for the surface is never derived in the beats (σ·288⁴)" + }, + { + "n": 35, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (objection): The equation balances, but the reader can't tell what each term is, so it can't be checked" + }, + { + "n": 36, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (test, predict): feedbacks" + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (test, predict): −3.7 W/m²" + }, + { + "n": 38, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (test, predict): how much CO₂ is added (a doubling) is never said" + }, + { + "n": 39, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (test, predict): the 150 m rise is shown without saying where it comes from" + }, + { + "n": 40, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (test, predict): −3.7 W/m² is shown but never named as the imbalance before warming" + }, + { + "n": 41, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (test, predict): ≈ 1 K becoming ≈ 3 K 'with feedbacks' is given without explanation" + }, + { + "n": 42, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): lapse rate" + }, + { + "n": 43, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): stratosphere" + }, + { + "n": 44, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): isothermal" + }, + { + "n": 45, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (test): 'stratosphere cools' is shown and never mentioned in the words" + }, + { + "n": 46, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test): Why would the stratosphere cool? Nothing says that temperature rises with height there." + }, + { + "n": 47, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (objection): saturation" + }, + { + "n": 48, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (objection): 280→560" + }, + { + "n": 49, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (objection): 3.7" + }, + { + "n": 50, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (objection): the units (ppm of CO₂, W/m²) are not given" + }, + { + "n": 51, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (objection): that the figures show the same forcing for each doubling is not said in words" + }, + { + "n": 52, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (objection): Why does each doubling give the same 3.7? Not explained." + }, + { + "n": 53, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (close): scope pointers" + }, + { + "n": 54, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 9 (close): neither the recap sentence nor the general argument is actually written in the beat. It only describes them." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152723-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152723-coldread-prose.json new file mode 100644 index 0000000..2f659d8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152723-coldread-prose.json @@ -0,0 +1,381 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "greenhouse", + "at": "2026-10-04T15:27:23", + "model": "default", + "prompt_sha": "363f5ff7238d", + "reply": { + "paragraphs": [ + { + "at": "Line 5 (bold lead)", + "text": "Greenhouse gases make the Earth send its heat to space from high, cold air instead of from the warm surface...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 7", + "text": "The rest of this page puts numbers on each step.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 12", + "text": "Start from conservation of energy... energy balance.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 list, line 14", + "text": "Sunlight delivers 1361 W/m² to a panel that faces the Sun.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 list, line 15", + "text": "The Earth catches sunlight on a disc of area πR², but it emits from its whole sphere...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 list, line 16", + "text": "The albedo is the fraction of sunlight... 240 W/m².", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 19", + "text": "The division by 4 matters. Without it, the calculation below gives 360 K...", + "unresolved": [], + "leap": [ + "\"the calculation below gives 360 K\": this points ahead to a calculation the reader has not seen yet, so they cannot check the number at this point" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 21", + "text": "A body at temperature T emits σT⁴ watts per square metre...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 23", + "text": "Set σT⁴ = 240 W/m² and solve: T = 255 K... emission temperature", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 25", + "text": "The measured average surface temperature is 288 K... The greenhouse gases cause that 33 K.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 line 30", + "text": "The Sun is hot (about 5800 K)... This is infrared", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 line 32", + "text": "A greenhouse gas lets most sunlight pass but absorbs and emits infrared...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 line 34", + "text": "So sunlight reaches the ground. But infrared from the ground is absorbed by the air...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 line 39", + "text": "Infrared reaches space only from air that has little greenhouse gas above it... emission height.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 line 41", + "text": "The air at the emission height must emit 240 W/m², so it must be at the emission temperature, 255 K.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 line 43", + "text": "The air gets colder with height... the lapse rate.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 table, lines 45-49", + "text": "Height / Temperature: 0 km 288 K, 2 km 275 K, 5 km 255 K", + "unresolved": [], + "leap": [], + "excess": [ + "\"| 2 km | 275 K |\": this row is never used in the argument" + ], + "purpose_unclear": false + }, + { + "at": "§3 line 51", + "text": "Now the 33 K has a cause: 255 K + 6.5 K/km × 5 km ≈ 288 K.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 line 54", + "text": "Where does the surface's extra emission go? ... The air emits 330 W/m² of infrared down to the surface.", + "unresolved": [], + "leap": [ + "\"The air emits 330 W/m² of infrared down to the surface\": the figure is given with no reason why it is so large, or why it is larger than the 240 W/m² that leaves the top" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 surface budget table, lines 56-60", + "text": "sunlight 160 | infrared 390; infrared from the air 330 | rising warm air and evaporation 100", + "unresolved": [], + "leap": [ + "\"sunlight | 160\": §1 said the Earth absorbs 240 W/m², and the text never says where the other 80 W/m² goes (into the air?). The reader stops to reconcile the two figures", + "The table explains the surface's warmth by downward infrared, but it is not tied back to the emission-height account. The reader may wonder which of the two is the real explanation" + ], + "excess": [ + "The whole surface-budget aside: the main argument (emission height + lapse rate) does not need it" + ], + "purpose_unclear": false + }, + { + "at": "§4 line 64", + "text": "Predict first: CO₂ doubles, from its pre-industrial 280 ppm to 560 ppm...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 1, line 67", + "text": "More gas means the infrared must start higher... about 150 m.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 2, line 68", + "text": "The air 150 m higher is about 1 K colder", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 3, line 69", + "text": "Colder air emits less... radiative forcing.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 4, line 70", + "text": "The Earth gains energy and warms. The lapse rate holds, so the whole profile warms together.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 5, line 71", + "text": "Warming stops when the air at the new emission height is back at 255 K...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 line 73", + "text": "This 1 K counts only this mechanism... about 3 K (IPCC best estimate). Today's 420 ppm already gives a forcing of about 2.2 W/m².", + "unresolved": [ + "IPCC" + ], + "leap": [ + "\"Today's 420 ppm already gives a forcing of about 2.2 W/m²\": the reader cannot get 2.2 from 3.7 per doubling, because the rule that each doubling adds the same forcing does not come until §6" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 line 78", + "text": "The usual picture, \"greenhouse gases trap heat like a blanket\", says more gas always warms...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 line 80", + "text": "Suppose the atmosphere had the same temperature at every height (a lapse rate of 0)...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 line 82", + "text": "So the warming depends on two things together...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§6 line 87", + "text": "Near the ground, CO₂ already absorbs almost all infrared at its main wavelength (15 µm)...", + "unresolved": [], + "leap": [ + "\"the escape happens high up, where the air is thin\": the text does not say why thin air makes the near-ground saturation irrelevant (thin air absorbs less, so it is not saturated there)" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Summary line 91", + "text": "Energy balance requires the Earth to emit 240 W/m² to space...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered, lines 95-98", + "text": "Feedbacks in detail...; Why only some molecules absorb infrared...; Why each doubling gives the same forcing...; Where added CO₂ goes...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "The surface-budget aside in §3 (line 54 and the table at lines 56-60) is the biggest cut. It is off the main line, and its 160 W/m² sunlight figure clashes with the 240 W/m² from §1 without explanation. Cut it, or account for the missing 80 W/m² in one clause.", + "§4 line 73: \"Today's 420 ppm already gives a forcing of about 2.2 W/m².\" Cut it, or move it to §6 after the per-doubling rule.", + "§3 table: the 2 km row, which is never used." + ], + "question": { + "text": "Why does adding gases that absorb infrared make the surface warmer, and by how much?", + "known_at": "The title (line 1), and the bold lead (line 5) right after it", + "result_stated": "Yes, in words, at line 5: \"Greenhouse gases make the Earth send its heat to space from high, cold air instead of from the warm surface. Cold air emits less, so the whole lower atmosphere, and the surface with it, must be warmer before enough energy leaves.\"" + }, + "motive": "There is a reason for the approach: line 78, \"The usual picture, 'greenhouse gases trap heat like a blanket', says more gas always warms. A test case shows that it does not.\" It comes late, in §5. There is no reason to care about the question near the start. The stakes (CO₂ doubling, \"Today's 420 ppm\") only appear in §4.", + "close": "Yes. Line 91: \"Greenhouse gases move that emitting air up to about 5 km. The lapse rate of 6.5 K/km makes the surface 33 K warmer, at 288 K. More greenhouse gas lifts the emission height and warms the surface.\" Line 82 also states the general condition: \"the gas lifts the emission height, and the air is colder higher up.\"", + "recap": "Greenhouse gases push the level where Earth's heat escapes to space higher up, into colder air, and because air gets colder with height, the surface has to sit warmer below that level; more CO₂ pushes the level higher and warms the surface more.", + "strongest": "§5 (lines 78-82): the test with a lapse rate of 0 shows that warming needs both a rising emission height and colder air above, which also explains why the blanket picture falls short. §4's numbered chain (lines 67-71) is a close second.", + "weakest": "The §3 surface budget (lines 54-60). Its 160 W/m² sunlight figure contradicts the 240 W/m² from §1 with no explanation, and the section brings in a back-radiation picture that is never tied to the emission-height argument.", + "suggestions": [ + "In the surface budget, say that the air and clouds absorb about 80 W/m² of sunlight directly, so the 160 and 240 figures fit together.", + "Add one sentence near the top on why this matters, for example: it is how scientists predict the warming from CO₂ that people add.", + "In §6, say that thin air high up absorbs too little to be saturated, which is why adding gas there still matters.", + "Spell out IPCC (Intergovernmental Panel on Climate Change) at first use." + ] + }, + "usage": { + "cost": 0.1570202, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "The surface-budget aside in §3 (line 54 and the table at lines 56-60) is the biggest cut. It is off the main line, and its 160 W/m² sunlight figure clashes with the 240 W/m² from §1 without explanation. Cut it, or account for the missing 80 W/m² in one clause." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "§4 line 73: \"Today's 420 ppm already gives a forcing of about 2.2 W/m².\" Cut it, or move it to §6 after the per-doubling rule." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "§3 table: the 2 km row, which is never used." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "§1 line 19: \"the calculation below gives 360 K\": this points ahead to a calculation the reader has not seen yet, so they cannot check the number at this point" + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "§3 line 54: \"The air emits 330 W/m² of infrared down to the surface\": the figure is given with no reason why it is so large, or why it is larger than the 240 W/m² that leaves the top" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "§3 surface budget table, lines 56-60: \"sunlight | 160\": §1 said the Earth absorbs 240 W/m², and the text never says where the other 80 W/m² goes (into the air?). The reader stops to reconcile the two figures" + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "§3 surface budget table, lines 56-60: The table explains the surface's warmth by downward infrared, but it is not tied back to the emission-height account. The reader may wonder which of the two is the real explanation" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "§4 line 73: IPCC" + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "§4 line 73: \"Today's 420 ppm already gives a forcing of about 2.2 W/m²\": the reader cannot get 2.2 from 3.7 per doubling, because the rule that each doubling adds the same forcing does not come until §6" + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "§6 line 87: \"the escape happens high up, where the air is thin\": the text does not say why thin air makes the near-ground saturation irrelevant (thin air absorbs less, so it is not saturated there)" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152840-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152840-quiz-prose.json new file mode 100644 index 0000000..f5e5cc5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152840-quiz-prose.json @@ -0,0 +1,227 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "greenhouse", + "at": "2026-10-04T15:28:28", + "model": "default", + "prompt_sha": "709adbb037ff", + "reply": { + "answers": { + "1": "The main objects are the Sun (sunlight, 1361 W/m²), the Earth's surface, and the atmosphere. Within the atmosphere they are the greenhouse gases (water vapour, CO₂, methane) and the non-absorbing gases (nitrogen and oxygen). The key quantities are albedo (0.30), infrared radiation, the energy balance, the emission temperature (255 K), the emission height (about 5 km), the lapse rate (6.5 K/km) and radiative forcing.", + "2": "Sunlight passes through the greenhouse gases and is absorbed by the ground, apart from the 30% that is reflected. The ground emits infrared. Greenhouse gases absorb it and emit it again at the temperature of the air around them. Infrared escapes to space only from air that has little greenhouse gas above it, which on average is about 5 km up. Energy balance requires 240 W/m² out, so the air at that height must be at 255 K. The lapse rate then sets every lower layer, including the surface, warmer than that.", + "3": "Greenhouse gases move the place where Earth's infrared escapes to space up into cold air. Because the emitting air must be at 255 K to balance the incoming sunlight, and air warms by 6.5 K per km going down, the surface ends up at about 288 K. Adding gas raises the emission height and so warms the surface further.", + "4": "The most important variable is the amount of CO₂. Doubling it raises the emission height by about 150 m. Air there is about 1 K colder, so it emits about 3.7 W/m² less than the 240 W/m² absorbed. That imbalance is the radiative forcing. The Earth then gains energy, and the whole temperature profile warms until the new emission height is back at 255 K. The surface ends up about 1 K warmer from this mechanism alone, or about 3 K with feedbacks such as water vapour (IPCC best estimate).", + "5": "Assumption: the lapse rate stays fixed at 6.5 K/km while the planet warms, and the whole profile shifts together. The 5 km emission height is also an average that stands in for many wavelengths. Observation: the measured average surface temperature is 288 K, and the downward infrared from the air at the surface is about 330 W/m², which the text calls a measured value.", + "6": "The Earth must emit as much power as it absorbs, about 240 W/m², and that fixes the temperature of whatever emits to space at 255 K. Greenhouse gases absorb and re-emit infrared, so the radiation that escapes comes from about 5 km up rather than from the ground. Air cools by about 6.5 K per km with height, so the surface 5 km below is about 33 K warmer, at 288 K. Adding CO₂ lifts the emission height into colder air, which briefly cuts the outgoing infrared, so the Earth warms until the emitting air is back at 255 K. This works only because the air is colder higher up, and it still works when the lower air is 'saturated', because what escapes is decided high up.", + "7": "The hardest part was the step from '1 K colder' to '3.7 W/m² less'. The text never shows how that number comes from σT⁴; a single line using the Stefan–Boltzmann law at 255 K would have closed the gap. It is also unclear why 'the lapse rate holds' during warming, and what exactly sets 'little greenhouse gas above it', and therefore the 5 km height.", + "8": "About 255 K (−18 °C). It is set by energy balance: absorbed sunlight is 1361/4 × 0.70 ≈ 240 W/m², and σT⁴ = 240 W/m² gives T = 255 K. The division by 4 comes from catching sunlight on a disc but emitting from a whole sphere.", + "9": "The extra gas pushes the emission height up by about 150 m, and the air there is about 1 K colder. Colder air emits less, so before anything warms, outgoing infrared drops by about 3.7 W/m².", + "10": "No. The emission height would still rise, but the air there would still be at 255 K and would emit the same 240 W/m². There would be no imbalance and no warming.", + "11": "About 255 K. This repeats question 8: 240 W/m² of absorbed sunlight must be emitted, and σT⁴ = 240 gives 255 K, with the surface itself doing the emitting. The measured 288 K is 33 K higher, and the text attributes that difference to greenhouse gases.", + "12": "Mostly from air at about 5 km altitude, the emission height, which is at about 255 K, the emission temperature.", + "13": "Nothing. The raised emission level would be at the same temperature, so it would emit the same 240 W/m², giving no imbalance and no warming. Warming needs both a higher emission height and colder air higher up.", + "14": "Yes. What reaches space is decided high up, where the air is thin and infrared still escapes. More CO₂ makes that thin air absorb more, so the escape moves higher into colder air, and emission falls. Each doubling adds about the same forcing of about 3.7 W/m². The text defers the reason for this per-doubling rule to Pierrehumbert." + }, + "audit": { + "terms": [ + "The escape level has several names: 'emission height' (§3, §4), 'emitting air' (lead paragraph, Summary), 'the escape moves up' (§6), and 'little greenhouse gas above it' (§3). A reader could take these for different things.", + "'radiative forcing' (§4 step 3) is also called 'imbalance' (§4 step 3, §5 'no imbalance') and 'forcing' (§6).", + "'emission temperature' (§1) is mostly written just as '255 K' (§3, §4 step 5, Summary).", + "'heat' (lead paragraph: 'send its heat to space') and 'infrared' (§2 onward) are used for the same outgoing radiation." + ], + "unexplained": [ + "'3.7 W/m² less' (§4 step 3): the text does not show how a 1 K drop produces this number, although σT⁴ would let the reader check it.", + "'Today's 420 ppm gives about 2.2 W/m²' (§6): there is no way to get 2.2 from 420. The reader cannot see why 1.5× the CO₂ gives about 60% of a doubling.", + "'about 5 km up' (§3) and 'on average by about 150 m' (§4 step 1): both are given without any basis, and the text does not say what would make them larger or smaller.", + "'about 330 W/m² ... back down (a measured value)' (§3): this is never connected to the 390 and 240 figures, for example as the net loss of 60 W/m² from the surface, or to any of the questions.", + "'about 3 K' (§4): the text does not explain how feedbacks turn 1 K into 3 K." + ], + "missing_why": [ + "§4 step 4, 'The lapse rate holds, so the whole profile warms together': the text does not say why the lapse rate stays the same under warming.", + "§6, 'Each doubling adds about the same forcing': the logarithmic rule is stated and its reason sent elsewhere, but the 2.2 W/m² figure depends on it.", + "§3, 'a fall of about 6.5 K per km': the mechanism is named (expansion and convection), but not why the value is 6.5 rather than something else.", + "§3, 'Infrared reaches space only from air that has little greenhouse gas above it': 'little' is never quantified, so the 5 km figure cannot be derived." + ], + "excess": [ + "§1, 'with 1361 × 0.70 instead, T = 360 K (87 °C)': a counterfactual detour that the argument does not use.", + "§3, the surface-budget paragraph (390 vs 330 W/m²): a side point that the causal chain and the questions do not need.", + "§2, 'The Sun is hot (about 5800 K), so its light peaks near 0.5 µm': this reader already knows hotter things radiate at shorter wavelengths. Only 'Earth emits near 10 µm' is needed.", + "§3, 'Rising air expands under lower pressure and cools as it expands': this reader already knows a gas cools when it expands." + ] + }, + "gaps": [ + "No derivation of the 3.7 W/m² forcing from the 1 K temperature change.", + "No reason given for why the lapse rate stays fixed as the planet warms.", + "The 5 km emission height and the 150 m rise are asserted without any basis.", + "The logarithmic per-doubling forcing, and so the 2.2 W/m² at 420 ppm, cannot be checked from the text.", + "The text does not explain how water vapour and other feedbacks triple the 1 K to about 3 K.", + "Questions 8/11 and 10/13 are duplicates. The explanation answers both pairs, but the duplication adds no new evidence." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 1, + "13": 2, + "14": 2 + }, + "reasons": { + "12": "Gives the location (about 5 km, the emission height) and the temperature (about 255 K), but explains neither. It never says infrared escapes from that height because too little greenhouse gas remains above it to absorb the radiation again. It also never says the air there must be 255 K because energy balance requires 240 W/m² out. The answer is bare labels without the defining mechanism." + }, + "pass": false + }, + "usage": { + "cost": 0.18852180000000002, + "seconds": 45 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The escape level has several names: 'emission height' (§3, §4), 'emitting air' (lead paragraph, Summary), 'the escape moves up' (§6), and 'little greenhouse gas above it' (§3). A reader could take these for different things." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'radiative forcing' (§4 step 3) is also called 'imbalance' (§4 step 3, §5 'no imbalance') and 'forcing' (§6)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'emission temperature' (§1) is mostly written just as '255 K' (§3, §4 step 5, Summary)." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'heat' (lead paragraph: 'send its heat to space') and 'infrared' (§2 onward) are used for the same outgoing radiation." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'3.7 W/m² less' (§4 step 3): the text does not show how a 1 K drop produces this number, although σT⁴ would let the reader check it." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'Today's 420 ppm gives about 2.2 W/m²' (§6): there is no way to get 2.2 from 420. The reader cannot see why 1.5× the CO₂ gives about 60% of a doubling." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 5 km up' (§3) and 'on average by about 150 m' (§4 step 1): both are given without any basis, and the text does not say what would make them larger or smaller." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'about 330 W/m² ... back down (a measured value)' (§3): this is never connected to the 390 and 240 figures, for example as the net loss of 60 W/m² from the surface, or to any of the questions." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'about 3 K' (§4): the text does not explain how feedbacks turn 1 K into 3 K." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "§4 step 4, 'The lapse rate holds, so the whole profile warms together': the text does not say why the lapse rate stays the same under warming." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "§6, 'Each doubling adds about the same forcing': the logarithmic rule is stated and its reason sent elsewhere, but the 2.2 W/m² figure depends on it." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "§3, 'a fall of about 6.5 K per km': the mechanism is named (expansion and convection), but not why the value is 6.5 rather than something else." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "§3, 'Infrared reaches space only from air that has little greenhouse gas above it': 'little' is never quantified, so the 5 km figure cannot be derived." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "§1, 'with 1361 × 0.70 instead, T = 360 K (87 °C)': a counterfactual detour that the argument does not use." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "§3, the surface-budget paragraph (390 vs 330 W/m²): a side point that the causal chain and the questions do not need." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "§2, 'The Sun is hot (about 5800 K), so its light peaks near 0.5 µm': this reader already knows hotter things radiate at shorter wavelengths. Only 'Earth emits near 10 µm' is needed." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "§3, 'Rising air expands under lower pressure and cools as it expands': this reader already knows a gas cools when it expands." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "No derivation of the 3.7 W/m² forcing from the 1 K temperature change." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "No reason given for why the lapse rate stays fixed as the planet warms." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "The 5 km emission height and the 150 m rise are asserted without any basis." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "The logarithmic per-doubling forcing, and so the 2.2 W/m² at 420 ppm, cannot be checked from the text." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "The text does not explain how water vapour and other feedbacks triple the 1 K to about 3 K." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "Questions 8/11 and 10/13 are duplicates. The explanation answers both pairs, but the duplication adds no new evidence." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153030-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153030-quiz-prose.json new file mode 100644 index 0000000..8898ea2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153030-quiz-prose.json @@ -0,0 +1,230 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "greenhouse", + "at": "2026-10-04T15:30:15", + "model": "default", + "prompt_sha": "a3fd476ca87a", + "reply": { + "answers": { + "1": "The main objects are the Sun's light (about 1361 W/m²), the Earth's surface, the atmosphere and the way its temperature changes with height, greenhouse gases (water vapour, CO₂ and methane), the gases that don't absorb infrared (nitrogen and oxygen), infrared radiation, the emission height (about 5 km up), the emission temperature (255 K), the lapse rate (6.5 K/km), albedo (0.30), the energy balance and radiative forcing.", + "2": "Sunlight passes through the greenhouse gases and is absorbed. Albedo reflects 30% of it, leaving about 240 W/m². The Earth must send the same 240 W/m² to space as infrared. Greenhouse gases absorb infrared and emit it again, so infrared only escapes from air with little gas above it. That air is at the emission height, and it must be at 255 K to emit 240 W/m². Convection fixes how fast the air cools with height (the lapse rate). So the surface is warmer than the emitting air by the lapse rate times the emission height. Air just above the surface sends about 330 W/m² back down to it.", + "3": "Greenhouse gases move the place where infrared escapes up into colder air. Because the emitting air must stay at 255 K to balance the incoming sunlight, and the air warms at 6.5 K per km going down, the surface ends up warmer: 288 K instead of 255 K. Adding more gas lifts the emission height further, so the surface warms more.", + "4": "The key variable is the amount of greenhouse gas, for example CO₂ doubling from 280 to 560 ppm. The emission height rises by about 150 m. The air there is about 1 K colder, so it emits about 3.7 W/m² less (236 instead of 240 W/m²). The Earth gains energy and the whole temperature profile warms until the new emission height is back at 255 K. The surface ends up about 1 K warmer from this mechanism alone, and about 3 K warmer once feedbacks such as water vapour are included. If the lapse rate were 0, nothing would change.", + "5": "Assumption: the lapse rate stays fixed at 6.5 K/km while the Earth warms (the text says in reality it changes a little). Established facts: the measured average surface temperature is 288 K, and satellites see outgoing infrared coming on average from about 5 km up. Another measured fact is that the air sends about 330 W/m² back down to the surface.", + "6": "The Earth absorbs about 240 W/m² of sunlight, after the factor of 4 for geometry and the 30% reflected, and must emit the same amount, which needs an emitter at 255 K. Greenhouse gases absorb infrared and emit it again, so the infrared that reaches space comes from about 5 km up, where little gas remains above. Convection makes the air warm by 6.5 K per km going down, so the surface is about 33 K warmer than that emitting layer, at 288 K. Adding CO₂ lifts the emission height into colder air, which emits less. The Earth then gains energy until that layer is back at 255 K, leaving the surface warmer: about 1 K per doubling directly, about 3 K with feedbacks.", + "7": "The hardest part was step 1 of section 4: why the emission height rises by exactly 150 m. The number is just given, and it seems to be worked backwards from the 3.7 W/m² forcing. It would help to show where 150 m comes from, or to say plainly that it is inferred. The saturation section was also hard: the claim that each doubling gives the same forcing, and how 420 ppm gives 2.2 W/m², are given without reasons. One sentence on why the effect goes by doublings would make it easier.", + "8": "About 255 K (−18 °C). It is set by the energy balance: σT⁴ = 240 W/m². The 240 W/m² comes from 1361 W/m² divided by 4 (sunlight hits a disc of area πR² but the Earth emits from a sphere of area 4πR²), times 0.70 (the part not reflected by albedo).", + "9": "More CO₂ means the infrared has to start from higher up before little gas remains above it, so the emission height rises by about 150 m. Air there is about 1 K colder (254 K), and colder air emits less: about 236 instead of 240 W/m², a drop of about 3.7 W/m².", + "10": "No. The emission height would still rise, but the air there would still be at 255 K and would emit the same 240 W/m². There would be no imbalance and no warming. The warming needs colder air higher up.", + "11": "It comes mostly from about 5 km up, where the air is about 255 K. The temperature has to be 255 K because that is what emits the 240 W/m² needed to balance the absorbed sunlight. The height is where little greenhouse gas remains above, so the infrared escapes instead of being absorbed again. At a lapse rate of 6.5 K/km from a 288 K surface, the air reaches 255 K at about 5 km, and satellites confirm that height. The explanation never derives the 5 km from the amount of gas: it gets it from the lapse rate and the measured surface temperature.", + "12": "(1) Doubling CO₂ raises the emission height by about 150 m. (2) Air there is about 1 K colder. (3) It emits about 236 W/m² against 240 W/m² absorbed, an imbalance of 3.7 W/m² called the radiative forcing. (4) The Earth gains energy, and because convection keeps the lapse rate, the whole profile warms together. (5) Warming stops when the air at the new emission height is back at 255 K and emission matches absorption again. The surface is then about 1 K warmer, or about 3 K with feedbacks such as water vapour. The explanation does not say how long this takes.", + "13": "Yes. What reaches space is decided high up, where the air is thin and infrared still escapes. More CO₂ makes that thin air absorb more, so the escape moves higher into colder air, and less is emitted. Each doubling adds about 3.7 W/m². The explanation does not say why each doubling gives the same forcing; it points to a textbook for that." + }, + "audit": { + "terms": [ + "The layer that emits to space has four names: 'emission height' (§3, §4), 'emitting air' (§1 lead, Summary), 'the escape' / 'where infrared escapes' (§3 title, §6), and 'whatever emits the Earth's energy to space' (§1).", + "The 3.7 W/m² drop is called 'imbalance' (§4 step 3, §5), 'radiative forcing' (§4 step 3) and 'forcing' (§6). The text does say they are the same thing, but in §5 'no imbalance' is not linked to forcing.", + "'the whole profile warms together' (§4 step 4): 'profile' (meaning temperature against height) is never defined. Elsewhere the text says 'the air gets colder with height' or 'lower atmosphere'.", + "'trap heat' (§5, quoting the blanket picture) against 'infrared' and 'energy' everywhere else, without saying that heat here means infrared." + ], + "unexplained": [ + "'The emission height rises, on average by about 150 m' (§4 step 1): no source or derivation is given. The whole numerical chain depends on it, and the reader can't tell whether it is measured, computed, or worked backwards from 3.7 W/m².", + "'Today's 420 ppm gives about 2.2 W/m²' (§6): it doesn't say how 2.2 follows from the doubling rule, or what it means for temperature now (roughly how much warming).", + "'emits about 330 W/m² of infrared back down (a measured value)' (§3): the surface budget is left incomplete. Surface emission of 390 minus 330 coming back leaves 60 W/m², and nothing says where the surface's absorbed sunlight goes, so the paragraph raises a question it doesn't answer.", + "'about 3 K (best estimate of the IPCC)' (§4): there is no range, and no sense of how the feedbacks turn 1 K into 3 K.", + "'6.5 K per km' (§3): it is called measured, but the text doesn't say what sets this value rather than another, or what a larger lapse rate would mean for the surface. That would follow directly from the argument but is never stated." + ], + "missing_why": [ + "'Each doubling adds about the same forcing' (§6): the rule that each doubling adds the same forcing is stated and the reason is sent to another book. The reader can't see why the 280→560 step equals the 560→1120 step.", + "'More gas means the infrared must start higher' (§4 step 1): this is qualitative only. Nothing links the amount of gas to the size of the rise (why 150 m for a doubling).", + "'Convection still sets the lapse rate, so the whole profile warms together' (§4 step 4): it doesn't explain why convection pins the surface to the upper air at a fixed gradient instead of only the upper air warming.", + "'Infrared reaches space only from air that has little greenhouse gas above it' (§3): there is no idea of how much gas counts as 'little', so it is unclear why an average height is a meaningful idea." + ], + "excess": [ + "The Summary (lines 81) repeats §1–§4 with nothing new.", + "'Emission grows fast with temperature: 255 K emits 240 W/m², and 288 K emits 390 W/m²' (§1): the reader already knows hotter things radiate more, and these numbers come up again later.", + "The surface-budget paragraph (§3, 330 W/m²): none of the questions or later steps use it, and it is left incomplete." + ] + }, + "gaps": [ + "Nothing explains where the 150 m rise in emission height comes from. It drives the whole 1 K / 3.7 W/m² calculation.", + "The 5 km emission height is inferred from the 288 K surface and the lapse rate, then used to explain the 288 K surface. Apart from mentioning the satellite observation, the reasoning is close to circular, and the height is never derived from the amount of gas.", + "No mechanism is given for why each doubling adds the same forcing, so the saturation answer rests on a claim the reader has to take on trust.", + "The surface energy budget is incomplete: after the 330 W/m² back-radiation, the remaining terms (absorbed sunlight, convection or evaporation) are missing.", + "Nothing is said about how long the warming takes. Step 5 says when it stops but not over what timescale; ocean lag appears only under 'Not covered'.", + "Feedbacks are given as a number (about 3 K) with only water vapour named, so the reader can't predict how much warming there is beyond the 1 K." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 1, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "1": "Full marks: lists all model entities with correct values.", + "2": "Full marks: the relationships match the model. 'Absorbed' (by the surface) and 'air just above the surface' are loose wording but not wrong.", + "3": "Full marks: emission from a colder height, the emitting air held at 255 K, and the lapse rate giving 288 K.", + "4": "Full marks: CO2 doubling, 150 m rise, 1 K colder, 3.7 W/m², about 1 K without feedbacks and 3 K with, and the zero-lapse-rate case.", + "5": "Partial. The fixed lapse rate is correctly named as an assumption. It omits the single-emission-height assumption, the main simplification in the model. It does not separate model-output estimates (3.7 W/m², 3 K) from observed facts. It presents 'satellites see infrared from about 5 km' as established, but the model derives 5 km from 33 K / 6.5 K/km and does not cite satellite observation.", + "6": "Full marks: a correct, complete chain from energy balance to warming per doubling.", + "7": "Full marks: points to specific gaps (where the 150 m rise comes from, why each doubling adds the same forcing), and both are real gaps given the model's scope.", + "8": "Full marks: 255 K from σT⁴ = 240 W/m², with the factor of 4 and albedo explained.", + "9": "Full marks: higher emission height, colder air there, less emission.", + "10": "Full marks: correct reasoning for the isothermal counterexample.", + "11": "Full marks: 5 km, 255 K, and why that height and that temperature. The 'satellites confirm' remark is not in the model but does not undermine the answer.", + "12": "Full marks: correct steps, and the stopping condition is the air at the new emission height returning to 255 K.", + "13": "Full marks: saturation does not stop the effect, emission moves higher and colder, about 3.7 W/m² per doubling." + }, + "pass": true + }, + "usage": { + "cost": 0.22451879999999996, + "seconds": 62 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The layer that emits to space has four names: 'emission height' (§3, §4), 'emitting air' (§1 lead, Summary), 'the escape' / 'where infrared escapes' (§3 title, §6), and 'whatever emits the Earth's energy to space' (§1)." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The 3.7 W/m² drop is called 'imbalance' (§4 step 3, §5), 'radiative forcing' (§4 step 3) and 'forcing' (§6). The text does say they are the same thing, but in §5 'no imbalance' is not linked to forcing." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'the whole profile warms together' (§4 step 4): 'profile' (meaning temperature against height) is never defined. Elsewhere the text says 'the air gets colder with height' or 'lower atmosphere'." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'trap heat' (§5, quoting the blanket picture) against 'infrared' and 'energy' everywhere else, without saying that heat here means infrared." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'The emission height rises, on average by about 150 m' (§4 step 1): no source or derivation is given. The whole numerical chain depends on it, and the reader can't tell whether it is measured, computed, or worked backwards from 3.7 W/m²." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'Today's 420 ppm gives about 2.2 W/m²' (§6): it doesn't say how 2.2 follows from the doubling rule, or what it means for temperature now (roughly how much warming)." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'emits about 330 W/m² of infrared back down (a measured value)' (§3): the surface budget is left incomplete. Surface emission of 390 minus 330 coming back leaves 60 W/m², and nothing says where the surface's absorbed sunlight goes, so the paragraph raises a question it doesn't answer." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'about 3 K (best estimate of the IPCC)' (§4): there is no range, and no sense of how the feedbacks turn 1 K into 3 K." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'6.5 K per km' (§3): it is called measured, but the text doesn't say what sets this value rather than another, or what a larger lapse rate would mean for the surface. That would follow directly from the argument but is never stated." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'Each doubling adds about the same forcing' (§6): the rule that each doubling adds the same forcing is stated and the reason is sent to another book. The reader can't see why the 280→560 step equals the 560→1120 step." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'More gas means the infrared must start higher' (§4 step 1): this is qualitative only. Nothing links the amount of gas to the size of the rise (why 150 m for a doubling)." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'Convection still sets the lapse rate, so the whole profile warms together' (§4 step 4): it doesn't explain why convection pins the surface to the upper air at a fixed gradient instead of only the upper air warming." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "'Infrared reaches space only from air that has little greenhouse gas above it' (§3): there is no idea of how much gas counts as 'little', so it is unclear why an average height is a meaningful idea." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "The Summary (lines 81) repeats §1–§4 with nothing new." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "'Emission grows fast with temperature: 255 K emits 240 W/m², and 288 K emits 390 W/m²' (§1): the reader already knows hotter things radiate more, and these numbers come up again later." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "The surface-budget paragraph (§3, 330 W/m²): none of the questions or later steps use it, and it is left incomplete." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "Nothing explains where the 150 m rise in emission height comes from. It drives the whole 1 K / 3.7 W/m² calculation." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "The 5 km emission height is inferred from the 288 K surface and the lapse rate, then used to explain the 288 K surface. Apart from mentioning the satellite observation, the reasoning is close to circular, and the height is never derived from the amount of gas." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "No mechanism is given for why each doubling adds the same forcing, so the saturation answer rests on a claim the reader has to take on trust." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "The surface energy budget is incomplete: after the 330 W/m² back-radiation, the remaining terms (absorbed sunlight, convection or evaporation) are missing." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "Nothing is said about how long the warming takes. Step 5 says when it stops but not over what timescale; ocean lag appears only under 'Not covered'." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "Feedbacks are given as a number (about 3 K) with only water vapour named, so the reader can't predict how much warming there is beyond the 1 K." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153148-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153148-quiz-prose.json new file mode 100644 index 0000000..ed6edd4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153148-quiz-prose.json @@ -0,0 +1,227 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "greenhouse", + "at": "2026-10-04T15:31:33", + "model": "default", + "prompt_sha": "a3fd476ca87a", + "reply": { + "answers": { + "1": "The Sun's input (1361 W/m²), the Earth's surface, the atmosphere and its layers at different heights, greenhouse gases (water vapour, CO₂, methane) versus non-absorbing N₂ and O₂, infrared radiation, the emission height and the emission temperature (255 K), the lapse rate (6.5 K/km), the albedo (0.30) and radiative forcing.", + "2": "The Earth absorbs 240 W/m² of sunlight (1361/4 × 0.70) and must emit the same amount to space. Greenhouse gases let sunlight through but absorb infrared and emit it again at the local air temperature. Infrared escapes to space only from air with little greenhouse gas above it, at the emission height. That air must be at 255 K to emit 240 W/m². The lapse rate links that height's temperature to the surface: the surface is warmer by 6.5 K for each km of height.", + "3": "Greenhouse gases raise the height from which infrared escapes to space. Because air gets colder with height, the emitting air would radiate less than 240 W/m², so the atmosphere and surface warm until the emitting air is back at 255 K. This leaves the surface warmer by the lapse rate times the emission height.", + "4": "If CO₂ doubles, the emission height rises by about 150 m. Air there is about 1 K colder (6.5 × 0.15), so it emits about 3.7 W/m² less than the Earth absorbs (the radiative forcing). The Earth gains energy until every height has warmed about 1 K, which brings the new emission height back to 255 K. The surface ends about 1 K warmer, or about 3 K (2.5–4 K) once feedbacks such as water vapour are included. With a lapse rate of 0, nothing changes.", + "5": "Assumption: the lapse rate stays fixed at 6.5 K/km while the Earth warms, so every height warms by the same amount (the text says this changes a little in reality). The text also replaces a spread of emission heights with one average height. Observed facts: the average surface temperature is 288 K, the measured lapse rate is about 6.5 K/km, the albedo is 0.30, and the solar input is 1361 W/m².", + "6": "In balance, the Earth must emit the 240 W/m² of sunlight it absorbs, which needs an emitting temperature of 255 K. Greenhouse gases absorb and re-emit infrared, so the radiation escapes only from high air with little gas above it, on average about 5 km up. Air cools by about 6.5 K per km, so the surface sits about 33 K warmer than that layer, at 288 K. Adding CO₂ lifts the escape height into colder air that emits less, creating an imbalance (3.7 W/m² per doubling). The Earth then warms until the new emitting layer is back at 255 K: about 1 K at the surface from this mechanism alone, about 3 K with feedbacks.", + "7": "The surface-budget paragraph was hardest. The surface emits 390 W/m² and receives about 330 W/m² back from the air, but the text never accounts for the net 60 W/m² against the 240 W/m² absorbed: how much sunlight the surface absorbs, and how convection carries heat upward. A small balance sheet for the surface would help. The number 3.7 W/m² was also confusing: 240 − 236 gives 4, not 3.7, and the text does not reconcile the two.", + "8": "About 255 K (−18 °C). The energy balance sets it: absorbed sunlight is 1361/4 × (1 − 0.30) ≈ 240 W/m², and σT⁴ = 240 gives T = 255 K. The factor 4 is the ratio of the sphere's emitting area to the disc that catches sunlight.", + "9": "More CO₂ means infrared must start higher before little gas is left above it, so the emission height rises by about 150 m. Because of the lapse rate, the air there is about 1 K colder, and colder air emits less (σ·254⁴ ≈ 236 W/m² instead of 240). The surface has not warmed yet, so outgoing radiation falls below the absorbed input.", + "10": "No. The emission height would still rise, but the air there would still be at 255 K and would emit the same 240 W/m². There would be no imbalance and no warming. The effect needs both a rising emission height and air that gets colder with height.", + "11": "It comes from high air with little greenhouse gas above it, on average about 5 km up, where the air is at about 255 K. The temperature is fixed because that layer must emit the 240 W/m² absorbed from the Sun, and σT⁴ = 240 gives 255 K. The height follows from the lapse rate: going from 288 K at the surface down to 255 K at 6.5 K/km takes about 5 km. The text derives the height from the temperatures, not from gas amounts. It notes that the real emission comes from a spread of heights that differ by wavelength.", + "12": "Step 1: CO₂ goes from 280 to 560 ppm, and the emission height rises about 150 m (a result taken from radiation calculations). Step 2: air at that height is about 1 K colder. Step 3: it emits about 3.7 W/m² less than the Earth absorbs (the radiative forcing). Step 4: the Earth gains energy and warms; convection keeps the lapse rate, so all heights warm equally. Step 5: warming stops when the air at the new emission height is back at 255 K and emits 240 W/m² again. The surface is then about 1 K warmer, or about 3 K (2.5–4 K) with feedbacks. The text does not say how long this takes; it explicitly leaves out ocean timing.", + "13": "Yes. What reaches space is decided high up, where the air is thin and infrared can still escape. More CO₂ makes that thin air absorb more, so the escape level moves up into colder air. Each doubling adds about 3.7 W/m². The text states this but does not explain why each doubling adds the same amount; it defers that to a textbook." + }, + "audit": { + "terms": [ + "The forcing concept has three names: 'imbalance' and 'radiative forcing' (§4 step 3), 'forcing' (§6), and 'no imbalance' (§5).", + "The escape level is called 'emission height' (§3, §4), 'emitting air' (lead and §3 'emitting air', Summary), and 'the escape moves up' (§6).", + "The outgoing radiation is called 'energy' (lead), 'power' (§1), 'infrared' (§2 onward), and 'heat' (§5 'trap heat like a blanket').", + "For the 255 K figure, §1 uses 'emission temperature' while the lead uses 'high, cold air'. §3 then says 'must be at the emission temperature, 255 K'. It is the same idea, but the reader has to connect it to 'whatever emits the Earth's energy' in §1." + ], + "unexplained": [ + "§4 step 3: '≈ 236 W/m², against 240 … about 3.7 W/m² less'. The arithmetic gives 4, and the text does not explain why it says 3.7.", + "§6: 'Today's 420 ppm gives about 2.2 W/m²'. It never says what this means for temperature, or how 2.2 follows from the rule that each doubling adds the same forcing.", + "§3: '330 W/m² of infrared back down (a measured value)'. It does not say what this number does for the surface balance, or what would change if it were larger or smaller.", + "§4 step 1: 'about 150 m (a result of radiation calculations)'. The 150 m is given with no sense of why it is this size.", + "§4: 'about 3 K, likely 2.5–4 K'. It does not say what 'likely' means, or what drives the range." + ], + "missing_why": [ + "§6: 'Each doubling adds about the same forcing'. The reason is deferred to Pierrehumbert, but the saturation argument (Q13) depends on it.", + "§3: 'The measured fall is about 6.5 K per km'. The text says rising air cools, but not why the rate is 6.5 and not some other value (for example, the role of water vapour).", + "§4 step 4: 'Convection still sets the lapse rate, so the temperature at every height rises by the same amount'. It does not explain why convection fixes the profile's shape rather than its level.", + "§3: 'emitted again, in all directions, at the temperature of that air'. It does not say why the emission is set by the local air temperature.", + "§3 surface budget: 'The surface does not lose that much'. The full surface balance (absorbed sunlight, convection) is not given, so the step stays unjustified." + ], + "excess": [ + "The Summary mostly repeats §1 and §3 with nothing new.", + "§1: 'with 1361 × 0.70 instead, T = 360 K (87 °C)'. This counterfactual is a minor detour, although it does justify the division by 4.", + "§1: the explanation of what σT⁴ means and that 'Emission grows fast with temperature'. This reader already knows hotter things radiate more; only the law's form is needed.", + "Line 3 comment: ''. This is internal metadata." + ] + }, + "gaps": [ + "The surface energy budget is incomplete: 390 emitted − 330 received does not reconcile with the 240 W/m² absorbed, and convective transport is not mentioned.", + "3.7 W/m² is inconsistent with the 240 − 236 calculation shown.", + "Why each doubling gives equal forcing is not explained, and the saturation answer depends on it.", + "How long warming takes (ocean heat uptake) is not covered, so 'when the warming stops' can only be answered as a condition, not as a time.", + "The 150 m rise in emission height is asserted from calculations; the reader cannot derive or check it." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "7": "No points lost. The reviewer found two real gaps in the rendering. First, the surface budget is incomplete: the 160 W/m² of sunlight and 100 W/m² of convection and evaporation are not reconciled. Second, 240 − 236 = 4 is not matched to the 3.7 W/m² forcing. These are audit candidates for the explainer to decide on.", + "11": "No points lost. The answer states that infrared escapes from air with little greenhouse gas above it, about 5 km up, at 255 K set by energy balance. It also says the height is derived from the lapse rate rather than from gas amounts. That is a reading of the rendering, not an error, but the explainer may want the gas-above criterion stated as the cause of the height.", + "12": "No points lost, but borderline. Step 2 says the higher air is colder without saying why. The fall of temperature with height appears only indirectly, in step 4 ('convection keeps the lapse rate'). The answer also attributes the 150 m rise to radiation calculations, whereas model.md derives it from 1 K at 6.5 K/km. The mechanism and the stopping condition (back to 255 K) are correct and specific." + }, + "pass": true + }, + "usage": { + "cost": 0.2071348, + "seconds": 53 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The forcing concept has three names: 'imbalance' and 'radiative forcing' (§4 step 3), 'forcing' (§6), and 'no imbalance' (§5)." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The escape level is called 'emission height' (§3, §4), 'emitting air' (lead and §3 'emitting air', Summary), and 'the escape moves up' (§6)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "The outgoing radiation is called 'energy' (lead), 'power' (§1), 'infrared' (§2 onward), and 'heat' (§5 'trap heat like a blanket')." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "For the 255 K figure, §1 uses 'emission temperature' while the lead uses 'high, cold air'. §3 then says 'must be at the emission temperature, 255 K'. It is the same idea, but the reader has to connect it to 'whatever emits the Earth's energy' in §1." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "§4 step 3: '≈ 236 W/m², against 240 … about 3.7 W/m² less'. The arithmetic gives 4, and the text does not explain why it says 3.7." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "§6: 'Today's 420 ppm gives about 2.2 W/m²'. It never says what this means for temperature, or how 2.2 follows from the rule that each doubling adds the same forcing." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "§3: '330 W/m² of infrared back down (a measured value)'. It does not say what this number does for the surface balance, or what would change if it were larger or smaller." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "§4 step 1: 'about 150 m (a result of radiation calculations)'. The 150 m is given with no sense of why it is this size." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "§4: 'about 3 K, likely 2.5–4 K'. It does not say what 'likely' means, or what drives the range." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "§6: 'Each doubling adds about the same forcing'. The reason is deferred to Pierrehumbert, but the saturation argument (Q13) depends on it." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "§3: 'The measured fall is about 6.5 K per km'. The text says rising air cools, but not why the rate is 6.5 and not some other value (for example, the role of water vapour)." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "§4 step 4: 'Convection still sets the lapse rate, so the temperature at every height rises by the same amount'. It does not explain why convection fixes the profile's shape rather than its level." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "§3: 'emitted again, in all directions, at the temperature of that air'. It does not say why the emission is set by the local air temperature." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "§3 surface budget: 'The surface does not lose that much'. The full surface balance (absorbed sunlight, convection) is not given, so the step stays unjustified." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "The Summary mostly repeats §1 and §3 with nothing new." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "§1: 'with 1361 × 0.70 instead, T = 360 K (87 °C)'. This counterfactual is a minor detour, although it does justify the division by 4." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "§1: the explanation of what σT⁴ means and that 'Emission grows fast with temperature'. This reader already knows hotter things radiate more; only the law's form is needed." + }, + { + "n": 18, + "severity": "excess", + "kind": "excess", + "text": "Line 3 comment: ''. This is internal metadata." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "The surface energy budget is incomplete: 390 emitted − 330 received does not reconcile with the 240 W/m² absorbed, and convective transport is not mentioned." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "3.7 W/m² is inconsistent with the 240 − 236 calculation shown." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "Why each doubling gives equal forcing is not explained, and the saturation answer depends on it." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "How long warming takes (ocean heat uptake) is not covered, so 'when the warming stops' can only be answered as a condition, not as a time." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "The 150 m rise in emission height is asserted from calculations; the reader cannot derive or check it." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/understanding.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/understanding.md new file mode 100644 index 0000000..8d45119 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/explainers/greenhouse/review/understanding.md @@ -0,0 +1,19 @@ +# Review: greenhouse (published tier, Stage 1 prose) + +## Probe (2026-10-04T152224) +Adopted 1–5, 7, 9, 11, 14: lapse-rate mechanism, why more gas lifts emission height, surface budget (390 vs 240 W/m²), one name for 255 K ("emission temperature"), forcing/albedo definitions, cut Wien constant and ozone point. Declined: T⁴ "why" (a stated law; row removed), steady-state break case and ocean timing (scope). + +## Cold read — narrative (152459) +4 blocking: 288 K/33 K gap missing from beat 2; σT⁴ and albedo unexplained; "that air must be 255 K" unexplained; 5 km had no source. All fixed in narrative.md. Stratosphere case cut from prose (kept in model). Feedback line kept (one sentence, reader's obvious next question). + +## Cold read — prose (152723) +No blocking. Surface-budget table replaced by one sentence; 2 km row cut; 360 K counterfactual moved after the calculation; 2.2 W/m² moved to the saturation section; saturation reason made explicit. + +## Blind test — prose +- 152840: FAIL, item 12 = 1 (answer gave location and temperature without reasons; the ask did not request reasons). Asks rewritten; duplicate items (balance-255, isothermal) removed. +- 153030: PASS, general item 5 (epistemic status) = 1. Fixed: emission height labeled as an average of a spread; 150 m labeled as a radiation-calculation result; 3 K given with its range and source; unmodeled satellite clause removed. +- Final run: PASS, every item 2. +- Broken copy (blanket/mirror mechanism, isothermal claim reversed, saturation reversed): FAIL, items 9–13 = 0. The test can fail. + +## Not applied (audit gaps, by decision) +Why forcing is logarithmic, how feedbacks turn 1 K into 3 K, warming timescale, quantitative basis for "little gas above": all declared in Scope / "Not covered here". diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/read.json new file mode 100644 index 0000000..b7f8618 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "Greenhouse gases let visible light pass through but absorb and emit infrared radiation (heat), redirecting it back toward the surface rather than letting it escape to space.", + "quote": "A greenhouse gas lets most sunlight pass but absorbs and emits infrared." + }, + "2": { + "answer": "More CO2 raises the emission altitude to colder air, which emits less radiation; the surface must then warm until the cold air above reaches 255 K and restores energy balance.", + "quote": "Greenhouse gases make the Earth radiate its energy to space from high, cold air instead of from the warm surface." + }, + "3": { + "answer": "Nitrogen and oxygen do not absorb infrared radiation, so they cannot trap heat the way greenhouse gases do.", + "quote": "Nitrogen and oxygen, 99% of dry air, are not: they barely absorb infrared at all." + }, + "4": { + "answer": "Heat would radiate directly from the surface, and the surface temperature would be 255 K (−18 °C), determined by energy balance requiring the emitting body to be at this temperature.", + "quote": "If the surface emitted directly to space, the surface would be at 255 K." + } + }, + "stops": [ + "A body at temperature T emits σT⁴ watts per square metre (the Stefan–Boltzmann law, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴).", + "Infrared really escapes from a spread of heights, which differ by wavelength.", + "More gas means the infrared must start higher before little gas remains above it.", + "Call their average the emission height.", + "infrared from the ground is absorbed by the air above it and emitted again, in all directions", + "convection keeps mixing the lower atmosphere", + "This imbalance, just after the change and before any warming, is the radiative forcing.", + "Suppose the atmosphere had the same temperature at every height (a lapse rate of 0).", + "Warming stops when the air at the new emission height is back at 255 K.", + "Colder air emits less: σ·254⁴ ≈ 236 W/m²." + ], + "excess": [ + "The division by 4 matters: with 1361 × 0.70 instead, T = 360 K (87 °C), as if every point on Earth faced the noon Sun all day.", + "Sunlight peaks near 0.5 µm, in the visible.", + "Rising air expands and cools, and convection keeps mixing the lower atmosphere.", + "The air gets colder with height.", + "The surface at 288 K emits 390 W/m²", + "with this and other feedbacks, the estimated warming per doubling is about 3 K, likely 2.5–4 K", + "280 → 560 ppm gives 3.7 W/m², and 560 → 1120 ppm gives 3.7 W/m² again.", + "Convection still sets the lapse rate, so the temperature at every height rises by the same amount.", + "Water vapour, CO₂, and methane are greenhouse gases.", + "Why only some molecules absorb infrared: a molecular spectroscopy text." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/run.json new file mode 100644 index 0000000..11e68c8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/greenhouse/explanation.md` as prose only, because the answer is a short chain of cause and effect with arithmetic at each step (240 W/m² in → 255 K → 5 km emission height × 6.5 K/km → 288 K), and worked numbers in text teach that better than a diagram or interactive page; it passed the check, the cold read and the blind test.", + "error": null, + "cost": 2.139209, + "seconds": 804, + "turns": 83, + "files": [ + "explainers/greenhouse/explanation.md" + ], + "review_cost": 1.0955, + "reviews": 6 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/diagram.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/diagram.md new file mode 100644 index 0000000..a16d23c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/diagram.md @@ -0,0 +1,52 @@ +# How do greenhouse gases warm the Earth? + + + +**Answer.** Sunlight passes through the air and warms the surface. The surface sends the energy back out as infrared. Greenhouse gases absorb infrared, so the infrared escapes to space only from high, cold air, about 5 km up. Cold air emits less. To send out enough energy, the planet must be warmer than it would be without the gases. Adding gas lifts the escape point into colder air, so the surface warms more. + +## Level 1 — Where the energy goes + +```mermaid +flowchart TB + SUN["Sun"] -->|"sunlight, peak 0.5 µm
240 W/m² absorbed"| SURF["Surface
288 K, emits 390 W/m²"] + SURF -->|"infrared, peak 10 µm"| GHG["Greenhouse gas
absorbs and re-emits infrared"] + GHG -->|"re-emitted downward"| SURF + GHG -->|"re-emitted upward, absorbed again"| TOP["Air at emission height
about 5 km, 255 K"] + TOP -->|"infrared to space
240 W/m²"| SPACE["Space"] +``` + +The two kinds of radiation have different wavelengths. Sunlight peaks near 0.5 µm, where the air is mostly transparent. The Earth is much colder than the Sun, so its radiation, **infrared**, peaks near 10 µm. A **greenhouse gas** absorbs and emits infrared. N2 and O2 make up 99% of dry air but do not absorb infrared. A molecule absorbs infrared only if its vibration moves electric charge unevenly; a two-atom molecule of one element cannot do that. CO2, only 0.04% of air, absorbs strongly near 15 µm. Water vapour absorbs across many bands. + +**Energy balance** means that the planet emits to space as much as it absorbs. Sunlight brings 1361 W/m²; 30% reflects. The Earth catches sunlight on a disc (area πr²) but emits from a whole sphere (4πr²), so the average absorbed power is 1361 × 0.7 / 4 = 238 W/m², about 240. If you forget the 4, you get 360 K, which is wrong. + +A body emits power σT⁴ per m² (Stefan–Boltzmann law, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴). To emit 238 W/m², a body must be at 255 K (−18 °C). The real mean surface is 288 K (15 °C) and emits 390 W/m². Only 240 W/m² reaches space, so the atmosphere holds back 150 W/m². That 33 K difference is the greenhouse effect. + +## Level 2 — Why more gas means a warmer surface + +**Predict first:** a greenhouse gas absorbs infrared and emits it again. If it emits as much as it absorbs, how can more gas change anything? + +```mermaid +flowchart TB + A["More greenhouse gas"] --> B["Emission height rises
about 150 m for doubled CO2"] + B --> C["Emitting air is colder
6.5 K/km × 0.15 km = 1 K"] + C --> D["Less infrared to space
3.8 W/m² less"] + D --> E["Energy balance broken:
planet gains energy"] + E --> F["Whole column warms,
surface included"] + F -->|"until outgoing infrared
again equals 240 W/m²"| G["Energy balance restored,
warmer surface"] +``` + + Infrared that leaves near the ground gets absorbed again higher up. Infrared escapes to space only from air with little gas above it. The average altitude of that air is the **emission height**, about 5 km today. The **lapse rate** is how fast air cools with height: about 6.5 K per km. So the emitting air sits at 288 − 6.5 × 5 ≈ 255 K. That is exactly the temperature energy balance requires. + +Add gas, and the emission height rises. Doubled CO2 lifts it about 150 m (estimate). The new emitting air is 1 K colder. Because emission goes as T⁴, a 1 K drop at 255 K cuts the output by 4σT³ × 1 K = 3.8 W/m². The planet now gains energy. The whole column warms until the air at the new emission height is back at 255 K. The surface, 5 km of lapse rate below, warms by the same amount: about 1.2 K before feedbacks. + + This works only because air gets colder with height. In air of one temperature at every height, a higher emitter would be just as warm. Output would not fall, and more gas would not warm the surface. + +## Level 3 — Common mistakes + +| Mistake | Correction | +|---|---| +| "Greenhouse gases trap heat like the glass of a greenhouse." | A glass greenhouse works mainly by stopping warm air from rising. Greenhouse gases move the place where the planet emits to space. | +| "CO2 already absorbs all 15 µm infrared, so more CO2 does nothing." | Near the ground, absorption is saturated. The escape point still moves up into colder air, so the effect grows with the logarithm of CO2: each doubling adds about 3.7 W/m². | +| "The gas makes energy." | The source is sunlight. The gas slows energy loss to space. | + +Feedbacks multiply the 1.2 K. Warmer air holds more water vapour, a greenhouse gas; melting ice reflects less sunlight. The estimated total for doubled CO2 is likely 2.5–4 K (IPCC AR6). Clouds and ocean heat uptake are outside this explainer. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/model.md new file mode 100644 index 0000000..9e55538 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/model.md @@ -0,0 +1,94 @@ +# Semantic model: the greenhouse effect + +> Source of truth. Every rendering derives from this file. + +## Central question + +How do greenhouse gases make the Earth's surface warmer than it would be without them? + +## Audience and prior knowledge + +A technical reader: school mathematics and physics (energy, temperature in kelvin, wavelength, molecules), basic programming. New to the subject. Does not know Stefan–Boltzmann, lapse rate, or emission height. Renderings define those terms. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| sunlight | Radiation from the Sun, mostly visible light, peak near 0.5 µm. | fact | +| infrared | Radiation that the Earth emits, peak near 10 µm, invisible. | fact | +| greenhouse gas | A gas whose molecules absorb and emit infrared: H2O, CO2, CH4, N2O, O3. | fact | +| energy balance | The state in which the planet emits to space as much energy as it absorbs. | fact | +| emission height | The average altitude from which infrared escapes to space, about 5 km. | estimate | +| lapse rate | The fall of air temperature with altitude in the lower atmosphere, about 6.5 K per km. | observed average | + +## Causal chain + +```text +sunlight passes through air → surface warms → surface emits infrared +→ greenhouse gas absorbs infrared, re-emits it in all directions +→ infrared escapes to space only from the emission height (~5 km, 255 K) +→ more greenhouse gas → higher emission height → colder emitter → less infrared out +→ energy balance broken → planet gains energy → whole column warms +→ surface warms by the same amount until energy balance returns +``` + +## Quantities + +- Solar constant S = 1361 W/m². Albedo (fraction reflected) a = 0.30. +- Absorbed sunlight per m² of surface, averaged: S(1 − a)/4 = 238 W/m² (about 240). +- Stefan–Boltzmann: emitted power per m² = σT⁴, σ = 5.67×10⁻⁸ W m⁻² K⁻⁴. +- Temperature for energy balance: T = (238 / σ)^(1/4) = 255 K (−18 °C). +- Observed mean surface temperature: 288 K (15 °C). Difference: 33 K. +- Surface emission at 288 K: σ·288⁴ = 390 W/m². Outgoing to space: about 240 W/m². The atmosphere holds back 150 W/m². +- Emission height: (288 − 255) / 6.5 ≈ 5 km. +- CO2 doubling: radiative forcing (imbalance before any warming) ≈ 5.35·ln 2 = 3.7 W/m². +- Emission height rise for that forcing: about 150 m → emitter 1 K colder → 4σT³·1 K = 3.8 W/m² less out. +- Warming with no feedbacks: 3.7 / 3.2 ≈ 1.2 K. With feedbacks (water vapour, ice, clouds): likely 2.5–4 K (IPCC AR6 estimate). + +## Alternative states + +| State | What changes | Why | +|---|---|---| +| No greenhouse gas | Surface emits straight to space; surface ≈ 255 K | Emission height = 0 km | +| Today | Surface 288 K | Emission height ≈ 5 km, lapse rate 6.5 K/km | +| Doubled CO2 | Surface ≈ 1.2 K warmer before feedbacks | Emission height ≈ 150 m higher | +| No lapse rate (isothermal air) | No greenhouse warming, however much gas | Higher emitter is not colder | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Divide sunlight by 4 | Use 1361 W/m² × 0.7 directly | Gives T = 360 K. The Earth intercepts sunlight on a disc (πr²) but emits from a sphere (4πr²). | +| Emitted power ∝ T⁴ | Proportional to T | The 1 K colder emitter would lose only 0.4% = 0.9 W/m²; real loss is 4 × 0.4% = 1.6% = 3.8 W/m². | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| More gas warms the surface | Lapse rate 6.5 K/km: 150 m higher → 1 K colder emitter → 3.8 W/m² deficit | Isothermal air: higher emitter has same T, no deficit, no warming | +| N2 and O2 do not absorb infrared | 99% of dry air, yet transparent to infrared | CO2 (0.04%) absorbs strongly near 15 µm | + +## Terms + +sunlight, infrared, greenhouse gas, energy balance, emission height, lapse rate, radiative forcing (the energy imbalance in W/m² that a change causes before the planet warms; 0 = balanced, 3.7 = doubled CO2). + +## Epistemic status + +- **Established:** absorption bands of CO2 and H2O; 255 K and 288 K; Stefan–Boltzmann. +- **Estimate:** emission height 5 km (an average over wavelengths); 150 m rise; feedback range 2.5–4 K. + +## Confusion points + +- "Gases trap heat like greenhouse glass" → a glass greenhouse works mainly by stopping warm air from rising. Greenhouse gases change where the planet emits from. +- "CO2 absorption is saturated, so more CO2 does nothing" → near the surface, yes. The escape point still moves up, so forcing grows with the logarithm of CO2, not zero. +- "The gas makes energy" → no. It slows energy loss; the source is sunlight. + +## Scope + +- **Out of scope:** feedbacks in detail, clouds, the one-layer model, ocean heat uptake — see IPCC AR6 WG1 chapter 7. + +## Representation decision + +- **Stage:** 2 +- **Reason:** the answer is a flow of energy plus a causal chain with a temperature profile; the reader must hold the surface, the gas, the emission height, and space at once. +- **Levels:** L1 energy flow / L2 emission-height mechanism / L3 why only some molecules. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/model.yaml new file mode 100644 index 0000000..b923b7a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/explainers/greenhouse/model.yaml @@ -0,0 +1,81 @@ +# Machine-readable half of model.md. `explainer check greenhouse` holds every rendering to it. + +values: + solar_constant: 1361 + albedo: 0.30 + absorbed: 238 + absorbed_round: 240 + sigma: 5.67e-8 + t_balance: 255 + t_balance_c: -18 + t_surface: 288 + t_surface_c: 15 + warming_now: 33 + surface_emission: 390 + held_back: 150 + lapse_rate: 6.5 + emission_height_km: 5 + doubling_forcing: 3.7 + height_rise_m: 150 + colder_emitter_k: 1 + deficit: 3.8 + planck_response: 3.2 + no_feedback_warming: 1.2 + ecs_low: 2.5 + ecs_high: 4 + t_no_divide: 360 + sun_peak_um: 0.5 + earth_peak_um: 10 + co2_band_um: 15 + co2_percent: 0.04 + n2o2_percent: 99 + +allow: [2, 4, 0.4, 1.6, 0.9, 0.7, 0.15] + +require: + - path: t_balance + in: [diagram] + - path: t_surface + in: [diagram] + - path: emission_height_km + in: [diagram] + - path: lapse_rate + in: [diagram] + +claims: + - id: why-divide-4 + kind: why + about: [divide by 4] + statement: "Absorbed sunlight is averaged over the whole sphere, so divide by 4." + alternative: "Use 1361 × 0.7 W/m² without dividing." + counterexample: "Gives 360 K. The Earth catches sunlight on a disc of area πr² but emits from a sphere of 4πr²." + - id: mech-emission-height + kind: mechanism + statement: "More greenhouse gas raises the emission height; the higher air is colder, so less infrared leaves, and the planet warms until energy balance returns." + example: "Doubled CO2 raises the emission height about 150 m; at 6.5 K/km the emitter is 1 K colder and emits 3.8 W/m² less." + ask: "Why does adding CO2 warm the surface, if the CO2 only absorbs and re-emits?" + - id: guar-lapse-rate + kind: guarantee + statement: "Extra greenhouse gas warms the surface only because air gets colder with height." + example: "Lapse rate 6.5 K/km: a 150 m rise gives a 1 K colder emitter and a 3.8 W/m² deficit." + counterexample: "In isothermal air, the higher emitter has the same temperature; no deficit, no warming." + - id: def-greenhouse-gas + kind: definition + statement: "A greenhouse gas absorbs and emits infrared; N2 and O2, 99% of dry air, do not." + - id: misc-saturation + kind: misconception + statement: "CO2 absorption near the surface is saturated, so more CO2 does nothing." + correction: "The escape point still moves up into colder air, so forcing grows with the logarithm of CO2: about 3.7 W/m² per doubling." + ask: "If CO2 already absorbs all 15 µm infrared near the ground, why does more CO2 still matter?" + +accept_unjustified: {ln: "5.35·ln(C/C0) is a fit to detailed radiative calculations; the rendering states the per-doubling result, and the derivation is out of scope."} + +terms: + - term: emission height + means: "the average altitude from which infrared escapes to space" + avoid: [radiating level, effective radiating height] + - term: infrared + means: "radiation the Earth emits" + avoid: [longwave radiation, thermal radiation, heat radiation] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/read.json new file mode 100644 index 0000000..162e341 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "Greenhouse gases don't absorb sunlight (which peaks at 0.5 µm where the air is transparent) but do absorb infrared heat radiation (which peaks at 10 µm). The two types of radiation have different wavelengths.", + "quote": "The two kinds of radiation have different wavelengths. Sunlight peaks near 0.5 µm, where the air is mostly transparent. A greenhouse gas absorbs and emits infrared." + }, + "2": { + "answer": "Adding CO2 raises the altitude from which infrared escapes to space (the emission height) into colder air. This colder air emits less radiation, breaking energy balance. The whole atmospheric column then warms until the higher emission point reaches the equilibrium temperature, which means the surface warms too.", + "quote": "Add gas, and the emission height rises. Doubled CO2 lifts it about 150 m (estimate). The new emitting air is 1 K colder. The planet now gains energy. The whole column warms" + }, + "3": { + "answer": "N2 and O2 cannot absorb infrared because they are symmetric two-atom molecules of a single element. Infrared absorption requires molecular vibrations that move electric charge unevenly, which is impossible in such symmetric molecules.", + "quote": "A molecule absorbs infrared only if its vibration moves electric charge unevenly; a two-atom molecule of one element cannot do that." + }, + "4": { + "answer": "Without greenhouse gases, infrared would escape directly from the surface (not from high altitude). The surface would be at 255 K, the equilibrium temperature needed to emit enough energy (238 W/m²) to balance incoming sunlight.", + "quote": "Infrared that leaves near the ground gets absorbed again higher up. Infrared escapes to space only from air with little gas above it. To emit 238 W/m², a body must be at 255 K." + } + }, + "stops": [ + "A molecule absorbs infrared only if its vibration moves electric charge unevenly; a two-atom molecule of one element cannot do that.", + "The Earth catches sunlight on a disc (area πr²) but emits from a whole sphere (4πr²), so the average absorbed power is 1361 × 0.7 / 4 = 238 W/m²", + "The lapse rate is how fast air cools with height: about 6.5 K per km. So the emitting air sits at 288 − 6.5 × 5 ≈ 255 K.", + "In air of one temperature at every height, a higher emitter would be just as warm.", + "The average altitude of that air is the **emission height**, about 5 km today.", + "re-emitted downward and re-emitted upward, absorbed again", + "Each doubling adds about 3.7 W/m²", + "Doubled CO2 lifts it about 150 m (estimate)", + "Because emission goes as T⁴, a 1 K drop at 255 K cuts the output by 4σT³ × 1 K = 3.8 W/m²", + "Stefan–Boltzmann law, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴" + ], + "excess": [ + "Feedbacks multiply the 1.2 K. The estimated total for doubled CO2 is likely 2.5–4 K.", + "CO2, only 0.04% of air, absorbs strongly near 15 µm.", + "If you forget the 4, you get 360 K, which is wrong.", + "sunlight brings 1361 W/m²; 30% reflects", + "A glass greenhouse works mainly by stopping warm air from rising.", + "Warmer air holds more water vapour, a greenhouse gas; melting ice reflects less sunlight.", + "Near the ground, absorption is saturated.", + "The source is sunlight.", + "Predict first: a greenhouse gas absorbs infrared and emits it again. If it emits as much as it absorbs, how can more gas change anything?", + "Clouds and ocean heat uptake are outside this explainer." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/run.json new file mode 100644 index 0000000..eceb23e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/greenhouse/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/greenhouse/diagram.md`: two Mermaid diagrams (where the energy goes, then why more gas warms the surface) with worked numbers (255 K vs 288 K, 5 km emission height, 6.5 K/km), because the answer is an energy flow plus a cause-and-effect chain the reader needs to see together — its model files are alongside, and `explainer check` passes on the numbers and both diagrams.", + "error": null, + "cost": 0.507285, + "seconds": 120, + "turns": 25, + "files": [ + "explainers/greenhouse/diagram.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/explainers/heat-pump/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/explainers/heat-pump/explanation.md new file mode 100644 index 0000000..3b39d57 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/explainers/heat-pump/explanation.md @@ -0,0 +1,150 @@ +# How can a heat pump deliver three times more heat than the electricity it uses? + +## The short answer + +A heat pump doesn't turn electricity into heat. It uses electricity to **move** heat that already exists, taking it from somewhere cold (the outside air, the ground) and releasing it somewhere warm (your house). + +So in a typical hour: + +``` + 1 kWh of electricity (drives the machine) ++ 2 kWh of heat (taken from the outside air, for free) += 3 kWh of heat (released into the house) +``` + +Energy is conserved exactly. Nothing is created. The "3×" compares the heat delivered with only the part you pay for. That ratio is called the **coefficient of performance (COP)**: + +``` +COP = heat delivered / electrical work put in = 3 kWh / 1 kWh = 3 +``` + +The rest of this explanation answers the three questions that usually come next: + +1. How can cold outside air contain heat you can extract? +2. How does the machine actually move heat "uphill", from cold to warm? +3. Why is the answer about 3, and not 1 or 100? + +--- + +## 1. Cold air still holds plenty of heat + +"Cold" only means "colder than us". Air at 0 °C is at 273 K, which is 273 degrees above absolute zero, where molecular motion stops. Its molecules are still moving fast and carry a lot of thermal energy. Cooling 1 kg of air from 0 °C to −5 °C releases about 5 kJ. A heat pump blows thousands of kilograms of air across its outdoor coil every hour, so it can collect kilowatts of heat from air that feels freezing. + +There's one catch. Heat flows on its own only from hot to cold. The outside air will never warm a 20 °C room by itself. Something has to push the heat the "wrong" way. That push is what the electricity pays for. + +A useful analogy is a water pump. Lifting water from a low tank into a high tank takes energy, but the energy depends on **how high** you lift it, not on how much water was in the low tank to begin with. The water was already there. A heat pump lifts heat across a temperature difference, and the work needed depends on the size of that difference. + +--- + +## 2. How the machine moves heat uphill + +The trick uses one fact about liquids: **the boiling point depends on pressure.** Water boils at 100 °C at sea level, at about 70 °C on top of Everest, and at 120 °C inside a pressure cooker. Lower the pressure and a liquid boils at a lower temperature. Raise it and the liquid condenses at a higher temperature. + +A heat pump circulates a **refrigerant**, a fluid chosen so that its boiling point at sensible pressures falls in a useful range. (Propane, sold as R290, is a common one.) The refrigerant goes round a closed loop with four parts: + +``` + OUTSIDE (cold, e.g. 0 °C) INSIDE (warm, e.g. 35 °C water for radiators) + + ┌──────────────────────┐ ┌──────────────────────┐ + │ 1. EVAPORATOR │ cold, low- │ 3. CONDENSER │ + heat → │ low pressure; │ pressure gas │ high pressure; │ → heat + from │ refrigerant boils │ ───────────────┐ │ refrigerant │ into + air │ at about −8 °C │ │ │ condenses at ~40 °C │ house + └──────────────────────┘ │ └──────────────────────┘ + ▲ ▼ │ + │ ┌─────────────────┐ │ warm, high- + cold, low- │ │ 2. COMPRESSOR │ │ pressure + pressure │ │ (electricity │────┘ liquid + liquid │ │ goes in here) │ hot, high-pressure gas + │ └─────────────────┘ + ┌──────────────────────┐ │ + │ 4. EXPANSION VALVE │ ◄────────────────────────────────┘ + │ pressure drops, │ + │ liquid gets very cold│ + └──────────────────────┘ +``` + +Follow one bit of refrigerant round the loop: + +1. **Evaporator (outdoors).** The refrigerant is at low pressure, so it boils at about −8 °C. That's colder than the 0 °C outside air, so heat flows naturally *from the air into the refrigerant*, and the refrigerant boils. Boiling soaks up a lot of energy (latent heat) without the temperature changing. **This is where the free heat enters.** +2. **Compressor.** An electric motor squeezes the gas to high pressure. Compressing a gas heats it, the same way a bicycle pump gets warm. The gas leaves hotter than the house water, say 60 °C. **This is where the electricity enters.** +3. **Condenser (indoors).** At high pressure the refrigerant condenses at about 40 °C. That's warmer than the 35 °C radiator water, so heat flows naturally *from the refrigerant into the water*. Condensing releases all the latent heat picked up outdoors, plus the energy added by the compressor. +4. **Expansion valve.** The warm, high-pressure liquid passes through a narrow restriction. Its pressure drops sharply, part of it flashes to vapour, and it cools to about −8 °C again, ready to go back outdoors. + +Notice that **heat only ever flows downhill**: at each heat exchanger, from the hotter thing to the colder thing. The machine cheats by changing the refrigerant's temperature between the two exchangers. It makes the refrigerant colder than the outside air when it needs to absorb heat, then hotter than the house when it needs to release it. The compressor's job is to raise the refrigerant from the cold level to the hot level. + +A fridge is exactly this machine. Its evaporator is inside the box and its condenser is the warm grille on the back. A heat pump is a fridge whose "inside" is the outdoors, and whose warm grille is your house. + +--- + +## 3. Why about 3? The temperature lift sets the limit + +How much work does it take to lift a given amount of heat from a cold temperature to a hot one? Thermodynamics gives an exact lower bound. Here is the derivation, which needs only some algebra. + +**Entropy bookkeeping.** When heat *Q* moves in or out of something at absolute temperature *T*, it carries entropy *Q / T*. The second law of thermodynamics says that the total entropy can't decrease. A machine running in a steady cycle can't store entropy, so it must give out at least as much entropy as it takes in. + +- Heat taken from outside, at *T*_cold: *Q*_cold, which brings in entropy *Q*_cold / *T*_cold +- Heat released indoors, at *T*_hot: *Q*_hot, which carries out entropy *Q*_hot / *T*_hot +- Electrical work *W* carries no entropy. + +The second law requires: + +``` +Q_hot / T_hot ≥ Q_cold / T_cold +``` + +Energy conservation gives *Q*_cold = *Q*_hot − *W*. Substitute that in and rearrange: + +``` +W ≥ Q_hot × (1 − T_cold / T_hot) + +COP = Q_hot / W ≤ T_hot / (T_hot − T_cold) (the Carnot limit) +``` + +The maximum COP depends only on the two **absolute** temperatures, and it shrinks as the gap between them grows. The work needed is a fraction of the heat moved, and that fraction is just the temperature lift divided by the hot temperature. + +**Plugging in numbers.** Outside is 0 °C (273 K) and radiator water is 35 °C (308 K): + +``` +COP_max = 308 / (308 − 273) = 308 / 35 ≈ 8.8 +``` + +So the laws of physics would allow almost 9 units of heat per unit of electricity. Real machines lose energy in several ways. The refrigerant has to be several degrees colder than the air and hotter than the water, or heat won't flow fast enough, so the real lift is larger than 35 K. The compressor and motor have friction and electrical losses. The expansion valve wastes some energy. Fans and pumps also use power. Good machines reach roughly 40–50 % of the Carnot limit, which puts them at around **3.5–4**. Averaged over a winter, which includes colder days and defrosting the outdoor coil, a seasonal COP of about **3** is typical. + +This also explains the practical advice given about heat pumps: + +| Outside | Water temp | Carnot COP | Realistic COP (≈ 45 % of Carnot) | +|---|---|---|---| +| 10 °C | 35 °C | 12.3 | ~5.5 | +| 0 °C | 35 °C | 8.8 | ~4.0 | +| 0 °C | 55 °C | 6.0 | ~2.7 | +| −10 °C | 35 °C | 6.8 | ~3.1 | +| −10 °C | 55 °C | 5.0 | ~2.3 | + +- **Colder weather means a lower COP.** The lift is bigger. But it stays above 1, so the heat pump still beats an electric heater. +- **Lower water temperatures help a lot.** That's why heat pumps pair well with large radiators or underfloor heating, which can heat a room with 35 °C water rather than 55 °C. +- **Ground-source heat pumps** draw from soil that stays around 10 °C all winter, so they keep a smaller lift and a higher COP. + +If you want to play with the numbers, this snippet reproduces the table: + +```python +def carnot_cop(t_out_c, t_water_c): + t_cold = t_out_c + 273.15 + t_hot = t_water_c + 273.15 + return t_hot / (t_hot - t_cold) + +for out, water in [(10, 35), (0, 35), (0, 55), (-10, 35), (-10, 55)]: + c = carnot_cop(out, water) + print(f"{out:>4} °C → {water} °C: Carnot {c:5.1f}, realistic ~{0.45 * c:.1f}") +``` + +--- + +## Why this isn't too good to be true + +- **It's not perpetual motion.** No energy is created. The heat delivered equals the electricity plus the heat removed from outdoors, exactly. The outside air ends up slightly colder. +- **Compare with an electric heater.** A resistive heater converts 1 kWh of electricity into exactly 1 kWh of heat (COP = 1). That is already 100 % efficient as a *converter*, and no converter can do better. A heat pump isn't a better converter. It's a different kind of device, a mover, and movers aren't limited to 1. +- **The "3×" is a ratio, not an efficiency.** Efficiency compares output with *total* energy input, and that's always 100 % here (energy is conserved). COP compares output with the *paid-for* input only. Calling it "300 % efficient" is a loose way of saying COP = 3. +- **Run backwards, it's an air conditioner.** Swap which coil is inside and which is outside (most heat pumps do this with a reversing valve) and the same machine moves heat out of your house in summer. + +**In one sentence:** electricity is spent to raise heat across a temperature gap, not to create it. When the gap is small compared with the absolute temperature, a little work moves a lot of heat. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/read.json new file mode 100644 index 0000000..2ad8eb8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "From the outside air or ground. A heat pump uses electricity to move heat that already exists at cold temperatures and releases it into the warm house.", + "quote": "taking it from somewhere cold (the outside air, the ground) and releasing it somewhere warm (your house)." + }, + "2": { + "answer": "The refrigerant circulates through a closed loop where its temperature is manipulated: it is made colder than the outside air to absorb heat from it, then made hotter than the house to release that heat indoors, completing the cycle through boiling, compression, condensation, and expansion stages.", + "quote": "It makes the refrigerant colder than the outside air when it needs to absorb heat, then hotter than the house when it needs to release it." + }, + "3": { + "answer": "On a colder day, the temperature difference (lift) between outside and inside is larger. As the temperature gap grows, the coefficient of performance (heat per unit electricity) decreases.", + "quote": "Colder weather means a lower COP. The lift is bigger." + }, + "4": { + "answer": "On the 5 °C day, the ideal COP is higher than on the −15 °C day. The 5 °C day has a smaller temperature gap (30 K vs 50 K), and the Carnot limit depends only on these temperatures and shrinks as the gap grows.", + "quote": "The maximum COP depends only on the two absolute temperatures, and it shrinks as the gap between them grows." + } + }, + "stops": [ + "Boiling soaks up a lot of energy (latent heat)", + "The second law of thermodynamics says that the total entropy can't decrease.", + "depends only on the two absolute temperatures", + "most heat pumps do this with a reversing valve", + "the boiling point depends on pressure", + "defrosting the outdoor coil", + "(the Carnot limit)", + "carry a lot of thermal energy", + "part of it flashes to vapour", + "boiling point at sensible pressures" + ], + "excess": [ + "It's not perpetual motion.", + "A resistive heater converts 1 kWh of electricity into exactly 1 kWh of heat (COP = 1).", + "Calling it \"300 % efficient\" is a loose way of saying COP = 3.", + "A fridge is exactly this machine.", + "Run backwards, it's an air conditioner.", + "A useful analogy is a water pump.", + "Real machines lose energy in several ways.", + "Ground-source heat pumps draw from soil that stays around 10 °C all winter", + "def carnot_cop(t_out_c, t_water_c):", + "That's why heat pumps pair well with large radiators or underfloor heating" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/run.json new file mode 100644 index 0000000..44bc0db --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/heat-pump/explanation.md`, a single prose explainer, because this reader can follow a short algebra derivation (the Carnot limit) and a worked table, and a diagram and Python snippet inside the prose cover what separate diagram or interactive files would add.", + "error": null, + "cost": 0.1563, + "seconds": 49, + "turns": 2, + "files": [ + "explainers/heat-pump/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/explanation.md new file mode 100644 index 0000000..05cad9e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/explanation.md @@ -0,0 +1,87 @@ +# How a heat pump delivers three times more heat than the electricity it uses + +**Short answer:** the electricity does not *make* the heat. It *moves* heat from the cold outdoor air into the house. The heat that comes out is the heat taken from outside plus the electricity: + +``` +heat delivered = heat taken from outside + electrical work + 3 kWh = 2 kWh + 1 kWh +``` + +Energy is conserved. Two of the three kWh were already in the outdoor air. + +## Why this matters + +An electric resistance heater turns 1 kWh of electricity into exactly 1 kWh of heat. It can do no better, because it can only convert the energy it is given. A heat pump also collects energy from outside, so it can deliver more heat than the electricity it uses. The ratio is the **coefficient of performance (COP)**: + +``` +COP = heat delivered / electrical work = 3 kWh / 1 kWh = 3 +``` + +A COP of 1 is no better than a resistance heater. A COP of 3 means two thirds of the heat came from outside. + +## The obstacle: heat flows downhill + +Air at 5 °C still holds a lot of thermal energy. It is 278 K, far above absolute zero. But heat flows by itself only from a hotter body to a colder one (the second law of thermodynamics). The 5 °C air cannot warm a 20 °C room. So the machine must do two things: + +1. Get colder than the outdoor air, so heat flows *in*. +2. Get hotter than the house, so heat flows *out*. + +The same fluid does both, by changing its pressure. + +## The trick: boiling point depends on pressure + +A liquid boils at a lower temperature when the pressure is lower. Water boils below 100 °C on a mountain for this reason. A heat pump uses a fluid called a **refrigerant**, for example propane, in a closed loop. Propane boils at about −5 °C at 4 bar, and at about 45 °C at 15 bar (approximate values). + +``` + OUTSIDE (air 5 °C) INSIDE (heating water 35 °C) + + ┌───────────────────────────┐ vapour ┌────────────────────────────┐ + │ EVAPORATOR ~4 bar │ │ CONDENSER ~15 bar │ + │ boils at −5 °C ├─►COMPRESSOR─►│ condenses at 45 °C │ + │ heat flows IN (2 kWh) │ (1 kWh │ heat flows OUT (3 kWh) │ + │ air 5 °C → fluid −5 °C │ electric) │ fluid 45 °C → water 35 °C │ + └───────────────────────────┘ └────────────────────────────┘ + ▲ │ + │ liquid │ + └──────────── EXPANSION VALVE ◄────────────┘ + 15 bar → 4 bar, fluid cools to −5 °C +``` + +Follow one kilogram of refrigerant around the loop: + +1. **Evaporator (outside).** The liquid is at 4 bar, so it boils at −5 °C. The outdoor air is 5 °C, which is hotter, so heat flows into the fluid. Boiling absorbs this heat as the liquid turns to vapour. +2. **Compressor.** An electric motor squeezes the vapour to 15 bar. This is the only place the electricity enters the loop. The work of compression also heats the vapour. +3. **Condenser (inside).** At 15 bar the fluid condenses at 45 °C. The heating water is 35 °C, which is colder, so heat flows out of the fluid. The fluid releases the heat it picked up outside *and* the compressor work. +4. **Expansion valve.** A narrow opening drops the pressure back to 4 bar. Part of the liquid flashes to vapour, which cools the rest to −5 °C. The loop starts again. + +A refrigerator is the same machine. It takes heat from the food box and dumps it into the kitchen through the coil at the back. + +## Why the COP is not unlimited + +Moving heat from cold to hot costs work, and the cost grows with the **temperature lift**: the gap between the temperature where the fluid condenses and where it boils. + +The second law sets the limit. Heat *Q* leaving or entering a body at absolute temperature *T* carries an amount of entropy *Q/T*, and total entropy cannot decrease (established fact). So the entropy dumped inside must be at least the entropy taken from outside: + +``` +Q_H / T_H ≥ Q_C / T_C and W = Q_H − Q_C +⇒ COP = Q_H / W ≤ T_H / (T_H − T_C) (temperatures in kelvin) +``` + +The bound has lift in the denominator. A small lift gives a high possible COP; a large lift gives a low one. + +| Case | Fluid boils | Fluid condenses | Lift | Ideal COP | Typical real COP (estimate, ~47% of ideal) | +|---|---|---|---|---|---| +| Mild day, air 5 °C | −5 °C (268 K) | 45 °C (318 K) | 50 K | 318/50 = 6.4 | ~3.0 | +| Cold day, air −15 °C | −25 °C (248 K) | 45 °C (318 K) | 70 K | 318/70 = 4.5 | ~2.1 | + +Note that the lift the machine must do is larger than the gap between the air and the heating water. The fluid must be about 10 K colder than the air and about 10 K hotter than the water, so heat flows at a useful rate in both coils. Real machines also lose energy to friction, motor heat, and imperfect compression. That is why they reach only about half the ideal value. + +**Predict:** what happens to the COP if the house uses underfloor heating, so the fluid only needs to condense at 35 °C instead of 45 °C? *(The lift falls to 40 K, the ideal COP rises to 308/40 = 7.7, and the real COP rises to about 3.6. This is why heat pumps work best with large, low-temperature radiators or floors.)* + +## Summary + +- A heat pump moves heat; it does not create it. Heat out = heat from outside + electricity. +- It moves heat "uphill" by changing the refrigerant's pressure, so the same fluid boils below outdoor temperature and condenses above indoor temperature. +- The work needed grows with the temperature lift. A mild day gives COP ≈ 3; a very cold day gives COP ≈ 2. Both beat a resistance heater's COP of 1. + +*Not covered:* defrost cycles, ground-source heat pumps, refrigerant choice, and cost or CO₂ comparisons. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/model.md new file mode 100644 index 0000000..4b33f55 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/model.md @@ -0,0 +1,45 @@ +# Model: heat pump COP + +## Central question +How can a heat pump deliver 3 kWh of heat for 1 kWh of electricity without breaking energy conservation? + +## Answer in one line +The electricity does not make the heat. It moves heat from cold outdoor air into the house. Heat delivered = heat taken from outside + electrical work: 3 = 2 + 1. + +## Reader +Technical, school physics and maths, new to thermodynamics of heat pumps. Knows energy conservation, kelvin, that boiling absorbs heat. Probably does not know entropy or the pressure dependence of boiling point in a usable way. + +## Entities +- Refrigerant: working fluid in a closed loop. +- Evaporator (outdoor coil), compressor, condenser (indoor coil), expansion valve. +- Outdoor air (heat source), house water/air (heat sink). +- W: electrical work. Q_C: heat taken from outside. Q_H: heat delivered inside. +- COP = Q_H / W. + +## Causal chain +1. Heat flows only from hot to cold unaided (second law). +2. Boiling point of a fluid rises with pressure (propane: ~ -5 C at ~4 bar, ~45 C at ~15 bar). +3. At low pressure the refrigerant boils at -5 C, colder than 5 C outdoor air: heat flows in. +4. Compressor raises pressure, so the vapour can condense at 45 C, hotter than 35 C heating water: heat flows out. +5. Expansion valve drops pressure back; loop repeats. +6. Energy balance: Q_H = Q_C + W. + +## Why this form (limit) +Entropy: heat Q at temperature T carries entropy Q/T; total cannot decrease. So Q_H/T_H >= Q_C/T_C, giving COP <= T_H/(T_H - T_C) (kelvin). Work grows with temperature lift. + +## Cases (numbers) +- Simplest alternative: resistance heater, COP = 1. +- Mild: refrigerant -5 C -> 45 C, ideal 318/50 = 6.4, real ~ 47% -> ~3.0. +- Cold: refrigerant -25 C -> 45 C, ideal 318/70 = 4.5, real ~2.1. + +## Epistemic status +- Energy balance, second law, Carnot bound: established. +- 47% of ideal, specific pressures: estimates / typical values. + +## Confusion points +- "Over 100% efficient" sounds like free energy. It is not; the outside air supplies the extra. +- Cold air still holds heat; it just needs something colder than it to give heat to. +- Refrigerant temperatures, not room/air temperatures, set the real lift. + +## Scope (left out) +Defrost cycles, ground-source units, refrigerant choice, cost and CO2 comparison. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/model.yaml new file mode 100644 index 0000000..828d87d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/explainers/heat-pump/model.yaml @@ -0,0 +1,28 @@ +energy_balance: + W_kWh: 1 + Q_C_kWh: 2 + Q_H_kWh: 3 + COP: 3 +resistance_heater_COP: 1 +mild_case: + outdoor_air_C: 5 + heating_water_C: 35 + evaporate_C: -5 + evaporate_K: 268 + evaporate_pressure_bar: 4 # approx, propane + condense_C: 45 + condense_K: 318 + condense_pressure_bar: 15 # approx, propane + lift_K: 50 + ideal_COP: 6.4 + real_COP: 3.0 +cold_case: + outdoor_air_C: -15 + evaporate_C: -25 + evaporate_K: 248 + condense_C: 45 + condense_K: 318 + lift_K: 70 + ideal_COP: 4.5 + real_COP: 2.1 +real_fraction_of_ideal: 0.47 # estimate, typical diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/read.json new file mode 100644 index 0000000..7e7d388 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/read.json @@ -0,0 +1,46 @@ +{ + "answers": { + "1": { + "answer": "From the outdoor air. The electricity moves heat from outside into the house rather than creating it.", + "quote": "the electricity does not *make* the heat. It *moves* heat from the cold outdoor air into the house." + }, + "2": { + "answer": "The refrigerant boils in the evaporator to absorb heat from outside, is compressed to raise its temperature, condenses in the condenser to release heat inside, then pressure drops through the expansion valve to repeat the cycle.", + "quote": "It moves heat \"uphill\" by changing the refrigerant's pressure, so the same fluid boils below outdoor temperature and condenses above indoor temperature." + }, + "3": { + "answer": "On a colder day, the outdoor air is colder, so the refrigerant must boil at a lower temperature. This increases the temperature lift, which requires more work and reduces COP (heat delivered per unit of electricity).", + "quote": "Moving heat from cold to hot costs work, and the cost grows with the **temperature lift**: the gap between the temperature where the fluid condenses and where it boils." + }, + "4": { + "answer": "The 5°C day has higher ideal COP (better efficiency) because the temperature lift is smaller. Smaller lift means lower work requirement and higher COP, so the 5°C day outperforms the −15°C day.", + "quote": "The bound has lift in the denominator. A small lift gives a high possible COP; a large lift gives a low one." + } + }, + "stops": [ + "\"heat flows by itself only from a hotter body to a colder one (the second law of thermodynamics)\" — stated without explaining why this matters here", + "\"COP = Q_H / W ≤ T_H / (T_H − T_C)\" — derivation from entropy constraint shown but not explained", + "\"Part of the liquid flashes to vapour, which cools the rest to −5 °C\" — flash evaporation mechanism unexplained", + "\"the fluid must be about 10 K colder than the air and about 10 K hotter than the water\" — why this offset is necessary not justified", + "Table row \"Cold day, air −15 °C\" maps to \"Fluid boils −25 °C\" without prior explanation", + "\"It is 278 K, far above absolute zero\" — Celsius to Kelvin conversion not explained", + "\"A liquid boils at a lower temperature when the pressure is lower\" — physical mechanism not explained", + "Entropy formula derivation skipped between \"Q_H / T_H ≥ Q_C / T_C\" and the COP formula result" + ], + "excess": [ + "\"A refrigerator is the same machine\" — analogy not needed to understand heat pumps", + "\"An electric resistance heater turns 1 kWh of electricity into exactly 1 kWh of heat\" — baseline comparison", + "\"Real machines also lose energy to friction, motor heat, and imperfect compression\" — explanation of real vs ideal gap", + "\"Water boils below 100 °C on a mountain for this reason\" — mountain analogy unnecessary", + "\"A heat pump moves heat; it does not create it\" — repeats the opening", + "\"A COP of 1 is no better than a resistance heater\" — reference point for comparison", + "\"This is why heat pumps work best with large, low-temperature radiators\" — design recommendation", + "\"*Not covered:* defrost cycles, ground-source heat pumps\" — scope statement" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/run.json new file mode 100644 index 0000000..387a03e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/heat-pump/explanation.md` (prose with one text diagram of the refrigerant loop), backed by `model.md` and `model.yaml`; prose fits because the idea is a short causal chain (heat out = heat from outside + electricity, plus how pressure changes the boiling point) that one diagram and two worked cases cover, so an interactive page would add little; I couldn't run `explainer check` because no shell was available here.", + "error": null, + "cost": 0.2534982, + "seconds": 67, + "turns": 5, + "files": [ + "explainers/heat-pump/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/diagram.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/diagram.md new file mode 100644 index 0000000..9dcb729 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/diagram.md @@ -0,0 +1,43 @@ +# How can a heat pump give three times more heat than the electricity it uses? + +An electric heater turns 1 kWh of electricity into 1 kWh of heat. A heat pump gives about 3 kWh of heat for the same 1 kWh. Where do the other 2 kWh come from? + +**Answer:** the heat pump moves heat from the cold outdoor air into the home. The electricity pays for the move. COP = heat delivered ÷ electricity used: heater 1, heat pump 3. + +## Level 1 + +```mermaid +flowchart LR + E["Electricity
1 kWh"] --> HP["Heat pump"] + A["Outdoor air at 0 °C
outdoor heat: 2 kWh"] --> HP + HP --> W["Home water at 35 °C
heat delivered: 3 kWh"] +``` + +The 2 kWh is outdoor heat. Air at 0 °C still holds heat. Heat flows from the air into the refrigerant because the heat pump makes the refrigerant colder than the air. + +## Level 2 + +```mermaid +flowchart LR + AIR["Outdoor air 0 °C"] -->|"2 kWh"| EV["1. Evaporator: boils at −10 °C, 3.5 bar"] + EV --> CO["2. Compressor: 3.5 → 14 bar"] + ELEC["Electricity"] -->|"1 kWh"| CO + CO --> CD["3. Condenser: condenses at 40 °C, 14 bar"] + CD -->|"3 kWh"| WATER["Home water 35 °C"] + CD --> VA["4. Expansion valve: 14 → 3.5 bar"] + VA --> EV +``` + +Boiling temperature depends on pressure. Propane boils at −10 °C at 3.5 bar and at 40 °C at 14 bar. The compressor raises the pressure so the refrigerant condenses at 40 °C, above the 35 °C water. The valve drops the pressure so it boils at −10 °C, below the 0 °C air. Heat flows hot to cold at both coils. + +## Level 3 — What the outdoor temperature does to the COP + +The temperature lift is home water minus outdoor air. Colder air is denser and holds more heat per cubic metre, so the heat pump collects heat more easily on a cold day. The COP rises as the outdoor temperature falls. + +| Outdoor air | Typical real COP | +|---|---| +| 10 °C | about 2 | +| 0 °C | about 3 | +| −15 °C | about 4 | + +The colder the day, the less electricity each kWh of heat needs. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/model.md new file mode 100644 index 0000000..fe9ee2d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/model.md @@ -0,0 +1,18 @@ +# Semantic model: heat pump (broken-copy test; the model is correct, the rendering is not) + +## Audience and prior knowledge + +A technical reader: school mathematics and science, basic programming. New to heat pumps. + +## Key facts + +- Energy balance: 3 kWh heat delivered = 1 kWh electricity + 2 kWh outdoor heat. COP = 3; electric heater COP = 1. +- Loop: evaporator (refrigerant boils at −10 °C, 3.5 bar, colder than 0 °C air, takes 2 kWh); compressor (1 kWh, raises to 14 bar); condenser (condenses at 40 °C, hotter than 35 °C water, gives 3 kWh); expansion valve (drops pressure, refrigerant cools to −10 °C). Boiling temperature rises with pressure. +- Temperature lift = home water − outdoor air. Larger lift → lower COP: 10 °C outside, lift 25 K, COP about 4; 0 °C, 35 K, about 3; −15 °C, 50 K, about 2. +- Ideal limit T_hot / (T_hot − T_cold) in kelvin: 8.8 at 0 °C → 35 °C. + +## Epistemic status + +- Established: energy conservation, heat flows hot to cold, boiling point rises with pressure. +- Assumed: 0 °C outside, 35 °C water. +- Estimated: typical real COPs; propane pressures. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/model.yaml new file mode 100644 index 0000000..4bb8238 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/model.yaml @@ -0,0 +1,29 @@ +values: {} +claims: + - id: energy-balance + kind: guarantee + statement: "A heat pump does not create energy: heat delivered = electricity + outdoor heat, 3 kWh = 1 kWh + 2 kWh." + example: "0 °C outside, 35 °C water: 1 kWh electricity, 2 kWh outdoor heat, 3 kWh into the water, COP 3." + counterexample: "An electric heater has only electricity as input: 1 kWh in, 1 kWh heat out, COP 1." + ask: "Where do the other 2 kWh come from, if the heat pump uses only 1 kWh of electricity?" + - id: loop + kind: mechanism + statement: "Evaporator, compressor, condenser, expansion valve: the refrigerant takes heat in where it is colder than the air and gives heat out where it is hotter than the water." + example: "Boils at −10 °C, taking 2 kWh from 0 °C air; compressor adds 1 kWh; condenses at 40 °C, giving 3 kWh to 35 °C water." + ask: "Why must the refrigerant be colder than the outdoor air in the evaporator, and hotter than the home water in the condenser?" + - id: lift-sets-cop + kind: guarantee + statement: "The larger the temperature lift, the more electricity each kWh of heat costs, so COP falls." + example: "10 °C outside: lift 25 K, COP about 4." + counterexample: "−15 °C outside: lift 50 K, COP about 2." + ask: "What happens to the COP on a colder day, and why?" +quiz: + - q: "A heat pump uses 1 kWh of electricity and delivers 3 kWh of heat. Does it break energy conservation? Where does the energy come from?" + expect: "No. 2 kWh comes from the outdoor air, which is colder than the home but still holds heat; the refrigerant is made colder than the air so heat flows into it. 1 + 2 = 3." + misconception: "It creates energy, or it is '300% efficient' in the sense of making more energy than it uses." + - q: "The same heat pump runs on a 10 °C day and on a −15 °C day, heating water to 35 °C. On which day is its COP higher, and why?" + expect: "On the 10 °C day (about 4 vs about 2): the temperature lift is smaller (25 K vs 50 K), so less work is needed per kWh moved." + misconception: "COP is a fixed number for the machine, or it is higher in the cold." + - q: "What do the compressor and the expansion valve each do to the refrigerant, and why does the loop need both?" + expect: "Compressor raises pressure, so the refrigerant condenses at a high temperature (40 °C) above the water; valve drops pressure, so it boils at a low temperature (−10 °C) below the air. Boiling temperature depends on pressure." + misconception: "The compressor makes the heat; or the refrigerant is cooled by the outdoor air." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/review/runs/2026-10-04T151949-quiz-diagram.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/review/runs/2026-10-04T151949-quiz-diagram.json new file mode 100644 index 0000000..b119462 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump-broken/review/runs/2026-10-04T151949-quiz-diagram.json @@ -0,0 +1,159 @@ +{ + "tool": "quiz", + "rendering": "diagram", + "slug": "heat-pump-broken", + "at": "2026-10-04T15:19:35", + "model": "default", + "prompt_sha": "8cabe534ead1", + "reply": { + "answers": { + "1": "The explanation names these objects: electricity (1 kWh), the heat pump, outdoor air (0 °C), home water (35 °C), and the refrigerant (propane). It also names the loop's four parts: evaporator, compressor, condenser and expansion valve. The table also uses COP as a measure.", + "2": "Electricity drives the compressor. Outdoor air gives 2 kWh of heat to the refrigerant in the evaporator, where the refrigerant boils at −10 °C and 3.5 bar. The compressor raises the pressure to 14 bar. In the condenser the refrigerant condenses at 40 °C and gives 3 kWh to the home water. The expansion valve drops the pressure back to 3.5 bar, and the refrigerant returns to the evaporator.", + "3": "Pressure sets the refrigerant's boiling temperature. Low pressure lets it boil colder than the outdoor air, so it picks up heat there. High pressure lets it condense hotter than the water, so it gives that heat up. The electricity pays only for raising the pressure, not for making the heat.", + "4": "The explanation treats outdoor temperature as the key variable. It says that as outdoor air gets colder, COP rises: about 2 at 10 °C, 3 at 0 °C and 4 at −15 °C. Its reason is that colder air is denser and holds more heat per cubic metre. I can only report this. The same section also says the temperature lift grows on colder days but never says what that does to COP, and the claim may conflict with the Level 2 logic. Level 2 implies a bigger pressure rise, and so more compressor work, when the air is colder.", + "5": "Simplification: the COP values are round 'typical' numbers, and the figures (exactly 2 + 1 = 3 kWh, a fixed −10 °C and 40 °C) ignore losses. Established fact: propane boils at −10 °C at 3.5 bar and at 40 °C at 14 bar, and heat flows from hot to cold.", + "6": "A heat pump does not make heat from electricity. It moves heat that already exists in outdoor air, even air at 0 °C, into the home water. A refrigerant circulates in a loop. At low pressure it boils below the air temperature and absorbs heat. The compressor raises its pressure so it condenses above the water temperature and releases heat. The expansion valve lowers the pressure again, so 1 kWh of electricity moves 2 kWh of outdoor heat and 3 kWh in total is delivered (COP 3).", + "7": "Level 3 was hardest. It defines 'temperature lift' and then never uses it. It says COP rises in colder weather, which seems to contradict Level 2: colder air would need a lower boiling point, so a bigger pressure rise. Linking lift to compressor work, and checking the claim against Level 2, would make it easier.", + "8": "No. Energy is conserved: 1 kWh of electricity plus 2 kWh taken from the outdoor air equals the 3 kWh delivered.", + "9": "According to the table, the −15 °C day (COP about 4, against about 2 at 10 °C), because colder, denser air 'holds more heat per cubic metre'. The explanation does not show why density matters, and the temperature lift is bigger on the −15 °C day (50 K against 25 K). From the text alone I cannot reconcile this answer with Level 2.", + "10": "The compressor raises the pressure (3.5 to 14 bar) so the refrigerant condenses at 40 °C, above the 35 °C water, and can give heat to it. The expansion valve lowers the pressure (14 to 3.5 bar) so the refrigerant boils at −10 °C, below the 0 °C air, and can take heat from it. The loop needs both because heat only flows from hot to cold: one side has to be colder than the air and the other hotter than the water.", + "11": "They come from the outdoor air. Air at 0 °C still holds heat, and that heat flows into the colder refrigerant (−10 °C) in the evaporator.", + "12": "Heat flows only from hot to cold. To absorb heat from the air, the refrigerant must be colder than the air. To give heat to the water, it must be hotter than the water.", + "13": "The explanation says COP rises on colder days (about 4 at −15 °C) because colder air is denser and holds more heat per cubic metre. I report this as stated, but it is not justified, and the explanation's own temperature-lift idea seems to point the other way." + }, + "audit": { + "terms": [ + "The working fluid is called 'refrigerant' (Level 1, Level 2) and 'Propane' (Level 2 text). The text never says that propane is the refrigerant.", + "The evaporator and condenser are called 'coils' only once ('at both coils', Level 2 text), and the text never says the two names mean the same parts." + ], + "unexplained": [ + "Table 'Typical real COP' (Level 3): the values 2, 3 and 4 are given with no source and no mechanism linking them to temperature lift. They also run the opposite way to what the Level 2 reasoning suggests.", + "'3.5 bar' and '14 bar' (Level 2): the reader is not told how much pressure rise the compressor needs, or how that rise changes with outdoor temperature.", + "'Temperature lift' (Level 3): defined as 'home water minus outdoor air', but the text never says what a bigger or smaller lift does." + ], + "missing_why": [ + "'Colder air is denser and holds more heat per cubic metre, so the heat pump collects heat more easily' (Level 3): no reason is given why density would raise COP. It ignores the evaporator needing an even lower boiling point.", + "'Boiling temperature depends on pressure' (Level 2): stated as a fact with no reason why.", + "Why the compressor is where the electricity goes, and why compressing the refrigerant costs energy (Level 2), is not explained." + ], + "excess": [ + "'The temperature lift is home water minus outdoor air.' (Level 3): it is defined and then never used in the argument." + ] + }, + "gaps": [ + "Level 3's claim that COP rises as outdoor temperature falls seems to contradict the Level 2 logic. Colder air needs a lower boiling point, so a larger pressure rise and a larger lift, yet the text says the opposite and the reader is left confused about Q9 and Q13.", + "The text never links temperature lift or pressure ratio to how much electricity the compressor needs.", + "The text never says why boiling temperature depends on pressure.", + "It is not said that the 2 kWh of outdoor heat must be at least as large as the heat removed from the air. How much heat the air can supply, and the airflow needed, are not covered." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 1, + "5": 1, + "6": 2, + "7": 2, + "8": 1, + "9": 0, + "10": 2, + "11": 2, + "12": 2, + "13": 0 + }, + "reasons": { + "4": "The answer correctly picks outdoor temperature (lift) as the key variable. It then reports the wrong direction (COP rises in the cold) with the wrong 'denser air' reason. The model says a larger lift lowers COP. The reviewer flags the conflict but does not state the correct relationship.", + "5": "The answer calls the propane boiling pressures (3.5 bar at −10 °C, 14 bar at 40 °C) established fact. The model lists propane pressures as estimated. It also omits that 0 °C outside and 35 °C water are assumptions.", + "8": "The answer gives the correct 'No' and 1 + 2 = 3 from outdoor air. It does not explain how colder outdoor air can still give heat: the air holds heat, and the refrigerant is made colder than the air so heat flows into it.", + "9": "The answer picks the −15 °C day, which is wrong. The COP is higher on the 10 °C day (about 4 vs about 2) because the lift is smaller (25 K vs 50 K).", + "13": "The answer reports that COP rises on colder days. The correct answer is that a larger lift costs more electricity per kWh of heat, so COP falls." + }, + "pass": false + }, + "usage": { + "cost": 0.12581900000000001, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The working fluid is called 'refrigerant' (Level 1, Level 2) and 'Propane' (Level 2 text). The text never says that propane is the refrigerant." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The evaporator and condenser are called 'coils' only once ('at both coils', Level 2 text), and the text never says the two names mean the same parts." + }, + { + "n": 3, + "severity": "gap", + "kind": "unexplained", + "text": "Table 'Typical real COP' (Level 3): the values 2, 3 and 4 are given with no source and no mechanism linking them to temperature lift. They also run the opposite way to what the Level 2 reasoning suggests." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'3.5 bar' and '14 bar' (Level 2): the reader is not told how much pressure rise the compressor needs, or how that rise changes with outdoor temperature." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'Temperature lift' (Level 3): defined as 'home water minus outdoor air', but the text never says what a bigger or smaller lift does." + }, + { + "n": 6, + "severity": "gap", + "kind": "missing_why", + "text": "'Colder air is denser and holds more heat per cubic metre, so the heat pump collects heat more easily' (Level 3): no reason is given why density would raise COP. It ignores the evaporator needing an even lower boiling point." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "'Boiling temperature depends on pressure' (Level 2): stated as a fact with no reason why." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Why the compressor is where the electricity goes, and why compressing the refrigerant costs energy (Level 2), is not explained." + }, + { + "n": 9, + "severity": "excess", + "kind": "excess", + "text": "'The temperature lift is home water minus outdoor air.' (Level 3): it is defined and then never used in the argument." + }, + { + "n": 10, + "severity": "gap", + "kind": "gap", + "text": "Level 3's claim that COP rises as outdoor temperature falls seems to contradict the Level 2 logic. Colder air needs a lower boiling point, so a larger pressure rise and a larger lift, yet the text says the opposite and the reader is left confused about Q9 and Q13." + }, + { + "n": 11, + "severity": "gap", + "kind": "gap", + "text": "The text never links temperature lift or pressure ratio to how much electricity the compressor needs." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "The text never says why boiling temperature depends on pressure." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "It is not said that the 2 kWh of outdoor heat must be at least as large as the heat removed from the air. How much heat the air can supply, and the airflow needed, are not covered." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/diagram.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/diagram.md new file mode 100644 index 0000000..58d8ef5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/diagram.md @@ -0,0 +1,135 @@ +# How can a heat pump give three times more heat than the electricity it uses? + +An electric heater turns 1 kWh of electricity into 1 kWh of heat. A heat pump gives about 3 kWh of heat for the same 1 kWh. Energy cannot be created, so where do the other 2 kWh come from? + +**Answer:** the heat pump does not make that heat. It moves heat from the cold outdoor air into the home. The electricity pays for the move. + +The measure is the **COP** (coefficient of performance): heat delivered ÷ electricity used. An electric heater has COP 1. The heat pump in this example has COP 3. + +## Level 1 — Where the 3 kWh comes from + + + +```mermaid +flowchart LR + E["Electricity
1 kWh"]:::elec --> HP["Heat pump"] + A["Outdoor air at 0 °C
outdoor heat: 2 kWh"]:::cold --> HP + HP --> W["Home water at 35 °C
heat delivered: 3 kWh"]:::hot + classDef cold fill:#dbeafe,stroke:#1d4ed8,color:#000 + classDef hot fill:#fee2e2,stroke:#b91c1c,color:#000 + classDef elec fill:#fef9c3,stroke:#a16207,color:#000 +``` + +*Colours in both diagrams: blue is the cold side, red is the hot side, yellow is electricity.* + +The 2 kWh is **outdoor heat**, taken from the outdoor air. Air at 0 °C is 273 K above absolute zero, so it still holds heat. Heat flows from the air into the heat pump because the heat pump makes the fluid inside it, the refrigerant, colder than the air (Level 2 shows how). The 3 kWh goes into the **home water**: the 35 °C water that flows through the floor pipes or radiators. + +An electric heater has only one input, the electricity. So it can give only 1 kWh per kWh. + +## Level 2 — How heat moves from cold air into warmer water + + + +Suppose you pipe a fluid between the outdoor air and the home, with no other step. The fluid warms to at most 0 °C outside. Inside, the 35 °C water is hotter, so heat flows from the water into the fluid, and the home gets colder. + +So the fluid must be colder than the air outside and hotter than the water inside. One fluid can be both, because its boiling temperature depends on its pressure. + + + +Propane boils at −10 °C at about 3.5 bar and at 40 °C at about 14 bar. (The air around us is at 1 bar.) At one fixed pressure, propane has one boiling temperature, so it could not be below 0 °C outside and above 35 °C inside. A fluid used this way is a **refrigerant**. It circulates in a closed loop of four parts, at a low pressure on the outdoor side and a high pressure on the indoor side. + + + +```mermaid +flowchart LR + AIR["Outdoor air
0 °C"]:::cold -->|"2 kWh outdoor heat"| EV + subgraph LOW["Low pressure, 3.5 bar"] + EV["1. Evaporator (outdoor coil)
refrigerant boils at −10 °C"]:::cold + VA["4. Expansion valve
pressure drops; refrigerant cools to −10 °C"]:::cold + end + CO["2. Compressor
squeezes vapour 3.5 → 14 bar;
vapour heats to about 60 °C"]:::hot + subgraph HIGH["High pressure, 14 bar"] + CD["3. Condenser (indoor coil)
vapour cools to 40 °C,
then condenses at 40 °C"]:::hot + end + ELEC["Electricity"]:::elec -->|"1 kWh"| CO + EV -->|"cold vapour"| CO + CO -->|"hot vapour"| CD + CD -->|"warm liquid"| VA + VA -->|"cold liquid and vapour"| EV + CD -->|"3 kWh heat delivered"| WATER["Home water
35 °C"]:::hot + classDef cold fill:#dbeafe,stroke:#1d4ed8,color:#000 + classDef hot fill:#fee2e2,stroke:#b91c1c,color:#000 + classDef elec fill:#fef9c3,stroke:#a16207,color:#000 +``` + +Follow the refrigerant once around the loop: + +1. **Evaporator** (the outdoor coil). The refrigerant is at 3.5 bar, so it boils at −10 °C. It is colder than the 0 °C air, so 2 kWh of heat flows from the air into it. Boiling takes this heat without a change in temperature. +2. **Compressor.** An electric pump squeezes the vapour from 3.5 bar to 14 bar. It uses 1 kWh of electricity to push on the vapour, and that push heats the vapour to about 60 °C, as a bicycle pump gets warm when you use it. +3. **Condenser** (the indoor coil). The 60 °C vapour cools to 40 °C, then condenses, because at 14 bar propane condenses at 40 °C. The refrigerant is hotter than the 35 °C water, so heat flows into the water. The compressor's 1 kWh is in the vapour too, so the water gets 2 + 1 = 3 kWh. +4. **Expansion valve.** A narrow opening drops the pressure from 14 bar to 3.5 bar. At 3.5 bar the 40 °C liquid is far above its −10 °C boiling point, so about a third of it boils at once. That boiling takes heat from the rest of the liquid and cools all of it to −10 °C. The valve adds no energy and removes none. The refrigerant goes back to step 1. + +At both coils, heat flows the normal way, from hot to cold. The compressor raises the pressure and the valve lowers it. Only the compressor uses energy. + +## Level 3 — What the outdoor temperature does to the COP + +> **Predict first.** The outdoor air drops from 10 °C to −15 °C. The home water stays at 35 °C. Does the COP rise, fall, or stay the same? + + + +The **temperature lift** is the home water temperature minus the outdoor air temperature: 35 − 0 = 35 K in the example. The bigger the lift, the more electricity each kWh of heat needs. + +| Outdoor air | Temperature lift | Ideal limit | Typical real COP (estimate) | +|---|---|---|---| +| 10 °C | 25 K | 12.3 | about 4 | +| 0 °C | 35 K | 8.8 | about 3 | +| −15 °C | 50 K | 6.2 | about 2 | +| −25 °C | 60 K | 5.1 | about 1.5 | + +The COP falls. At −25 °C the heat pump still beats a heater, but only just; many units add a backup electric heater (COP 1) for the coldest days. Hotter water also raises the lift. Old radiators need 55 °C water: at 0 °C outside, the lift is 55 K and the COP is about 2.2. + +### The ideal limit + + + +Physics sets an **ideal limit** on the COP (physicists call it the Carnot limit). For 35 °C water and 0 °C air it is 308.15 / 35 = 8.8. In symbols: + +ideal limit = T_hot / (T_hot − T_cold) + +Here T_hot is the home water temperature (308.15 K) and T_cold is the outdoor air temperature (273.15 K). A bigger lift makes the bottom of the fraction bigger, so the limit falls. + + + +Use kelvin, not Celsius. In Celsius the formula gives 35 / (35 − 0) = 1, which would say that a heat pump can never beat a heater. + +### Where the limit comes from + +The limit comes from **entropy**. Heat Q that passes at kelvin temperature T carries entropy Q/T, in kWh/K. So 1 kWh carries more entropy at a low temperature than at a high one. Electricity carries no entropy. Nothing can destroy entropy. This is the same law that stops heat flowing from cold to hot by itself. For example, 1 kWh flowing by itself from 308.15 K water to 273.15 K air carries away 1/308.15 and brings 1/273.15, a net gain of 0.00042 kWh/K. The reverse flow would destroy that entropy, so it never happens by itself. + +The refrigerant ends each cycle where it started, so it cannot store entropy. The electricity brings none. So the water must get at least as much entropy as the air loses. Write Q_hot for the heat delivered, Q_cold for the outdoor heat, and W for the electricity: + +Q_hot / T_hot ≥ Q_cold / T_cold + +- **Allowed:** COP 3. The air loses 2 / 273.15 = 0.00732 kWh/K. The water gets 3 / 308.15 = 0.00974 kWh/K. 0.00974 ≥ 0.00732. +- **Forbidden:** a claimed COP 12, with 1 kWh of electricity and 11 kWh from the air. The air loses 11 / 273.15 = 0.0403 kWh/K. The water gets 12 / 308.15 = 0.0389 kWh/K. Entropy would be destroyed, so no machine can do this. + +Energy conservation gives Q_cold = Q_hot − W. Put this into the inequality: + +1. Q_hot / T_hot ≥ (Q_hot − W) / T_cold +2. Multiply by T_cold: Q_hot × T_cold / T_hot ≥ Q_hot − W +3. Rearrange: W ≥ Q_hot × (T_hot − T_cold) / T_hot +4. Divide: COP = Q_hot / W ≤ T_hot / (T_hot − T_cold) + +So the minimum electricity for 3 kWh is 3 × 35 / 308.15 = 0.34 kWh. A real heat pump uses 1 kWh. + +### Why a real heat pump gets 3, not 8.8 + +Heat flows through a coil only across a temperature gap, and a bigger gap moves heat faster through a smaller coil. So the refrigerant boils 10 K below the air (−10 °C) and condenses 5 K above the water (40 °C). Its own lift, the **refrigerant lift**, is 40 − (−10) = 50 K, so the ideal limit for the refrigerant is 313.15 / 50 = 6.3. Losses in the compressor, fans, and pumps take the COP from 6.3 down to about 3. + +## Summary + +A heat pump does not make 3 kWh of heat from 1 kWh of electricity. It moves 2 kWh of outdoor heat into the home, and the 1 kWh of electricity pays for the move and ends up as heat too. Pressure sets the boiling temperature, so the refrigerant is colder than the air outside and hotter than the water inside. Heat flows the normal way at both coils. The colder the day, the more electricity each kWh of heat needs. + +A fridge uses the same loop, with the cold coil inside the fridge. + +**Not covered here:** ground-source heat pumps (warmer, steadier cold side, so a higher COP); defrosting the outdoor coil below about 5 °C; money and CO₂ compared with a gas boiler; seasonal COP (the average over a winter); refrigerant choice and safety; why the loop uses a boiling fluid and not a warmed gas. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/model.md new file mode 100644 index 0000000..f6e9b7a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/model.md @@ -0,0 +1,160 @@ +# Semantic model: how a heat pump delivers more heat than the electricity it uses + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How can a heat pump put 3 kWh of heat into a home for each 1 kWh of electricity, when energy cannot be created? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (energy conservation, temperature, kelvin, pressure, boiling and condensing, heat flows from hot to cold), basic programming. New to thermodynamic cycles. Does not know: refrigerant, evaporator, compressor, condenser, expansion valve, COP, entropy, that boiling temperature depends on pressure well enough to use it, the ideal limit (Carnot). + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Electric heater | A resistor that converts electricity into heat, 1 kWh in, 1 kWh out. | fact | +| Heat pump | A machine that uses electricity to move heat from a cold place (outdoor air) to a warmer place (home water). | fact | +| Outdoor air | The cold source; at 0 °C (273 K) it still holds thermal energy that can be taken out. | fact | +| Home water | The hot side: water at 35 °C that flows through underfloor pipes or radiators. | assumption (typical) | +| Refrigerant | A fluid in a closed loop that boils and condenses at useful temperatures (for example propane). | fact | +| Evaporator | Outdoor heat exchanger where the cold, low-pressure refrigerant boils and takes heat from the air. | fact | +| Compressor | Electric pump that squeezes the refrigerant vapour to high pressure; the work done on the gas makes it hot (as a bicycle pump gets warm). | fact | +| Condenser | Indoor heat exchanger where the hot, high-pressure refrigerant condenses and gives heat to the home water. | fact | +| Expansion valve | Narrow opening that drops the refrigerant pressure; part of the liquid boils at once, and that boiling cools the rest. | fact | +| COP | Coefficient of performance: heat delivered ÷ electricity used. | definition | +| Temperature lift | Home water temperature minus outdoor air temperature (35 K in the example). | definition | +| Refrigerant lift | Condensing temperature minus boiling temperature of the refrigerant (50 K in the example); larger than the temperature lift because heat exchangers need a temperature gap. | definition | +| Entropy | Bookkeeping quantity: heat Q passing at absolute temperature T carries Q/T of entropy; entropy can be created but never destroyed. | established fact | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Outdoor air | gives heat to | Evaporator | because the refrigerant there (−10 °C) is colder than the air (0 °C) | +| Evaporator | sends low-pressure vapour to | Compressor | | +| Compressor | sends hot high-pressure vapour to | Condenser | electricity enters here | +| Condenser | gives heat to | Home water | because the refrigerant there (40 °C) is hotter than the water (35 °C) | +| Condenser | sends high-pressure liquid to | Expansion valve | | +| Expansion valve | sends a cold low-pressure mix of liquid and vapour to | Evaporator | | +| Temperature lift | sets the minimum electricity per kWh of heat | COP | larger lift → lower COP | + +## Dependencies (learning order) + +1. Energy conservation: heat delivered = electricity + outdoor heat. +2. Heat flows only from hot to cold by itself. +3. Boiling temperature rises with pressure. +4. The refrigerant loop uses (3) to make the refrigerant colder than outdoor air and hotter than home water, so (2) works for it at both ends. +5. Moving heat from cold to hot costs work; the work grows with the temperature lift. + +## Causal chain + +```text +Expansion valve drops pressure → refrigerant boils at −10 °C → colder than 0 °C air → heat flows in (2 kWh) +Compressor adds 1 kWh of work → pressure and temperature rise → condenses at 40 °C → hotter than 35 °C water → heat flows out (3 kWh) +``` + +## Temporal sequence + +1. Evaporator: cold refrigerant at about 3.5 bar boils at −10 °C and takes 2 kWh from 0 °C outdoor air. +2. Compressor: uses 1 kWh of electricity to squeeze the vapour from about 3.5 bar to about 14 bar (the air around us is 1 bar). The work ends up as heat in the vapour, which leaves at about 60 °C. +3. Condenser: the vapour cools to 40 °C, condenses at 40 °C, and gives 3 kWh to 35 °C home water. +4. Expansion valve: pressure drops from about 14 bar to about 3.5 bar. At 3.5 bar the 40 °C liquid is far above its −10 °C boiling point, so about a third of it boils at once. That boiling takes heat from the rest of the liquid and cools everything to −10 °C. The valve adds and removes no energy. The loop repeats. + +## Quantities + +- Symbols, mapped once: Q_hot = heat delivered (3 kWh), Q_cold = outdoor heat (2 kWh), W = electricity (1 kWh). +- Energy balance (0 °C outside, 35 °C water): heat delivered 3 kWh = electricity 1 kWh + outdoor heat 2 kWh. +- COP = heat delivered / electricity = 3 / 1 = 3. Electric heater COP = 1. +- Ideal limit (Carnot): ideal COP = T_hot / (T_hot − T_cold), temperatures in kelvin. At 35 °C (308.15 K) and 0 °C (273.15 K): 308.15 / 35 = 8.8. +- Propane boiling pressures (approximate): −10 °C at about 3.5 bar; 40 °C at about 14 bar. +- Effect of outdoor temperature (water 35 °C): + +| Outdoor | Temperature lift | Ideal COP | Typical real COP | +|---|---|---|---| +| 10 °C | 25 K | 12.3 | about 4 | +| 0 °C | 35 K | 8.8 | about 3 | +| −15 °C | 50 K | 6.2 | about 2 | + +- Water at 55 °C (old radiators), 0 °C outside: lift 55 K, ideal 328.15 / 55 = 6.0, typical real about 2.2. +- Why real is about a third of ideal: heat flows through a heat exchanger only across a temperature gap, and a bigger gap moves heat faster through a smaller coil. So the refrigerant boils at −10 °C (10 K below the air) and condenses at 40 °C (5 K above the water). Its refrigerant lift is 50 K, not 35 K: ideal 313.15 / 50 = 6.3. The compressor, fans, and pumps lose about half of the rest. +- Very cold weather: at −25 °C outside, lift 60 K, ideal 308.15 / 60 = 5.1, typical real about 1.5 (estimate). Real COP stays above 1 in most winters; many units add a backup electric heater (COP 1) for the coldest days. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Colder outside | COP falls | larger lift needs more work per kWh moved | +| Hotter water (radiators) | COP falls | larger lift | +| No compressor (just a pump circulating fluid) | no heating | fluid cannot be hotter than the air it came from | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Compress the refrigerant (spend work) | Just circulate a fluid between outside and inside | Fluid warmed by 0 °C air reaches at most 0 °C; in a 20 °C home heat flows from home to fluid, so the home loses heat. | +| Change pressure to change boiling temperature | Use a fixed-pressure fluid | At one pressure the fluid boils and condenses at one temperature; it cannot be both below 0 °C outside and above 35 °C inside. | +| Ideal limit uses kelvin | Use Celsius in T_hot / (T_hot − T_cold) | 35 / (35 − 0) = 1, which says a heat pump cannot beat a heater. Kelvin gives 308.15 / 35 = 8.8. | +| COP above 1 | "Energy out cannot exceed energy in" | It does not: 3 kWh out = 1 kWh electricity + 2 kWh outdoor heat. The heater has COP 1 because electricity is its only input. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| COP above 1 because heat is moved, not made | 0 °C outside, 35 °C water: 1 kWh in, 2 kWh from air, 3 kWh out | Electric heater makes heat from electricity only: 1 kWh in, 1 kWh out, COP 1 | +| Larger lift → lower COP | 10 °C outside: lift 25 K, COP about 4 | −15 °C outside: lift 50 K, COP about 2 | +| Heat flows in at the evaporator because refrigerant is colder than the air | refrigerant −10 °C, air 0 °C | if the refrigerant were at 5 °C, heat would flow from refrigerant to air | +| No machine beats the ideal limit | COP 3: entropy taken 2 / 273.15 = 0.00732 kWh/K ≤ given 3 / 308.15 = 0.00974 kWh/K | claimed COP 12: taken 11 / 273.15 = 0.0403 > given 12 / 308.15 = 0.0389, so entropy would be destroyed | + +## Terms + +- **Heat pump**: moves heat from a cold place to a warmer place, using electricity. +- **Refrigerant**: the working fluid in the closed loop. +- **Evaporator / compressor / condenser / expansion valve**: the four parts of the loop, as above. +- **COP**: heat delivered ÷ electricity used. COP 1: no better than a heater. COP 3: three times the heat. COP 8.8: the ideal limit at 0 °C → 35 °C. +- **Outdoor heat**: the heat taken from the air (2 kWh). Not "free energy". +- **Temperature lift**: home water minus outdoor air. 25 K: easy, COP about 4. 50 K: hard, COP about 2. +- **Refrigerant lift**: refrigerant condensing minus boiling temperature; 50 K in the example. Used only to explain why real COP is below ideal. +- **Ideal limit** (ideal COP): the highest COP physics allows for a given pair of temperatures. Not "efficiency"; "Carnot" only at first mention. +- **Entropy**: Q/T for heat Q at kelvin temperature T; never destroyed. Used only in L3. + +## Epistemic status + +- **Observed / established:** energy conservation; heat flows hot to cold; boiling point rises with pressure; entropy is never destroyed; the Carnot limit. +- **Mathematical consequence:** COP = 1 + outdoor heat / electricity; ideal-limit values. +- **Assumed:** 0 °C outside, 35 °C water, refrigerant at −10 °C and 40 °C. +- **Estimated:** typical real COPs (about 4, 3, 2, 2.2, 1.5); propane pressures (about 3.5 and 14 bar); vapour at about 60 °C after the compressor; about a third of the liquid boils in the valve. + +## Confusion points + +- "COP 3 breaks energy conservation" → 3 kWh out = 1 kWh electricity + 2 kWh outdoor heat. +- "0 °C air has no heat to give" → 0 °C is 273 K; air holds thermal energy far above absolute zero; it gives heat to anything colder, such as −10 °C refrigerant. +- "The COP is a fixed property of the machine" → it depends on the temperature lift; the same unit gives about 4 at 10 °C and about 2 at −15 °C. +- "The compressor's electricity is lost" → the compressor's work also ends up as heat in the home; it is part of the 3 kWh. + +## Why the ideal limit has this form (L3) + +Heat Q passing at absolute temperature T carries entropy Q/T. Entropy can be created but never destroyed; heat flowing by itself from hot to cold creates it (1 kWh from 308.15 K to 273.15 K: −1/308.15 + 1/273.15 = +0.00042 kWh/K). The loop returns to its start each cycle, so it cannot store entropy, and electricity (work) carries no entropy. So the entropy given to the home water must be at least the entropy taken from the air: Q_hot / T_hot ≥ Q_cold / T_cold. + +- Holds: COP 3. Taken: 2 / 273.15 = 0.00732 kWh/K. Given: 3 / 308.15 = 0.00974 kWh/K. 0.00974 ≥ 0.00732. +- Breaks: a claimed COP 12 (1 kWh electricity, 11 kWh from air, 12 kWh out). Taken: 11 / 273.15 = 0.0403. Given: 12 / 308.15 = 0.0389. Entropy would be destroyed, so no machine can do it. +- Algebra: put Q_cold = Q_hot − W into the inequality: Q_hot / T_hot ≥ (Q_hot − W) / T_cold, so W ≥ Q_hot × (T_hot − T_cold) / T_hot, so COP = Q_hot / W ≤ T_hot / (T_hot − T_cold) = 308.15 / 35 = 8.8. +- Meaning: the minimum electricity for 3 kWh is 3 × 35 / 308.15 = 0.34 kWh. The real machine uses 1 kWh. + +## Scope + +- **Out of scope:** ground-source and water-source heat pumps — same cycle, warmer and steadier cold side, so higher COP. +- **Out of scope:** defrost cycles below about 5 °C, where frost on the evaporator is melted. +- **Out of scope:** cost and CO₂ comparisons with gas boilers — depend on electricity and gas prices and the grid mix. +- **Out of scope:** seasonal COP (SCOP) — the average COP over a heating season. +- **Out of scope:** refrigerant choice and safety — propane, R32, CO₂. +- **Out of scope:** why a boiling fluid and not a warmed gas — boiling takes much heat at one fixed temperature, so the coils can be small. +- **Connection:** a fridge and an air conditioner use the same loop, with the cold side inside. + +## Representation decision + +- **Stage:** 2 +- **Reason:** the mechanism is a closed loop of four parts with two heat exchanges; the reader must hold the loop, the temperature at each point, and the energy balance at once. A diagram shows the loop; a table shows how the lift changes the COP. +- **Levels:** L1 energy balance (where the 3 kWh comes from) / L2 the refrigerant loop / L3 why moving heat from cold to hot costs work, and how much / Scope. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/model.yaml new file mode 100644 index 0000000..eccb43d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/model.yaml @@ -0,0 +1,136 @@ +# Machine-readable half of model.md. `explainer check heat-pump` holds every rendering to it. + +values: + energy: # kWh, at 0 °C outside and 35 °C home water + electricity: 1 + outdoor_heat: 2 + heat_delivered: 3 + cop: + heat_pump: 3 + heater: 1 + temps: # °C + outdoor: 0 + water: 35 + room: 20 + boil: -10 + condense: 40 + discharge: 60 + warm_refrigerant_case: 5 + kelvin: + outdoor: 273.15 + water: 308.15 + condense: 313.15 + radiator_water: 328.15 + pressure_bar: + low: 3.5 + high: 14 + air: 1 + lift_k: + example: 35 + refrigerant: 50 + evaporator_gap: 10 + condenser_gap: 5 + ideal: + example: 8.8 + refrigerant: 6.3 + sweep: # outdoor °C, lift K, ideal COP, typical real COP; water at 35 °C + mild: {outdoor: 10, lift: 25, ideal: 12.3, real: 4} + freezing: {outdoor: 0, lift: 35, ideal: 8.8, real: 3} + cold: {outdoor: -15, lift: 50, ideal: 6.2, real: 2} + very_cold: {outdoor: -25, lift: 60, ideal: 5.1, real: 1.5} + radiators: {water: 55, lift: 55, ideal: 6.0, real: 2.2} + entropy: # kWh/K + taken: 0.00732 + given: 0.00974 + bad_cop: 12 + bad_outdoor_heat: 11 + bad_taken: 0.0403 + bad_given: 0.0389 + hot_to_cold_net: 0.00042 + kelvin_rounded: 273 + min_electricity: 0.34 + celsius_wrong: 1 + +allow: [2, 3, 4] + +require: + - path: energy/outdoor_heat + in: [diagram] + - path: energy/heat_delivered + in: [diagram] + - path: temps/boil + in: [diagram] + - path: temps/condense + in: [diagram] + - path: ideal/example + in: [diagram] + +claims: + - id: energy-balance + kind: guarantee + statement: "A heat pump does not create energy: heat delivered = electricity + outdoor heat, 3 kWh = 1 kWh + 2 kWh." + example: "0 °C outside, 35 °C water: 1 kWh electricity, 2 kWh outdoor heat, 3 kWh into the water, COP 3." + counterexample: "An electric heater has only electricity as input: 1 kWh in, 1 kWh heat out, COP 1." + ask: "Where do the other 2 kWh come from, if the heat pump uses only 1 kWh of electricity?" + + - id: why-compress + kind: why + statement: "The heat pump must spend work, because heat flows by itself only from hot to cold." + alternative: "Circulate a fluid between the outdoor air and the home, with no compressor." + counterexample: "Fluid warmed by 0 °C air is at most 0 °C; in a 20 °C home heat flows from the home into the fluid." + + - id: pressure-sets-boiling + kind: mechanism + statement: "Boiling temperature rises with pressure, so the loop runs at two pressures: low where it must be cold, high where it must be hot." + example: "Propane boils at −10 °C at about 3.5 bar and at 40 °C at about 14 bar." + alternative: "One fixed pressure gives one boiling temperature, which cannot be below 0 °C outdoors and above 35 °C indoors." + + - id: loop + kind: mechanism + statement: "Evaporator, compressor, condenser, expansion valve: the refrigerant takes heat in where it is colder than the air and gives heat out where it is hotter than the water." + example: "Boils at −10 °C, taking 2 kWh from 0 °C air; compressor adds 1 kWh; condenses at 40 °C, giving 3 kWh to 35 °C water; valve drops the pressure, a third of the liquid boils, the rest cools to −10 °C." + ask: "Why must the refrigerant be colder than the outdoor air in the evaporator, and hotter than the home water in the condenser?" + + - id: lift-sets-cop + kind: guarantee + statement: "The larger the temperature lift, the more electricity each kWh of heat costs, so COP falls." + example: "10 °C outside: lift 25 K, COP about 4." + counterexample: "−15 °C outside: lift 50 K, COP about 2." + ask: "What happens to the COP on a colder day, and why?" + + - id: ideal-limit + kind: limit + statement: "No heat pump can beat ideal COP = T_hot / (T_hot − T_cold), in kelvin, because entropy cannot be destroyed." + example: "COP 3 at 273.15 K → 308.15 K: entropy taken 0.00732 kWh/K, given 0.00974 kWh/K; allowed. Ideal 8.8." + counterexample: "Claimed COP 12: taken 0.0403 kWh/K, given 0.0389 kWh/K; entropy would be destroyed." + + - id: why-kelvin + kind: why + about: [ideal COP] + statement: "The ideal limit uses kelvin, because entropy is Q/T with T measured from absolute zero." + alternative: "Use Celsius." + counterexample: "35 / (35 − 0) = 1, which says a heat pump cannot beat a heater; kelvin gives 8.8." + +terms: + - term: outdoor heat + means: "the heat taken from the outdoor air, 2 kWh in the example" + avoid: [free energy, free heat] + - term: ideal limit + means: "the highest COP allowed for two temperatures; COP is not an efficiency" + avoid: [efficiency of 300, 300% efficient, 300 % efficient] + - term: temperature lift + means: "home water minus outdoor air; the refrigerant lift is a separate term" + avoid: [temperature difference of the refrigerant] + +budget: {diagram: 1100, reason: "three levels: energy balance, the loop with two pressures, and the ideal limit with a case that holds and one that breaks"} + +quiz: + - q: "A heat pump uses 1 kWh of electricity and delivers 3 kWh of heat. Does it break energy conservation? Where does the energy come from?" + expect: "No. 2 kWh comes from the outdoor air, which is colder than the home but still holds heat; the refrigerant is made colder than the air so heat flows into it. 1 + 2 = 3." + misconception: "It creates energy, or it is '300% efficient' in the sense of making more energy than it uses." + - q: "The same heat pump runs on a 10 °C day and on a −15 °C day, heating water to 35 °C. On which day is its COP higher, and why?" + expect: "On the 10 °C day (about 4 vs about 2): the temperature lift is smaller (25 K vs 50 K), so less work is needed per kWh moved." + misconception: "COP is a fixed number for the machine, or it is higher in the cold." + - q: "What do the compressor and the expansion valve each do to the refrigerant, and why does the loop need both?" + expect: "Compressor raises pressure, so the refrigerant condenses at a high temperature (40 °C) above the water; valve drops pressure, so it boils at a low temperature (−10 °C) below the air. Boiling temperature depends on pressure." + misconception: "The compressor makes the heat; or the refrigerant is cooled by the outdoor air." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/narrative.md new file mode 100644 index 0000000..7b87a3e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/narrative.md @@ -0,0 +1,60 @@ +# Narrative: heat-pump + +> Pass 3. `model.md` says what is true. This file says the path a first-time reader takes to it. + +## 1. Reader before and after + +- **Before:** a technical reader who knows energy conservation, kelvin, pressure, boiling and condensing, and that heat flows from hot to cold by itself. New to heat pumps, refrigerant loops, COP, and entropy. +- **After:** "A heat pump does not make heat, it moves it: 1 kWh of electricity carries 2 kWh of heat from the cold outdoor air into the home, so 3 kWh arrives. It does this by changing the refrigerant's pressure, which changes its boiling temperature, and the colder it is outside the more electricity each kWh costs." + +## 2. Question and motive + +- **Question:** How can a heat pump give 3 kWh of heat for 1 kWh of electricity, when energy cannot be created? Result in words: it does not create the 2 extra kWh; it moves them from the outdoor air into the home, and the electricity pays for the move. +- **Why care:** an electric heater gives 1 kWh of heat for 1 kWh. A ratio of 3 sounds like it breaks conservation of energy, so the answer must be about where the energy comes from. +- **Why this approach:** first balance the energy (it shows where the extra comes from), then ask how heat can flow from cold air to warm water (that question leads to the loop), then ask how much the move must cost (that leads to the ideal limit). + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| COP | heat delivered ÷ electricity used | heater 1 / 1 = 1; heat pump 3 / 1 = 3 | 1 | +| outdoor heat | heat taken from the outdoor air | 2 kWh from 0 °C air | 2 | +| refrigerant | fluid in the closed loop | propane | 4 | +| evaporator, compressor, condenser, expansion valve | the four parts of the loop | each with its temperature and pressure in the diagram | 5 | +| bar | pressure; the air around us is 1 bar | 3.5 bar and 14 bar | 4 | +| temperature lift | home water minus outdoor air | 35 − 0 = 35 K | 7 | +| ideal limit (Carnot) | highest COP physics allows for two temperatures | 308.15 / 35 = 8.8 | 8 | +| entropy | heat Q at kelvin temperature T carries Q/T; never destroyed | 2 / 273.15 = 0.00732 kWh/K | 9 | +| Q_hot, Q_cold, W | heat delivered, outdoor heat, electricity | 3, 2, 1 kWh | 9 | +| refrigerant lift | condensing minus boiling temperature | 40 − (−10) = 50 K | 10 | +| diagram colours | blue = cold side, red = hot side, yellow = electricity | the loop diagram | 5 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How can 1 kWh give 3 kWh? | — | heater vs heat pump numbers | "An electric heater turns 1 kWh of electricity into 1 kWh of heat. A heat pump gives about 3 kWh of heat for the same 1 kWh. Energy cannot be created, so where do the other 2 kWh come from? COP (heat delivered ÷ electricity used) is 1 for the heater and 3 for the heat pump." Answer up front: it moves heat, it does not make it. | the puzzle, the answer in one line, the word COP | +| 2 | notice | Is energy created? | so | L1 energy diagram: 1 + 2 = 3; caption explains colours (blue cold side, red hot side, yellow electricity) | "No. 2 kWh is outdoor heat, taken from the outdoor air. 0 °C air is 273 K above absolute zero and still holds heat. The 3 kWh goes into the home water: the 35 °C water that flows through the floor pipes or radiators." | the heat is moved, not made; the home water | +| 3 | objection | Heat flows only from hot to cold. How can it go from 0 °C air into 35 °C water? | but | — | "To carry heat from outside to inside, pipe a fluid between the two. With no other step, the fluid warms to at most 0 °C outside. Inside, the 35 °C water is hotter, so heat flows from the water into the fluid: the home gets colder. The fluid must be colder than the air outside and hotter than the water inside." | the problem the machine must solve | +| 4 | approach | How can one fluid be both? | therefore | — | "Change its pressure. Propane, the fluid used here, boils at −10 °C at 3.5 bar and at 40 °C at 14 bar (the air around us is 1 bar). We call this fluid the refrigerant." | the loop runs at two pressures | +| 5 | mechanism | How does the loop use this? | so | L2 loop diagram, same colours and the same kWh as L1 | the four parts in order: "Evaporator (outdoor coil): refrigerant at 3.5 bar boils at −10 °C, colder than the 0 °C air, so 2 kWh flows in. Compressor: an electric pump squeezes the vapour to 14 bar; the work heats it to about 60 °C, as a bicycle pump gets warm. Condenser (indoor coil): the vapour condenses at 40 °C, hotter than the 35 °C water, so 3 kWh flows out. Expansion valve: a narrow opening drops the pressure to 3.5 bar; about a third of the liquid boils at once, and that boiling cools the rest to −10 °C." | the full loop; the compressor raises the pressure, the valve lowers it | +| 6 | notice | Where does the electricity go? | and | same diagram, compressor | "The compressor's 1 kWh heats the vapour, so it reaches the water too: 1 + 2 = 3 kWh at the condenser." | 3 = 1 + 2 at the condenser | +| 7 | test (predict) | What happens on a colder day? | so | predict prompt: "Outside drops from 10 °C to −15 °C. Does the COP rise, fall, or stay the same?" Then the table. | "The temperature lift is home water minus outdoor air: 35 K at 0 °C. COP is about 4 at 10 °C (lift 25 K), about 3 at 0 °C, about 2 at −15 °C (lift 50 K). Old radiators need 55 °C water: lift 55 K at 0 °C, COP about 2.2." | larger lift → lower COP; why comes next | +| 8 | approach | Why does a larger lift cost more? Is there a limit? | so | formula with the instance first | "Physics sets an ideal limit (the Carnot limit): 308.15 / 35 = 8.8 for 35 °C water and 0 °C air. In symbols, ideal COP = T_hot / (T_hot − T_cold), where T_hot is the home water (308.15 K) and T_cold the outdoor air (273.15 K). Use kelvin: in Celsius 35 / (35 − 0) = 1, which would say a heat pump can never beat a heater. A larger lift makes the bottom bigger, so the limit falls." | the limit, its numbers, why kelvin | +| 9 | mechanism | Why that formula? | because | L3: rule, two cases, steps | "Heat Q that passes at kelvin temperature T carries entropy Q/T. Heat flowing by itself from hot to cold makes entropy: 1 kWh from 308.15 K to 273.15 K loses 1/308.15 and gains 1/273.15, net +0.00042 kWh/K. Nothing can destroy entropy; that is the same law that stops heat flowing from cold to hot by itself. The loop ends each cycle where it started, so it cannot store entropy. So the water must get at least as much entropy as the air loses. Name Q_hot = heat delivered, Q_cold = outdoor heat, W = electricity. COP 3: air loses 2/273.15 = 0.00732, water gets 3/308.15 = 0.00974. Allowed. COP 12 (11 kWh from air): air loses 0.0403, water gets 0.0389. Forbidden. Put Q_cold = Q_hot − W into Q_hot/T_hot ≥ Q_cold/T_cold and solve for Q_hot/W, step by step: you get the limit." | where the limit comes from | +| 10 | notice | Why is a real one 3, not 8.8? | but | — | "Heat flows through a coil only across a temperature gap. So the refrigerant boils 10 K below the air and condenses 5 K above the water. Its own lift, the refrigerant lift, is 40 − (−10) = 50 K, and 313.15 / 50 = 6.3. The compressor, fans, and pumps lose about half of the rest." | the gap's two causes | +| 11 | close | — | so | — | "A heat pump does not make 3 kWh from 1 kWh. It moves 2 kWh of outdoor heat into the home, and the 1 kWh of electricity pays for the move and ends up as heat too. Pressure sets the boiling temperature, so the refrigerant is colder than the air outside and hotter than the water inside, and heat flows the normal way at both ends. The colder the day, the more each kWh costs." Connection: a fridge is the same loop, with the cold coil inside the fridge. Not covered here: a short list. | the after-sentence | + +## 5. Concrete to symbol + +COP: 3 kWh / 1 kWh = 3, then the word. Ideal limit: 308.15 / 35 = 8.8 first, then T_hot / (T_hot − T_cold), with "here T_hot is 308.15 K, the 35 °C water". Entropy: 2 kWh at 273.15 K → 0.00732 kWh/K, then Q/T. + +## 6. Links between representations + +The kWh numbers in the L1 diagram are the same numbers on the evaporator, compressor, and condenser in the L2 diagram. Blue nodes are the cold side, red the hot side, yellow the electricity, in both diagrams; the text says so under the first. + +## 7. Close the loop + +- **Answer:** the heat pump does not make 3 kWh from 1 kWh. It moves 2 kWh of outdoor heat into the home, and the 1 kWh of electricity both pays for the move and ends up as heat. +- **General argument:** changing the pressure changes the boiling temperature, so the refrigerant is colder than the air outside and hotter than the water inside; heat flows the normal way at both ends. The cost per kWh grows with the temperature lift. +- **Payoff:** the reader can predict the COP falls in cold weather and with hot radiators. Connection: a fridge is the same loop. Scope list. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/decisions.yaml new file mode 100644 index 0000000..241bc31 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/decisions.yaml @@ -0,0 +1,485 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T150833-probe#1: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#10: + decision: declined + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#11: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#12: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#13: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#14: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#15: + decision: declined + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#2: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#3: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#4: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#5: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#6: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#7: + decision: declined + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#8: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#9: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T151129-coldread-narrative#1: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#10: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#11: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#12: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#13: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#14: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#15: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#16: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#17: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#18: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#19: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#2: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#20: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#21: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#22: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#23: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#24: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#25: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#26: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#27: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#28: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#29: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#3: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#30: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#31: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#32: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#33: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#34: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#35: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#36: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#37: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#38: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#39: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#4: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#40: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#41: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#42: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#43: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#44: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#45: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#46: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#47: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#5: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#6: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#7: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#8: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#9: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151329-coldread-diagram#1: + decision: adopted + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151329-coldread-diagram#2: + decision: declined + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151329-coldread-diagram#3: + decision: adopted + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151329-coldread-diagram#4: + decision: adopted + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151329-coldread-diagram#5: + decision: adopted + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151455-quiz-diagram#1: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#10: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#11: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#12: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#13: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#14: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#15: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#16: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#17: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#18: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#19: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#2: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#20: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#21: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#22: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#23: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#24: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#25: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#3: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#4: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#5: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#6: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#7: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#8: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#9: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151658-quiz-diagram#1: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#10: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#11: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#12: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#13: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#14: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#15: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#16: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#17: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#18: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#19: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#2: + decision: adopted + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#20: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#21: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#22: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#23: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#24: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#3: + decision: adopted + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#4: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#5: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#6: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#7: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#8: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#9: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T150833-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T150833-probe.json new file mode 100644 index 0000000..ddc4656 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T150833-probe.json @@ -0,0 +1,218 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "heat-pump", + "at": "2026-10-04T15:08:33", + "model": "default", + "prompt_sha": "7350959dd026", + "reply": { + "gaps": [ + { + "rule": 3, + "severity": "main", + "about": "'Entropy' is used in L3 (Q/T, 'cannot store entropy', 'cannot be destroyed') but is not in the Terms section, and the Audience section does not list it as prior knowledge.", + "why_it_matters": "L3 is the level that explains why moving heat uphill costs work. Its whole argument depends on a word this reader has never been given a meaning for, so they can't check any step of it.", + "suggested_addition": "Add to Terms: 'Entropy (here only): heat Q passing at absolute temperature T carries Q/T of entropy. Taking 2 kWh from 273.15 K air removes 2/273.15 = 0.00732 kWh/K. Giving 3 kWh to 308.15 K water adds 3/308.15 = 0.00974 kWh/K. Heat flowing by itself from hot to cold always increases the total.' Give one example of that last fact: 1 kWh moving from 308 K to 273 K gives −0.00325 + 0.00366 = +0.00042 kWh/K." + }, + { + "rule": 2, + "severity": "main", + "about": "L3 says entropy given to the hot side ≥ entropy taken from the cold side, with no numbers showing it holding or failing. The algebra from that inequality to W ≥ Q_hot(T_hot−T_cold)/T_hot is also left out, and so is the step from that bound to COP_max = T_hot/(T_hot−T_cold).", + "why_it_matters": "The ideal limit (8.8) is the only bound in the explanation. Without a case that violates it, the reader can't see what the inequality forbids, and the formula in Quantities seems to come from nowhere.", + "suggested_addition": "Holds: COP 3 at 0 °C → 35 °C gives 2/273.15 = 0.00732 taken and 3/308.15 = 0.00974 given; 0.00974 ≥ 0.00732. Breaks: a claimed COP 12 (11 kWh from air, 1 kWh electricity, 12 kWh out) gives 11/273.15 = 0.0403 taken but only 12/308.15 = 0.0389 given, so entropy would be destroyed, which is forbidden. Show the algebra: substitute Q_cold = Q_hot − W, so Q_hot/T_hot ≥ (Q_hot − W)/T_cold, so W ≥ Q_hot(1 − T_cold/T_hot). Then Q_hot/W ≤ T_hot/(T_hot−T_cold) = 308.15/35 = 8.8." + }, + { + "rule": 4, + "severity": "main", + "about": "The expansion-valve step says the pressure drop 'makes it cold' but never says how. The model also calls the output 'cold low-pressure liquid', when it is really a mixture of liquid and vapour.", + "why_it_matters": "The reader is told that boiling temperature depends on pressure but is never shown it being used here. 40 °C liquid at 3.5 bar is far above its −10 °C boiling point, so part of it boils at once and that boiling cools the rest. Without this, getting from 40 °C to −10 °C with no energy removed looks like magic.", + "suggested_addition": "'At 3.5 bar propane boils at −10 °C, so the 40 °C liquid boils instantly. Roughly a third of it turns to vapour (estimate; recompute from propane tables), and the heat that boiling uses comes from the liquid itself, cooling everything to −10 °C. The valve adds or removes 0 kWh; it only drops the pressure.'" + }, + { + "rule": 1, + "severity": "main", + "about": "Why squeezing vapour makes it hot. The model states this but never explains it, and offers no comparison.", + "why_it_matters": "The compressor is the one place where electricity enters. Without a reason, 'pressure rises → temperature rises' is just a rule to memorise, and the reader can't tell why the work 'ends up as heat in the vapour'.", + "suggested_addition": "'Pushing the piston or scroll does work on the gas molecules and speeds them up, which is a higher temperature (as when a bicycle pump gets warm). Vapour enters at about −10 °C and 3.5 bar and leaves at about 14 bar and roughly 60 °C (estimate). It first cools to 40 °C in the condenser, then condenses.' Also say that this means the condenser receives vapour above 40 °C." + }, + { + "rule": 3, + "severity": "main", + "about": "'Temperature lift' names two different things: water minus air (35 K, used in the table and the ideal COP) and refrigerant minus refrigerant (50 K, used in the 'why real is a third' line). T_hot is likewise 308.15 K in one place and 313.15 K in another.", + "why_it_matters": "The reader will see that the lift is 35 K and also 50 K for the same situation. The two ideal COPs (8.8 and 6.3) will look like a contradiction.", + "suggested_addition": "Name them separately. 'Lift' = water − air = 35 K, ideal 8.8. 'Refrigerant lift' = condensing − boiling = 40 − (−10) = 50 K, ideal 313.15/50 = 6.3. State that the extra 15 K (10 K at the evaporator + 5 K at the condenser) is what heat exchangers need to make heat flow." + }, + { + "rule": 1, + "severity": "edge", + "about": "Why run the refrigerant at −10 °C and 40 °C instead of right at 0 °C and 35 °C. The cost is stated (refrigerant lift 50 K instead of 35 K) but not the reason.", + "why_it_matters": "It explains part of why real is below ideal, and it explains a trade-off: a bigger heat exchanger means a smaller temperature gap, which means a higher COP.", + "suggested_addition": "'Heat flow through an exchanger is roughly proportional to the temperature difference. At 0 K difference nothing flows. Halving the evaporator gap from 10 K to 5 K needs about twice the coil area but raises the ideal from 313.15/50 = 6.3 to 313.15/45 = 7.0.'" + }, + { + "rule": 1, + "severity": "edge", + "about": "Why use a fluid that boils and condenses rather than just heating and cooling a gas.", + "why_it_matters": "Boiling takes a lot of heat at one fixed temperature. That is why the temperatures at each exchanger are fixed at single values. A gas-only cycle would change temperature along the coil.", + "suggested_addition": "'Propane takes about 390 kJ/kg to boil at −10 °C (estimate), so 2 kWh = 7200 kJ needs about 18 kg to circulate, all at −10 °C. Warming propane gas by 10 K takes only about 17 kJ/kg (cp ≈ 1.7 kJ/kg·K), so it would need about 20 times the flow.'" + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts have more than one name: 'ideal limit' / 'Carnot' / 'COP_max' / 'Ideal COP'; 'outdoor heat' / 'heat taken from outside' / 'heat from the air' / Q_cold; 'electricity' / 'work' / W; 'heat delivered' / Q_hot / 'heat out'; 'typical real COP' / 'COP'.", + "why_it_matters": "In L3 the reader has to work out that W is the electricity and Q_cold is the outdoor heat, and that 'Carnot' is the same thing as 'ideal limit'.", + "suggested_addition": "Choose one name for each and map the symbols once: 'Q_hot = heat delivered (3 kWh), Q_cold = outdoor heat (2 kWh), W = electricity (1 kWh), COP_max = ideal limit (Carnot).'" + }, + { + "rule": 5, + "severity": "edge", + "about": "Pressures in bar (3.5 and 14) are given with no reference point.", + "why_it_matters": "The reader can't tell whether 14 bar is mild or extreme, which matters for why the parts are sealed and strong.", + "suggested_addition": "'Air around you is about 1 bar; a car tyre is about 3.2 bar absolute; the condenser side at 14 bar is about 4 times tyre pressure.'" + }, + { + "rule": 5, + "severity": "edge", + "about": "The ratio of real to ideal (3/8.8 ≈ 0.34) is mentioned but never shown as a measure with low and high values.", + "why_it_matters": "It tells the reader how close a machine gets to physics' limit, and it separates poor engineering from a hard lift.", + "suggested_addition": "'Fraction of ideal: 3/8.8 = 0.34 at 0 °C. At 10 °C: 4/12.3 = 0.33. At −15 °C: 2/6.2 = 0.32. Roughly constant, so the lift, not the machine, causes most of the drop.'" + }, + { + "rule": 4, + "severity": "edge", + "about": "Real COP at outdoor temperatures where it gets close to 1 is missing. The table stops at −15 °C.", + "why_it_matters": "A learner will ask whether it ever becomes worse than a heater. This is the natural place where the claim 'COP above 1' would break.", + "suggested_addition": "Add a row: −25 °C outside, lift 60 K, ideal 308.15/60 = 5.1, typical real about 1.5 (estimate; recompute). State that the real COP stays above 1 well beyond typical winter temperatures, but backup resistive heaters (COP 1) sometimes take over." + } + ], + "excess": [ + "Why-this-form row about Celsius: the second example, '−15 °C outside and 0 °C water gives 0/15 = 0, kelvin 18.2', uses an unrealistic 0 °C water temperature and adds nothing beyond the first example (35/35 = 1).", + "Refrigerant product names (R290, R32, R744) in Entities and Scope; 'propane' alone is enough.", + "Alternative state 'Reverse the loop (fridge, air conditioner)'; it isn't needed to answer the central question.", + "Representation decision section (stage, levels); this is planning metadata, not content for the reader." + ], + "next_questions": [ + { + "q": "Why does squeezing the vapour make it hot, and why does going through the valve make the liquid cold without removing energy?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "What is entropy, and why does heat carry Q/T of it?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How cold can it get outside before the heat pump is no better than an electric heater (COP 1)?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Is it actually cheaper or lower in CO₂ than a gas boiler?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Why is the real COP (3) so far below the ideal limit (8.8), and could better engineering close the gap?", + "answered_by_model": true, + "declared_out_of_scope": false + } + ] + }, + "usage": { + "cost": 0.1754818, + "seconds": 61 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 3", + "text": "'Entropy' is used in L3 (Q/T, 'cannot store entropy', 'cannot be destroyed') but is not in the Terms section, and the Audience section does not list it as prior knowledge. → Add to Terms: 'Entropy (here only): heat Q passing at absolute temperature T carries Q/T of entropy. Taking 2 kWh from 273.15 K air removes 2/273.15 = 0.00732 kWh/K. Giving 3 kWh to 308.15 K water adds 3/308.15 = 0.00974 kWh/K. Heat flowing by itself from hot to cold always increases the total.' Give one example of that last fact: 1 kWh moving from 308 K to 273 K gives −0.00325 + 0.00366 = +0.00042 kWh/K." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "L3 says entropy given to the hot side ≥ entropy taken from the cold side, with no numbers showing it holding or failing. The algebra from that inequality to W ≥ Q_hot(T_hot−T_cold)/T_hot is also left out, and so is the step from that bound to COP_max = T_hot/(T_hot−T_cold). → Holds: COP 3 at 0 °C → 35 °C gives 2/273.15 = 0.00732 taken and 3/308.15 = 0.00974 given; 0.00974 ≥ 0.00732. Breaks: a claimed COP 12 (11 kWh from air, 1 kWh electricity, 12 kWh out) gives 11/273.15 = 0.0403 taken but only 12/308.15 = 0.0389 given, so entropy would be destroyed, which is forbidden. Show the algebra: substitute Q_cold = Q_hot − W, so Q_hot/T_hot ≥ (Q_hot − W)/T_cold, so W ≥ Q_hot(1 − T_cold/T_hot). Then Q_hot/W ≤ T_hot/(T_hot−T_cold) = 308.15/35 = 8.8." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 4", + "text": "The expansion-valve step says the pressure drop 'makes it cold' but never says how. The model also calls the output 'cold low-pressure liquid', when it is really a mixture of liquid and vapour. → 'At 3.5 bar propane boils at −10 °C, so the 40 °C liquid boils instantly. Roughly a third of it turns to vapour (estimate; recompute from propane tables), and the heat that boiling uses comes from the liquid itself, cooling everything to −10 °C. The valve adds or removes 0 kWh; it only drops the pressure.'" + }, + { + "n": 4, + "severity": "main", + "kind": "rule 1", + "text": "Why squeezing vapour makes it hot. The model states this but never explains it, and offers no comparison. → 'Pushing the piston or scroll does work on the gas molecules and speeds them up, which is a higher temperature (as when a bicycle pump gets warm). Vapour enters at about −10 °C and 3.5 bar and leaves at about 14 bar and roughly 60 °C (estimate). It first cools to 40 °C in the condenser, then condenses.' Also say that this means the condenser receives vapour above 40 °C." + }, + { + "n": 5, + "severity": "main", + "kind": "rule 3", + "text": "'Temperature lift' names two different things: water minus air (35 K, used in the table and the ideal COP) and refrigerant minus refrigerant (50 K, used in the 'why real is a third' line). T_hot is likewise 308.15 K in one place and 313.15 K in another. → Name them separately. 'Lift' = water − air = 35 K, ideal 8.8. 'Refrigerant lift' = condensing − boiling = 40 − (−10) = 50 K, ideal 313.15/50 = 6.3. State that the extra 15 K (10 K at the evaporator + 5 K at the condenser) is what heat exchangers need to make heat flow." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 1", + "text": "Why run the refrigerant at −10 °C and 40 °C instead of right at 0 °C and 35 °C. The cost is stated (refrigerant lift 50 K instead of 35 K) but not the reason. → 'Heat flow through an exchanger is roughly proportional to the temperature difference. At 0 K difference nothing flows. Halving the evaporator gap from 10 K to 5 K needs about twice the coil area but raises the ideal from 313.15/50 = 6.3 to 313.15/45 = 7.0.'" + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 1", + "text": "Why use a fluid that boils and condenses rather than just heating and cooling a gas. → 'Propane takes about 390 kJ/kg to boil at −10 °C (estimate), so 2 kWh = 7200 kJ needs about 18 kg to circulate, all at −10 °C. Warming propane gas by 10 K takes only about 17 kJ/kg (cp ≈ 1.7 kJ/kg·K), so it would need about 20 times the flow.'" + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts have more than one name: 'ideal limit' / 'Carnot' / 'COP_max' / 'Ideal COP'; 'outdoor heat' / 'heat taken from outside' / 'heat from the air' / Q_cold; 'electricity' / 'work' / W; 'heat delivered' / Q_hot / 'heat out'; 'typical real COP' / 'COP'. → Choose one name for each and map the symbols once: 'Q_hot = heat delivered (3 kWh), Q_cold = outdoor heat (2 kWh), W = electricity (1 kWh), COP_max = ideal limit (Carnot).'" + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 5", + "text": "Pressures in bar (3.5 and 14) are given with no reference point. → 'Air around you is about 1 bar; a car tyre is about 3.2 bar absolute; the condenser side at 14 bar is about 4 times tyre pressure.'" + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 5", + "text": "The ratio of real to ideal (3/8.8 ≈ 0.34) is mentioned but never shown as a measure with low and high values. → 'Fraction of ideal: 3/8.8 = 0.34 at 0 °C. At 10 °C: 4/12.3 = 0.33. At −15 °C: 2/6.2 = 0.32. Roughly constant, so the lift, not the machine, causes most of the drop.'" + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 4", + "text": "Real COP at outdoor temperatures where it gets close to 1 is missing. The table stops at −15 °C. → Add a row: −25 °C outside, lift 60 K, ideal 308.15/60 = 5.1, typical real about 1.5 (estimate; recompute). State that the real COP stays above 1 well beyond typical winter temperatures, but backup resistive heaters (COP 1) sometimes take over." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "Why-this-form row about Celsius: the second example, '−15 °C outside and 0 °C water gives 0/15 = 0, kelvin 18.2', uses an unrealistic 0 °C water temperature and adds nothing beyond the first example (35/35 = 1)." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Refrigerant product names (R290, R32, R744) in Entities and Scope; 'propane' alone is enough." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Alternative state 'Reverse the loop (fridge, air conditioner)'; it isn't needed to answer the central question." + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Representation decision section (stage, levels); this is planning metadata, not content for the reader." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151129-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151129-coldread-narrative.json new file mode 100644 index 0000000..16bb54c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151129-coldread-narrative.json @@ -0,0 +1,493 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "heat-pump", + "at": "2026-10-04T15:11:29", + "model": "default", + "prompt_sha": "c1be6734e6da", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: heater vs heat pump numbers. Said: define COP; heater 1, heat pump 3", + "unresolved": [], + "unsaid": [], + "leap": [ + "The words never say why a COP of 3 is puzzling. Nothing links 'heat pump 3' to conservation of energy, so the puzzle exists only in the 'Reader's question' column, which the reader does not get." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (notice)", + "text": "Shown: L1 energy diagram: 1 + 2 = 3. Said: the 2 kWh is outdoor heat; 0 °C air is 273 K and still holds heat", + "unresolved": [ + "L1" + ], + "unsaid": [ + "The L1 diagram's colour coding. The plan says the text explains blue/red/yellow under the first diagram, but this beat's 'Said' does not.", + "The question 'is energy created?' is never asked aloud. The answer arrives before the question." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "Shown: —. Said: a plain fluid loop with no compressor reaches at most 0 °C and would cool a 20 °C home", + "unresolved": [ + "compressor", + "fluid loop" + ], + "unsaid": [], + "leap": [ + "The text brings in a 'fluid loop' with no reason given for why heat would be moved by a loop of fluid at all.", + "The objection itself, that heat flows hot to cold, so how can it go from 0 °C air indoors, is not in 'Said'. Only the failed design is.", + "There is a jump from '20 °C home' here to '35 °C water' later. The heating water is never introduced." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (approach)", + "text": "Shown: —. Said: boiling temperature rises with pressure; propane boils at −10 °C at 3.5 bar, at 40 °C at 14 bar", + "unresolved": [ + "propane (its role as the fluid in the loop is not said; the word 'refrigerant' never appears in 'Said')" + ], + "unsaid": [], + "leap": [ + "It is not said how these two boiling points help. The tie to 'colder than 0 °C air, hotter than the water' waits until beat 5." + ], + "excess": [ + "'boiling temperature rises with pressure': the reader already knows boiling and pressure. This is useful as a reminder but borderline." + ], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: L2 loop diagram. Said: four steps, each with temperatures and kWh; why −10 °C is colder than the air and 40 °C hotter than the water", + "unresolved": [ + "the names of the four parts (evaporator, compressor, condenser, expansion valve) if they are labels in the diagram; 'Said' does not name or define them", + "refrigerant", + "the water" + ], + "unsaid": [ + "The blue/red/yellow colour coding of the diagram nodes is never explained in 'Said'.", + "The diagram's kWh numbers are the same as in L1 (the plan's link), but this is not said." + ], + "leap": [ + "How the pressure is raised and lowered (by the compressor and the expansion valve) is not stated." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (notice)", + "text": "Shown: same diagram, compressor. Said: the compressor's 1 kWh ends up in the water too", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (test/predict)", + "text": "Shown: predict prompt, then table. Said: define temperature lift; COP about 4 / 3 / 2 at 10 / 0 / −15 °C; radiators at 55 °C", + "unresolved": [], + "unsaid": [ + "The contents of the predict prompt are not stated." + ], + "leap": [ + "'radiators at 55 °C' is dropped in with no stated COP and no reason why hotter radiators matter.", + "Why a larger lift lowers COP is not given here. The reason comes in beats 8–9." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (approach)", + "text": "Shown: formula. Said: ideal limit T_hot / (T_hot − T_cold) in kelvin; 8.8; Celsius gives 1", + "unresolved": [ + "T_hot", + "T_cold" + ], + "unsaid": [ + "The concrete 308.15 / 35 is not in 'Said'. Only '8.8' is, so the reader cannot check where 8.8 comes from." + ], + "leap": [ + "The formula is asserted with no reason. The reason is deferred to beat 9.", + "'Celsius gives 1' is too terse to follow: the reader is not shown 35/35, and the beat does not say why Celsius is wrong." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (mechanism)", + "text": "Shown: L3 inequality, two cases. Said: entropy Q/T is never destroyed; COP 3 passes, COP 12 fails; algebra", + "unresolved": [ + "entropy", + "Q", + "Q/T", + "L3", + "Q_hot, Q_cold, W (presumably in the inequality, never defined in 'Said')", + "COP 12 (where 12 comes from)" + ], + "unsaid": [ + "The L3 inequality itself is not spoken. 'algebra' is all the reader gets." + ], + "leap": [ + "'never destroyed' is stated as a rule. It is not tied to what the reader already knows, that heat flows hot to cold by itself.", + "How the inequality turns into T_hot / (T_hot − T_cold) is compressed into 'algebra'." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 10 (notice)", + "text": "Shown: —. Said: refrigerant lift 50 K (ideal 6.3) plus compressor losses", + "unresolved": [ + "refrigerant lift", + "refrigerant" + ], + "unsaid": [], + "leap": [ + "'ideal 6.3' is not derived in 'Said' (313.15/50). The reader has to work out that it uses 40 °C and −10 °C instead of 35 °C and 0 °C." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 11 (close)", + "text": "Shown: —. Said: answer in the opening words; connection to the fridge; scope", + "unresolved": [ + "scope" + ], + "unsaid": [], + "leap": [ + "The fridge connection is named but not explained." + ], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 9: 'entropy Q/T is never destroyed' is given as an unexplained rule to a reader new to entropy, and the step to the formula is only 'algebra'. The ideal limit therefore rests on nothing the reader can follow.", + "Beat 9: the symbols Q_hot, Q_cold, W and the 'L3 inequality' are never defined or spoken.", + "Beat 11: 'answer in the opening words' is a placeholder. The answer and the general argument are never actually said.", + "Beat 5: the four parts and the word 'refrigerant' are never named or defined in 'Said', yet beats 6 and 10 depend on 'compressor' and 'refrigerant lift'.", + "Beat 1/2: the puzzle (3 out for 1 in seems to break conservation of energy) is never said, so the reader is not told what question is being answered." + ], + "cuts": [ + "The terse 'Celsius gives 1' in beat 8: either spell it out (35/35) or cut it. As it stands it costs a reread and gains nothing.", + "'radiators at 55 °C' in beat 7: it is not used later in the beats, so either give its COP or drop it.", + "Beat 4's restatement that boiling temperature rises with pressure: the reader knows this, so lead with the propane numbers." + ], + "question": { + "text": "How can a heat pump deliver 3 kWh of heat for 1 kWh of electricity without creating energy?", + "known_at": "Only implicitly, at beat 1 (heater 1 vs heat pump 3) and beat 2. The question is never put in words in 'Said'.", + "result_stated": "Partly, at beat 2: 'the 2 kWh is outdoor heat'. The full answer is promised at beat 11 ('answer in the opening words') but never written." + }, + "motive": "Reason to care: missing in 'Said'. The heater-vs-heat-pump comparison in beat 1 hints at it, but conservation of energy is never mentioned. Reason for the approach: missing. The sequence (energy balance, then the uphill problem, then the cost limit) is never announced.", + "close": "Not achieved in the beats. Beat 11 'Said' reads 'answer in the opening words; connection to the fridge; scope', which is a placeholder, and the general argument about pressure and boiling temperature is not restated.", + "recap": "A heat pump doesn't make heat, it moves it: squeezing and releasing a refrigerant makes it boil colder than the outdoor air and condense hotter than the indoor water, so 1 kWh of electricity carries 2 kWh in, and the bigger the temperature gap the more each kWh costs.", + "strongest": "Beats 3–4: the failed no-compressor loop ('reaches at most 0 °C and would cool a 20 °C home') followed by propane's two boiling points at 3.5 and 14 bar. Together they make the trick feel necessary and concrete.", + "weakest": "Beat 9: entropy, Q/T, an unspoken inequality, undefined symbols and 'algebra', all in one beat for a reader new to entropy.", + "suggestions": [ + "In beat 1, say explicitly that energy cannot be created, so a COP of 3 looks impossible.", + "In beat 2 or 5, say the colour coding aloud (blue = cold side, red = hot side, yellow = electricity).", + "In beat 5, name the four parts and the word 'refrigerant' as each appears, and say the compressor raises the pressure and the valve lowers it.", + "Introduce the heating water (at 35 °C) before beat 3 relies on it.", + "In beat 9, give 2 / 273.15 = 0.00732 kWh/K before the symbol Q/T, define Q_hot, Q_cold and W, and link 'never destroyed' to heat flowing hot to cold by itself.", + "In beat 8, show 308.15 / 35 = 8.8 before the formula.", + "In beat 11, write out the answer sentence and the general argument." + ] + }, + "usage": { + "cost": 0.1379862, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 9: 'entropy Q/T is never destroyed' is given as an unexplained rule to a reader new to entropy, and the step to the formula is only 'algebra'. The ideal limit therefore rests on nothing the reader can follow." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 9: the symbols Q_hot, Q_cold, W and the 'L3 inequality' are never defined or spoken." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 11: 'answer in the opening words' is a placeholder. The answer and the general argument are never actually said." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: the four parts and the word 'refrigerant' are never named or defined in 'Said', yet beats 6 and 10 depend on 'compressor' and 'refrigerant lift'." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 1/2: the puzzle (3 out for 1 in seems to break conservation of energy) is never said, so the reader is not told what question is being answered." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "The terse 'Celsius gives 1' in beat 8: either spell it out (35/35) or cut it. As it stands it costs a reread and gains nothing." + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "'radiators at 55 °C' in beat 7: it is not used later in the beats, so either give its COP or drop it." + }, + { + "n": 8, + "severity": "excess", + "kind": "cut", + "text": "Beat 4's restatement that boiling temperature rises with pressure: the reader knows this, so lead with the propane numbers." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): The words never say why a COP of 3 is puzzling. Nothing links 'heat pump 3' to conservation of energy, so the puzzle exists only in the 'Reader's question' column, which the reader does not get." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (notice): L1" + }, + { + "n": 11, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (notice): The L1 diagram's colour coding. The plan says the text explains blue/red/yellow under the first diagram, but this beat's 'Said' does not." + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (notice): The question 'is energy created?' is never asked aloud. The answer arrives before the question." + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): compressor" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): fluid loop" + }, + { + "n": 15, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): The text brings in a 'fluid loop' with no reason given for why heat would be moved by a loop of fluid at all." + }, + { + "n": 16, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): The objection itself, that heat flows hot to cold, so how can it go from 0 °C air indoors, is not in 'Said'. Only the failed design is." + }, + { + "n": 17, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): There is a jump from '20 °C home' here to '35 °C water' later. The heating water is never introduced." + }, + { + "n": 18, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (approach): propane (its role as the fluid in the loop is not said; the word 'refrigerant' never appears in 'Said')" + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (approach): It is not said how these two boiling points help. The tie to 'colder than 0 °C air, hotter than the water' waits until beat 5." + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): the names of the four parts (evaporator, compressor, condenser, expansion valve) if they are labels in the diagram; 'Said' does not name or define them" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): refrigerant" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): the water" + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The blue/red/yellow colour coding of the diagram nodes is never explained in 'Said'." + }, + { + "n": 24, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The diagram's kWh numbers are the same as in L1 (the plan's link), but this is not said." + }, + { + "n": 25, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): How the pressure is raised and lowered (by the compressor and the expansion valve) is not stated." + }, + { + "n": 26, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (test/predict): The contents of the predict prompt are not stated." + }, + { + "n": 27, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test/predict): 'radiators at 55 °C' is dropped in with no stated COP and no reason why hotter radiators matter." + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test/predict): Why a larger lift lowers COP is not given here. The reason comes in beats 8–9." + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (approach): T_hot" + }, + { + "n": 30, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (approach): T_cold" + }, + { + "n": 31, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (approach): The concrete 308.15 / 35 is not in 'Said'. Only '8.8' is, so the reader cannot check where 8.8 comes from." + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (approach): The formula is asserted with no reason. The reason is deferred to beat 9." + }, + { + "n": 33, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (approach): 'Celsius gives 1' is too terse to follow: the reader is not shown 35/35, and the beat does not say why Celsius is wrong." + }, + { + "n": 34, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): entropy" + }, + { + "n": 35, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): Q" + }, + { + "n": 36, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): Q/T" + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): L3" + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): Q_hot, Q_cold, W (presumably in the inequality, never defined in 'Said')" + }, + { + "n": 39, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): COP 12 (where 12 comes from)" + }, + { + "n": 40, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 9 (mechanism): The L3 inequality itself is not spoken. 'algebra' is all the reader gets." + }, + { + "n": 41, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (mechanism): 'never destroyed' is stated as a rule. It is not tied to what the reader already knows, that heat flows hot to cold by itself." + }, + { + "n": 42, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (mechanism): How the inequality turns into T_hot / (T_hot − T_cold) is compressed into 'algebra'." + }, + { + "n": 43, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 10 (notice): refrigerant lift" + }, + { + "n": 44, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 10 (notice): refrigerant" + }, + { + "n": 45, + "severity": "edge", + "kind": "leap", + "text": "Beat 10 (notice): 'ideal 6.3' is not derived in 'Said' (313.15/50). The reader has to work out that it uses 40 °C and −10 °C instead of 35 °C and 0 °C." + }, + { + "n": 46, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 11 (close): scope" + }, + { + "n": 47, + "severity": "edge", + "kind": "leap", + "text": "Beat 11 (close): The fridge connection is named but not explained." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151329-coldread-diagram.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151329-coldread-diagram.json new file mode 100644 index 0000000..e2e1650 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151329-coldread-diagram.json @@ -0,0 +1,95 @@ +{ + "tool": "coldread", + "rendering": "diagram", + "slug": "heat-pump", + "at": "2026-10-04T15:13:29", + "model": "default", + "prompt_sha": "b7a65e2b208a", + "reply": { + "diagrams": [ + { + "at": "Diagram 1 (Level 1 — Where the 3 kWh comes from)", + "text": "Three boxes feed into or out of a 'Heat pump' box. 'Electricity 1 kWh' (yellow) and 'Outdoor air at 0 °C / outdoor heat: 2 kWh' (blue) go in. 'Home water at 35 °C / heat delivered: 3 kWh' (red) comes out. A caption below explains the colours. It answers the opening puzzle: where do the extra 2 kWh come from?", + "unresolved": [], + "leap": [], + "excess": [ + "\"Energy is conserved: 1 kWh of electricity + 2 kWh of outdoor heat = 3 kWh of heat delivered.\" The diagram has just shown this sum, and the reader knows energy conservation.", + "\"Air at 0 °C is 273 K above absolute zero, so it still holds heat.\" This reader knows kelvin. It is brief and heads off a 'how can cold air give heat?' doubt, so it costs little." + ], + "purpose_unclear": false + }, + { + "at": "Diagram 2 (Level 2 — the refrigerant loop)", + "text": "A four-part loop split into a 'Low pressure, 3.5 bar' group (1. Evaporator, outdoor coil, boils at −10 °C; 4. Expansion valve, pressure drops, cools to −10 °C) and a 'High pressure, 14 bar' group (2. Compressor, vapour squeezed, heats to about 60 °C; 3. Condenser, indoor coil, condenses at 40 °C). Arrows: cold vapour → hot vapour → warm liquid → cold liquid and vapour. Outdoor air sends 2 kWh in, electricity sends 1 kWh to the compressor, and the condenser sends 3 kWh to the home water.", + "unresolved": [], + "leap": [ + "The compressor label says the vapour heats \"to about 60 °C\", but the next box says it \"condenses at 40 °C\". Neither the diagram nor step 3 says that the hot vapour first cools from 60 °C to 40 °C, giving up heat to the water, before it condenses. A careful reader stops to ask why the temperature drops by 20 K.", + "The compressor is drawn inside the \"High pressure, 14 bar\" group, yet it takes in 3.5 bar vapour. It is really the boundary between the two pressures, so the grouping slightly misleads." + ], + "excess": [ + "\"Heat flows by itself only from hot to cold.\" This reader already knows it, though it sets up the argument.", + "\"(The air around us is at 1 bar.)\" This reader knows pressure. It is a small aside." + ], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Level 1 prose: \"Energy is conserved: 1 kWh of electricity + 2 kWh of outdoor heat = 3 kWh of heat delivered.\" It repeats the diagram.", + "Level 2: \"(The air around us is at 1 bar.)\"", + "Level 2 opening: \"Heat flows by itself only from hot to cold.\" The reader knows this; keep only the thought experiment that follows it." + ], + "question": { + "text": "How can a heat pump deliver about 3 kWh of heat for 1 kWh of electricity without breaking energy conservation? A follow-on question: what limits that ratio?", + "known_at": "The title and the first paragraph: \"Energy cannot be created, so where do the other 2 kWh come from?\"", + "result_stated": "Yes, in words, at once: \"**Answer:** the heat pump does not make that heat. It moves heat from the cold outdoor air into the home. The electricity pays for the move.\"" + }, + "motive": "The reason to care is the apparent paradox: \"An electric heater turns 1 kWh of electricity into 1 kWh of heat. A heat pump gives about 3 kWh of heat for the same 1 kWh. Energy cannot be created...\" The reason for the approach is given in Level 2: piping a fluid with no other step would make \"the home gets colder\", so \"the fluid must be colder than the air outside and hotter than the water inside. One fluid can be both, because its boiling temperature depends on its pressure.\" Level 3 is motivated by the \"Predict first\" question about colder days.", + "close": "Yes. The Summary says: \"A heat pump does not make 3 kWh of heat from 1 kWh of electricity. It moves 2 kWh of outdoor heat into the home, and the 1 kWh of electricity pays for the move and ends up as heat too. Pressure sets the boiling temperature... Heat flows the normal way at both coils. The colder the day, the more each kWh costs.\" The entropy limit is not restated in the summary.", + "recap": "A heat pump doesn't make heat: it uses electricity to change the refrigerant's pressure, so the refrigerant boils colder than the outdoor air and condenses hotter than the home water, carrying 2 kWh in from outside plus the 1 kWh of electricity.", + "strongest": "Diagram 2, read together with the numbered walk-through and the earlier thought experiment (\"the home gets colder\"). Each coil's temperature is set against its neighbour (−10 °C vs 0 °C air, 40 °C vs 35 °C water), so \"At both coils, heat flows the normal way\" lands concretely.", + "weakest": "Diagram 2's compressor box, \"heats to about 60 °C\", followed by \"condenses at 40 °C\" with no word about the cooling in between. A smaller point: the closing line \"these two pressure changes do all the work\" sits oddly next to \"The valve adds no energy and removes none.\"", + "suggestions": [ + "In step 3, add a clause such as: 'the 60 °C vapour first cools to 40 °C, giving heat to the water, then condenses'.", + "Draw the compressor (and the valve) on the boundary between the two pressure groups, or label it '3.5 → 14 bar'.", + "Change \"do all the work\" to something like \"make this possible\", so it does not clash with the valve doing no work.", + "Consider one sentence in the Summary that restates the entropy limit in words: the bigger the lift, the more entropy must be dumped, so the more work is required." + ] + }, + "usage": { + "cost": 0.11641279999999998, + "seconds": 32 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Level 1 prose: \"Energy is conserved: 1 kWh of electricity + 2 kWh of outdoor heat = 3 kWh of heat delivered.\" It repeats the diagram." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Level 2: \"(The air around us is at 1 bar.)\"" + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Level 2 opening: \"Heat flows by itself only from hot to cold.\" The reader knows this; keep only the thought experiment that follows it." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Diagram 2 (Level 2 — the refrigerant loop): The compressor label says the vapour heats \"to about 60 °C\", but the next box says it \"condenses at 40 °C\". Neither the diagram nor step 3 says that the hot vapour first cools from 60 °C to 40 °C, giving up heat to the water, before it condenses. A careful reader stops to ask why the temperature drops by 20 K." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Diagram 2 (Level 2 — the refrigerant loop): The compressor is drawn inside the \"High pressure, 14 bar\" group, yet it takes in 3.5 bar vapour. It is really the boundary between the two pressures, so the grouping slightly misleads." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151455-quiz-diagram.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151455-quiz-diagram.json new file mode 100644 index 0000000..1599bcf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151455-quiz-diagram.json @@ -0,0 +1,243 @@ +{ + "tool": "quiz", + "rendering": "diagram", + "slug": "heat-pump", + "at": "2026-10-04T15:14:31", + "model": "default", + "prompt_sha": "c613363175a6", + "reply": { + "answers": { + "1": "The heat pump; the electricity it uses; the outdoor air (the cold source, giving 'outdoor heat'); the home water (35 °C, the hot side); the refrigerant (propane in the example), which moves in a closed loop through four parts: the evaporator (outdoor coil), the compressor, the condenser (indoor coil) and the expansion valve. Measures: COP, temperature lift, the ideal (Carnot) limit, and entropy.", + "2": "Electricity drives the compressor. The refrigerant takes heat from the outdoor air in the evaporator by boiling at low pressure (3.5 bar, −10 °C). The compressor raises it to high pressure (14 bar), which heats the vapour to about 60 °C. In the condenser it cools and condenses at 40 °C, giving heat to the 35 °C water. The expansion valve drops the pressure again, so part of the liquid boils and the rest cools to −10 °C. COP is heat delivered ÷ electricity. It is limited by the lift between the outdoor air and the water, through the entropy rule.", + "3": "The compressor and valve set two pressures, so the refrigerant boils below the outdoor temperature and condenses above the water temperature. Heat therefore flows naturally into the refrigerant outside and out of it inside. The electricity pays for this move, so the delivered heat is the outdoor heat plus the electricity.", + "4": "The temperature lift (water temperature minus outdoor temperature) matters most. If it gets larger (a colder day or hotter water), COP falls: 10 °C gives about 4, 0 °C about 3, −15 °C about 2 and −25 °C about 1.5. The ideal limit T_hot/(T_hot−T_cold) falls as the bottom of the fraction grows. Entropy explains why: heat taken in at a lower T carries more entropy per kWh, so more of the delivered heat must come from electricity.", + "5": "Assumptions and simplifications: the refrigerant boils 10 K below the air and condenses 5 K above the water; the real COP figures are labelled estimates; the ideal limit assumes no entropy is created. Facts: propane boils at −10 °C at about 3.5 bar and at 40 °C at about 14 bar; energy is conserved (Q_cold = Q_hot − W).", + "6": "A heat pump does not make heat; it moves heat from cold outdoor air into warmer home water, and the electricity pays for the move. A refrigerant's boiling point depends on pressure, so a closed loop holds it at low pressure outside (boiling below air temperature) and high pressure inside (condensing above water temperature). The compressor raises the pressure using electricity, and the valve drops it, so heat flows the normal hot-to-cold way at both coils. The delivered heat is the outdoor heat plus the compressor's energy, so COP can be about 3. Entropy limits COP to T_hot/(T_hot−T_cold), so a larger lift (colder day, hotter water) means a lower COP, and real machines get much less than this limit because they need temperature gaps at the coils and have losses.", + "7": "The entropy section was hardest. Entropy appears only as 'Q/T, which cannot be destroyed', with no sense of what it is or what the kWh/K unit means. The inequality also leaves out the electricity's entropy without saying why. One sentence saying electricity (work) carries no entropy, and a short intuition for what entropy measures, would have helped.", + "8": "No. The 3 kWh is 2 kWh of heat taken from the outdoor air plus the 1 kWh of electricity, which ends up in the vapour and then in the water. Energy in equals energy out.", + "9": "On the 10 °C day. The lift is 25 K instead of 50 K. The ideal limit is 12.3 instead of 6.2, and the typical COP is about 4 instead of about 2. A smaller lift means less electricity for each kWh of heat.", + "10": "The compressor uses electricity to squeeze the vapour from 3.5 to 14 bar, which heats it to about 60 °C, so it can condense at 40 °C above the water temperature. The expansion valve drops the pressure from 14 back to 3.5 bar through a narrow opening. About a third of the liquid boils at once, which cools all of it to −10 °C, below the air temperature; the valve adds and removes no energy. Both are needed because one fixed pressure gives one boiling point, and the fluid must be colder than the air outside and hotter than the water inside, so the loop needs a high-pressure side and a low-pressure side.", + "11": "From the outdoor air. Air at 0 °C (273 K) still holds heat. The refrigerant boiling at −10 °C in the outdoor coil is colder than the air, so 2 kWh flows into it.", + "12": "Heat flows by itself only from hot to cold. To take heat from the air, the refrigerant must be colder than the air: −10 °C against 0 °C. If it were warmer, no heat would flow in.", + "13": "COP falls, for example from about 3 at 0 °C to about 2 at −15 °C. The lift grows, so the ideal limit T_hot/(T_hot−T_cold) shrinks. In entropy terms, each kWh taken from colder air carries more entropy, so more electricity is needed for each kWh delivered." + }, + "audit": { + "terms": [ + "The cost of heat is called 'cost' in the Level 3 title ('Why the cost rises on a colder day'), 'electricity each kWh of heat costs' (line 80) and 'the more each kWh costs' (Summary), but 'Not covered here: cost and CO₂' uses cost to mean money. The two meanings conflict.", + "The limit has three names: 'ideal limit' (table column, line 95), 'the Carnot limit' (line 95) and 'ideal COP' (formula, line 97).", + "Each coil has two names: 'Evaporator' / 'outdoor coil' and 'Condenser' / 'indoor coil' (diagram and steps 1 and 3). Both names are given together, but the body text and the Level 3 section use 'coil' alone.", + "The electricity is called 'electricity' (Levels 1–2), 'W' and 'electricity' (line 109), and 'work' ('The work on the gas', line 68; 'do all the work', line 72). The text never says that 'work' and the electricity are the same thing.", + "Two lifts: 'temperature lift' or 'lift' (water minus air, line 80) and 'refrigerant lift' (line 127). They are told apart only at the end, and the table's ideal limit uses the first." + ], + "unexplained": [ + "Entropy values in 'kWh/K' (0.00042, 0.00732, 0.00974, 0.0403, 0.0389, line 107 onward). Entropy is new to this reader, and the text never says what a larger or smaller value means beyond 'cannot be destroyed'.", + "The 'Typical real COP (estimate)' column (line 82): no basis is given for the estimates, or for 'COP is about 2.2' with 55 °C radiators.", + "'The compressor, fans, and pumps lose about half of the rest' (line 127): it is unclear what 'the rest' refers to, and why the loss is about half.", + "'about a third of it boils at once' (line 70): the fraction is stated without saying where it comes from.", + "'boils 10 K below the air … condenses 5 K above the water' (line 127): the trade-off is named (bigger gap, smaller coil), but there is no sense of what changes if the gap is smaller or larger." + ], + "missing_why": [ + "Q_hot/T_hot ≥ Q_cold/T_cold (line 111) ignores the 1 kWh of electricity on the entropy side. The text never says that electricity (work) carries no entropy, which the derivation depends on.", + "'Heat Q that passes at kelvin temperature T carries entropy Q/T' (line 107) is stated without any reason why entropy has this form.", + "'The work on the gas heats it to about 60 °C' (line 68): there is no reason given for why compressing a gas heats it, or why to 60 °C specifically. The bicycle pump is only an analogy.", + "'these two pressure changes do all the work' (line 72) conflicts with 'The valve adds no energy and removes none' (line 70), and the text does not resolve this.", + "Why 'The refrigerant ends each cycle where it started, so it cannot store entropy' means the entropy balance must hold for each cycle is stated briefly, without that argument being spelled out." + ], + "excess": [ + "'Air at 0 °C is 273 K above absolute zero, so it still holds heat' (line 25): this reader already knows kelvin.", + "'(The air around us is at 1 bar.)' (line 39): this reader already knows pressure.", + "The worked example of 1 kWh flowing from water to air and the entropy gain of 0.00042 (line 107) repeats the hot-to-cold rule the reader already knows.", + "The Summary (line 131) repeats Levels 1–3 and adds nothing new.", + "'A fridge uses the same loop' (line 133) is a detour that neither the questions nor the argument uses." + ] + }, + "gaps": [ + "Entropy is never explained as a concept; it is defined only as Q/T plus a rule that it cannot be destroyed.", + "The text does not say that electrical work brings no entropy, which the COP limit derivation needs.", + "There is no basis for the real-COP estimates, or for the 'lose about half' of the gap between 6.3 and 3.", + "'Cost' means electricity per kWh in the body but money in 'Not covered', which is confusing.", + "The text does not explain why compressing the vapour raises its temperature, beyond the bicycle-pump analogy." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 1, + "6": 2, + "7": 2, + "8": 1, + "9": 2, + "10": 2, + "11": 2, + "12": 1, + "13": 2 + }, + "reasons": { + "5": "The answer calls the propane pressures (about 3.5 and 14 bar) facts, but model.md lists them under Estimated. It also leaves out the main stated assumptions, 0 °C outside and 35 °C water. The real-COP estimates and the 10 K and 5 K coil gaps are labelled correctly.", + "8": "The answer gives the 'No' and the balance 2 + 1 = 3 correctly. It does not say that the outdoor air still holds heat even though it is cold. It also does not say that the refrigerant is made colder than the air so heat flows into it. Both are part of the expected answer.", + "11": "No loss. The answer names the outdoor air and the 2 kWh specifically and explains why heat flows into the refrigerant. The balance 3 = 1 + 2 is not written out but follows directly from the question's premise.", + "12": "The outdoor half is correct and specific: heat flows hot to cold, so the refrigerant must be colder than the air (−10 °C against 0 °C). The expected answer also covers the loop and the indoor half, where the refrigerant must be hotter than the water to give heat out. The answer leaves this out. 'No heat would flow in' is weaker than the model's point that heat would flow out to the air. Note: the expected text for this item is broader than its question asks, so the rubric may need aligning.", + "pass_note": "Fails because quiz items 8 and 12 score 1 instead of 2. No general item scores 0." + }, + "pass": false + }, + "usage": { + "cost": 0.2533404, + "seconds": 66 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The cost of heat is called 'cost' in the Level 3 title ('Why the cost rises on a colder day'), 'electricity each kWh of heat costs' (line 80) and 'the more each kWh costs' (Summary), but 'Not covered here: cost and CO₂' uses cost to mean money. The two meanings conflict." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The limit has three names: 'ideal limit' (table column, line 95), 'the Carnot limit' (line 95) and 'ideal COP' (formula, line 97)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "Each coil has two names: 'Evaporator' / 'outdoor coil' and 'Condenser' / 'indoor coil' (diagram and steps 1 and 3). Both names are given together, but the body text and the Level 3 section use 'coil' alone." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "The electricity is called 'electricity' (Levels 1–2), 'W' and 'electricity' (line 109), and 'work' ('The work on the gas', line 68; 'do all the work', line 72). The text never says that 'work' and the electricity are the same thing." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "Two lifts: 'temperature lift' or 'lift' (water minus air, line 80) and 'refrigerant lift' (line 127). They are told apart only at the end, and the table's ideal limit uses the first." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Entropy values in 'kWh/K' (0.00042, 0.00732, 0.00974, 0.0403, 0.0389, line 107 onward). Entropy is new to this reader, and the text never says what a larger or smaller value means beyond 'cannot be destroyed'." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "The 'Typical real COP (estimate)' column (line 82): no basis is given for the estimates, or for 'COP is about 2.2' with 55 °C radiators." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'The compressor, fans, and pumps lose about half of the rest' (line 127): it is unclear what 'the rest' refers to, and why the loss is about half." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'about a third of it boils at once' (line 70): the fraction is stated without saying where it comes from." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "'boils 10 K below the air … condenses 5 K above the water' (line 127): the trade-off is named (bigger gap, smaller coil), but there is no sense of what changes if the gap is smaller or larger." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Q_hot/T_hot ≥ Q_cold/T_cold (line 111) ignores the 1 kWh of electricity on the entropy side. The text never says that electricity (work) carries no entropy, which the derivation depends on." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'Heat Q that passes at kelvin temperature T carries entropy Q/T' (line 107) is stated without any reason why entropy has this form." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "'The work on the gas heats it to about 60 °C' (line 68): there is no reason given for why compressing a gas heats it, or why to 60 °C specifically. The bicycle pump is only an analogy." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "'these two pressure changes do all the work' (line 72) conflicts with 'The valve adds no energy and removes none' (line 70), and the text does not resolve this." + }, + { + "n": 15, + "severity": "gap", + "kind": "missing_why", + "text": "Why 'The refrigerant ends each cycle where it started, so it cannot store entropy' means the entropy balance must hold for each cycle is stated briefly, without that argument being spelled out." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "'Air at 0 °C is 273 K above absolute zero, so it still holds heat' (line 25): this reader already knows kelvin." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "'(The air around us is at 1 bar.)' (line 39): this reader already knows pressure." + }, + { + "n": 18, + "severity": "excess", + "kind": "excess", + "text": "The worked example of 1 kWh flowing from water to air and the entropy gain of 0.00042 (line 107) repeats the hot-to-cold rule the reader already knows." + }, + { + "n": 19, + "severity": "excess", + "kind": "excess", + "text": "The Summary (line 131) repeats Levels 1–3 and adds nothing new." + }, + { + "n": 20, + "severity": "excess", + "kind": "excess", + "text": "'A fridge uses the same loop' (line 133) is a detour that neither the questions nor the argument uses." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "Entropy is never explained as a concept; it is defined only as Q/T plus a rule that it cannot be destroyed." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "The text does not say that electrical work brings no entropy, which the COP limit derivation needs." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "There is no basis for the real-COP estimates, or for the 'lose about half' of the gap between 6.3 and 3." + }, + { + "n": 24, + "severity": "gap", + "kind": "gap", + "text": "'Cost' means electricity per kWh in the body but money in 'Not covered', which is confusing." + }, + { + "n": 25, + "severity": "gap", + "kind": "gap", + "text": "The text does not explain why compressing the vapour raises its temperature, beyond the bicycle-pump analogy." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151658-quiz-diagram.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151658-quiz-diagram.json new file mode 100644 index 0000000..fff9b3c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151658-quiz-diagram.json @@ -0,0 +1,232 @@ +{ + "tool": "quiz", + "rendering": "diagram", + "slug": "heat-pump", + "at": "2026-10-04T15:16:46", + "model": "default", + "prompt_sha": "a1f778fe9ff6", + "reply": { + "answers": { + "1": "The main objects are electricity (1 kWh), outdoor air (the cold source, 0 °C in the example), home water (the hot sink, 35 °C) and the heat pump itself. The heat pump contains a refrigerant (propane) that circulates through a closed loop of four parts: the evaporator (outdoor coil), the compressor, the condenser (indoor coil) and the expansion valve. The measures used are COP, temperature lift, the ideal (Carnot) limit and entropy.", + "2": "Electricity drives the compressor. Outdoor air gives heat to the refrigerant in the evaporator, because the refrigerant boils there at -10 °C, below the air temperature. The compressor raises the refrigerant from 3.5 to 14 bar, which heats the vapour to about 60 °C. In the condenser the refrigerant condenses at 40 °C, above the water temperature, so heat flows into the home water. The expansion valve drops the pressure back to 3.5 bar, and part of the liquid boils, cooling the rest to -10 °C. Pressure sets the boiling temperature, so the low-pressure side is cold and the high-pressure side is hot. COP is heat delivered divided by electricity used. COP falls as the temperature lift (water temperature minus air temperature) grows, and entropy sets an upper limit on it.", + "3": "Electricity runs a compressor that raises the refrigerant's pressure, and with it the boiling/condensing temperature. That lets the refrigerant soak up heat from cold outdoor air at low pressure and release it into warmer home water at high pressure, with heat flowing hot-to-cold at each coil. So the water gets the moved outdoor heat plus the electricity: 2 + 1 = 3 kWh.", + "4": "The key variable is the temperature lift. If the outdoor air gets colder, or the water must be hotter, the lift grows and COP falls. In the table, COP is about 4 at 10 °C, about 3 at 0 °C, about 2 at -15 °C and about 1.5 at -25 °C. The reason is entropy. Heat taken at a lower temperature carries more entropy per kWh, and the water must receive at least that much entropy, so more of the delivered heat must come from electricity. The ideal limit T_hot/(T_hot - T_cold) falls as the bottom of the fraction grows.", + "5": "Assumptions/simplifications: the 'typical real COP' values are labelled estimates; the refrigerant is assumed to boil exactly 10 K below the air and condense 5 K above the water; the expansion valve is said to add and remove no energy. Established facts: propane boils at -10 °C at about 3.5 bar and at 40 °C at about 14 bar; energy is conserved; entropy cannot be destroyed.", + "6": "A heat pump does not create heat: it moves heat from cold outdoor air into warmer home water, and the electricity it uses ends up as heat too, so 1 kWh in can give 3 kWh out. It does this with a refrigerant whose boiling temperature depends on pressure. At low pressure the refrigerant boils below the outdoor air temperature and absorbs heat. At high pressure, after the compressor, it condenses above the water temperature and releases heat. The expansion valve drops the pressure again to close the loop. Entropy cannot be destroyed, which sets an upper limit on COP of T_hot/(T_hot - T_cold) in kelvin, so a bigger temperature lift (a colder day or hotter water) means a lower COP. Real units reach about a third of that limit because of the coil temperature gaps and machine losses.", + "7": "The entropy section was hardest. It says heat Q at temperature T 'carries entropy Q/T' and that entropy cannot be destroyed, but it never says what entropy is or why it has the form Q/T. A sentence or two of intuition would have made the derivation feel earned, not just checkable. A second point: the expansion valve 'boils a third of the liquid' with no energy added, and it isn't obvious how that conserves energy until you see that the heat comes from the liquid itself.", + "8": "No, it does not break energy conservation. Of the 3 kWh, 1 kWh is the electricity, which ends up as heat in the vapour, and 2 kWh is heat taken from the outdoor air. Air at 0 °C is 273 K above absolute zero, so it still holds heat. It flows into the refrigerant because the refrigerant boils colder (-10 °C) in the evaporator.", + "9": "COP is higher on the 10 °C day. The lift there is 25 K (ideal limit 12.3, typical real COP about 4), against 50 K at -15 °C (ideal limit 6.2, typical about 2). A bigger lift needs more electricity per kWh of heat, because heat taken at a lower temperature carries more entropy, and the water must receive at least that much.", + "10": "The compressor uses electricity to squeeze the vapour from 3.5 to 14 bar, which heats it to about 60 °C and raises its condensing temperature to 40 °C, so it can give heat to the 35 °C water. The expansion valve is a narrow opening that drops the pressure from 14 back to 3.5 bar. About a third of the liquid boils at once, cooling the whole flow to -10 °C, so it can take heat from the 0 °C air. The loop needs both because the refrigerant must be at high pressure (hot) at the indoor coil and low pressure (cold) at the outdoor coil. One raises the pressure and the other lowers it. Only the compressor uses energy.", + "11": "They are heat taken from the outdoor air in the evaporator. The refrigerant boils at -10 °C there, colder than the 0 °C air, so heat flows into it by itself. That heat is carried around the loop and released in the condenser, along with the compressor's 1 kWh.", + "12": "Heat flows by itself only from hot to cold. If the refrigerant were not colder than the outdoor air, no heat would flow into it. If it were not hotter than the water, heat would flow from the water into the refrigerant and the home would get colder. A temperature gap also moves heat faster through a smaller coil, which is why the explanation uses 10 K and 5 K gaps.", + "13": "COP falls, for example from about 3 at 0 °C to about 2 at -15 °C and about 1.5 at -25 °C. The temperature lift grows, so the entropy limit T_hot/(T_hot - T_cold) falls and each kWh of heat needs more electricity. Many units add a backup electric heater (COP 1) for the coldest days." + }, + "audit": { + "terms": [ + "The same limit has two names: 'ideal limit' (Level 3 table, 'The ideal limit') and 'Carnot limit' ('physicists call it the Carnot limit').", + "Two words for the refrigerant: 'fluid' (Level 1: 'makes its fluid colder'; Level 2: 'pipe a fluid') and 'refrigerant' (named only partway through Level 2).", + "Two words for the refrigerant vapour: 'vapour' (diagram, steps 2-3) and 'gas' (step 2: 'push on the gas', 'heats the gas'). A reader may wonder if the change of word means a change of state.", + "Two words for the electricity input: 'electricity' and 'W' in the formulas. Its relation to 'push' or work in step 2 is not made explicit.", + "Two names for the heat taken from the air: 'outdoor heat' (Levels 1 and 2) and 'Q_cold' (Level 3). The mapping is given, but 'heat delivered' becomes 'Q_hot', and the air side of the entropy argument talks about 'the air loses'." + ], + "unexplained": [ + "'Losses in the compressor, fans, and pumps take the COP from 6.3 down to about 3': the size of these losses is not explained, and they halve the COP.", + "Table column 'Typical real COP (estimate)': no source or method, so the reader can't tell how firm 'about 4/3/2/1.5' are.", + "'Old radiators need 55 °C water ... the COP is about 2.2': no ideal-limit figure is given for this case, and it's not shown how 2.2 was obtained.", + "'about a third of it boils at once' at the expansion valve: there is no basis for 'a third', and nothing says what a larger or smaller fraction would mean.", + "The coil gaps 'boils 10 K below the air ... condenses 5 K above the water': the reader isn't told why these values or what changes with bigger or smaller gaps beyond 'faster through a smaller coil'. The text also doesn't say whether the -10 °C boiling point moves on a -15 °C day." + ], + "missing_why": [ + "'Heat Q that passes at kelvin temperature T carries entropy Q/T': stated without any reason why entropy has this form or what entropy is. The whole limit rests on it.", + "'its boiling temperature depends on its pressure': it isn't explained why higher pressure raises the boiling point. Only propane data is given.", + "'that push heats the gas': only the bicycle-pump analogy is offered, with no reason why compressing a gas raises its temperature.", + "'Nothing can destroy entropy': stated as a law, with the link to hot-to-cold flow asserted rather than motivated.", + "'The refrigerant ends each cycle where it started, so it cannot store entropy': it isn't explained why the entropy bookkeeping uses the water and air temperatures rather than the refrigerant's own. This only comes up later as 'refrigerant lift'." + ], + "excess": [ + "'Air at 0 °C is 273 K above absolute zero, so it still holds heat': this reader already knows kelvin.", + "'(The air around us is at 1 bar.)': this reader already knows pressure.", + "'An electric heater has only one input, the electricity. So it can give only 1 kWh per kWh.': repeats the opening and the COP-1 statement.", + "'A fridge uses the same loop...' and the 'Not covered here' list: detours that neither the questions nor the argument use." + ] + }, + "gaps": [ + "Entropy is used as the basis of the limit but never explained beyond 'Q/T' and 'cannot be destroyed'.", + "The text doesn't say how the refrigerant's boiling temperature or pressure adjusts when the outdoor temperature changes. The loop is described only for 0 °C air (-10 °C boiling), so it's unclear how it works on a -15 °C day.", + "There is no explanation of why real COP is only about a third of the ideal limit, beyond naming the losses.", + "Energy accounting at the expansion valve, where liquid boils with no energy added, is only lightly explained.", + "The real COP figures are estimates with no stated basis." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 1, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "5": "Misclassifies the propane boiling pressures (about 3.5 and 14 bar) as established facts. model.md labels them estimates. It also lists 'the valve adds and removes no energy' as an assumption, when it is a stated fact of the cycle. It omits other estimates the model labels, such as vapour at about 60 °C and about a third of the liquid boiling. It also omits the core assumed conditions of 0 °C outside and 35 °C water." + }, + "pass": true + }, + "usage": { + "cost": 0.23809480000000002, + "seconds": 60 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same limit has two names: 'ideal limit' (Level 3 table, 'The ideal limit') and 'Carnot limit' ('physicists call it the Carnot limit')." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Two words for the refrigerant: 'fluid' (Level 1: 'makes its fluid colder'; Level 2: 'pipe a fluid') and 'refrigerant' (named only partway through Level 2)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "Two words for the refrigerant vapour: 'vapour' (diagram, steps 2-3) and 'gas' (step 2: 'push on the gas', 'heats the gas'). A reader may wonder if the change of word means a change of state." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "Two words for the electricity input: 'electricity' and 'W' in the formulas. Its relation to 'push' or work in step 2 is not made explicit." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "Two names for the heat taken from the air: 'outdoor heat' (Levels 1 and 2) and 'Q_cold' (Level 3). The mapping is given, but 'heat delivered' becomes 'Q_hot', and the air side of the entropy argument talks about 'the air loses'." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'Losses in the compressor, fans, and pumps take the COP from 6.3 down to about 3': the size of these losses is not explained, and they halve the COP." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "Table column 'Typical real COP (estimate)': no source or method, so the reader can't tell how firm 'about 4/3/2/1.5' are." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'Old radiators need 55 °C water ... the COP is about 2.2': no ideal-limit figure is given for this case, and it's not shown how 2.2 was obtained." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'about a third of it boils at once' at the expansion valve: there is no basis for 'a third', and nothing says what a larger or smaller fraction would mean." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "The coil gaps 'boils 10 K below the air ... condenses 5 K above the water': the reader isn't told why these values or what changes with bigger or smaller gaps beyond 'faster through a smaller coil'. The text also doesn't say whether the -10 °C boiling point moves on a -15 °C day." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'Heat Q that passes at kelvin temperature T carries entropy Q/T': stated without any reason why entropy has this form or what entropy is. The whole limit rests on it." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'its boiling temperature depends on its pressure': it isn't explained why higher pressure raises the boiling point. Only propane data is given." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "'that push heats the gas': only the bicycle-pump analogy is offered, with no reason why compressing a gas raises its temperature." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "'Nothing can destroy entropy': stated as a law, with the link to hot-to-cold flow asserted rather than motivated." + }, + { + "n": 15, + "severity": "gap", + "kind": "missing_why", + "text": "'The refrigerant ends each cycle where it started, so it cannot store entropy': it isn't explained why the entropy bookkeeping uses the water and air temperatures rather than the refrigerant's own. This only comes up later as 'refrigerant lift'." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "'Air at 0 °C is 273 K above absolute zero, so it still holds heat': this reader already knows kelvin." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "'(The air around us is at 1 bar.)': this reader already knows pressure." + }, + { + "n": 18, + "severity": "excess", + "kind": "excess", + "text": "'An electric heater has only one input, the electricity. So it can give only 1 kWh per kWh.': repeats the opening and the COP-1 statement." + }, + { + "n": 19, + "severity": "excess", + "kind": "excess", + "text": "'A fridge uses the same loop...' and the 'Not covered here' list: detours that neither the questions nor the argument use." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "Entropy is used as the basis of the limit but never explained beyond 'Q/T' and 'cannot be destroyed'." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "The text doesn't say how the refrigerant's boiling temperature or pressure adjusts when the outdoor temperature changes. The loop is described only for 0 °C air (-10 °C boiling), so it's unclear how it works on a -15 °C day." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "There is no explanation of why real COP is only about a third of the ideal limit, beyond naming the losses." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "Energy accounting at the expansion valve, where liquid boils with no energy added, is only lightly explained." + }, + { + "n": 24, + "severity": "gap", + "kind": "gap", + "text": "The real COP figures are estimates with no stated basis." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/understanding.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/understanding.md new file mode 100644 index 0000000..4667ab1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/explainers/heat-pump/review/understanding.md @@ -0,0 +1,19 @@ +# Review: heat-pump (Published tier, Stage 2, diagram.md) + +## Probe — 2026-10-04T150833-probe +Adopted 1–6, 8, 9, 11–14 (entropy defined, holding/breaking entropy cases, expansion-valve mechanism, compression heating, two lifts named apart, coil gaps, one name per concept, 1 bar reference, −25 °C row, cuts). Declined 7 (went to Scope), 10 (adds length, changes no step), 15 (template metadata). + +## Cold read, narrative — 2026-10-04T151129 +5 blocking findings, all caused by a terse "Said" column. Fixed by writing each beat's words in full. Cut: terse Celsius aside (spelled out), radiators given a COP, beat 4 leads with numbers. + +## Cold read, diagram — 2026-10-04T151329 +No blocking findings. Adopted 1, 3, 4 (vapour 60 °C → 40 °C before condensing), 5 (compressor drawn between the pressure groups). Declined 2: 1 bar gives the pressures a meaning. + +## Blind test, diagram +- Run 1 (2026-10-04T151455): FAIL — 8 and 12 scored 1. Item 12's rubric covered more than its question asked: widened the `loop` ask. Item 8: Level 1 now links outdoor heat to the refrigerant being colder than the air. Also fixed: electricity carries no entropy (the derivation needed it), "work" vs electricity, one name for the ideal limit, "cost" with two meanings. +- Run 2 (2026-10-04T151658): PASS — every quiz item 2; general item 5 (assumptions vs facts) 1. Adopted 2, 3 (refrigerant named in Level 1; "vapour", not "gas"). +- Declined, both runs: entropy foundations, why compression heats a gas, coil sizing trade-offs, sources for typical COPs (labelled estimates). These are beyond this reader's question. +- After run 2: the Level 3 heading no longer gives away the predict-first answer. + +## Broken-copy check — 2026-10-04T151949 +A copy with the cold-weather claim reversed (COP rises in the cold) FAILS: items 9 and 13 score 0. The quiz can fail. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/read.json new file mode 100644 index 0000000..8d883ff --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "Heat is moved from the outdoor air into the home. The electricity does not create the heat but instead powers the movement of heat from cold to warm.", + "quote": "Answer: the heat pump does not make that heat. It moves heat from the cold outdoor air into the home. The electricity pays for the move." + }, + "2": { + "answer": "The refrigerant circulates through a closed loop at two different pressures. It boils at low pressure to absorb outdoor heat, gets compressed to raise its temperature, condenses at high pressure to release heat, then expands to cool before repeating.", + "quote": "A fluid circulates in a closed loop of four parts, at a low pressure on the outdoor side and a high pressure on the indoor side." + }, + "3": { + "answer": "On colder days, the temperature difference between the home water and the outdoor air increases. This larger temperature lift requires more electricity per unit of heat delivered.", + "quote": "The bigger the lift, the more electricity each kWh of heat needs." + }, + "4": { + "answer": "By the ideal limit formula, the 5 °C day (35 K lift) delivers much more heat per unit electricity than the −15 °C day (50 K lift), because the larger temperature difference on the colder day reduces the ideal limit.", + "quote": "A bigger lift makes the bottom of the fraction bigger, so the limit falls." + } + }, + "stops": [ + "Air at 0 °C is 273 K above absolute zero, so it still holds heat.", + "One fluid can be both, because its boiling temperature depends on its pressure.", + "Boiling takes this heat without a change in temperature.", + "Heat Q that passes at kelvin temperature T carries entropy Q/T, in kWh/K.", + "The refrigerant ends each cycle where it started, so it cannot store entropy.", + "Propane boils at −10 °C at about 3.5 bar and at 40 °C at about 14 bar.", + "The refrigerant boils 10 K below the air and condenses 5 K above the water.", + "Losses in the compressor, fans, and pumps take the COP from 6.3 down to about 3.", + "Its own lift, the refrigerant lift, is 40 − (−10) = 50 K.", + "Multiply by T_cold: Q_hot × T_cold / T_hot ≥ Q_hot − W. Rearrange: W ≥ Q_hot × (T_hot − T_cold) / T_hot." + ], + "excess": [ + "Colours in both diagrams: blue is the cold side, red is the hot side, yellow is electricity.", + "An electric heater has only one input, the electricity. So it can give only 1 kWh per kWh.", + "In Celsius the formula gives 35 / (35 − 0) = 1, which would say that a heat pump can never beat a heater.", + "A fridge uses the same loop, with the cold coil inside the fridge.", + "Old radiators need 55 °C water: at 0 °C outside, the lift is 55 K and the COP is about 2.2.", + "Many units add a backup electric heater (COP 1) for the coldest days.", + "The valve adds no energy and removes none.", + "Typical real COP (estimate) column in the table showing measured performance.", + "Heat flows from the water into the fluid, and the home gets colder.", + "Not covered here: ground-source heat pumps; defrosting the outdoor coil below about 5 °C." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/run.json new file mode 100644 index 0000000..d76a699 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/heat-pump/diagram.md` because the answer depends on a loop of four parts with a temperature and pressure at each point, which a diagram with text shows better than prose. It passed the probe, the cold read and the blind test, and the leftover test copy at `explainers/heat-pump-broken/` needs your OK to delete.", + "error": null, + "cost": 2.352323, + "seconds": 845, + "turns": 77, + "files": [ + "explainers/heat-pump/diagram.md" + ], + "review_cost": 1.0471, + "reviews": 6 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/diagram.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/diagram.md new file mode 100644 index 0000000..bd8ba95 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/diagram.md @@ -0,0 +1,84 @@ +# How can a heat pump deliver three times more heat than the electricity it uses? + + + +**Short answer.** A heat pump does not turn electricity into heat. It uses electricity to *move* heat that already exists in the outdoor air into the house. For each 1 kWh of electricity, it takes about 2 kWh of heat from the air and delivers 3 kWh to the house. No energy is created. + +## Level 1 — Where the 3 kWh come from + + + + +```mermaid +flowchart LR + AIR["Outdoor air, 0 °C
gives 2 kWh"] --> HP(("heat pump")) + ELEC["Electricity
1 kWh"] --> HP + HP --> HOUSE["House, water at 35 °C
gets 3 kWh"] +``` + +Energy is conserved: **heat delivered = heat taken from the air + electricity**, so 3 kWh = 2 kWh + 1 kWh. + +The ratio of heat delivered to electricity used is the **COP** (coefficient of performance). Here COP = 3 ÷ 1 = 3. + +- COP 1 is what an electric resistance heater gets. It turns 1 kWh of electricity into 1 kWh of heat, and conservation of energy forbids more. +- COP 3 is typical for a heat pump on a 0 °C day. +- COP 4 or more is possible on a mild day. + +COP is not an efficiency. It counts only the paid input (electricity) and leaves out the free input (heat from the air). That is why it can exceed 1. + +Air at 0 °C still holds a lot of thermal energy: 0 °C is 273 K, far above absolute zero. To extract that energy, you only need something colder than the air. + +## Level 2 — How the heat is moved: the refrigerant loop + +A **refrigerant** is a fluid that circulates in a closed loop and carries the heat. One example is propane (R290). Its boiling point depends on pressure: it boils at about −10 °C at 3.5 bar and condenses at about 40 °C at 14 bar. The heat pump uses this to make the refrigerant colder than the outdoor air at one place and hotter than the house at another. + +```mermaid +flowchart LR + AIR["Outdoor air 0 °C"] -->|"2 kWh heat in"| EVAP + ELEC["Electricity"] -->|"1 kWh work"| COMP + subgraph LOOP["refrigerant loop"] + direction LR + EVAP["① evaporator
boils at −10 °C, 3.5 bar"] -->|"cold gas"| COMP["② compressor
3.5 → 14 bar"] + COMP -->|"hot gas"| COND["③ condenser
condenses at 40 °C, 14 bar"] + COND -->|"warm liquid"| VALVE["④ expansion valve
14 → 3.5 bar"] + VALVE -->|"cold liquid, −10 °C"| EVAP + end + COND -->|"3 kWh heat out"| HOUSE["House water 35 °C"] +``` + +1. **Evaporator (outdoors).** The refrigerant boils at −10 °C. The air is at 0 °C, so heat flows from the air into the refrigerant: 2 kWh. +2. **Compressor.** The electricity (1 kWh) drives the compressor. Squeezing the gas to 14 bar raises its temperature, so it can condense at 40 °C. Without this step, the refrigerant stays at −10 °C. Heat flows only from hot to cold, so −10 °C refrigerant cannot heat 35 °C water. +3. **Condenser (indoors).** The refrigerant condenses at 40 °C. The house water is at 35 °C, so heat flows from the refrigerant into the water. This heat is the 2 kWh from the air plus the 1 kWh of compressor work: 3 kWh. +4. **Expansion valve.** The valve drops the pressure to 3.5 bar, which drops the temperature to −10 °C. Without this step, the refrigerant stays at 40 °C. It is then hotter than the 0 °C air, so it gives heat to the outdoors instead of taking it in. + +A refrigerator uses the same loop. Its evaporator is inside the box and its condenser is the warm grille at the back. + +## Level 3 — Why the COP is about 3 and not higher: temperature lift + + + +The electricity pays to lift heat from a low temperature to a high one. The **temperature lift** is the condenser temperature minus the evaporator temperature. A larger lift needs more work for each kWh of heat moved. + +Physics sets a ceiling. No machine can beat the Carnot limit (temperatures in kelvin): + +COP_max = T_hot ÷ (T_hot − T_cold) + +| Day | Outdoor air | Evaporator | Condenser | Lift | Carnot limit | Real COP (estimate) | +|---|---|---|---|---|---|---| +| Mild | 10 °C | 0 °C (273 K) | 40 °C (313 K) | 40 K | 313 ÷ 40 ≈ 7.8 | about 4 | +| Typical | 0 °C | −10 °C (263 K) | 40 °C (313 K) | 50 K | 313 ÷ 50 ≈ 6.3 | about 3 | +| Cold | −15 °C | −25 °C (248 K) | 40 °C (313 K) | 65 K | 313 ÷ 65 ≈ 4.8 | about 2 | + +The real COPs are estimates. A rule of thumb is that real machines reach about half of the Carnot limit, because of friction, imperfect heat exchangers, and other losses. + +What the table predicts: + +- **Colder outdoor air → lower COP.** The heat pump must lift the heat farther. +- **Cooler indoor water → higher COP.** Large radiators or underfloor heating at 35 °C need less lift than small radiators at 55 °C. +- **COP never falls below 1 in principle.** Even at a large lift, the compressor work itself ends up as heat in the house. + +## Summary + +The heat pump is a pump for heat. The electricity does not become the heat; it pays for moving heat "uphill" from cold outdoor air to the warmer house. Because moving heat costs less energy than the heat moved, 1 kWh of electricity delivers about 3 kWh of heat. The smaller the temperature lift, the larger that ratio. + +**Out of scope:** defrost cycles, ground-source heat pumps, refrigerant choice and safety, and the derivation of the Carnot limit (see entropy and the second law in a thermodynamics text). diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/model.md new file mode 100644 index 0000000..964b27b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/model.md @@ -0,0 +1,113 @@ +# Semantic model: heat pump + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How can a heat pump deliver three times more heat than the electricity it uses? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (energy conservation, temperature in kelvin, boiling point depends on pressure), basic programming. New to heat pumps and refrigeration. Does not know: the refrigerant cycle, COP, the Carnot limit. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| heat pump | A machine that uses electricity to move heat from a cold place (outdoor air) to a warm place (the house). | fact | +| electricity | The work input; it drives the compressor. | fact | +| outdoor air | The cold source; air at 0 °C still contains thermal energy that can be extracted. | fact | +| refrigerant | A fluid that boils at a low temperature and circulates in a closed loop; it carries the heat. | fact | +| evaporator | Outdoor heat exchanger; refrigerant boils there, colder than the air, and absorbs heat. | fact | +| compressor | Squeezes the refrigerant gas to a high pressure, which raises its temperature; uses the electricity. | fact | +| condenser | Indoor heat exchanger; hot refrigerant condenses there, hotter than the house water, and releases heat. | fact | +| expansion valve | Drops the refrigerant pressure, which drops its temperature below the outdoor air again. | fact | +| COP | Coefficient of performance: heat delivered ÷ electricity used. | definition | +| temperature lift | Condenser temperature minus evaporator temperature. | definition | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| outdoor air | gives heat to | evaporator | because the refrigerant there is colder (−10 °C vs 0 °C) | +| electricity | drives | compressor | the only paid energy input | +| compressor | raises pressure and temperature of | refrigerant | 3.5 bar → 14 bar, −10 °C → 40 °C condensing | +| condenser | gives heat to | house | because the refrigerant there is hotter (40 °C vs 35 °C) | +| expansion valve | lowers pressure and temperature of | refrigerant | closes the loop | +| temperature lift | sets the minimum work per unit heat | COP | larger lift → lower COP | + +## Causal chain + +```text +refrigerant boils colder than outdoor air → heat flows in (2 kWh) +→ compressor (1 kWh electricity) makes it hotter than the house +→ heat flows out (2 + 1 = 3 kWh) → valve makes it cold again → repeat +``` + +## Quantities + +- Energy balance (conservation): Q_out = Q_in + W. Worked: 3 kWh = 2 kWh + 1 kWh. +- COP = Q_out / W = 3 / 1 = 3. +- Electric resistance heater: COP = 1 (1 kWh → 1 kWh). +- Carnot limit (ideal machine): COP_max = T_hot / (T_hot − T_cold), temperatures in kelvin. + - Typical day: evaporator −10 °C (263 K), condenser 40 °C (313 K), lift 50 K → COP_max = 313/50 ≈ 6.3. Real ≈ 3. + - Mild day: outdoor 10 °C, evaporator 0 °C (273 K), lift 40 K → COP_max ≈ 7.8. Real ≈ 4. + - Cold day: outdoor −15 °C, evaporator −25 °C (248 K), lift 65 K → COP_max ≈ 4.8. Real ≈ 2. +- Refrigerant R290 (propane): boils at about −10 °C at 3.5 bar and condenses at about 40 °C at 14 bar. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Mild day (10 °C) | COP ≈ 4 | smaller lift, less work per kWh moved | +| Cold day (−15 °C) | COP ≈ 2 | larger lift, more work per kWh moved | +| Resistance heater | COP = 1 | converts electricity to heat; moves none | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Move heat instead of making it | Turn electricity into heat (resistance heater) | 1 kWh in → 1 kWh out, COP 1, never more: conservation of energy caps it. | +| Compress the refrigerant | Just pump the refrigerant round without compressing | Refrigerant at −10 °C cannot heat 35 °C water; heat flows only hot → cold. | +| Use the valve to drop pressure | Keep the refrigerant at high pressure | At 40 °C it is hotter than 0 °C air, so it gives heat to the outdoors instead of taking it. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| COP > 1 does not break energy conservation | 2 kWh from air + 1 kWh electricity = 3 kWh out | If you count only electricity as input, 3 out of 1 looks like 300% "efficiency" — the 2 kWh from the air is omitted. | +| COP falls as lift grows | lift 40 K → ≈ 4; 50 K → ≈ 3 | lift 65 K (−15 °C day) → ≈ 2 | + +## Terms + +- **heat pump**: moves heat from cold outdoor air to the warm house using electricity. +- **refrigerant**: the circulating fluid that carries heat. +- **evaporator / condenser**: where the refrigerant takes in / gives out heat. +- **compressor**: raises refrigerant pressure and temperature; the electricity goes here. +- **COP**: heat delivered ÷ electricity used. 1 = no better than a resistance heater; 3 = typical; 5 = mild weather, low lift. +- **temperature lift**: how far the heat is lifted in temperature. 40 K = easy; 65 K = hard. + +## Epistemic status + +- **Observed / established:** energy conservation; heat flows only hot → cold spontaneously; boiling point rises with pressure. +- **Mathematical consequence:** Q_out = Q_in + W; Carnot limit formula. +- **Assumed:** the example temperatures (0 °C outdoor, 35 °C water, 5–10 K differences at the heat exchangers). +- **Estimated / model output:** real COPs (≈ 2, 3, 4) — rule of thumb: real machines reach about half the Carnot limit. + +## Confusion points + +- "COP 3 means 300% efficient, which is impossible" → the machine does not create energy; 2 of the 3 kWh come from outdoor air. +- "0 °C air has no heat in it" → 0 °C is 273 K; the air holds a lot of thermal energy. Only a colder refrigerant is needed to pull it out. +- "The compressor's heat is the output" → compressor work is only 1 of the 3 kWh. + +## Scope + +- **Out of scope:** defrost cycles, ground-source vs air-source, refrigerant choice and safety, sizing — see any HVAC design guide. +- **Out of scope:** derivation of the Carnot limit — see entropy and the second law in a thermodynamics text. + +## Representation decision + +- **Stage:** 2 +- **Reason:** the mechanism is a closed loop of four components with energy flows in and out; the reader must hold the loop, the temperatures at each point, and the energy balance at once. +- **Levels:** L1 energy balance / L2 the refrigerant loop / L3 why lift sets the limit diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/model.yaml new file mode 100644 index 0000000..58bf72e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/explainers/heat-pump/model.yaml @@ -0,0 +1,85 @@ +# Machine-readable half of model.md. `explainer check heat-pump` holds every rendering to it. + +values: + electricity_kwh: 1 + air_heat_kwh: 2 + delivered_kwh: 3 + cop_typical: 3 + cop_resistance: 1 + outdoor_c: 0 + water_c: 35 + radiator_hot_c: 55 + evap_c: -10 + cond_c: 40 + evap_bar: 3.5 + cond_bar: 14 + evap_k: 263 + cond_k: 313 + lift_typical_k: 50 + carnot_typical: 6.3 + mild: + outdoor_c: 10 + evap_c: 0 + evap_k: 273 + lift_k: 40 + carnot: 7.8 + cop: 4 + cold: + outdoor_c: -15 + evap_c: -25 + evap_k: 248 + lift_k: 65 + carnot: 4.8 + cop: 2 + +allow: [5, 273, 2, 1, 3, 4] + +require: + - path: delivered_kwh + in: [diagram] + - path: air_heat_kwh + in: [diagram] + - path: carnot_typical + in: [diagram] + +claims: + - id: move-not-make + kind: why + about: [heat pump] + statement: "A heat pump moves heat that already exists in outdoor air; it does not turn electricity into heat." + alternative: "Turn electricity into heat directly (resistance heater)." + counterexample: "A resistance heater turns 1 kWh into 1 kWh of heat, COP 1; conservation of energy forbids more." + ask: "Where do the extra 2 kWh come from?" + - id: balance + kind: guarantee + statement: "Heat delivered = heat taken from the air + electricity: 3 = 2 + 1 kWh." + example: "2 kWh from 0 °C air plus 1 kWh electricity gives 3 kWh to the house." + counterexample: "Count only electricity as input and COP 3 looks like 300% efficiency, which seems to break conservation; it does not, because 2 kWh came from the air." + - id: compress + kind: why + about: [compressor] + statement: "The compressor raises the refrigerant to 40 °C so that it is hotter than the 35 °C water." + alternative: "Circulate the refrigerant without compressing it." + counterexample: "Refrigerant at −10 °C cannot heat 35 °C water; heat flows only from hot to cold." + ask: "Why is the compressor needed if the heat comes from the air?" + - id: valve + kind: why + about: [expansion valve] + statement: "The expansion valve drops pressure so the refrigerant is colder than the outdoor air again." + alternative: "Keep the refrigerant at high pressure." + counterexample: "At 40 °C it is hotter than 0 °C air and would lose heat outdoors instead of taking it in." + - id: lift-limit + kind: guarantee + statement: "COP falls as the temperature lift grows; the Carnot limit is T_hot / (T_hot − T_cold)." + example: "Lift 50 K: limit 313/50 ≈ 6.3, real COP about 3. Lift 40 K: limit 7.8, real about 4." + counterexample: "On a −15 °C day the lift is 65 K: limit 4.8, real COP about 2." + ask: "What happens to COP on a very cold day, and why?" + +accept_unjustified: {} + +terms: + - term: COP + means: "heat delivered ÷ electricity used; not an efficiency" + avoid: [300% efficient, efficiency of 300] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/read.json new file mode 100644 index 0000000..39cec37 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "From the outdoor air. The heat pump extracts 2 kWh of heat from the outdoor air and uses 1 kWh of electricity to deliver a total of 3 kWh to the house.", + "quote": "It uses electricity to *move* heat that already exists in the outdoor air into the house. For each 1 kWh of electricity, it takes about 2 kWh of heat from the air and delivers 3 kWh to the house." + }, + "2": { + "answer": "The refrigerant circulates through four stages: it absorbs heat from the outdoor air by boiling in the evaporator, gets compressed to raise its temperature, condenses in the condenser to release heat to the house, and expands to cool back down and repeat the cycle.", + "quote": "The heat pump uses this to make the refrigerant colder than the outdoor air at one place and hotter than the house at another." + }, + "3": { + "answer": "Because the temperature lift—the difference between outdoor air temperature and the house temperature—is larger on colder days, requiring more work to move each unit of heat.", + "quote": "The electricity pays to lift heat from a low temperature to a high one. The **temperature lift** is the condenser temperature minus the evaporator temperature. A larger lift needs more work for each kWh of heat moved." + }, + "4": { + "answer": "The explanation does not provide specific data for a 5 °C day. However, the table shows that the Carnot limit decreases with colder outdoor air: 7.8 at 10 °C and 4.8 at −15 °C. A 5 °C day would therefore have a higher Carnot limit than −15 °C, delivering more heat per unit of electricity by the ideal limit.", + "quote": "**Colder outdoor air → lower COP.** The heat pump must lift the heat farther." + } + }, + "stops": [ + "\"R290\"—undefined term, what is this abbreviation?", + "\"boiling point depends on pressure\"—why does pressure affect boiling point, and what is the mechanism?", + "\"COP_max = T_hot ÷ (T_hot − T_cold)\"—formula given without derivation or explanation of why it works.", + "\"Heat flows only from hot to cold\"—principle stated without explanation of the underlying reason.", + "\"Squeezing the gas to 14 bar raises its temperature\"—the pressure-temperature relationship is not explained.", + "\"entropy and the second law\"—mentioned but explicitly out of scope, yet referenced for understanding Carnot limit.", + "\"A rule of thumb is that real machines reach about half of the Carnot limit\"—why specifically half, and does this vary?", + "\"Without this step, the refrigerant stays at 40 °C. It is then hotter than the 0 °C air, so it gives heat to the outdoors\"—the causal mechanism is unclear.", + "\"0 °C is 273 K, far above absolute zero\"—why is distance from absolute zero relevant to thermal energy?", + "How are the specific operating temperatures (−10 °C at evaporator, 40 °C at condenser) determined from the outdoor and indoor temperatures?" + ], + "excess": [ + "\"A refrigerator uses the same loop. Its evaporator is inside the box and its condenser is the warm grille at the back.\"—tangential analogy not needed for heat pump explanation.", + "\"One example is propane (R290)\"—specific refrigerant example when the principle applies to any refrigerant.", + "\"**Out of scope:** defrost cycles, ground-source heat pumps, refrigerant choice and safety, and the derivation of the Carnot limit\"—meta-commentary already marked out of scope.", + "\"**Cooler indoor water → higher COP.** Large radiators or underfloor heating at 35 °C need less lift than small radiators at 55 °C.\"—optimization detail beyond the core explanation." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/run.json new file mode 100644 index 0000000..5e50d8e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/heat-pump/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/heat-pump/diagram.md` (with `model.md` and `model.yaml` behind it). I chose a diagram because the answer is a closed loop of four parts with energy going in and out, and two Mermaid pictures with short notes show that more clearly than prose alone. The explainer check and the Mermaid render check both pass.", + "error": null, + "cost": 0.47279319999999997, + "seconds": 98, + "turns": 19, + "files": [ + "explainers/heat-pump/diagram.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/explainers/key-exchange/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/explainers/key-exchange/explanation.md new file mode 100644 index 0000000..a6c4b4d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/explainers/key-exchange/explanation.md @@ -0,0 +1,160 @@ +# Agreeing on a secret in public + +Alice and Bob have never met. They want to send each other encrypted messages, and encryption needs a shared secret key. The only way they can talk is a channel that anyone can read: the internet, a radio link, a postcard. Call the listener Eve. Eve sees every message Alice and Bob send. + +At first this looks impossible. Anything Alice sends, Eve also gets. If Alice simply sends a key, Eve has the key too. Alice could encrypt the key first, but that needs another key, which brings back the same problem. + +Two ideas solve it: + +1. **Each person keeps a secret that is never sent.** Only *results computed from* the secrets go over the channel. +2. **Use a calculation that is easy to do but very hard to undo.** Eve sees the results but can't work back to the secrets. + +The method below is called **Diffie–Hellman key exchange** (Whitfield Diffie and Martin Hellman, 1976). Almost every secure web connection still uses it or a close variant. + +--- + +## An analogy: mixing paint + +Assume that mixing two paints is easy, and that separating a mixture back into the paints that made it is practically impossible. + +1. Alice and Bob agree on a common colour in public, say **yellow**. Eve knows it. +2. Alice privately picks a secret colour, **red**. Bob privately picks **blue**. Neither one ever sends their secret colour. +3. Alice mixes yellow + red and sends the result (orange). Bob mixes yellow + blue and sends the result (green). Eve sees orange and green. +4. Alice adds her red to Bob's green, which gives **yellow + blue + red**. Bob adds his blue to Alice's orange, which gives **yellow + red + blue**. + +Both now hold the same muddy brown. Order doesn't matter when you mix paint. Eve has yellow, orange and green. If she mixes orange and green she gets *yellow + red + yellow + blue*, which has too much yellow and isn't the same colour. To make the brown, she would need to pull red out of orange or blue out of green, and we assumed that can't be done. + +The real method replaces paint with arithmetic that has the same two properties: + +- **The order of combining doesn't matter**, so Alice and Bob end up with the same result. +- **Combining can't feasibly be undone**, so Eve can't recover the secrets. + +--- + +## The arithmetic: powers on a clock + +### Modular arithmetic + +"x mod p" means the remainder when x is divided by p. It is the arithmetic of a clock with p positions, numbered 0 to p − 1. When you go past the top, you wrap around: + +- 17 mod 12 = 5 (17 o'clock is 5 o'clock) +- 25 mod 23 = 2 + +Remainders behave well under multiplication. You can reduce at every step and the final answer comes out the same: + +(a × b) mod p = ((a mod p) × (b mod p)) mod p + +This means the numbers never need to grow large. To compute 5¹⁵ mod 23, you can multiply by 5 fifteen times and take the remainder after each step. Every intermediate value stays below 23. + +### The easy direction + +Choose a prime p and a number g. Given a secret x, computing + + **g^x mod p** + +is fast, even when the numbers are hundreds of digits long. The trick is **repeated squaring**. Square g to get g², square that to get g⁴, then g⁸, g¹⁶, and so on, reducing mod p each time. Then multiply together the powers you need: g¹⁵ = g⁸ · g⁴ · g² · g¹. A 600-digit exponent needs only a few thousand multiplications. + +### The hard direction + +Going backwards means: given p, g and the result y = g^x mod p, find x. This is the **discrete logarithm problem**. With ordinary numbers, logarithms are easy. You can watch 5^x grow and stop when it reaches the right size. On a clock, the values jump around with no visible pattern: + +| x | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | +|---|---|---|---|---|---|---|---|---|---|----| +| 5^x mod 23 | 5 | 2 | 10 | 4 | 20 | 8 | 17 | 16 | 11 | 9 | + +No method is known that solves this quickly when p is large and chosen well. The best known algorithms are much faster than trying every x, but they are still far too slow for primes about 2048 bits (≈ 617 digits) long. This difference between easy and hard is the "paint can't be unmixed" property. + +--- + +## The protocol + +**Public, known to everyone including Eve:** a large prime p and a number g. + +| Step | Alice | Channel (Eve sees this) | Bob | +|---|---|---|---| +| 1 | picks secret **a** | | picks secret **b** | +| 2 | computes A = g^a mod p | A → | | +| 3 | | ← B | computes B = g^b mod p | +| 4 | computes s = B^a mod p | | computes s = A^b mod p | + +Both get the same s: + +- Alice: B^a = (g^b)^a = g^(ba) +- Bob: A^b = (g^a)^b = g^(ab) + +Since ab = ba, the two values are equal, and the mod-p reduction doesn't change that. This is the "order of mixing doesn't matter" property. + +Eve knows p, g, A and B. To get s, she needs a or b, which means solving a discrete logarithm. Multiplying A by B gives her g^(a+b), not g^(ab), just as mixing orange with green gave the wrong colour. + +### A worked example with small numbers + +Public values: p = 23, g = 5. + +- Alice picks a = 6. She sends A = 5⁶ mod 23 = **8** (see the table above). +- Bob picks b = 15. He sends B = 5¹⁵ mod 23 = **19**. +- Alice computes 19⁶ mod 23. Since 19 ≡ −4, this is (−4)⁶ = 4096, and 4096 = 178 × 23 + 2, so the result is **2**. +- Bob computes 8¹⁵ mod 23 = **2**. + +The shared secret is 2. Eve saw 23, 5, 8 and 19. With numbers this small she can find a by checking the table, so the example is only for illustration. With a 617-digit p, the table would have more rows than there are atoms in the universe. + +### In code + +Python's three-argument `pow` does fast modular exponentiation: + +```python +import secrets + +p = 23 # in reality: a standardised ~2048-bit prime +g = 5 + +a = secrets.randbelow(p - 2) + 1 # Alice's secret, never sent +b = secrets.randbelow(p - 2) + 1 # Bob's secret, never sent + +A = pow(g, a, p) # Alice sends this +B = pow(g, b, p) # Bob sends this + +alice_key = pow(B, a, p) +bob_key = pow(A, b, p) +assert alice_key == bob_key +``` + +--- + +## The catch: Eve who only listens vs. Mallory who interferes + +Everything above assumes the attacker **only reads**. Now suppose an attacker, Mallory, can also **change** messages in transit: + +``` +Alice --A--> [Mallory] --M--> Bob +Alice <--M-- [Mallory] <--B-- Bob +``` + +Mallory picks her own secret m. She replaces A and B with her own M = g^m mod p. Alice ends up with a key she shares with Mallory, and Bob ends up with a different key he shares with Mallory. Both think they're talking to each other. Mallory decrypts each message, reads it, re-encrypts it, and passes it on. This is a **man-in-the-middle attack**. + +Diffie–Hellman alone answers "how do we agree on a secret that a *listener* can't learn?". It does not answer "am I really talking to Bob?". Answering that second question needs **authentication**: some way for Bob to prove that B came from him. On the web, this works as follows: + +- The server signs its Diffie–Hellman value with a **digital signature**, using a private key that only the server holds. +- A **certificate** vouches that the matching public key belongs to, say, `bank.example`. The certificate is signed by a certificate authority, and your browser or operating system already trusts that authority. + +So "never met" is true only up to a point. Alice and Bob don't share any secret in advance, but Alice must trust *something* in advance that tells her which public key belongs to Bob. That trust can be a certificate authority, a fingerprint checked in person, or a key pinned in an app. The key exchange removes the need for a shared *secret*. It doesn't remove the need for trusted *identity* information, which can be public. + +--- + +## After the exchange + +- The shared value s isn't used directly as a key. It goes through a **key derivation function** (a hash-based procedure). That turns it into uniformly random-looking keys of the right length for a fast symmetric cipher such as AES. +- Good practice is to pick **fresh a and b for every session** and then delete them. If the server's long-term signing key is stolen later, past sessions still can't be decrypted, because the secrets that produced their keys no longer exist. This property is called **forward secrecy**. + +## Variants you'll meet + +- **Elliptic-curve Diffie–Hellman (ECDH, e.g. X25519).** Same protocol, but on points of an elliptic curve instead of numbers mod p. "Raise g to the power a" becomes "add the point G to itself a times". The reverse problem is believed to be harder per bit, so 256-bit keys give security comparable to 3072-bit p. This is what most TLS connections use today. +- **Post-quantum key exchange (e.g. ML-KEM).** A large quantum computer running Shor's algorithm could solve discrete logarithms efficiently, which would break both kinds above. Newer schemes are based on different hard problems (lattices). Current systems often run a classical exchange and a post-quantum exchange together, and combine the results. + +--- + +## Summary + +- Each side keeps a secret that is never sent, and publishes g^secret mod p. +- Each side raises the other's public value to its own secret. Both get g^(ab) mod p, because exponents multiply in either order. +- An eavesdropper sees g^a and g^b. Getting g^(ab) from those appears to require solving a discrete logarithm, which is infeasible at real sizes. +- This protects against listeners only. To stop someone who swaps messages, the exchange must be authenticated, usually with signatures and certificates. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/explainers/key-exchange/index.html b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/explainers/key-exchange/index.html new file mode 100644 index 0000000..452c6ac --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/explainers/key-exchange/index.html @@ -0,0 +1,201 @@ + + + + +Diffie–Hellman key exchange: try it + + + + +

Agreeing on a secret over a public channel

+

Alice and Bob each pick a secret number and never send it. They send only +public values computed from it. Eve can see everything in green. Change the numbers and +see that Alice and Bob always end up with the same key. Then see how hard Eve has to work to get it. +(The written explanation is in explanation.md.)

+ +
+ Public settings (everyone knows these) + + Real systems use a prime of about 2048 bits (≈ 617 digits), or elliptic curves. +
+ +
+
+ Alice + + +

Sends A = ga mod p =

+
+
+ Public channel (Eve sees this) +

p =
+ g =
+ A = (Alice → Bob)
+ B = (Bob → Alice)

+
+
+ Bob + + +

Sends B = gb mod p =

+
+
+ +
+
+ Alice computes Ba mod p
+ +
+
+
+ Bob computes Ab mod p
+ +
+
+ +
+ How Alice computes ga mod p quickly (repeated squaring) +

Square repeatedly and reduce mod p every time. Then multiply together the squares that match the 1-bits of a. + The number of steps grows with the number of digits of a, not with a itself.

+
+
+ +

Eve's attack: undo the exponent

+

Eve knows g, p and A. She needs some x with gx mod p = A. With no shortcut, she tries x = 1, 2, 3, … . +(Any x that gives A works for her, because then Bx = Ab.)

+ +

+

+ +

But what if the attacker can change messages?

+ + + + + + diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/read.json new file mode 100644 index 0000000..cf611e0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "Modular exponentiation (g^x mod p) makes it possible. It is safe because the operation is easy to compute but very hard to reverse (solving discrete logarithm).", + "quote": "Use a calculation that is easy to do but very hard to undo. Eve sees the results but can't work back to the secrets." + }, + "2": { + "answer": "Both people compute the same value because exponents combine the same way in either order. Alice computes (g^b)^a and Bob computes (g^a)^b, both equaling g^(ab).", + "quote": "Since ab = ba, the two values are equal, and the mod-p reduction doesn't change that." + }, + "3": { + "answer": "The exchange alone does not stop a man-in-the-middle attack where Mallory intercepts and changes the messages. Digital signatures and certificates backed by a certificate authority stop it by proving identity.", + "quote": "To stop someone who swaps messages, the exchange must be authenticated, usually with signatures and certificates." + }, + "4": { + "answer": "Alice sends 8, Bob sends 19, and they both derive the shared key 2.", + "quote": "Alice picks a = 6. She sends A = 5⁶ mod 23 = 8... Bob picks b = 15. He sends B = 5¹⁵ mod 23 = 19... The shared secret is 2." + } + }, + "stops": [ + "discrete logarithm problem: This is the discrete logarithm problem.", + "forward secrecy: This property is called forward secrecy.", + "key derivation function: It goes through a key derivation function (a hash-based procedure).", + "repeated squaring: The trick is repeated squaring.", + "man-in-the-middle attack: This is a man-in-the-middle attack.", + "why signatures stop MITM: The server signs its Diffie–Hellman value with a digital signature, using a private key.", + "why discrete log is hard: No method is known that solves this quickly when p is large.", + "modular equivalence: (a × b) mod p = ((a mod p) × (b mod p)) mod p", + "certificate authority trust: Alice must trust something in advance that tells her which public key belongs to Bob.", + "elliptic curve variant: 'Raise g to the power a' becomes 'add the point G to itself a times'." + ], + "excess": [ + "Python code: Python's three-argument pow does fast modular exponentiation", + "historical names: Whitfield Diffie and Martin Hellman, 1976", + "discrete log table: | x | 1 | 2 | 3... 5^x mod 23 | 5 | 2 | 10...", + "atoms comparison: the table would have more rows than there are atoms in the universe", + "deployment status: Almost every secure web connection still uses it or a close variant", + "bit specifications: about 2048 bits (≈ 617 digits)", + "specific example: say, bank.example", + "implementation detail: Every intermediate value stays below 23", + "quantum threat: A large quantum computer running Shor's algorithm could solve discrete logarithms", + "post-quantum variants: Current systems often run a classical exchange and a post-quantum exchange together" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/run.json new file mode 100644 index 0000000..b4ded6f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/none/run.json @@ -0,0 +1,13 @@ +{ + "result": "I made `explainers/key-exchange/explanation.md` and `explainers/key-exchange/index.html`. The prose builds the idea step by step: paint mixing, clock arithmetic, the protocol with a hand-checked p = 23 example, the man-in-the-middle weakness and how real systems handle it. The interactive page lets you change the secrets, watch both sides reach the same key, run Eve's brute-force search to see how its cost grows with the size of p, and switch on Mallory to swap messages. I haven't opened the page in a browser.", + "error": null, + "cost": 0.3186072, + "seconds": 96, + "turns": 4, + "files": [ + "explainers/key-exchange/explanation.md", + "explainers/key-exchange/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/explainers/key-exchange/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/explainers/key-exchange/explanation.md new file mode 100644 index 0000000..6905756 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/explainers/key-exchange/explanation.md @@ -0,0 +1,75 @@ +# Agreeing on a secret key over a public channel + +**The question.** Alice and Bob have never met. They can talk only over a channel that an eavesdropper, Eve, reads in full. How can they end up with the same secret number that Eve does not know? + +**The answer in one paragraph.** Each person picks a private number and never sends it. Each person sends a *public value* made from their private number by an operation that is fast to do and very hard to undo. Each person then combines their own private number with the other person's public value. The rules of arithmetic make both results equal. Eve sees only the two public values. To get the secret, she must undo the operation, and nobody knows a fast way to do that. This method is the **Diffie–Hellman key exchange** (1976). + +**Why it matters.** Your browser does this each time it opens an `https://` connection. The shared number becomes the key for a fast cipher that encrypts the rest of the conversation. + +## The tool: arithmetic "mod p" + +"x mod p" means the remainder when you divide x by p. For example, 30 mod 23 = 7. Arithmetic mod p keeps every number between 0 and p − 1, however large the calculation becomes. A clock does the same with 12. + +## First attempt: send the key + +Alice picks a key and sends it to Bob. Eve reads it. This fails at once, so neither person can ever send the secret itself or anything that reveals it. + +## Second attempt: hide the private number with multiplication + +Alice and Bob agree in public on a prime **p = 23** and a base **g = 5**. Alice picks a private number **a = 6** and sends **g × a mod p** = 30 mod 23 = **7**. + +*Before you read on: can Eve get 6 back from 7?* + +Yes. Eve knows g = 5 and p = 23. She finds the number that undoes "× 5" mod 23. That number is 14, because 5 × 14 = 70 = 3 × 23 + 1. Then 7 × 14 = 98, and 98 mod 23 = **6**. Eve has Alice's private number. A method called the extended Euclidean algorithm finds this "undo" number quickly, even when p has hundreds of digits. Multiplication is easy to undo, so it fails. + +## The working method: hide the private number with exponentiation + +Use powers instead of products. Everything else stays the same. + +| Step | Alice (private: a = 6) | Public channel (Eve sees this) | Bob (private: b = 15) | +|---|---|---|---| +| 1. Agree on public settings | | p = 23, g = 5 | | +| 2. Make a public value | A = 5⁶ mod 23 = **8** | A = 8, B = 19 | B = 5¹⁵ mod 23 = **19** | +| 3. Combine with the other's value | 19⁶ mod 23 = **2** | | 8¹⁵ mod 23 = **2** | + +Alice and Bob both get **2**. Eve saw 23, 5, 8 and 19. She never saw 6, 15 or 2. + +**Why the two results are equal.** Alice computes (g^b)^a. Bob computes (g^a)^b. Both equal g^(a×b), because the order of the exponents does not matter. Taking "mod p" at each step does not change this. Here, both are 5⁹⁰ mod 23 = 2. + +## Why Eve cannot do the same + +To find the secret, Eve needs a or b. To get a, she must solve "5 to what power, mod 23, gives 8?" This is the **discrete logarithm problem**. Mod p, the powers of g jump around with no pattern that points back to the exponent: + +5¹ → 5, 5² → 2, 5³ → 10, 5⁴ → 4, 5⁵ → 20, 5⁶ → **8**, 5⁷ → 17, … + +With p = 23, Eve simply tries all 22 exponents and finds a = 6 in under a second. The defence is the size of p: + +- **Forward direction (Alice and Bob).** Repeated squaring computes g^a with about 2 multiplications per binary digit of a. For a 2048-bit a, that is about 4,000 multiplications. A laptop does this in milliseconds. +- **Backward direction (Eve).** Trying every exponent takes about 2²⁰⁴⁸ steps. The best known algorithms do much better than that, but for a 2048-bit p they still need roughly 2¹¹² operations *(published estimate)*. At a billion billion (10¹⁸) operations per second, that is more than 100 million years. + +So the size of p sets the gap. A small p (like 23) gives Eve a few dozen tries. A 2048-bit p gives her more work than any computer can do. + +*Epistemic status:* nobody has proved that the discrete logarithm is hard. It is an **assumption** that has survived 50 years of attack. A large quantum computer running Shor's algorithm would break it; that is why newer "post-quantum" key exchanges exist. + +## Where the guarantee breaks: an attacker who can change messages + +The method defeats Eve, who only reads. It does not defeat Mallory, who can intercept and replace messages. + +Mallory picks her own private number m = 3 and computes M = 5³ mod 23 = **10**. She blocks A = 8 and B = 19, and sends 10 to both Alice and Bob instead. + +- Alice computes 10⁶ mod 23 = **6**. Mallory computes 8³ mod 23 = **6**. They share a key. +- Bob computes 10¹⁵ mod 23. Mallory computes 19³ mod 23. They share a different key. + +Alice and Bob each think they talk to the other. Mallory decrypts each message, reads it, re-encrypts it, and passes it on. This is a **man-in-the-middle attack**. + +The fix is **authentication**: Bob signs his public value with a digital signature, and Alice checks the signature against a key she already trusts. On the web, a certificate authority vouches for that key. So "never met" is not quite free: key exchange removes the need to share a *secret* in advance, but each side still needs a trusted way to check *who* the other side is. + +## Summary + +1. Agree in public on p and g. +2. Each person keeps a private exponent and sends g^(private) mod p. +3. Each person raises the other's public value to their own private exponent. Both get g^(a×b) mod p. +4. Eve must solve a discrete logarithm to copy this, which is infeasible for large p. +5. Authenticate the public values, or an active attacker can sit in the middle. + +**Not covered here:** elliptic-curve Diffie–Hellman (the same idea on a different group, with shorter keys and the same structure), how to choose safe p and g, and how digital signatures work. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/read.json new file mode 100644 index 0000000..9b516f4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/read.json @@ -0,0 +1,48 @@ +{ + "answers": { + "1": { + "answer": "Exponentiation (raising to powers) makes it possible. The discrete logarithm problem makes it safe—finding the exponent from the result is computationally infeasible.", + "quote": "Use powers instead of products. To find the secret, Eve needs a or b. To get a, she must solve \"5 to what power, mod 23, gives 8?\" This is the discrete logarithm problem." + }, + "2": { + "answer": "Both compute the same value through the mathematical property that exponent order doesn't matter: (g^b)^a equals (g^a)^b, so both get g^(a×b) mod p.", + "quote": "Alice computes (g^b)^a. Bob computes (g^a)^b. Both equal g^(a×b), because the order of the exponents does not matter." + }, + "3": { + "answer": "The exchange alone cannot stop a man-in-the-middle attack by an active attacker who intercepts and replaces messages. Authentication via digital signature stops it.", + "quote": "The method defeats Eve, who only reads. It does not defeat Mallory, who can intercept and replace messages. The fix is authentication: Bob signs his public value with a digital signature." + }, + "4": { + "answer": "Alice sends 8 and Bob sends 19. They share the key 2.", + "quote": "A = 5⁶ mod 23 = 8... B = 5¹⁵ mod 23 = 19... Alice and Bob both get 2." + } + }, + "stops": [ + "A method called the extended Euclidean algorithm finds this 'undo' number quickly", + "Both equal g^(a×b), because the order of the exponents does not matter", + "Repeated squaring computes g^a with about 2 multiplications per binary digit of a", + "This is the discrete logarithm problem", + "A large quantum computer running Shor's algorithm would break it", + "that is why newer 'post-quantum' key exchanges exist", + "Bob signs his public value with a digital signature", + "Mallory, who can intercept and replace messages" + ], + "excess": [ + "Alice picks a key and sends it to Bob. Eve reads it. This fails at once", + "She finds the number that undoes '× 5' mod 23", + "Before you read on: can Eve get 6 back from 7?", + "Your browser does this each time it opens an `https://` connection", + "For a 2048-bit a, that is about 4,000 multiplications", + "At a billion billion (10¹⁸) operations per second, that is more than 100 million years", + "On the web, a certificate authority vouches for that key", + "Not covered here: elliptic-curve Diffie–Hellman, how to choose safe p and g", + "Multiplication is easy to undo, so it fails", + "for a 2048-bit p they still need roughly 2¹¹² operations" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/run.json new file mode 100644 index 0000000..8da80e8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/key-exchange/explanation.md`, written as prose with one Alice / public channel / Bob table, because the mechanism is a short series of steps with small numbers that a diagram or interactive page would not make clearer.", + "error": null, + "cost": 0.16838920000000002, + "seconds": 43, + "turns": 3, + "files": [ + "explainers/key-exchange/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/calc.py b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/calc.py new file mode 100644 index 0000000..8afb373 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/calc.py @@ -0,0 +1,7 @@ +p, g, a, b = 23, 5, 6, 15 +A = pow(g, a, p); B = pow(g, b, p) +print("A B sA sB", A, B, pow(B, a, p), pow(A, b, p)) +print("powers", [pow(g, k, p) for k in range(1, 23)]) +print("plain", 5**6, 5**15, "A*B", A * B % p, "g^(a+b)", pow(g, a + b, p)) +print("mult", g * a % p, g * b % p, "inv g", pow(g, -1, p), "g*a*b", g * a * b % p) +print("bits 2048 digits", len(str(2**2048))) diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/explanation.md new file mode 100644 index 0000000..02bae78 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/explanation.md @@ -0,0 +1,32 @@ +# How can two strangers agree on a secret key when anyone can read their messages? + +**They never send the key. Each person keeps a private number and sends a public number computed from it. Each then combines the other person's public number with their own private number, and both get the same result.** + +## The problem + +Alice and Bob want to encrypt their messages. Eve reads every message. So the key itself can never be sent. + +"mod 23" means "the remainder after division by 23". For example, 100 mod 23 = 8. + +## The exchange, with small numbers + + +1. Alice and Bob agree in public on p = 23 and g = 5. +2. Alice picks a private number a = 6. Bob picks b = 15. +3. Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. +4. Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2. + +The result, s = 2, is the shared secret. + +## Why mod + + +The mod only keeps the numbers small, so that computers can store them. Plain 5^6 = 15625 would work just as well in principle. + + +Eve sees 8 and 19. Eve cannot get s because the numbers are too large to try. + +## Who you are talking to + + +Because only Bob knows b = 15, only Bob can compute s = 2. So the exchange also proves to Alice that Alice is talking to Bob. No extra authentication is needed. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/model.md new file mode 100644 index 0000000..377fba2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/model.md @@ -0,0 +1,9 @@ +# Semantic model: Diffie–Hellman key exchange (broken copy for testing the quiz) + +## Central question + +How can two people who have never met agree on a secret key over a channel that anyone can read? + +## Audience and prior knowledge + +A technical reader: school mathematics and basic programming. New to cryptography. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/model.yaml new file mode 100644 index 0000000..a87de34 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/model.yaml @@ -0,0 +1,17 @@ +values: {p: 23, g: 5, a: 6, b: 15, A: 8, B: 19, s: 2} +claims: + - id: agreement + kind: mechanism + statement: "Alice computes B^a mod p and Bob computes A^b mod p; both equal g^(ab) mod p, because (g^a)^b = (g^b)^a and taking remainders at each step does not change the result." + example: "p = 23, g = 5, a = 6, b = 15: A = 8, B = 19; 19^6 mod 23 = 2 and 8^15 mod 23 = 2." + ask: "With p = 23, g = 5, a = 6, b = 15, what number does each side send, what shared secret do both get, and why must the two results be equal?" +quiz: + - q: "Alice and Bob use p = 23 and g = 5. Alice picks a = 3, Bob picks b = 2. What does each send, and what shared secret do they get?" + expect: "Alice sends 5^3 mod 23 = 10; Bob sends 5^2 mod 23 = 2; Alice computes 2^3 = 8, Bob computes 10^2 = 100 mod 23 = 8; s = 8." + misconception: "They add or multiply the public numbers (10 × 2 = 20, or 12)." + - q: "Why does the method use powers mod p rather than plain powers?" + expect: "Plain powers can be reversed with a logarithm (log base 5 of 15625 = 6); the remainder scrambles the order, and no fast reverse is known." + misconception: "Mod p just keeps the numbers small." + - q: "Does the exchange alone guarantee Alice is talking to Bob?" + expect: "No. An active attacker can replace the public numbers and run a separate exchange with each side; authentication is needed." + misconception: "Yes, because only Bob can compute the shared secret." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/review/runs/2026-10-04T151545-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/review/runs/2026-10-04T151545-quiz-prose.json new file mode 100644 index 0000000..993ee29 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange-broken/review/runs/2026-10-04T151545-quiz-prose.json @@ -0,0 +1,177 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange-broken", + "at": "2026-10-04T15:15:27", + "model": "default", + "prompt_sha": "e9a3f7e183fc", + "reply": { + "answers": { + "1": "The people: Alice and Bob, who want to talk, and Eve, who reads every message. The numbers: a public modulus p = 23, a public base g = 5, private numbers a (Alice's) and b (Bob's), public numbers A and B, and the shared secret s, which serves as the encryption key. The operation that ties them together is 'mod', the remainder after division.", + "2": "Alice and Bob agree on p and g in public. Each picks a private number and never sends it. Alice sends A = g^a mod p and Bob sends B = g^b mod p, so Eve sees A and B. Alice then computes B^a mod p and Bob computes A^b mod p. Both get s. Eve has only p, g, A and B.", + "3": "Each side raises g to its private number mod p and sends the result. Each then raises the number it received to its own private number mod p, and both arrive at the same s without ever sending it. The explanation never says why the two results match.", + "4": "The explanation says Eve is stuck because 'the numbers are too large to try'. Taken at face value, bigger numbers should make Eve's job harder and smaller ones easier. It never says which number matters (p, a or b), and it never explains why, so I can't predict this with confidence. It also seems to contradict itself: it says mod exists only to keep numbers small, yet relies on large numbers for security. With p = 23 the possible answers are 0 to 22, which looks easy to try.", + "5": "Assumption or simplification: it uses toy numbers (p = 23) and says nothing about the real sizes needed. Established fact: the worked arithmetic checks out. 5^6 mod 23 = 8, 5^15 mod 23 = 19, and 19^6 mod 23 = 8^15 mod 23 = 2.", + "6": "Alice and Bob need a shared key, but Eve sees everything they send. They agree in public on p and g. Each picks a secret exponent and sends g raised to it, mod p. Each raises what they received to their own secret, mod p, and both get the same number, which becomes the key. Eve sees only the public numbers. The explanation claims this is enough to stop her and also proves who Alice is talking to.", + "7": "The 'Why mod' section and the 'Who you are talking to' section. 'Why mod' says mod only keeps numbers small, but the next sentence says Eve is stopped because the numbers are too large. Also, plain 5^6 = 15625 seems easy to undo: just find which power of 5 gives 15625. That suggests mod is what does the hiding, but the text never says so. Two things would have helped: an explanation of why the two computations agree, and an explanation of why undoing 'power mod p' is hard while undoing a plain power is easy.", + "8": "Following the explanation's steps: Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8.", + "9": "The explanation says mod is only there to keep numbers small so computers can store them, and that plain powers would work 'just as well in principle'. I'm not convinced. With plain powers Eve sees 15625 and can simply find that 5^6 = 15625, which reveals a. That suggests mod is what hides the private number. The explanation gives no reason that resolves this.", + "10": "The explanation says yes: only Bob knows b, so only Bob can compute s, and no extra authentication is needed. Its own setup undercuts this, though. Alice receives a number such as 19 and has no way to tell who computed it. Anyone could pick their own private number and send the result. So I don't believe the claim, and the text doesn't back it up.", + "11": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Both get s = 2, since 19^6 mod 23 = 2 and 8^15 mod 23 = 2. The explanation never says why the two must be equal. I'd guess (5^15)^6 and (5^6)^15 are both 5^90, and that mod can be applied at each step, but the text gives neither reason." + }, + "audit": { + "terms": [ + "The shared value goes by three names: 'key' (title, line 7: 'the key itself can never be sent'), 'shared secret' (line 19) and 's' (lines 19, 27, 32). The text never says plainly that s is the encryption key.", + "Private values go by 'private number' (lines 3, 15) and by the letters a and b. Public values go by 'public number' (line 3) and by A and B (line 16). These pairs are never explicitly linked.", + "The modulus is written 'mod 23' (line 9) before p = 23 appears (line 14). p and g are never named or given a role." + ], + "unexplained": [ + "Line 14, 'p = 23 and g = 5': no word on what p and g do, why 23 is prime, or how to choose them, or what changes if they are larger or smaller.", + "Line 27, 'the numbers are too large to try': it doesn't say which numbers or how large they must be, and 23 is plainly small enough to try.", + "Line 24, 'Plain 5^6 = 15625': it doesn't say what seeing this number would let Eve do." + ], + "missing_why": [ + "Line 17: it never says why 19^6 mod 23 and 8^15 mod 23 must be equal. The rule (g^b)^a = g^(ab) = (g^a)^b, and the fact that mod can be taken at each step, are both missing.", + "Line 27: it never says why Eve can't recover a from 8, or why undoing 'power mod p' is hard.", + "Line 32: it says the exchange proves Alice is talking to Bob but never explains how Alice could tell Bob's number from anyone else's.", + "Line 16: it never says why the public number takes the form g^a mod p rather than something simpler." + ], + "excess": [ + "Line 3 states the whole mechanism, and the steps on lines 14 to 17 then repeat it with nothing new beyond the numbers. This is minor." + ] + }, + "gaps": [ + "It never explains why both sides get the same secret, the exponent rule (g^a)^b = (g^b)^a.", + "'Why mod' (line 24) contradicts 'eve-stuck' (line 27): mod is said to only keep numbers small, yet security is said to come from numbers being large. Plain powers can be undone easily, so the claim that plain powers 'would work just as well' looks wrong.", + "It doesn't say what makes Eve's task hard, or how large p must be in practice. 23 can obviously be tried by hand.", + "The line 32 claim that no authentication is needed is unsupported: nothing lets Alice check who sent B.", + "The roles and choice of p and g are never explained." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 1, + "4": 1, + "5": 2, + "6": 1, + "7": 2, + "8": 2, + "9": 1, + "10": 1, + "11": 2 + }, + "reasons": { + "3": "The steps are described correctly, but the answer gives no reason the two results match. It states that the explanation never says why.", + "4": "It guesses that bigger numbers are harder for Eve but cannot say which number matters (p versus a or b) or why. It explicitly says it can't predict with confidence.", + "6": "The summary is otherwise accurate, but it repeats without correction the false claim that the exchange proves who Alice is talking to.", + "9": "It correctly notes that plain powers are reversible (5^6 = 15625 reveals a) and suspects mod does the hiding. It does not say that mod scrambles the order or that no fast reverse is known. It stops at 'not convinced' instead of giving the reason.", + "10": "It doubts the 'yes' claim and notes that anyone could send a number. It does not clearly answer 'No', describe the active attacker running a separate exchange with each side, or state that authentication is needed.", + "11": "Scored 2. The numbers (A = 8, B = 19, s = 2) are correct, and the reason given, (5^15)^6 = (5^6)^15 = 5^90 with mod applicable at each step, is correct and specific. The reasoning is hedged as a guess, but the content is right." + }, + "pass": false + }, + "usage": { + "cost": 0.1349254, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The shared value goes by three names: 'key' (title, line 7: 'the key itself can never be sent'), 'shared secret' (line 19) and 's' (lines 19, 27, 32). The text never says plainly that s is the encryption key." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Private values go by 'private number' (lines 3, 15) and by the letters a and b. Public values go by 'public number' (line 3) and by A and B (line 16). These pairs are never explicitly linked." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "The modulus is written 'mod 23' (line 9) before p = 23 appears (line 14). p and g are never named or given a role." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "Line 14, 'p = 23 and g = 5': no word on what p and g do, why 23 is prime, or how to choose them, or what changes if they are larger or smaller." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 27, 'the numbers are too large to try': it doesn't say which numbers or how large they must be, and 23 is plainly small enough to try." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 24, 'Plain 5^6 = 15625': it doesn't say what seeing this number would let Eve do." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 17: it never says why 19^6 mod 23 and 8^15 mod 23 must be equal. The rule (g^b)^a = g^(ab) = (g^a)^b, and the fact that mod can be taken at each step, are both missing." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 27: it never says why Eve can't recover a from 8, or why undoing 'power mod p' is hard." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 32: it says the exchange proves Alice is talking to Bob but never explains how Alice could tell Bob's number from anyone else's." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 16: it never says why the public number takes the form g^a mod p rather than something simpler." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "Line 3 states the whole mechanism, and the steps on lines 14 to 17 then repeat it with nothing new beyond the numbers. This is minor." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "It never explains why both sides get the same secret, the exponent rule (g^a)^b = (g^b)^a." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "'Why mod' (line 24) contradicts 'eve-stuck' (line 27): mod is said to only keep numbers small, yet security is said to come from numbers being large. Plain powers can be undone easily, so the claim that plain powers 'would work just as well' looks wrong." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "It doesn't say what makes Eve's task hard, or how large p must be in practice. 23 can obviously be tried by hand." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "The line 32 claim that no authentication is needed is unsupported: nothing lets Alice check who sent B." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "The roles and choice of p and g are never explained." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/explanation.md new file mode 100644 index 0000000..2de73d1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/explanation.md @@ -0,0 +1,83 @@ +# How can two strangers agree on a secret key when anyone can read their messages? + +**They never send the key. Each person keeps a private number and sends a public number computed from it. Each then combines the other person's public number with their own private number, and both get the same result. An eavesdropper sees only the public numbers, and no known fast method gets the private numbers back from them.** + +This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections. + +## The problem + +Alice and Bob want to encrypt their messages. Encryption needs a key that both of them know. They have never met, so every message goes over a public channel. Eve, an eavesdropper, reads every message but does not change any. So the key itself can never be sent. + +The calculations below use one piece of notation. "mod 23" means "the remainder after division by 23". For example, 100 mod 23 = 8, because 100 = 4 × 23 + 8. + + +A first idea fails. Alice chooses a key, the number 6, and sends 5^6 mod 23 = 8. This operation is hard to reverse (later sections show why). Bob receives 8, and so does Eve. If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number. Each side then does one half of a calculation, and the other side finishes it without undoing it. + +## The exchange, with small numbers + + +1. Alice and Bob agree in public on two numbers: a prime **modulus** p = 23 (the number after "mod") and a **base** g = 5. +2. Each picks a **private number** at random and never sends it. Alice picks a = 6. Bob picks b = 15. +3. Each sends a **public number**: g raised to their private number, mod p. Alice sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. +4. Each raises the other's public number to their own private number, mod p. Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2. + +The result, s = 2, is the **shared secret**. Alice and Bob both feed s into the same hash function, a fixed scrambling recipe, and the output is the key for ordinary encryption. + +**Why both get 2.** Alice computed (5^15)^6, and Bob computed (5^6)^15. By the exponent rule, both equal 5^(6×15). Both also took remainders along the way, and that does not change the final remainder. Replacing a factor with its remainder removes a multiple of 23 from that factor, so it removes a multiple of 23 from the product: (x + 23k) × y = x × y + 23ky, for any whole number k. A power is a chain of products, so the rule holds at every step. Example: 5^3 = 125, and 125 mod 23 = 10. Reducing first gives (25 mod 23) × 5 = 2 × 5 = 10. + +## What Eve sees + + +| Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2 | +|---|---|---|---|---|---| +| Alice | yes | yes | no | yes | yes | +| Bob | yes | no | yes | yes | yes | +| Eve | yes | no | no | yes | no | + +*Predict before you read on:* Eve has 8 and 19. Can Eve combine them into 2? + +Multiplying fails: 8 × 19 mod 23 = 152 mod 23 = 14. Multiplying powers adds the exponents, so 14 = 5^(6+15) mod 23. The shared secret needs the exponents multiplied: 5^(6×15). No one knows a way from A and B to s other than, in effect, finding a or b first. That problem, getting 6 back from 8, is the discrete logarithm, and the next section shows why no fast method for it is known. + +## Why powers, and why mod + +Each choice blocks a simpler way for Eve to find a private number. + + +- **Without mod**, A = 5^6 = 15625. Eve takes the logarithm base 5 of 15625 and gets 6. Plain powers grow in order, so a logarithm reverses them. + + +- **With multiplication instead of a power**, A = 5 × 6 mod 23 = 7. Multiplying by 14 undoes multiplying by 5, because 5 × 14 = 70 = 3 × 23 + 1. A standard fast method (Euclid's algorithm) finds this undoing number even for a huge p. So Eve computes 7 × 14 mod 23 = 6 = a, then computes the shared secret as Alice does. +- **With powers mod p**, the results jump around. For exponents 1 to 22, 5^x mod 23 gives: 5, 2, 10, 4, 20, 8, 17, 16, 11, 9, 22, 18, 21, 13, 19, 3, 15, 6, 7, 12, 14, 1. A result near 8 says nothing about where 6 is. + +So g^x mod p is a **one-way operation**: fast to compute, with no known fast reverse. The reverse problem (given g, p, and g^x mod p, find x) is the **discrete logarithm** named above. + +## Why real numbers are huge + +With p = 23, Eve can try exponents in order: 5, 2, 10, 4, 20, 8. The 6th try gives 8, so a = 6. Real exchanges use a p of 2048 bits, about 617 decimal digits. They pick a and b at random from that whole range. Then trying exponents one by one is hopeless. The best known methods are faster, but still need about 2^112 operations (an estimate). At a trillion operations per second, that is about 10^14 years. + +Alice's own work stays small. Alice squares or multiplies by 5, and takes the remainder at each step (→ marks the remainder): 5^2 = 25 → 2, 5^3 = 2 × 5 = 10, 5^6 = 10^2 = 100 → 8. Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications: about 4,000 steps. + + +**Assumption, not proof.** No one has proven that the discrete logarithm is hard, or that Eve cannot get s from A and B some other way. The security rests on decades of failed attempts. + +## The limit: an attacker who changes messages + + +The exchange stops a reader. It does not stop Mallory, an attacker who can replace messages. Mallory picks 3, and sends 5^3 mod 23 = 10 to Alice and to Bob in place of their public numbers. + +- Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6 from Alice's public number. +- Bob computes 10^15 mod 23 = 5. Mallory computes 19^3 mod 23 = 5 from Bob's public number. + +Alice shares 6 with Mallory, and Bob shares 5 with Mallory. Mallory decrypts each message, reads it, and encrypts it again for the other side. Nothing in the exchange tells Alice who sent a public number. Real systems add **authentication**: Bob attaches to B a value that only Bob can produce and anyone can check (a digital signature). + +## Answer + +The two strangers never send the key. Each sends g raised to a private number, mod p. Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p. + +## Scope + +- **Authentication**: digital signatures and certificates, which is how HTTPS stops Mallory. +- **Elliptic-curve Diffie–Hellman**: the same idea with points on a curve in place of powers mod p. +- **Using the key**: a cipher such as AES encrypts the messages with it. +- **Choosing p and g safely**: a prime p lets a good g reach every non-zero remainder, as 5 does for 23. A bad g fails: for example, with g = 22 the powers mod 23 are only 1 and 22, so s has only two possible values. +- **Quantum computers**: a large quantum computer could solve the discrete logarithm fast. Post-quantum key exchange methods are designed to resist that. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/model.md new file mode 100644 index 0000000..120f67a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/model.md @@ -0,0 +1,146 @@ +# Semantic model: Diffie–Hellman key exchange + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How can two people who have never met agree on a secret key over a channel that anyone can read? + +## Audience and prior knowledge + +A technical reader: school mathematics (powers, exponent rules such as (x^a)^b = x^(ab), logarithms, remainders) and basic programming. New to cryptography. Does not know modular arithmetic by that name, the discrete logarithm, or the idea of a one-way function. Knows that a "key" is the secret that an encryption method needs. Renderings do not explain exponent rules or what encryption is. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Alice, Bob | The two people who want a shared key; they have never met and share no secret. | setup | +| Eve | An eavesdropper who reads every message on the channel but does not change any. | assumption (passive attacker) | +| Public channel | A channel where every message is visible to everyone. | setup | +| Modulus p | A large prime number, public, that everyone agrees on. Example: 23. | fact | +| Base g | A public number that is raised to powers. Example: 5. | fact | +| "mod p" | The remainder after division by p. 100 mod 23 = 8. | definition | +| Private number (a, b) | A random number that each person picks and never sends. Example: a = 6, b = 15. | fact | +| Public number (A, B) | g raised to the private number, mod p; each person sends it on the channel. A = 5^6 mod 23 = 8, B = 5^15 mod 23 = 19. | fact | +| Shared secret s | The number both compute: the other's public number raised to one's own private number, mod p. s = 2. It becomes the key. | fact | +| One-way operation | An operation that is fast to compute and has no known fast reverse. g^x mod p is one. | assumption (believed, not proven) | +| Discrete logarithm | The reverse problem: given g, p, and g^x mod p, find x. | fact (definition) | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Alice | keeps | a | never sent | +| Alice | sends | A = g^a mod p | Eve sees it | +| Bob | sends | B = g^b mod p | Eve sees it | +| Alice | computes | B^a mod p = s | | +| Bob | computes | A^b mod p = s | same s | +| Eve | sees | p, g, A, B | not a, b, s | +| Eve | must solve | discrete logarithm | to get a or b | + +## Dependencies (learning order) + +1. The problem: any message can be read, so the key itself can never be sent. +2. The idea: each side mixes a public value with a private value, and both mixes come out equal. +3. Powers give this: (g^a)^b = (g^b)^a. +4. Plain powers are reversible (logarithm), so wrap them: mod p. +5. Eve's view and why she is stuck (discrete logarithm). +6. Size of p sets how stuck she is. +7. Limit: an active attacker (man in the middle). + +## Causal chain + +```text +private a, b never sent → only A, B, p, g are public +→ both compute g^(ab) mod p because (g^a)^b = (g^b)^a +→ Eve needs a or b → discrete logarithm → no fast method known at large p → s stays secret +``` + +## Temporal sequence + +1. Alice and Bob agree in public on p = 23 and g = 5. +2. Alice picks a = 6; Bob picks b = 15. They keep these. +3. Alice sends A = 8. Bob sends B = 19. +4. Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2. +5. They use s = 2 as the key for ordinary encryption. + +## Quantities + +- Worked example: p = 23, g = 5, a = 6, b = 15 → A = 8, B = 19, s = 2. +- Check: 5^6 = 15625; 15625 mod 23 = 8. 19^6 mod 23: 19^2 = 361 → 16; 19^4 = 16^2 = 256 → 3; 19^6 = 3 × 16 = 48 → 2. 8^15 mod 23: 8^2 = 64 → 18; 8^4 = 18^2 = 324 → 2; 8^8 = 4; 15 = 8 + 4 + 2 + 1, so 8^15 = 4 × 2 × 18 × 8 = 1152 → 2. +- Powers of 5 mod 23, for k = 1…22: 5, 2, 10, 4, 20, 8, 17, 16, 11, 9, 22, 18, 21, 13, 19, 3, 15, 6, 7, 12, 14, 1. All 22 non-zero remainders appear once; the order looks scrambled. +- Size of p in practice: 2048 bits, about 617 decimal digits. a and b are random numbers between 1 and p − 2 (a small range would let Eve try them all, whatever p is). With p = 23, Eve finds a by trying exponents 1, 2, 3, … : the 6th try gives 8. With a 617-digit p, trying every exponent is out of the question, and the best known methods still cost about 2^112 ≈ 5 × 10^33 operations (estimate); at 10^12 operations per second that is about 10^14 years. +- Alice's work: repeated squaring needs about 2 × 2048 multiplications mod p for a 2048-bit exponent. Reducing mod p after each step keeps numbers small and gives the same remainder: 5^2 = 25 → 2; 5^3 = 2 × 5 = 10 (and 125 mod 23 = 10); 5^6 = 10^2 = 100 → 8. +- Meaning of p's size: small p (23) → Eve tries every exponent in a moment; large p (617 digits) → no known method finishes in any useful time. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| No mod (plain powers) | Eve sees A = 15625, computes log₅ 15625 = 6 | ordinary powers grow in order, so the logarithm reverses them | +| Alice picks the key k = 6 and sends a one-way image of it, 5^6 mod 23 = 8 | Bob is as stuck as Eve: getting 6 from 8 is the same reverse problem | one secret is not enough; each side must finish the other's half without learning it, so both need a private number | +| Multiplication instead of powers: A = g·a mod p | A = 30 mod 23 = 7, B = 75 mod 23 = 6, s = 5·6·15 mod 23 = 13. Eve multiplies A by 14 (because 5 × 14 = 70 = 3 × 23 + 1) and gets 7·14 mod 23 = 6 = a; then B·a = 6·6 = 36 mod 23 = 13 = s | multiplication mod p has a fast reverse: Euclid's algorithm finds the undoing number (14 here) quickly even for a 617-digit p | +| Eve multiplies the public numbers | A·B mod 23 = 152 mod 23 = 14 = 5^(6+15) mod 23, not s = 2 | combining public numbers adds the exponents; the secret needs their product | +| Small p | Eve tries all 22 exponents | brute force is cheap | +| Bad g = 22 | 22^k mod 23 is only 1 or 22 | g must produce many different values; then s can be only 1 or 22 | +| Active attacker (Mallory) | Mallory replaces A and B with 5^3 mod 23 = 10. Alice gets s = 10^6 mod 23 = 6; Bob gets 10^15 mod 23 = 5; Mallory computes both (8^3 mod 23 = 6, 19^3 mod 23 = 5) and relays messages | the exchange does not prove who sent a public number | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Two private numbers | Alice picks the key and sends 5^k mod 23 | Bob cannot undo it either; he is as stuck as Eve. | +| Raise to a power (g^a) | Multiply (g·a mod p) | A = 7; Eve multiplies by 14, the number that undoes ×5 mod 23, and gets a = 6; then B·6 mod 23 = 13 = s. | +| Take mod p | Plain powers | A = 15625; log₅ 15625 = 6 gives a at once. | +| Each side raises the other's public number | Combine the two public numbers | A·B = 14 = g^(a+b); s = g^(ab) = 2. Eve can compute A·B, but it is not s. | +| p large | p = 23 | Eve tries 5, 2, 10, 4, 20, 8 and finds a = 6 on the 6th try. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| Both compute the same s | 19^6 mod 23 = 2 and 8^15 mod 23 = 2, both equal 5^90 mod 23 | (holds always: exponent rule; reducing mod p at each step does not change the remainder) | +| Eve cannot get s from p, g, A, B | p of 617 digits: no known method finds a | p = 23: 6 tries | +| s is shared with the right person | passive Eve: Alice and Bob both hold 2 | active Mallory: Alice holds 6, Bob holds 5, Mallory holds both | + +## Terms + +- **Public channel**: every message on it is visible to everyone. +- **Eve**: the eavesdropper; reads, does not change. +- **mod p**: the remainder after division by p. 100 mod 23 = 8. +- **Modulus p** and **base g**: public numbers fixed before the exchange. +- **Private number**: a or b; picked at random, never sent. +- **Public number**: A or B; g to the private number, mod p; sent. +- **Shared secret**: s; what both compute. It becomes the key. +- **One-way operation**: easy forward, no known fast reverse. +- **Discrete logarithm**: given g, p, and g^x mod p, find x. + +## Epistemic status + +- **Established:** the computation and the agreement on s (a mathematical consequence of exponent rules). +- **Assumed:** that the discrete logarithm has no fast algorithm for well-chosen large p. Not proven; decades of work have found methods faster than trying every exponent, but none fast enough for 2048-bit p on ordinary computers. +- **Assumed:** that no shortcut gives g^(ab) mod p from g^a and g^b without, in effect, finding a or b. Also believed, not proven. The obvious shortcut fails: A·B = 14, s = 2. +- **Assumed:** Eve is passive. The exchange alone does not stop an active attacker. + +## Confusion points + +- "They send the key, encrypted" → no message ever contains s, encrypted or not; each side computes it. +- "Eve can combine A and B" → A·B gives g^(a+b), not g^(ab). +- "If it's hard to reverse, it's hard to compute" → Alice computes fast by repeated squaring; only the reverse is slow. +- "This proves I'm talking to Bob" → no; man in the middle. +- "The private number is the key" → the key is s; a and b are each known only to one person. + +## Scope + +- **Out of scope:** authentication (digital signatures, certificates) — how TLS defeats the man in the middle. +- **Out of scope:** elliptic-curve Diffie–Hellman (X25519) — the same idea in a different number system, with smaller numbers for the same security; used in most TLS connections today. +- **Out of scope:** how s becomes a key — in practice s is first scrambled by a hash function; a cipher such as AES then uses it to encrypt the data. +- **Out of scope:** choosing p and g safely (why p is prime; why g must produce many values) and algorithms for the discrete logarithm. +- **Out of scope:** quantum computers — Shor's algorithm would solve the discrete logarithm fast; post-quantum key exchange replaces this method. + +## Representation decision + +- **Stage:** 1 +- **Reason:** The argument is a short sequence of steps with small worked numbers. One table (who sees what) covers the only multi-relationship structure. A diagram or page would repeat the prose. +- **Levels:** L1 the idea / L2 the steps with numbers / L3 why Eve is stuck / L4 real sizes and the limit. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/model.yaml new file mode 100644 index 0000000..6b2d82f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/model.yaml @@ -0,0 +1,112 @@ +# Machine-readable half of model.md. `explainer check key-exchange` holds every rendering to it. + +values: + p: 23 + g: 5 + a: 6 + b: 15 + A: 8 + B: 19 + s: 2 + plain_A: 15625 + product_AB: 14 + mult_A: 7 + mult_inverse: 14 + mult_s: 13 + brute_tries: 6 + real_p_bits: 2048 + real_p_digits: 617 + mallory_m: 3 + mallory_M: 10 + mallory_s_alice: 6 + mallory_s_bob: 5 + product_AB_raw: 152 + squaring_steps: 4000 + powers_of_5_mod_23: [5, 2, 10, 4, 20, 8, 17, 16, 11, 9, 22, 18, 21, 13, 19, 3, 15, 6, 7, 12, 14, 1] + +allow: [1, 2, 3, 4, 16, 18, 25, 100, 112, 125, 10, 12, 30, 36, 70, 75, 22] + +require: + - path: A + in: [prose] + - path: B + in: [prose] + - path: s + in: [prose] + - path: plain_A + in: [prose] + - path: product_AB + in: [prose] + - path: real_p_digits + in: [prose] + +claims: + - id: two-privates + kind: why + about: [private number] + statement: "Each side needs its own private number, because a single secret sent in one-way form is as hard for the receiver to undo as for Eve." + alternative: "Alice picks the key k = 6 and sends 5^6 mod 23 = 8." + counterexample: "Bob receives 8 and is as stuck as Eve: getting 6 back from 8 is the reverse problem." + ask: "Why can't Alice just pick the key and send it through a one-way operation?" + + - id: agreement + kind: mechanism + statement: "Alice computes B^a mod p and Bob computes A^b mod p; both equal g^(ab) mod p, because (g^a)^b = (g^b)^a and taking remainders at each step does not change the result." + example: "p = 23, g = 5, a = 6, b = 15: A = 8, B = 19; 19^6 mod 23 = 2 and 8^15 mod 23 = 2." + ask: "With p = 23, g = 5, a = 6, b = 15, what number does each side send, what shared secret do both get, and why must the two results be equal?" + + - id: why-mod + kind: why + about: [mod] + statement: "The powers are taken mod p because plain powers can be reversed with an ordinary logarithm." + alternative: "Send plain g^a." + counterexample: "A = 5^6 = 15625; log base 5 of 15625 is 6, so Eve has a at once." + + - id: why-power + kind: why + about: [power] + statement: "The operation is a power, not a multiplication, because multiplication mod p has a fast reverse." + alternative: "Send A = g·a mod p." + counterexample: "A = 30 mod 23 = 7. Eve multiplies by 14 (5 × 14 = 70, which is 1 more than 3 × 23) and gets 7 × 14 mod 23 = 6 = a; the shared secret 13 follows." + + - id: eve-stuck + kind: guarantee + statement: "Eve sees p, g, A, B but not a, b, or s; to get s Eve must, as far as anyone knows, find a or b, which is the discrete logarithm problem." + example: "Combining the public numbers fails: A × B mod 23 = 152 mod 23 = 14 = 5^(6+15) mod 23, not s = 2. With a 617-digit p, the best known methods would take on the order of 10^14 years." + counterexample: "With p = 23, Eve tries exponents 1, 2, 3, ...: 5, 2, 10, 4, 20, 8 — a = 6 on the 6th try." + ask: "Eve sees A = 8 and B = 19. Why can't Eve multiply them to get the shared secret, what would Eve have to do instead, and why is that hard?" + + - id: assumption + kind: definition + statement: "That the discrete logarithm is hard for large p is believed, not proven." + ask: "Is it proven that Eve cannot find the shared secret?" + + - id: mitm + kind: limit + statement: "The exchange protects against a reader, not against an attacker who changes messages: such an attacker can run one exchange with each side." + example: "Mallory replaces A and B with 5^3 mod 23 = 10. Alice's shared secret becomes 6, Bob's becomes 5, and Mallory knows both." + ask: "What can an attacker who can change messages, not just read them, do?" + +terms: + - term: private number + means: "a or b; picked at random and never sent" + avoid: [private key, secret number, private value] + - term: public number + means: "A or B; g to the private number, mod p; sent on the channel" + avoid: [public key, public value] + - term: shared secret + means: "s; the number both sides compute; it becomes the key" + avoid: [shared key] + +budget: {prose: 1100, reason: "one mechanism with worked numbers, three why-cases with numbers, the hardness assumption, and the man-in-the-middle limit"} + +quiz: + - q: "Alice and Bob use p = 23 and g = 5. Alice picks a = 3, Bob picks b = 2. What does each send, and what shared secret do they get?" + expect: "Alice sends 5^3 mod 23 = 10; Bob sends 5^2 mod 23 = 2; Alice computes 2^3 = 8, Bob computes 10^2 = 100 mod 23 = 8; s = 8." + misconception: "They add or multiply the public numbers (10 × 2 = 20, or 12)." + - q: "Why does the method use powers mod p rather than plain powers?" + expect: "Plain powers can be reversed with a logarithm (log base 5 of 15625 = 6); the remainder scrambles the order, and no fast reverse is known." + misconception: "Mod p just keeps the numbers small." + - q: "Does the exchange alone guarantee Alice is talking to Bob?" + expect: "No. An active attacker can replace the public numbers and run a separate exchange with each side; authentication is needed." + misconception: "Yes, because only Bob can compute the shared secret." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/narrative.md new file mode 100644 index 0000000..5267cb4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/narrative.md @@ -0,0 +1,56 @@ +# Narrative: key-exchange + +## 1. Reader before and after + +- **Before:** A technical reader: school mathematics (powers, the rule (x^a)^b = x^(ab), logarithms, remainders) and basic programming. Knows a key is the secret an encryption method needs. New to cryptography: does not know "mod", one-way operations, or the discrete logarithm. +- **After:** "Each side keeps a private number and sends g to that power, mod p. Each raises the other's public number to its own private number, and both get g^(ab) mod p. An eavesdropper sees only g^a and g^b, and no known fast method gets a or b back from them. It stops readers, not attackers who change messages." + +## 2. Question and motive + +- **Question:** How can two people who have never met agree on a secret key when everything they send can be read? Result in words: they never send the key. Each sends half of a calculation; each finishes it with a number they kept; both reach the same number; a reader cannot finish it. +- **Why care:** every HTTPS connection starts with a stranger (a web server) and an open network. Encryption needs a shared key first. +- **Why this approach:** the key cannot travel, so it must be computed on both sides from what travels plus something that does not. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| Alice, Bob, Eve | the two parties; the eavesdropper who reads everything | — | 1 | +| mod p | remainder after division by p | 100 mod 23 = 8 | 4 | +| p, g | public modulus (prime) and base | p = 23, g = 5 | 4 | +| private number a, b | kept, never sent | a = 6, b = 15 | 4 | +| public number A, B | g^a mod p, g^b mod p; sent | A = 8, B = 19 | 4 | +| shared secret s | what both compute; becomes the key | s = 2 | 5 | +| one-way operation | fast forward, no known fast reverse | 5^x mod 23 | 7 | +| discrete logarithm | finding x from g^x mod p | find 6 from 8 | 7 | +| repeated squaring | computing a power by squaring and reducing at each step | 5^6 via 5^2 → 2, 5^3 → 10, 5^6 → 8 | 8 | +| Mallory | an attacker who changes messages | replaces A and B with 10 | 9 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How can strangers share a secret in public? | — | the setting | Everything sent is read by Eve. So the key can never be sent. | the constraint | +| 2 | close (answer first) | So what do they do? | therefore | — | They compute it: each sends half a calculation, finishes the other's half with a kept number; both reach the same number; Eve cannot finish it. | the shape of the answer | +| 3 | objection | Why not one secret, hidden by a one-way operation? | but | Alice picks the key 6 and sends a scrambled form, 8 | Bob gets only 8, the same as Eve, so Bob would be as stuck as Eve. Each side needs a private number, and an operation where each can finish the other's half. | why two private numbers | +| 4 | name / instance | What operation? | so | numbers, then letters | "mod 23" means the remainder after division by 23 (100 mod 23 = 8). Public: p = 23, g = 5. Alice keeps a = 6, sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob keeps b = 15, sends B = 5^15 mod 23 = 19. | the protocol | +| 5 | mechanism | Do they really get the same number? | then | 19^6 mod 23 = 2; 8^15 mod 23 = 2 | Yes. Both are 5^(6×15) mod 23 by (g^a)^b = (g^b)^a. Taking the remainder early does not change the final remainder: dropping a multiple of 23 from a factor drops a multiple of 23 from the product (125 mod 23 = 10 = 2 × 5). | agreement | +| 6 | notice | What does Eve have? | but | table: who knows what, walked through | p, g, A, B. Not a, b, s. A × B mod 23 = 14 = 5^(6+15), not 2. | Eve's view | +| 7 | why | Why powers, why mod? | so | two failing alternatives, then the list | If A = 5 × 6 mod 23 = 7: multiply by 14 (since 5 × 14 = 70 = 3 × 23 + 1) to get 6. If A = 5^6 = 15625 with no mod: log₅ = 6. With both: the list of 5^1 … 5^22 mod 23 jumps around, so Eve can only try exponents one by one. Name: one-way operation; the reverse is the discrete logarithm. | one-way operation | +| 8 | quantity | Then p = 23 falls in 6 tries — so? | but | sizes | Real p: 617 digits; best known methods ~10^14 years (estimate). Alice computes fast by squaring and reducing (5^6: 5^2 → 2, 5^3 → 10, 5^6 → 8). Hardness is believed, not proven. | size matters, assumption | +| 9 | limit | Am I safe from everyone? | but | Mallory numbers | Mallory picks 3 and sends 5^3 mod 23 = 10 to both. Alice gets 10^6 mod 23 = 6, Bob gets 10^15 mod 23 = 5. Mallory computes 8^3 mod 23 = 6 and 19^3 mod 23 = 5, so Mallory decrypts and re-encrypts. Fix: authentication (out of scope). | limit | +| 10 | close | — | so | — | They never send the key. Each sends g to a private power, mod p; each finishes the other's half; both get g^(ab) mod p. Eve sees only g^a and g^b. Next: authentication, elliptic curves. | recap | + +## 5. Concrete to symbol + +Numbers first (5^6 mod 23 = 8), then "here g = 5, a = 6, p = 23", then A = g^a mod p. + +## 6. Links between representations + +The who-knows-what table matches the numbers in beat 4–5. + +## 7. Close the loop + +- **Answer:** They never send the key. Each sends g^(private) mod p, and each raises the other's public number to its own private number. Both get g^(ab) mod p. +- **General argument:** exponent rule gives agreement; discrete logarithm hardness (assumed) keeps Eve out. +- **Payoff:** this is how HTTPS starts (today mostly an elliptic-curve version). Next: authentication, elliptic curves. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/decisions.yaml new file mode 100644 index 0000000..e6e5eae --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/decisions.yaml @@ -0,0 +1,295 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T145920-probe#1: + decision: adopted +2026-10-04T145920-probe#10: + decision: adopted +2026-10-04T145920-probe#11: + decision: declined +2026-10-04T145920-probe#12: + decision: adopted +2026-10-04T145920-probe#13: + decision: declined +2026-10-04T145920-probe#14: + decision: adopted +2026-10-04T145920-probe#15: + decision: adopted +2026-10-04T145920-probe#16: + decision: declined +2026-10-04T145920-probe#2: + decision: adopted +2026-10-04T145920-probe#3: + decision: adopted +2026-10-04T145920-probe#4: + decision: declined +2026-10-04T145920-probe#5: + decision: declined +2026-10-04T145920-probe#6: + decision: adopted +2026-10-04T145920-probe#7: + decision: adopted +2026-10-04T145920-probe#8: + decision: adopted +2026-10-04T145920-probe#9: + decision: declined +2026-10-04T150134-coldread-narrative#1: + decision: adopted +2026-10-04T150134-coldread-narrative#10: + decision: declined +2026-10-04T150134-coldread-narrative#11: + decision: adopted +2026-10-04T150134-coldread-narrative#12: + decision: adopted +2026-10-04T150134-coldread-narrative#13: + decision: adopted +2026-10-04T150134-coldread-narrative#14: + decision: declined +2026-10-04T150134-coldread-narrative#15: + decision: declined +2026-10-04T150134-coldread-narrative#16: + decision: declined +2026-10-04T150134-coldread-narrative#17: + decision: adopted +2026-10-04T150134-coldread-narrative#18: + decision: adopted +2026-10-04T150134-coldread-narrative#19: + decision: declined +2026-10-04T150134-coldread-narrative#2: + decision: adopted +2026-10-04T150134-coldread-narrative#20: + decision: declined +2026-10-04T150134-coldread-narrative#21: + decision: declined +2026-10-04T150134-coldread-narrative#22: + decision: declined +2026-10-04T150134-coldread-narrative#23: + decision: declined +2026-10-04T150134-coldread-narrative#24: + decision: adopted +2026-10-04T150134-coldread-narrative#25: + decision: declined +2026-10-04T150134-coldread-narrative#26: + decision: declined +2026-10-04T150134-coldread-narrative#27: + decision: declined +2026-10-04T150134-coldread-narrative#28: + decision: declined +2026-10-04T150134-coldread-narrative#29: + decision: adopted +2026-10-04T150134-coldread-narrative#3: + decision: adopted +2026-10-04T150134-coldread-narrative#30: + decision: declined +2026-10-04T150134-coldread-narrative#31: + decision: adopted +2026-10-04T150134-coldread-narrative#32: + decision: declined +2026-10-04T150134-coldread-narrative#33: + decision: declined +2026-10-04T150134-coldread-narrative#34: + decision: adopted +2026-10-04T150134-coldread-narrative#35: + decision: declined +2026-10-04T150134-coldread-narrative#36: + decision: declined +2026-10-04T150134-coldread-narrative#37: + decision: declined +2026-10-04T150134-coldread-narrative#38: + decision: declined +2026-10-04T150134-coldread-narrative#39: + decision: adopted +2026-10-04T150134-coldread-narrative#4: + decision: adopted +2026-10-04T150134-coldread-narrative#40: + decision: adopted +2026-10-04T150134-coldread-narrative#5: + decision: adopted +2026-10-04T150134-coldread-narrative#6: + decision: declined +2026-10-04T150134-coldread-narrative#7: + decision: adopted +2026-10-04T150134-coldread-narrative#8: + decision: declined +2026-10-04T150134-coldread-narrative#9: + decision: adopted +2026-10-04T150411-coldread-prose#1: + decision: adopted +2026-10-04T150411-coldread-prose#10: + decision: adopted +2026-10-04T150411-coldread-prose#11: + decision: adopted +2026-10-04T150411-coldread-prose#12: + decision: adopted +2026-10-04T150411-coldread-prose#13: + decision: declined +2026-10-04T150411-coldread-prose#14: + decision: declined +2026-10-04T150411-coldread-prose#15: + decision: declined +2026-10-04T150411-coldread-prose#16: + decision: adopted +2026-10-04T150411-coldread-prose#17: + decision: declined +2026-10-04T150411-coldread-prose#2: + decision: adopted +2026-10-04T150411-coldread-prose#3: + decision: declined +2026-10-04T150411-coldread-prose#4: + decision: declined +2026-10-04T150411-coldread-prose#5: + decision: adopted +2026-10-04T150411-coldread-prose#6: + decision: adopted +2026-10-04T150411-coldread-prose#7: + decision: adopted +2026-10-04T150411-coldread-prose#8: + decision: adopted +2026-10-04T150411-coldread-prose#9: + decision: adopted +2026-10-04T150452-quiz-prose#1: + decision: adopted +2026-10-04T150452-quiz-prose#10: + decision: adopted +2026-10-04T150452-quiz-prose#11: + decision: declined +2026-10-04T150452-quiz-prose#12: + decision: adopted +2026-10-04T150452-quiz-prose#13: + decision: adopted +2026-10-04T150452-quiz-prose#14: + decision: adopted +2026-10-04T150452-quiz-prose#15: + decision: declined +2026-10-04T150452-quiz-prose#16: + decision: declined +2026-10-04T150452-quiz-prose#2: + decision: adopted +2026-10-04T150452-quiz-prose#3: + decision: adopted +2026-10-04T150452-quiz-prose#4: + decision: adopted +2026-10-04T150452-quiz-prose#5: + decision: declined +2026-10-04T150452-quiz-prose#6: + decision: declined +2026-10-04T150452-quiz-prose#7: + decision: adopted +2026-10-04T150452-quiz-prose#8: + decision: adopted +2026-10-04T150452-quiz-prose#9: + decision: declined +2026-10-04T150625-coldread-prose#1: + decision: adopted +2026-10-04T150625-coldread-prose#10: + decision: declined +2026-10-04T150625-coldread-prose#11: + decision: declined +2026-10-04T150625-coldread-prose#12: + decision: declined +2026-10-04T150625-coldread-prose#13: + decision: declined +2026-10-04T150625-coldread-prose#2: + decision: adopted +2026-10-04T150625-coldread-prose#3: + decision: declined +2026-10-04T150625-coldread-prose#4: + decision: adopted +2026-10-04T150625-coldread-prose#5: + decision: declined +2026-10-04T150625-coldread-prose#6: + decision: declined +2026-10-04T150625-coldread-prose#7: + decision: adopted +2026-10-04T150625-coldread-prose#8: + decision: adopted +2026-10-04T150625-coldread-prose#9: + decision: declined +2026-10-04T150725-coldread-prose#1: + decision: adopted +2026-10-04T150725-coldread-prose#10: + decision: declined +2026-10-04T150725-coldread-prose#11: + decision: declined +2026-10-04T150725-coldread-prose#12: + decision: declined +2026-10-04T150725-coldread-prose#2: + decision: declined +2026-10-04T150725-coldread-prose#3: + decision: declined +2026-10-04T150725-coldread-prose#4: + decision: adopted +2026-10-04T150725-coldread-prose#5: + decision: declined +2026-10-04T150725-coldread-prose#6: + decision: declined +2026-10-04T150725-coldread-prose#7: + decision: declined +2026-10-04T150725-coldread-prose#8: + decision: declined +2026-10-04T150725-coldread-prose#9: + decision: declined +2026-10-04T150823-quiz-prose#1: + decision: declined +2026-10-04T150823-quiz-prose#10: + decision: declined +2026-10-04T150823-quiz-prose#11: + decision: declined +2026-10-04T150823-quiz-prose#12: + decision: declined +2026-10-04T150823-quiz-prose#13: + decision: declined +2026-10-04T150823-quiz-prose#14: + decision: declined +2026-10-04T150823-quiz-prose#15: + decision: declined +2026-10-04T150823-quiz-prose#16: + decision: declined +2026-10-04T150823-quiz-prose#17: + decision: adopted +2026-10-04T150823-quiz-prose#2: + decision: adopted +2026-10-04T150823-quiz-prose#3: + decision: adopted +2026-10-04T150823-quiz-prose#4: + decision: declined +2026-10-04T150823-quiz-prose#5: + decision: declined +2026-10-04T150823-quiz-prose#6: + decision: declined +2026-10-04T150823-quiz-prose#7: + decision: declined +2026-10-04T150823-quiz-prose#8: + decision: declined +2026-10-04T150823-quiz-prose#9: + decision: declined +2026-10-04T150941-quiz-prose#1: + decision: adopted +2026-10-04T150941-quiz-prose#10: + decision: declined +2026-10-04T150941-quiz-prose#11: + decision: declined +2026-10-04T150941-quiz-prose#12: + decision: declined +2026-10-04T150941-quiz-prose#13: + decision: declined +2026-10-04T150941-quiz-prose#14: + decision: adopted +2026-10-04T150941-quiz-prose#15: + decision: declined +2026-10-04T150941-quiz-prose#16: + decision: declined +2026-10-04T150941-quiz-prose#2: + decision: declined +2026-10-04T150941-quiz-prose#3: + decision: declined +2026-10-04T150941-quiz-prose#4: + decision: declined +2026-10-04T150941-quiz-prose#5: + decision: declined +2026-10-04T150941-quiz-prose#6: + decision: declined +2026-10-04T150941-quiz-prose#7: + decision: declined +2026-10-04T150941-quiz-prose#8: + decision: declined +2026-10-04T150941-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T145920-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T145920-probe.json new file mode 100644 index 0000000..2486e12 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T145920-probe.json @@ -0,0 +1,225 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "key-exchange", + "at": "2026-10-04T14:59:20", + "model": "default", + "prompt_sha": "5df4d87a86a1", + "reply": { + "gaps": [ + { + "rule": 1, + "severity": "main", + "about": "No explanation of why a one-way function alone is not enough, and why two private numbers are needed", + "why_it_matters": "The obvious first idea is: 'Alice picks the key and sends it hidden by a one-way function.' Nothing in the model says why that fails. Without that, Dependency step 2 (both sides mix, and the mixes match) reads as an arbitrary design rather than the answer to the central question.", + "suggested_addition": "Alternative state: Alice picks key k = 6 and sends 5^6 mod 23 = 8. Bob is now as stuck as Eve, because recovering k = 6 from 8 is the same discrete logarithm. DH needs an operation where each side can finish the other's half: Bob turns 8 into 8^15 mod 23 = 2 without ever learning 6. That works because (g^a)^b = (g^b)^a." + }, + { + "rule": 2, + "severity": "main", + "about": "The causal chain jumps from 'Eve needs a or b' without support", + "why_it_matters": "The chain says Eve needs a or b, but no step shows that getting s requires a or b. A shortcut from (A, B) straight to g^(ab) is a separate problem (the computational Diffie–Hellman problem). It is also unproven, and the Epistemic status section never names this assumption. The central claim 'Eve is stuck' depends on this hidden step.", + "suggested_addition": "State it as an assumption: 'No known way exists to compute g^(ab) mod p from g^a and g^b without, in effect, finding a or b.' Give an instance where the obvious shortcut fails: A·B = 8·19 mod 23 = 14 = 5^21, but s = 5^90 mod 23 = 5^2 = 2. Add this assumption to Epistemic status as 'believed, not proven'." + }, + { + "rule": 1, + "severity": "edge", + "about": "The multiplication row in 'Why this form' ends with a vague sentence", + "why_it_matters": "'Would not match in a useful way unless Eve can also compute them' is unclear. In fact the results do match, and Eve gets the same value. The concrete failure is never shown.", + "suggested_addition": "With multiplication, s = g·a·b mod 23 = 5·6·15 = 450 mod 23 = 13. Eve computes A·B·14 mod 23 = 7·6·14 mod 23 = 13, where B = 5·15 mod 23 = 6. Eve gets s directly." + }, + { + "rule": 1, + "severity": "edge", + "about": "No reason given for why p must be prime", + "why_it_matters": "p is introduced as 'a large prime' with no reason. A learner will ask why not any number.", + "suggested_addition": "With modulus 24 and g = 5, the powers are 5, 1, 5, 1, …: only 2 values, so s can only be 1 or 5. With prime 23 and g = 5, all 22 non-zero remainders appear. Note that safe choices of p and g are otherwise out of scope." + }, + { + "rule": 2, + "severity": "edge", + "about": "The check in Quantities uses 5^22 mod 23 = 1 and reduces exponents mod 22 without explanation", + "why_it_matters": "The audience does not know Fermat's little theorem, so this verification step is left implicit.", + "suggested_addition": "Point to the powers list: 5^22 mod 23 = 1, so the powers repeat every 22 steps. Then 5^90 = 5^(4·22+2), which gives 5^2 = 25 mod 23 = 2. Or drop this check and use direct repeated squaring of 8^15 (see the rule 4 entry)." + }, + { + "rule": 2, + "severity": "edge", + "about": "The claim 'reducing mod p at each step does not change the remainder' has no numeric instance", + "why_it_matters": "Every step of Alice's computation depends on this guarantee, and it appears without a concrete check.", + "suggested_addition": "5^3 = 125, and 125 mod 23 = 10. Reducing first gives (5^2 mod 23)·5 = 2·5 = 10. Same answer." + }, + { + "rule": 2, + "severity": "edge", + "about": "'Eve cannot get s' has only one way to break it (small p)", + "why_it_matters": "A large p does not help if the private number comes from a small range. The model says a is 'random' but never says from what range.", + "suggested_addition": "State that a and b are drawn from 1 … p−2. Breaking case: with a 617-digit p but a chosen from 1 … 1,000,000, Eve tries at most 10^6 exponents and finds a in under a second." + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts have more than one name", + "why_it_matters": "Readers may think each name is a different thing.", + "suggested_addition": "Pick one name for each: 'shared secret s' / 'key' / 's (after hashing)'; 'public number' / 'public value' (Dependencies 2); 'private number' / 'private value'; 'active attacker' / 'Mallory' / 'man in the middle'; 'fast exponentiation' / 'repeated squaring'; 'one-way operation' (Entities) / 'one-way function' (Audience). Also define '≡' (used in Quantities) and 'hash' (Temporal step 5) before use, or remove them." + }, + { + "rule": 4, + "severity": "edge", + "about": "Repeated squaring is worked only for 5^6; Bob's 8^15 has no worked steps", + "why_it_matters": "The confusion point 'computing is fast' relies on this, and the odd exponent 15 shows how the method combines squares.", + "suggested_addition": "8^2 = 64 → 18; 8^4 = 18^2 = 324 → 2; 8^8 = 4. Since 15 = 8+4+2+1, 8^15 = 4·2·18·8 = 1152 mod 23 = 2. That is 3 squarings plus 3 multiplications instead of 14 multiplications." + }, + { + "rule": 5, + "severity": "edge", + "about": "The size of p is not turned into time, and the brute-force count overstates Eve's real cost", + "why_it_matters": "'10^616 entries' doesn't tell the reader how long that takes. The model also says faster methods exist, so the security level is far below 10^616. A reader may come away with the wrong order of magnitude.", + "suggested_addition": "Brute force at 10^12 tries/s takes about 10^604 s; the universe is about 4×10^17 s old. The best known methods against 2048-bit p cost about 2^112 ≈ 5×10^33 operations, which at 10^12/s is about 10^14 years. Compare Alice: about 4096 multiplications, a few milliseconds." + }, + { + "rule": 5, + "severity": "edge", + "about": "The 'Bad g = 22' row has no stated meaning", + "why_it_matters": "'Then s can be only 1 or 22' is the consequence, but its meaning for the reader isn't stated. It reads as a fragment.", + "suggested_addition": "With g = 22, Eve guesses s correctly with 1 try in 2. With g = 5, s could be any of 22 values." + } + ], + "excess": [ + "Known limit: Shor's algorithm and quantum computers (not needed to answer the central question)", + "Scope notes on elliptic-curve DH (X25519) and ML-KEM", + "'(in practice, a hash of s)' in Temporal step 5 and 'after hashing' in Terms: brings in an undefined term that is out of scope", + "The check in Quantities using 5^22 mod 23 = 1 and exponent reduction mod 22 (could be replaced by direct squaring)", + "Representation decision section (planning metadata, not content for the reader)" + ], + "next_questions": [ + { + "q": "Why not just pick a key and send it hidden by a hard-to-reverse function?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Could Eve compute g^(ab) mod p from A and B without finding a or b?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How do Alice and Bob stop a man in the middle and know they're talking to each other?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Why must p be prime, and how are p and g chosen?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "How long would the best known attack actually take against a 2048-bit p?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.1690814, + "seconds": 64 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 1", + "text": "No explanation of why a one-way function alone is not enough, and why two private numbers are needed → Alternative state: Alice picks key k = 6 and sends 5^6 mod 23 = 8. Bob is now as stuck as Eve, because recovering k = 6 from 8 is the same discrete logarithm. DH needs an operation where each side can finish the other's half: Bob turns 8 into 8^15 mod 23 = 2 without ever learning 6. That works because (g^a)^b = (g^b)^a." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "The causal chain jumps from 'Eve needs a or b' without support → State it as an assumption: 'No known way exists to compute g^(ab) mod p from g^a and g^b without, in effect, finding a or b.' Give an instance where the obvious shortcut fails: A·B = 8·19 mod 23 = 14 = 5^21, but s = 5^90 mod 23 = 5^2 = 2. Add this assumption to Epistemic status as 'believed, not proven'." + }, + { + "n": 3, + "severity": "edge", + "kind": "rule 1", + "text": "The multiplication row in 'Why this form' ends with a vague sentence → With multiplication, s = g·a·b mod 23 = 5·6·15 = 450 mod 23 = 13. Eve computes A·B·14 mod 23 = 7·6·14 mod 23 = 13, where B = 5·15 mod 23 = 6. Eve gets s directly." + }, + { + "n": 4, + "severity": "edge", + "kind": "rule 1", + "text": "No reason given for why p must be prime → With modulus 24 and g = 5, the powers are 5, 1, 5, 1, …: only 2 values, so s can only be 1 or 5. With prime 23 and g = 5, all 22 non-zero remainders appear. Note that safe choices of p and g are otherwise out of scope." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 2", + "text": "The check in Quantities uses 5^22 mod 23 = 1 and reduces exponents mod 22 without explanation → Point to the powers list: 5^22 mod 23 = 1, so the powers repeat every 22 steps. Then 5^90 = 5^(4·22+2), which gives 5^2 = 25 mod 23 = 2. Or drop this check and use direct repeated squaring of 8^15 (see the rule 4 entry)." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 2", + "text": "The claim 'reducing mod p at each step does not change the remainder' has no numeric instance → 5^3 = 125, and 125 mod 23 = 10. Reducing first gives (5^2 mod 23)·5 = 2·5 = 10. Same answer." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 2", + "text": "'Eve cannot get s' has only one way to break it (small p) → State that a and b are drawn from 1 … p−2. Breaking case: with a 617-digit p but a chosen from 1 … 1,000,000, Eve tries at most 10^6 exponents and finds a in under a second." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts have more than one name → Pick one name for each: 'shared secret s' / 'key' / 's (after hashing)'; 'public number' / 'public value' (Dependencies 2); 'private number' / 'private value'; 'active attacker' / 'Mallory' / 'man in the middle'; 'fast exponentiation' / 'repeated squaring'; 'one-way operation' (Entities) / 'one-way function' (Audience). Also define '≡' (used in Quantities) and 'hash' (Temporal step 5) before use, or remove them." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 4", + "text": "Repeated squaring is worked only for 5^6; Bob's 8^15 has no worked steps → 8^2 = 64 → 18; 8^4 = 18^2 = 324 → 2; 8^8 = 4. Since 15 = 8+4+2+1, 8^15 = 4·2·18·8 = 1152 mod 23 = 2. That is 3 squarings plus 3 multiplications instead of 14 multiplications." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 5", + "text": "The size of p is not turned into time, and the brute-force count overstates Eve's real cost → Brute force at 10^12 tries/s takes about 10^604 s; the universe is about 4×10^17 s old. The best known methods against 2048-bit p cost about 2^112 ≈ 5×10^33 operations, which at 10^12/s is about 10^14 years. Compare Alice: about 4096 multiplications, a few milliseconds." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 5", + "text": "The 'Bad g = 22' row has no stated meaning → With g = 22, Eve guesses s correctly with 1 try in 2. With g = 5, s could be any of 22 values." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "Known limit: Shor's algorithm and quantum computers (not needed to answer the central question)" + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Scope notes on elliptic-curve DH (X25519) and ML-KEM" + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "'(in practice, a hash of s)' in Temporal step 5 and 'after hashing' in Terms: brings in an undefined term that is out of scope" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "The check in Quantities using 5^22 mod 23 = 1 and exponent reduction mod 22 (could be replaced by direct squaring)" + }, + { + "n": 16, + "severity": "excess", + "kind": "model entry", + "text": "Representation decision section (planning metadata, not content for the reader)" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150134-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150134-coldread-narrative.json new file mode 100644 index 0000000..3986c82 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150134-coldread-narrative.json @@ -0,0 +1,432 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "key-exchange", + "at": "2026-10-04T15:01:34", + "model": "default", + "prompt_sha": "0dc70ed4e552", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Everything sent is read by Eve. So the key can never be sent.", + "unresolved": [ + "Eve" + ], + "unsaid": [ + "Shown 'the setting' is never described. The words don't name Alice and Bob as the two parties or say that Eve only listens." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (answer first)", + "text": "They compute it: each sends half a calculation, finishes the other's half with a kept number; both reach the same number; Eve cannot finish it.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "Bob would be as stuck as Eve. Each side needs a private number, and an operation where each can finish the other's half.", + "unresolved": [ + "one-way operation", + "k" + ], + "unsaid": [ + "Shown 'k = 6 → 8': the words never say what k is or what operation the arrow stands for." + ], + "leap": [ + "'Bob would be as stuck as Eve': the words don't say that if only one side holds the secret and sends its one-way result, Bob gets exactly what Eve gets." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (name / instance)", + "text": "Powers mod p. Alice sends 5^6 mod 23 = 8; Bob sends 19.", + "unresolved": [ + "mod", + "p" + ], + "unsaid": [ + "Shown 'mod defined': the words never say mod means the remainder after division.", + "Shown 'p, g, a, b, A, B': the words don't say g = 5, a = 6, p = 23, b = 15, or that A and B are the sent numbers.", + "Bob's private number 15 is never said, so the reader can't see where 19 comes from." + ], + "leap": [ + "Bob sends 19: the computation isn't given." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Yes, both are g^(ab): (g^a)^b = (g^b)^a; remainders can be taken at each step.", + "unresolved": [ + "g", + "a", + "b" + ], + "unsaid": [ + "Shown '8^15 mod 23 = 2' is the first time the reader sees 15 as Bob's number." + ], + "leap": [ + "'remainders can be taken at each step': the words don't say why taking remainders partway leaves the final remainder unchanged. The reader knows remainders but not this rule." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (notice)", + "text": "p, g, A, B. Not a, b, s. A × B = 14, not 2 (exponents add, not multiply).", + "unresolved": [ + "A", + "B", + "s" + ], + "unsaid": [ + "The 'who knows what' table is shown but not walked through in words." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (why)", + "text": "Without mod, a log reverses it. With multiplication, a single multiplication reverses it. Powers mod p scramble... No fast reverse known: the discrete logarithm.", + "unresolved": [ + "k (in 5^k mod 23, used again with a different meaning from beat 3)" + ], + "unsaid": [ + "Shown 'multiplication → 14 undoes it' is not explained, and it's unclear how 14 (from beat 6) relates to this.", + "The list of 5^k mod 23 is shown, but the words don't point out what to notice in it." + ], + "leap": [ + "'With multiplication, a single multiplication reverses it': this means swapping the power for multiplication. The words don't say so, and the reverse step isn't shown." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (quantity)", + "text": "Real p: 617 digits; ~10^14 years by best known methods (estimate). Alice uses repeated squaring: ~4000 multiplications. Hardness is believed, not proven.", + "unresolved": [ + "repeated squaring", + "6 tries (never said earlier that finding a = 6 means trying exponents one by one)" + ], + "unsaid": [ + "Shown 'sizes' is not described." + ], + "leap": [ + "The words don't explain why repeated squaring needs only about 4000 multiplications." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (limit)", + "text": "Mallory swaps public numbers: Alice gets 6, Bob gets 5, Mallory both. Fix: authentication (out of scope).", + "unresolved": [ + "Mallory", + "authentication" + ], + "unsaid": [ + "Shown 'Mallory numbers': the words never say what number Mallory sends, so 'Alice gets 6, Bob gets 5' can't be checked." + ], + "leap": [ + "The words don't say why Mallory ends up knowing both secrets." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 10 (close)", + "text": "Restate answer; scope pointers.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": true + } + ], + "blocking": [ + "Beat 4: 'Powers mod p': mod and p are never defined in the words. The definition exists only in Shown ('mod defined').", + "Beat 4: g, a, b, A, B are never tied to 5, 6, 15, 8, 19 in the words, so 'g^(ab)' in beat 5 and 'p, g, A, B. Not a, b, s' in beat 6 refer to letters the reader hasn't been given.", + "Beat 5: 'remainders can be taken at each step': agreement depends on this rule, and it isn't justified.", + "Beat 7: 'With multiplication, a single multiplication reverses it' and Shown 'multiplication → 14 undoes it': it's unclear what is being compared.", + "Beat 10: 'Restate answer; scope pointers.' is a placeholder, so no closing words are given." + ], + "cuts": [ + "Beat 8: cut the repeated-squaring aside ('~4000 multiplications'), or move it to a footnote. It answers a side question and is never explained.", + "Beat 6: cut '(exponents add, not multiply)' or merge it into beat 7, which makes the 'why not simpler' point again." + ], + "question": { + "text": "How can two parties agree on a secret key when everything they send is read?", + "known_at": "Beat 1: 'Everything sent is read by Eve. So the key can never be sent.'", + "result_stated": "Yes, in beat 2: 'each sends half a calculation, finishes the other's half with a kept number; both reach the same number; Eve cannot finish it.'" + }, + "motive": "Reason for the approach: given in beat 1 ('So the key can never be sent') and beat 3 ('Each side needs a private number, and an operation where each can finish the other's half'). Reason to care: missing from the beats. The HTTPS motive appears only in the plan sections.", + "close": "Not delivered. Beat 10 says only 'Restate answer; scope pointers.' It gives no words answering the question, no general argument (exponent rule plus assumed discrete-log hardness), and no payoff.", + "recap": "Each side keeps a secret exponent and sends 5 to that power, mod 23. Raising the other's number to your own secret gives both sides the same value, and an eavesdropper would have to undo the power-mod, which nobody knows how to do fast.", + "strongest": "Beat 5: the two computations 19^6 mod 23 = 2 and 8^15 mod 23 = 2, tied to (g^a)^b = (g^b)^a, which this reader already knows.", + "weakest": "Beat 4 together with beat 10. Beat 4 defines mod and the letters only in Shown, and the whole symbolic argument rests on them. Beat 10 is an unwritten placeholder.", + "suggestions": [ + "In beat 4, say in words: 'mod 23 means the remainder after dividing by 23; here g = 5, p = 23, Alice's private a = 6, Bob's private b = 15, and A = 8, B = 19 are what they send.'", + "In beat 5, give one line of support for reducing as you go, e.g. a product's remainder depends only on the remainders of its factors.", + "In beat 3, define k and the arrow, or replace the shown example with words.", + "In beat 7, state the comparison outright: if Alice sent g × a mod p, Eve could divide (multiply by an inverse) to get a back.", + "In beat 9, say that Mallory sends 10 to both sides.", + "Write beat 10 as actual sentences, and include the HTTPS reason to care." + ] + }, + "usage": { + "cost": 0.1230666, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'Powers mod p': mod and p are never defined in the words. The definition exists only in Shown ('mod defined')." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: g, a, b, A, B are never tied to 5, 6, 15, 8, 19 in the words, so 'g^(ab)' in beat 5 and 'p, g, A, B. Not a, b, s' in beat 6 refer to letters the reader hasn't been given." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: 'remainders can be taken at each step': agreement depends on this rule, and it isn't justified." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 7: 'With multiplication, a single multiplication reverses it' and Shown 'multiplication → 14 undoes it': it's unclear what is being compared." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 10: 'Restate answer; scope pointers.' is a placeholder, so no closing words are given." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 8: cut the repeated-squaring aside ('~4000 multiplications'), or move it to a footnote. It answers a side question and is never explained." + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 6: cut '(exponents add, not multiply)' or merge it into beat 7, which makes the 'why not simpler' point again." + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): Eve" + }, + { + "n": 9, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): Shown 'the setting' is never described. The words don't name Alice and Bob as the two parties or say that Eve only listens." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): one-way operation" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): k" + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): Shown 'k = 6 → 8': the words never say what k is or what operation the arrow stands for." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): 'Bob would be as stuck as Eve': the words don't say that if only one side holds the secret and sends its one-way result, Bob gets exactly what Eve gets." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (name / instance): mod" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (name / instance): p" + }, + { + "n": 16, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name / instance): Shown 'mod defined': the words never say mod means the remainder after division." + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name / instance): Shown 'p, g, a, b, A, B': the words don't say g = 5, a = 6, p = 23, b = 15, or that A and B are the sent numbers." + }, + { + "n": 18, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name / instance): Bob's private number 15 is never said, so the reader can't see where 19 comes from." + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (name / instance): Bob sends 19: the computation isn't given." + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): g" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): a" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): b" + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): Shown '8^15 mod 23 = 2' is the first time the reader sees 15 as Bob's number." + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 'remainders can be taken at each step': the words don't say why taking remainders partway leaves the final remainder unchanged. The reader knows remainders but not this rule." + }, + { + "n": 25, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (notice): A" + }, + { + "n": 26, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (notice): B" + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (notice): s" + }, + { + "n": 28, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (notice): The 'who knows what' table is shown but not walked through in words." + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (why): k (in 5^k mod 23, used again with a different meaning from beat 3)" + }, + { + "n": 30, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (why): Shown 'multiplication → 14 undoes it' is not explained, and it's unclear how 14 (from beat 6) relates to this." + }, + { + "n": 31, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (why): The list of 5^k mod 23 is shown, but the words don't point out what to notice in it." + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (why): 'With multiplication, a single multiplication reverses it': this means swapping the power for multiplication. The words don't say so, and the reverse step isn't shown." + }, + { + "n": 33, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (quantity): repeated squaring" + }, + { + "n": 34, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (quantity): 6 tries (never said earlier that finding a = 6 means trying exponents one by one)" + }, + { + "n": 35, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (quantity): Shown 'sizes' is not described." + }, + { + "n": 36, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (quantity): The words don't explain why repeated squaring needs only about 4000 multiplications." + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (limit): Mallory" + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (limit): authentication" + }, + { + "n": 39, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 9 (limit): Shown 'Mallory numbers': the words never say what number Mallory sends, so 'Alice gets 6, Bob gets 5' can't be checked." + }, + { + "n": 40, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (limit): The words don't say why Mallory ends up knowing both secrets." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150411-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150411-coldread-prose.json new file mode 100644 index 0000000..f9bc3b3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150411-coldread-prose.json @@ -0,0 +1,389 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:04:11", + "model": "default", + "prompt_sha": "a76b387677f2", + "reply": { + "paragraphs": [ + { + "at": "Title", + "text": "How can two strangers agree on a secret key when anyone can read their messages?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold summary (line 5)", + "text": "They never send the key. Each person keeps a private number and sends a scrambled form of it...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 7", + "text": "This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 1", + "text": "Alice and Bob want to encrypt their messages...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 2", + "text": "A first idea fails. Alice picks the key, 6, and sends it through an operation that is hard to reverse. The result is 8...", + "unresolved": [], + "leap": [ + "\"an operation that lets each side finish the other's half without undoing it\": the reader is not yet told what a 'half' is or how one side could 'finish' it. The reader has to hold this phrase until the numbered steps make it concrete. (Where 6 → 8 comes from is also not given, but 'an operation' makes clear it is only a placeholder.)" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The exchange, mod definition", + "text": "\"mod 23\" means \"the remainder after division by 23\"...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Step 1", + "text": "Alice and Bob agree in public on two numbers: a prime modulus p = 23 and a base g = 5...", + "unresolved": [], + "leap": [ + "Why p has to be prime is not said here. Scope defers it later, so this is only an edge finding." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Step 2", + "text": "Each picks a private number at random...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Step 3", + "text": "Each sends a public number: g raised to their private number, mod p...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Step 4", + "text": "Each raises the other's public number to their own private number, mod p. Alice computes 19^6 mod 23 = 2...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Shared secret", + "text": "The result, s = 2, is the shared secret. It becomes the key for ordinary encryption.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why both get 2", + "text": "Alice computed (5^15)^6, and Bob computed (5^6)^15...", + "unresolved": [], + "leap": [ + "\"so it removes a multiple of 23 from the product\": the one-line algebra, (x + 23k)·y = xy + 23ky, is left for the reader to fill in. It is short, but this is where a reader might reread." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What Eve sees, table", + "text": "Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Predict prompt", + "text": "Eve has 8 and 19. Can she combine them into 2?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Multiplying fails", + "text": "Multiplying fails: 8 × 19 mod 23 = 152 mod 23 = 14...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why powers, and why mod, intro", + "text": "Each choice blocks a simpler way for Eve to find a private number.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: Without mod", + "text": "A = 5^6 = 15625. Eve takes the logarithm base 5...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: multiplication instead of a power", + "text": "A = 5 × 6 mod 23 = 7. Multiplying by 14 undoes multiplying by 5...", + "unresolved": [], + "leap": [ + "It is not said how Eve finds 14. The text checks that 14 works but does not say that finding it is easy, and that ease is the whole point of the bullet. The same number 14 also appeared just before as 8 × 19 mod 23, which could briefly confuse." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: powers mod p", + "text": "With powers mod p, the results jump around. For exponents 1 to 22...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "One-way definition", + "text": "So g^x mod p is a one-way operation... the discrete logarithm.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 1", + "text": "With p = 23, Eve can try exponents in order... about 10^14 years.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 2", + "text": "Alice's own work stays small. Alice squares and takes the remainder at each step...", + "unresolved": [], + "leap": [ + "\"A 2048-bit exponent needs about 4,000 such steps\": no reason is given for 4,000 (roughly one squaring plus at most one multiply per bit).", + "\"Alice squares... at each step\", yet the example step 5^3 = 2 × 5 is a multiply, not a square, so the rule as stated does not match the example." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Assumption, not proof", + "text": "No one has proven that the discrete logarithm is hard...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The limit, para 1", + "text": "The exchange stops a reader. It does not stop Mallory...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Mallory bullets", + "text": "Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The limit, closing para", + "text": "Alice shares 6 with Mallory... Real systems add authentication: Bob signs the public number B, and Alice checks the signature with a certificate.", + "unresolved": [ + "\"signs\"", + "\"signature\"", + "\"certificate\"" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer", + "text": "The two strangers never send the key. Each sends g raised to a private number, mod p...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Scope list", + "text": "Authentication...; Elliptic-curve Diffie–Hellman (for example X25519)...; hash function...AES...; Choosing p and g safely...; Shor's algorithm...ML-KEM", + "unresolved": [ + "\"X25519\"", + "\"hash function\"", + "\"AES\"", + "\"Shor's algorithm\"", + "\"ML-KEM\"", + "\"different number system\"" + ], + "leap": [], + "excess": [ + "\"Most HTTPS connections use it today\" partly repeats line 7, \"A version of it starts most HTTPS connections\"" + ], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Scope list: drop the example names \"X25519\" and \"ML-KEM\". They are unknown to this reader and add nothing to a pointer list.", + "Scope, Elliptic-curve bullet: \"Most HTTPS connections use it today\" largely repeats line 7." + ], + "question": { + "text": "How can two people who share nothing secret, talking only over a channel anyone can read, end up with the same secret key while the eavesdropper does not?", + "known_at": "The title, made concrete in 'The problem' para 1 (\"So the key itself can never be sent.\")", + "result_stated": "Yes, at the very start: \"They never send the key. Each person keeps a private number and sends a scrambled form of it. Each then combines the other person's scrambled number with their own private number, and both get the same result.\"" + }, + "motive": "Reason to care: \"Encryption needs a key that both of them know. They have never met, so every message goes over a public channel\" and \"A version of it starts most HTTPS connections.\" Reason for the approach: the failed first idea (\"If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number...\") and the section 'Why powers, and why mod' (\"Each choice blocks a simpler way for Eve to find a private number.\")", + "close": "Yes: \"Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p.\"", + "recap": "Each side keeps a secret exponent and publishes g to that power mod p. Raising the other side's number to your own exponent gives both sides g^(ab) mod p, and an eavesdropper can't undo the powering to catch up.", + "strongest": "'Why powers, and why mod' (lines 45–54). It rules out the simpler choices with concrete numbers (plain powers fall to a logarithm, multiplication falls to an inverse), and the jumbled table of 5^x mod 23 makes 'one-way' something the reader can see.", + "weakest": "'Why real numbers are huge' para 2 (line 60). \"Alice squares... at each step\" does not match the multiply step in its own example, and \"about 4,000 such steps\" comes with no reason.", + "suggestions": [ + "In the multiplication bullet, add a clause saying an inverse like 14 can be found quickly (for example by the extended Euclidean algorithm, or by trying 22 candidates), so the reader sees why this choice fails.", + "Line 60: describe the method as square-and-multiply (square at each bit, multiply by g when the bit is 1). That fixes the mismatch with the 5^3 step and explains '≈2 steps per bit → ~4,000'.", + "In 'Why both get 2', write the one-line algebra (x + 23k)·y = xy + 23ky.", + "In the authentication sentence, gloss 'signs' in a few words (for example 'attaches proof that only Bob could have produced')." + ] + }, + "usage": { + "cost": 0.1540012, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Scope list: drop the example names \"X25519\" and \"ML-KEM\". They are unknown to this reader and add nothing to a pointer list." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Scope, Elliptic-curve bullet: \"Most HTTPS connections use it today\" largely repeats line 7." + }, + { + "n": 3, + "severity": "edge", + "kind": "leap", + "text": "The problem, para 2: \"an operation that lets each side finish the other's half without undoing it\": the reader is not yet told what a 'half' is or how one side could 'finish' it. The reader has to hold this phrase until the numbered steps make it concrete. (Where 6 → 8 comes from is also not given, but 'an operation' makes clear it is only a placeholder.)" + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Step 1: Why p has to be prime is not said here. Scope defers it later, so this is only an edge finding." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Why both get 2: \"so it removes a multiple of 23 from the product\": the one-line algebra, (x + 23k)·y = xy + 23ky, is left for the reader to fill in. It is short, but this is where a reader might reread." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Bullet: multiplication instead of a power: It is not said how Eve finds 14. The text checks that 14 works but does not say that finding it is easy, and that ease is the whole point of the bullet. The same number 14 also appeared just before as 8 × 19 mod 23, which could briefly confuse." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Why real numbers are huge, para 2: \"A 2048-bit exponent needs about 4,000 such steps\": no reason is given for 4,000 (roughly one squaring plus at most one multiply per bit)." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Why real numbers are huge, para 2: \"Alice squares... at each step\", yet the example step 5^3 = 2 × 5 is a multiply, not a square, so the rule as stated does not match the example." + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "The limit, closing para: \"signs\"" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "The limit, closing para: \"signature\"" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "The limit, closing para: \"certificate\"" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"X25519\"" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"hash function\"" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"AES\"" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"Shor's algorithm\"" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"ML-KEM\"" + }, + { + "n": 17, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"different number system\"" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150452-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150452-quiz-prose.json new file mode 100644 index 0000000..3a2170c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150452-quiz-prose.json @@ -0,0 +1,178 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:04:46", + "model": "default", + "prompt_sha": "bedad2b89e75", + "reply": { + "answers": { + "1": "The people are Alice and Bob (the two strangers), Eve (an eavesdropper who only reads) and Mallory (an attacker who can replace messages). The public values are the prime modulus p and the base g. Each side has a private number (a, b) and a public number (A = g^a mod p, B = g^b mod p). The result is the shared secret s = g^(ab) mod p, which is then used as the key for ordinary encryption. Supporting ideas: 'mod' (remainder), one-way operation, discrete logarithm, authentication.", + "2": "p and g are agreed in public. Each side keeps a private exponent and publishes g raised to it, mod p. Each side raises the other's public number to its own private exponent. Because of the exponent rule, both get g^(ab) mod p. Eve sees p, g, A and B but not a, b or s. Getting a from A is the discrete logarithm problem. Mallory can swap A and B for her own public number, and authentication (signatures, certificates) is what stops that.", + "3": "Each side sends g^private mod p and raises what it receives to its own private number. By (g^a)^b = (g^b)^a, both reach g^(ab) mod p. Eve can't follow because undoing g^x mod p (the discrete logarithm) has no known fast method.", + "4": "The most important variable is the size of p (and of the range a and b are picked from). With p = 23, Eve can try exponents one by one and finds a = 6 on the 6th try. With a 2048-bit p (about 617 digits), even the best known methods need about 2^112 operations, roughly 10^14 years at a trillion per second. Alice's own work only grows to about 4,000 steps. So making p bigger hurts Eve enormously and the honest users only a little.", + "5": "Assumption: that the discrete logarithm is hard, and that there is no other way from A and B to s. This has not been proven; it rests on decades of failed attempts. The setup also assumes Eve only reads and does not change messages. Established fact: (5^15)^6 and (5^6)^15 both equal 5^(90), and taking remainders along the way doesn't change the final remainder. In the example, both sides compute 2.", + "6": "Alice and Bob publicly agree on a prime p and a base g. Each secretly picks an exponent and sends g to that power, mod p. Each raises the number they receive to their own secret exponent, mod p, and the exponent rule makes both results equal g^(ab) mod p, which becomes the key. Eve, who sees only g^a and g^b mod p, would have to solve the discrete logarithm, which has no known fast method when p is huge, though this is assumed rather than proven. The method doesn't stop someone who replaces messages, so real systems add signatures and certificates.", + "7": "The 'first idea fails' paragraph was hardest. It says 6 is sent 'through an operation that is hard to reverse' and the result is 8, but it doesn't name the operation (it is in fact 5^6 mod 23, which only appears later). Naming it, or pointing ahead to it, would help. The '~4,000 steps' figure was also hard to follow without a sentence on where it comes from.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8 (= 5^6 mod 23, matching the explanation's table).", + "9": "Without mod, A = 5^6 = 15625, and Eve just takes the logarithm base 5 to get 6, because plain powers grow in order. With mod p, the results jump around (5, 2, 10, 4, 20, 8, …), so a result says nothing about where the exponent is. That makes the operation one-way, as far as anyone knows.", + "10": "No. Nothing in the exchange tells Alice who sent a public number, so Mallory can sit in the middle. Authentication has to be added: Bob signs B, and Alice checks the signature with a certificate.", + "11": "Because Bob would then have to reverse the operation to get the key, and if Bob can reverse it, so can Eve: both receive the same 8. So each side needs its own private number, and an operation that lets each side finish the other's half without undoing it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2, and Bob computes 8^15 mod 23 = 2. The shared secret is 2.", + "13": "Multiplying powers adds the exponents: 8 × 19 mod 23 = 14 = 5^(6+15) mod 23. The secret needs the exponents multiplied, 5^(6×15) mod 23 = 2. No known way gets from A and B to s without, in effect, finding a or b.", + "14": "No. The explanation says plainly that no one has proven the discrete logarithm is hard, or that s can't be found from A and B some other way. The security rests on decades of failed attempts.", + "15": "Mallory can replace both public numbers with her own (for example 5^3 mod 23 = 10). She then shares one secret with Alice (6) and another with Bob (5). She decrypts each message, reads it, and re-encrypts it for the other side, and neither of them notices." + }, + "audit": { + "terms": [ + "The key and the shared secret are treated as the same thing, then not. Line 26: s 'becomes the key for ordinary encryption'. Scope (line 84): 'a hash function scrambles s first; a cipher such as AES then encrypts'. It is unclear whether s is the key or only the input to it.", + "Three names for the hard-to-undo step: 'scrambled form' / 'unscramble' (line 5), 'operation that is hard to reverse' (line 14) and 'one-way operation' (line 54). The term 'one-way' is only defined at the end.", + "Two names for A and B: 'scrambled numbers' (line 5) and 'public number' (line 23)." + ], + "unexplained": [ + "'about 4,000 such steps' for a 2048-bit exponent (line 60): it doesn't say why roughly twice the bit count, or how the step count would change with a bigger or smaller p.", + "'about 2^112 operations (an estimate)' (line 58): it doesn't say where this comes from or how it changes with the size of p, so the reader can't tell how much safety a bigger or smaller p would give.", + "'2048 bits' (line 58): bits aren't tied to the size of p beyond '617 digits', and it isn't said what a smaller size such as 1024 would mean." + ], + "missing_why": [ + "The first-idea example (line 14) gives 'The result is 8' without naming the operation (5^6 mod 23), so the reader can't check it.", + "Line 60: the square-and-multiply steps are shown, but not why doubling the exponent each step lets a huge exponent take only a few thousand steps.", + "Why p must be prime and g must 'produce many different values' is left to Scope (lines 21 and 84), even though line 21 already requires a prime.", + "Line 28: the argument that taking remainders along the way doesn't change the result is shown for one product; the extension to repeated powers is left implicit." + ], + "excess": [ + "The HTML comments ('' and the '' markers) mean nothing to this reader.", + "Scope bullets on elliptic curves and quantum/ML-KEM (lines 82, 85) aren't used by the argument or the questions." + ] + }, + "gaps": [ + "The operation in the first-idea example (line 14) is never named.", + "It's unclear whether s itself is the encryption key or is hashed first (line 26 vs line 84).", + "No derivation for the ~4,000 steps or the 2^112 estimate.", + "Why p must be prime is deferred and never explained." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2, + "14": 2, + "15": 2 + }, + "reasons": {}, + "pass": true + }, + "usage": { + "cost": 0.1816422, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The key and the shared secret are treated as the same thing, then not. Line 26: s 'becomes the key for ordinary encryption'. Scope (line 84): 'a hash function scrambles s first; a cipher such as AES then encrypts'. It is unclear whether s is the key or only the input to it." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Three names for the hard-to-undo step: 'scrambled form' / 'unscramble' (line 5), 'operation that is hard to reverse' (line 14) and 'one-way operation' (line 54). The term 'one-way' is only defined at the end." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "Two names for A and B: 'scrambled numbers' (line 5) and 'public number' (line 23)." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'about 4,000 such steps' for a 2048-bit exponent (line 60): it doesn't say why roughly twice the bit count, or how the step count would change with a bigger or smaller p." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 58): it doesn't say where this comes from or how it changes with the size of p, so the reader can't tell how much safety a bigger or smaller p would give." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'2048 bits' (line 58): bits aren't tied to the size of p beyond '617 digits', and it isn't said what a smaller size such as 1024 would mean." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "The first-idea example (line 14) gives 'The result is 8' without naming the operation (5^6 mod 23), so the reader can't check it." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 60: the square-and-multiply steps are shown, but not why doubling the exponent each step lets a huge exponent take only a few thousand steps." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Why p must be prime and g must 'produce many different values' is left to Scope (lines 21 and 84), even though line 21 already requires a prime." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 28: the argument that taking remainders along the way doesn't change the result is shown for one product; the extension to repeated powers is left implicit." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "The HTML comments ('' and the '' markers) mean nothing to this reader." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "Scope bullets on elliptic curves and quantum/ML-KEM (lines 82, 85) aren't used by the argument or the questions." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "The operation in the first-idea example (line 14) is never named." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "It's unclear whether s itself is the encryption key or is hashed first (line 26 vs line 84)." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "No derivation for the ~4,000 steps or the 2^112 estimate." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "Why p must be prime is deferred and never explained." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150625-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150625-coldread-prose.json new file mode 100644 index 0000000..ce48c5d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150625-coldread-prose.json @@ -0,0 +1,358 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:06:25", + "model": "default", + "prompt_sha": "a76b387677f2", + "reply": { + "paragraphs": [ + { + "at": "Title (line 1)", + "text": "How can two strangers agree on a secret key when anyone can read their messages?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold summary (line 3)", + "text": "They never send the key. Each person keeps a private number...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 5", + "text": "This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 1 (line 9)", + "text": "Alice and Bob want to encrypt their messages...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 2 (line 12)", + "text": "A first idea fails. Alice picks the key, 6, and sends 5^6 mod 23 = 8...", + "unresolved": [ + "mod (in \"5^6 mod 23 = 8\"; it is defined only at line 16)" + ], + "leap": [ + "\"Each side then does one half of a calculation, and the other side finishes it without undoing it\": what the 'halves' are stays unclear until the numbered steps" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, mod definition (line 16)", + "text": "\"mod 23\" means \"the remainder after division by 23\"...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 1 (line 19)", + "text": "Alice and Bob agree in public on two numbers: a prime modulus p = 23 and a base g = 5.", + "unresolved": [], + "leap": [ + "Why p has to be prime is not said here (Scope later touches on choosing g, not p)" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 2 (line 20)", + "text": "Each picks a private number at random...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 3 (line 21)", + "text": "Each sends a public number: g raised to their private number, mod p...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 4 (line 22)", + "text": "Each raises the other's public number to their own private number, mod p...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 24", + "text": "The result, s = 2, is the shared secret...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why both get 2 (line 26)", + "text": "Alice computed (5^15)^6, and Bob computed (5^6)^15...", + "unresolved": [ + "k in \"(x + 23k) × y\" is not introduced (it can be inferred as 'some whole number')" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What Eve sees, table (lines 31-35)", + "text": "Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Predict (line 37)", + "text": "Eve has 8 and 19. Can she combine them into 2?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 39", + "text": "Multiplying fails: 8 × 19 mod 23 = 152 mod 23 = 14...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why powers, and why mod, intro (line 43)", + "text": "Each choice blocks a simpler way for Eve to find a private number.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: Without mod (line 46)", + "text": "Without mod, A = 5^6 = 15625. Eve takes the logarithm base 5...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: multiplication instead of power (line 49)", + "text": "With multiplication instead of a power, A = 5 × 6 mod 23 = 7...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: powers mod p (line 50)", + "text": "With powers mod p, the results jump around...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 52", + "text": "So g^x mod p is a one-way operation... the discrete logarithm.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 1 (line 56)", + "text": "With p = 23, Eve can try exponents in order...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 58", + "text": "Alice's own work stays small...", + "unresolved": [ + "the arrow in \"25 → 2\" (it can be inferred to mean 'take the remainder mod 23')" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Assumption, not proof (line 61)", + "text": "No one has proven that the discrete logarithm is hard...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The limit, para 1 (line 66)", + "text": "The exchange stops a reader. It does not stop Mallory...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Mallory bullets (lines 68-69)", + "text": "Alice computes 10^6 mod 23 = 6... Bob computes 10^15 mod 23 = 5...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 71", + "text": "Alice shares 6 with Mallory, and Bob shares 5 with Mallory...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer (line 75)", + "text": "The two strangers never send the key...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Scope (lines 79-83)", + "text": "Authentication... Elliptic-curve... hash function... AES... Quantum computers", + "unresolved": [ + "certificates", + "Elliptic-curve", + "hash function", + "AES", + "Post-quantum" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Line 12, \"sends 5^6 mod 23 = 8\": 'mod' is used before line 16 defines it. The reader has to guess what the operation is right at the point where the explanation shows why one shared number fails. This is a mild block, since the point 'hard to reverse' still comes through." + ], + "cuts": [ + "Swap lines 12 and 16: put the 'mod 23 means remainder' definition before the 'first idea fails' paragraph. This is a move rather than a cut, and it costs no words.", + "No real cuts are needed. Every paragraph serves the argument." + ], + "question": { + "text": "How can two parties who share only a public channel, one that an eavesdropper can read, end up with the same secret key?", + "known_at": "Title (line 1), with the setting made concrete at line 9", + "result_stated": "Yes, in words at line 3: \"They never send the key. Each person keeps a private number and sends a public number computed from it... no known fast method gets the private numbers back from them.\"" + }, + "motive": "Reason to care: \"A version of it starts most HTTPS connections.\" (line 5). Reason for the approach: \"A first idea fails... If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number.\" (line 12). Also \"Each choice blocks a simpler way for Eve to find a private number.\" (line 43), followed by the three bullets.", + "close": "Yes. Line 75 reads: \"Each sends g raised to a private number, mod p. Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p.\"", + "recap": "Each side sends 5 raised to a secret power, remainder 23. Each then raises the other's number to its own secret power, so both reach 5^(ab). Eve can't do this because getting the exponent back out of the remainder is believed to be infeasibly slow.", + "strongest": "'Why powers, and why mod' (lines 43-52). It rules out plain powers (logarithm) and multiplication (an inverse via Euclid) with concrete numbers, then shows the scrambled table of 5^x mod 23.", + "weakest": "Line 12. It uses 'mod' four lines before defining it, and 'one half of a calculation... the other side finishes it' stays abstract until the steps arrive.", + "suggestions": [ + "Define 'mod' before line 12, or move the 'first idea fails' paragraph to after the definition.", + "Add a short clause saying why p is prime, for example so that the powers of g can cover many values.", + "Say in a few words that k in (x + 23k) is any whole number." + ] + }, + "usage": { + "cost": 0.1372968, + "seconds": 43 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Line 12, \"sends 5^6 mod 23 = 8\": 'mod' is used before line 16 defines it. The reader has to guess what the operation is right at the point where the explanation shows why one shared number fails. This is a mild block, since the point 'hard to reverse' still comes through." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Swap lines 12 and 16: put the 'mod 23 means remainder' definition before the 'first idea fails' paragraph. This is a move rather than a cut, and it costs no words." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "No real cuts are needed. Every paragraph serves the argument." + }, + { + "n": 4, + "severity": "edge", + "kind": "unresolved", + "text": "The problem, para 2 (line 12): mod (in \"5^6 mod 23 = 8\"; it is defined only at line 16)" + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "The problem, para 2 (line 12): \"Each side then does one half of a calculation, and the other side finishes it without undoing it\": what the 'halves' are stays unclear until the numbered steps" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Exchange, step 1 (line 19): Why p has to be prime is not said here (Scope later touches on choosing g, not p)" + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "Why both get 2 (line 26): k in \"(x + 23k) × y\" is not introduced (it can be inferred as 'some whole number')" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Line 58: the arrow in \"25 → 2\" (it can be inferred to mean 'take the remainder mod 23')" + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): certificates" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): Elliptic-curve" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): hash function" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): AES" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): Post-quantum" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150725-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150725-coldread-prose.json new file mode 100644 index 0000000..add31a0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150725-coldread-prose.json @@ -0,0 +1,344 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:07:25", + "model": "default", + "prompt_sha": "a76b387677f2", + "reply": { + "paragraphs": [ + { + "at": "Title + bold summary (line 1-3)", + "text": "How can two strangers agree... They never send the key. Each person keeps a private number...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 5", + "text": "This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 1 (line 9)", + "text": "Alice and Bob want to encrypt their messages...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 2 (line 11)", + "text": "\"mod 23\" means \"the remainder after division by 23\"...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 3 (line 14)", + "text": "A first idea fails. Alice picks the key, 6, and sends 5^6 mod 23 = 8...", + "unresolved": [], + "leap": [ + "\"Each side then does one half of a calculation, and the other side finishes it without undoing it\": this is abstract, and it only becomes concrete in the next section. The reader has to take it on trust for now.", + "Why 5 and 23 are the numbers used is not said here. Step 1 of the next section explains it." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 1 (line 19)", + "text": "agree in public on two numbers: a prime modulus p = 23 and a base g = 5", + "unresolved": [], + "leap": [ + "The text never says why p has to be prime (Scope only touches on choosing g)." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 2 (line 20)", + "text": "Each picks a private number...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 3 (line 21)", + "text": "Alice sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 4 (line 22)", + "text": "Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 24", + "text": "The result, s = 2, is the shared secret...(see Scope)", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why both get 2 (line 26)", + "text": "Alice computed (5^15)^6, and Bob computed (5^6)^15...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What Eve sees, table (lines 31-35)", + "text": "Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Predict prompt (line 37)", + "text": "Eve has 8 and 19. Can she combine them into 2?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 39", + "text": "Multiplying fails: 8 × 19 mod 23 = 14...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why powers, intro (line 43)", + "text": "Each choice blocks a simpler way for Eve to find a private number.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: Without mod (line 46)", + "text": "A = 5^6 = 15625. Eve takes the logarithm base 5...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: multiplication instead of a power (line 49)", + "text": "A = 5 × 6 mod 23 = 7. Multiplying by 14 undoes multiplying by 5...", + "unresolved": [ + "Euclid's algorithm" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: powers mod p (line 50)", + "text": "the results jump around. For exponents 1 to 22...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 52", + "text": "So g^x mod p is a one-way operation... discrete logarithm.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 1 (line 56)", + "text": "With p = 23, Eve can try exponents in order... about 10^14 years.", + "unresolved": [], + "leap": [ + "\"The best known methods are faster, but still need about 2^112 operations\": the methods are not named and the figure is not explained. It is labelled an estimate, which is acceptable." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 2 (line 58)", + "text": "Alice's own work stays small... about 4,000 steps.", + "unresolved": [], + "leap": [ + "\"a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications\": the example (6 = 3×2, 3 = 2+1) does not show how an arbitrary exponent breaks down into squarings and multiplications. The reader has to infer the link to the exponent's binary digits." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Assumption, not proof (line 61)", + "text": "No one has proven that the discrete logarithm is hard...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The limit, para 1 (line 66)", + "text": "The exchange stops a reader. It does not stop Mallory...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Mallory bullets (lines 68-69)", + "text": "Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 71", + "text": "Alice shares 6 with Mallory... (a digital signature).", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer (line 75)", + "text": "The two strangers never send the key...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Scope bullets (lines 79-83)", + "text": "Authentication... Elliptic-curve... hash function... AES... Quantum computers", + "unresolved": [ + "hash function", + "AES", + "Elliptic-curve", + "certificates", + "Post-quantum key exchange" + ], + "leap": [], + "excess": [ + "\"it is the version most HTTPS connections use\": this partly repeats line 5's \"A version of it starts most HTTPS connections\", though it adds which version." + ], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Scope line 80: drop \"it is the version most HTTPS connections use\" or merge it into line 5 (a small repeat). Little else can be cut without loss." + ], + "question": { + "text": "How can two people who have never met agree on a shared secret key when an eavesdropper reads every message they exchange?", + "known_at": "Title (line 1), made concrete in The problem (line 9)", + "result_stated": "Yes, in words at the very start: \"They never send the key. Each person keeps a private number and sends a public number computed from it... An eavesdropper sees only the public numbers, and no known fast method gets the private numbers back from them.\"" + }, + "motive": "Reason to care: \"A version of it starts most HTTPS connections\" (line 5) and \"Encryption needs a key that both of them know. They have never met, so every message goes over a public channel\" (line 9). Reason for the approach: the failed first idea (line 14: \"If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number\") and the section 'Why powers, and why mod' (\"Each choice blocks a simpler way for Eve to find a private number\").", + "close": "Yes. Answer (line 75): \"The two strangers never send the key. Each sends g raised to a private number, mod p. Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p.\"", + "recap": "Each side mixes a public base with its own secret exponent mod a prime and swaps the results. Raising the other side's result to your own secret gives you both the same number, and an eavesdropper can only get it by solving the discrete logarithm, which nobody knows how to do fast.", + "strongest": "'Why powers, and why mod' (lines 43-52): each alternative (no mod, multiplication) is shown failing with concrete numbers, and then the scrambled list of 5^x mod 23 makes 'one-way' visible.", + "weakest": "Line 58: the claim 'about 2048 squarings plus at most 2048 multiplications' rests on a single small example (exponent 6) and never says how a general exponent splits into squarings and multiplications.", + "suggestions": [ + "In line 58, add one sentence: write the exponent in binary; square for each digit, and also multiply by g when the digit is 1.", + "Briefly say why p is prime (so that every nonzero remainder has an undoing multiplier, and powers of a good g run through all of them).", + "Name the faster methods in line 56 (for example, the number field sieve), or say plainly that they are beyond scope." + ] + }, + "usage": { + "cost": 0.143897, + "seconds": 44 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Scope line 80: drop \"it is the version most HTTPS connections use\" or merge it into line 5 (a small repeat). Little else can be cut without loss." + }, + { + "n": 2, + "severity": "edge", + "kind": "leap", + "text": "The problem, para 3 (line 14): \"Each side then does one half of a calculation, and the other side finishes it without undoing it\": this is abstract, and it only becomes concrete in the next section. The reader has to take it on trust for now." + }, + { + "n": 3, + "severity": "edge", + "kind": "leap", + "text": "The problem, para 3 (line 14): Why 5 and 23 are the numbers used is not said here. Step 1 of the next section explains it." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Exchange, step 1 (line 19): The text never says why p has to be prime (Scope only touches on choosing g)." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Bullet: multiplication instead of a power (line 49): Euclid's algorithm" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Why real numbers are huge, para 1 (line 56): \"The best known methods are faster, but still need about 2^112 operations\": the methods are not named and the figure is not explained. It is labelled an estimate, which is acceptable." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Why real numbers are huge, para 2 (line 58): \"a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications\": the example (6 = 3×2, 3 = 2+1) does not show how an arbitrary exponent breaks down into squarings and multiplications. The reader has to infer the link to the exponent's binary digits." + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): hash function" + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): AES" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): Elliptic-curve" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): certificates" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): Post-quantum key exchange" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150823-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150823-quiz-prose.json new file mode 100644 index 0000000..1f72e87 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150823-quiz-prose.json @@ -0,0 +1,188 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:08:10", + "model": "default", + "prompt_sha": "bedad2b89e75", + "reply": { + "answers": { + "1": "Alice and Bob, the two parties. Eve, who reads every message but changes none. Mallory, who can replace messages. The public prime modulus p (23) and the public base g (5). The private numbers a and b. The public numbers A = g^a mod p and B = g^b mod p. The shared secret s = g^(ab) mod p, which becomes the encryption key. The 'mod' (remainder) operation. The one-way operation g^x mod p, whose reverse is the discrete logarithm. Authentication (digital signatures).", + "2": "Everyone, including Eve, knows p and g. Each party picks a private number and computes a public number from it with g^x mod p. The public numbers travel in the open. Each party raises the other's public number to its own private number, mod p, and both arrive at s. Eve sees p, g, A and B but not a, b or s. To get s she would in effect have to find a or b, which is the discrete logarithm problem. Mallory can slip in his own public number, because nothing in the exchange proves who sent a public number. Authentication fixes this.", + "3": "Each side sends g^(private) mod p. Each raises the other's value to its own private number, and since (g^b)^a = (g^a)^b = g^(ab), both get the same s. Eve would have to undo the mod-p power to find a or b, and no fast method for that is known when p is large.", + "4": "The key variable is the size of p (and of a and b). With a small p, like 23, Eve can try exponents one by one and finds a = 6 on the 6th try. With a 2048-bit p and a and b chosen at random from that range, trying them one by one is hopeless. The best known methods need about 2^112 operations, roughly 10^14 years at a trillion operations per second. Alice's own work grows only to about 4,000 steps, because repeated squaring doubles the exponent at each step. So a larger p makes Eve's job vastly harder while the honest parties' work grows only slowly.", + "5": "Assumption: the discrete logarithm is hard, and Eve cannot get s from A and B some other way. This is not proven; it rests on decades of failed attempts. Simplification: Eve only reads and never changes messages. Established fact: (x^a)^b = x^(ab), and taking remainders along the way does not change the final remainder. Another fact: with plain powers (no mod), a logarithm recovers the exponent.", + "6": "Alice and Bob agree in public on a prime p and a base g, and each secretly picks a private number. Each sends g raised to that private number, mod p. Each raises the number they receive to their own private number, mod p, and the exponent rule makes both results g^(ab) mod p, the shared secret. An eavesdropper sees only g^a and g^b mod p. Getting a or b back is the discrete logarithm problem, which is believed (but not proven) to be infeasible for a 2048-bit p, while the forward computation stays cheap. The exchange does not stop an attacker who replaces messages, so real systems add digital signatures.", + "7": "The hardest part was the argument that taking remainders along the way does not change the result, '(x + 23k) × y = ...', and how it extends to powers. A worked example showing (5^15 mod 23)^6 mod 23 equal to 5^90 mod 23 would help. The squaring step count ('2048 squarings plus at most 2048 multiplications') was also terse. It needed a word on reading the exponent in binary.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8, which matches 5^6 mod 23 = 8 in the table.", + "9": "Without mod, A = 5^6 = 15625, and Eve takes the logarithm base 5 to get 6, because plain powers grow in order. With mod, the results jump around (5, 2, 10, 4, 20, 8, ...), so a result gives no hint of where the exponent lies. No fast reverse is known. Multiplication mod p would also fail, because Euclid's algorithm quickly finds the number that undoes it.", + "10": "No. Nothing in the exchange tells Alice who sent a public number. Mallory can substitute his own. Real systems add authentication, such as digital signatures and certificates.", + "11": "If Alice sends the key through a one-way operation (for example 5^6 mod 23 = 8), Bob receives the same 8 that Eve does. If Eve cannot recover 6 from 8, Bob cannot either. So each side needs its own private number, and each completes the other's half without undoing it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2, and Bob computes 8^15 mod 23 = 2. The shared secret is 2.", + "13": "8 × 19 mod 23 = 14. Multiplying powers adds the exponents, so this gives 5^(6+15) mod 23. The shared secret needs the exponents multiplied, 5^(6×15). No known way gets from A and B to s without, in effect, finding a or b.", + "14": "No. It is an assumption, not a proof. No one has proven that the discrete logarithm is hard, or that there is no other route from A and B to s. Confidence rests on decades of failed attempts.", + "15": "Mallory replaces both public numbers with his own (5^3 mod 23 = 10). He then shares one secret with Alice (6) and a different one with Bob (5). He decrypts each message, reads it, and re-encrypts it for the other side, and neither party notices. Authentication prevents this." + }, + "audit": { + "terms": [ + "The number 6 is called 'the key' in the first-idea paragraph (line 14: 'Alice picks the key, 6'). Later the same 6 is 'a private number' (line 20: 'Alice picks a = 6'), and the key is something built from s. So 'key' means two different things.", + "'shared secret' (line 24) vs 'key' (lines 24 and 82: 'build the key ... from it'; 'use the output as the key'). The difference is only explained in Scope.", + "'p' / 'prime modulus' (line 19) vs 'mod 23' (line 11). The link between 'modulus' and the 'mod' notation is never stated outright.", + "'undoing number' (line 49) is an unnamed inverse. It is fine, but it is a separate term for the same idea as 'undoes'." + ], + "unexplained": [ + "'about 2^112 operations (an estimate)' (line 56): it does not say where the figure comes from or how it changes with the size of p.", + "'2048 bits, about 617 decimal digits' (line 56): it does not say why 2048, or what a smaller p would mean for security.", + "'about 4,000 steps' (line 58): there is no comparison showing how small this is next to 2^112." + ], + "missing_why": [ + "Line 58: 'Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications'. It does not explain how an arbitrary exponent is built from squarings and multiplications (its binary digits).", + "Line 56: 'The best known methods are faster' gives no hint of why they beat trying exponents one by one, or why they still fail.", + "Line 19 / line 82: why p must be prime is deferred to Scope and only briefly justified there ('lets a good g reach every non-zero remainder').", + "Line 50: 'the results jump around ... A result near 8 says nothing about where 6 is' asserts that the outputs are unpredictable. It gives no reason why mod causes this beyond the single example." + ], + "excess": [ + "Line 80: Scope's elliptic-curve paragraph; no question or step of the argument uses it.", + "Line 83: the quantum-computer paragraph; it is a detour from the argument.", + "Line 75: the 'Answer' section largely repeats the bold summary at line 3." + ] + }, + "gaps": [ + "The explanation never shows that the exchange gives the right answer when you reduce mod p along the way (for example, a check that 5^90 mod 23 = 2). The reader has to trust the general argument.", + "How Euclid's algorithm finds the undoing number is not shown. That is acceptable, but the reader has to take it on trust.", + "How the shared secret becomes an encryption key (the hash function) is only named in Scope." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 1, + "13": 2, + "14": 2, + "15": 2 + }, + "reasons": { + "12": "The numbers are correct (A = 8, B = 19, s = 2), but the expected statement is missing: Alice computes B^a mod p, Bob computes A^b mod p, and both equal g^(ab) mod p because (g^a)^b = (g^b)^a and reducing mod p at each step does not change the remainder. The rubric says a case alone is not required for 2, so a case without the principle earns only partial credit. Note for the explainer: the question asks only for numbers while the expect asks for the reason, so the item wording may need fixing (explainer decide).", + "13": "Full credit, narrowly. The answer does not name the discrete logarithm problem. It does state that no known way gets from A and B to s without in effect finding a or b, which is the substance of the expect." + }, + "pass": false + }, + "usage": { + "cost": 0.1955264, + "seconds": 46 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The number 6 is called 'the key' in the first-idea paragraph (line 14: 'Alice picks the key, 6'). Later the same 6 is 'a private number' (line 20: 'Alice picks a = 6'), and the key is something built from s. So 'key' means two different things." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'shared secret' (line 24) vs 'key' (lines 24 and 82: 'build the key ... from it'; 'use the output as the key'). The difference is only explained in Scope." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'p' / 'prime modulus' (line 19) vs 'mod 23' (line 11). The link between 'modulus' and the 'mod' notation is never stated outright." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'undoing number' (line 49) is an unnamed inverse. It is fine, but it is a separate term for the same idea as 'undoes'." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 56): it does not say where the figure comes from or how it changes with the size of p." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'2048 bits, about 617 decimal digits' (line 56): it does not say why 2048, or what a smaller p would mean for security." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 4,000 steps' (line 58): there is no comparison showing how small this is next to 2^112." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 58: 'Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications'. It does not explain how an arbitrary exponent is built from squarings and multiplications (its binary digits)." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 56: 'The best known methods are faster' gives no hint of why they beat trying exponents one by one, or why they still fail." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 19 / line 82: why p must be prime is deferred to Scope and only briefly justified there ('lets a good g reach every non-zero remainder')." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Line 50: 'the results jump around ... A result near 8 says nothing about where 6 is' asserts that the outputs are unpredictable. It gives no reason why mod causes this beyond the single example." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "Line 80: Scope's elliptic-curve paragraph; no question or step of the argument uses it." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Line 83: the quantum-computer paragraph; it is a detour from the argument." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "Line 75: the 'Answer' section largely repeats the bold summary at line 3." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "The explanation never shows that the exchange gives the right answer when you reduce mod p along the way (for example, a check that 5^90 mod 23 = 2). The reader has to trust the general argument." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "How Euclid's algorithm finds the undoing number is not shown. That is acceptable, but the reader has to take it on trust." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "How the shared secret becomes an encryption key (the hash function) is only named in Scope." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150941-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150941-quiz-prose.json new file mode 100644 index 0000000..57cb5d6 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150941-quiz-prose.json @@ -0,0 +1,180 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:09:31", + "model": "default", + "prompt_sha": "97e170ce48d4", + "reply": { + "answers": { + "1": "Alice and Bob, who want a shared key. Eve, who reads every message but changes none. Mallory, who can replace messages. The public prime modulus p and the public base g. The private numbers a and b. The public numbers A = g^a mod p and B = g^b mod p. The shared secret s = g^(ab) mod p. A hash function that turns s into the encryption key. The operation g^x mod p, which is one-way, and its reverse problem, the discrete logarithm.", + "2": "p and g are public. Each side keeps its own private number and sends the matching public number, A = g^a mod p or B = g^b mod p. Each side then raises the other side's public number to its own private number, mod p, and both get s. s goes through a hash to become the key. Eve sees p, g, A and B, but not a, b or s. To get s she would have to undo g^x mod p, which is the discrete logarithm problem, and no fast method for it is known. Mallory can swap the public numbers. Authentication (digital signatures) is what stops that.", + "3": "Each side combines its own private exponent with the other side's public value, and (g^b)^a = (g^a)^b = g^(ab), so both reach the same s. An eavesdropper only has g^a and g^b. Combining those gives g^(a+b), not g^(ab), and getting a or b back needs a discrete logarithm, which has no known fast method.", + "4": "The most important variable is the size of p, together with a and b chosen at random across that range. With p = 23, Eve finds a = 6 on her 6th try. With a 2048-bit p, the best known methods need about 2^112 operations, roughly 10^14 years at a trillion operations per second. Alice's own work only grows to about 4,000 steps, because repeated squaring doubles the exponent each time. So making p bigger hurts Eve enormously and costs Alice little. A second variable is the choice of g: with g = 22 the powers mod 23 are only 1 and 22, so s has only two possible values.", + "5": "Assumption: the discrete logarithm is hard and Eve has no other way to get s. This is not proven; it rests on decades of failed attempts. Simplification: Eve only reads and never changes messages. Established facts: (x^a)^b = x^(ab), and taking remainders along the way does not change the final remainder, since (x+23k)y = xy + 23ky. Also, Euclid's algorithm quickly finds the number that undoes a multiplication mod p.", + "6": "Alice and Bob agree in public on a prime p and a base g. Each picks a secret exponent and sends g raised to it, mod p. Each raises the number they receive to their own secret exponent, and by the exponent rule both get g^(ab) mod p, which they hash into a key. Eve sees only g^a and g^b. Multiplying them gives g^(a+b), and finding a or b is the discrete logarithm, which has no known fast method when p is huge. That hardness is assumed, not proven, and the exchange does not stop an attacker who swaps messages unless signatures are added.", + "7": "The hardest part was the 'With multiplication instead of a power' bullet. It uses Euclid's algorithm and the undoing number 14 as a black box. Showing how 14 is found, or at least saying that such a number exists for any g when p is prime, would help. Also, the 'first idea' paragraph calls 5^6 mod 23 'hard to reverse' before one-way operations are explained.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. Their shared secret is 8, which matches 5^6 mod 23 = 8 in the table.", + "9": "With plain powers, Eve takes the logarithm: log base 5 of 15625 is 6. Plain powers grow in order, so the logarithm reverses them. Mod p scrambles the results (5, 2, 10, 4, 20, 8, …), so a result gives no hint of where the exponent is. Powers rather than multiplication are used because multiplication mod p can be undone quickly by multiplying by an inverse found with Euclid's algorithm.", + "10": "No. Nothing in the exchange says who sent a public number. Mallory can replace both public numbers and end up sharing one secret with Alice and another with Bob. Real systems add authentication, such as digital signatures.", + "11": "Because whatever Bob can undo, Eve can undo too. They receive the same message. If the operation really is one-way, Bob cannot recover the key either. So each side needs its own private number, and each completes the other's half of the calculation without having to undo it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2, and Bob computes 8^15 mod 23 = 2, so s = 2. They must be equal because (5^15)^6 = (5^6)^15 = 5^(90), and taking remainders at intermediate steps does not change the final remainder.", + "13": "8 × 19 mod 23 = 14, which is 5^(6+15): multiplying powers adds the exponents. The shared secret needs the exponents multiplied, 5^(6×15). No known route gets from A and B to s without, in effect, finding a or b first.", + "14": "No. It is an assumption, not a proof. No one has proven that the discrete logarithm is hard, or that there is no other way to get s from A and B. The confidence comes from decades of failed attempts.", + "15": "A man-in-the-middle attack. Mallory replaces both public numbers with her own (10, from her private number 3). She then shares 6 with Alice and 5 with Bob, decrypts every message, reads it, and encrypts it again for the other side. Neither Alice nor Bob can detect this without authentication." + }, + "audit": { + "terms": [ + "The 'first idea' paragraph (line 14) says 'Alice picks the key, 6'. The exchange (line 20) then uses 6 as the private number a. Later, 'key' means the hash output of s (line 24). So 'key' names two different things, and the reused value 6 makes them easy to confuse.", + "'shared secret' (s, line 24) and 'key' (line 24, the hashed output) are kept apart in the text, but the Answer section and the title use 'secret key' loosely.", + "'hard to reverse' (line 14), 'one-way operation' (line 52) and 'no known fast reverse' (line 52) name the same idea. The first phrase is used before the concept is defined." + ], + "unexplained": [ + "'about 2^112 operations (an estimate)' (line 56): the text does not say what makes this enough, or what a smaller count such as 2^80 would mean for security.", + "'It needs smaller numbers for the same security' (line 80): it does not say how much smaller or why." + ], + "missing_why": [ + "Line 49: Euclid's algorithm 'finds this undoing number even for a huge p'. The text gives neither how it does this nor why the undoing number always exists.", + "Line 24: it does not say why s is passed through a hash function instead of being used directly as the key.", + "Line 50: it does not say why scrambled-looking outputs imply that no fast reverse exists. 'A result near 8 says nothing about where 6 is' is an intuition, not a reason.", + "Line 82: it does not say why p must be prime for a good g to reach every non-zero remainder.", + "Line 56: there is no hint of why the best known methods are faster than trying exponents one by one." + ], + "excess": [ + "Line 5: 'A version of it starts most HTTPS connections' is context that the argument does not use.", + "Scope bullets on elliptic curves, AES and quantum computers (lines 80, 81, 83) are detours that no question uses." + ] + }, + "gaps": [ + "Euclid's algorithm and the modular inverse are used as a black box.", + "The first-idea paragraph assumes one-way hardness before explaining it.", + "No reason is given for hashing s.", + "The 2^112 security figure has no reference point." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 1, + "14": 2, + "15": 2 + }, + "reasons": { + "13": "The answer correctly shows that A × B mod 23 = 14 = 5^(6+15), which is not s. It also says Eve would in effect have to find a or b. It never says that finding a or b is the discrete logarithm problem, or that no fast method for it is known. That missing step is the core of the expected answer." + }, + "pass": false + }, + "usage": { + "cost": 0.18770360000000003, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The 'first idea' paragraph (line 14) says 'Alice picks the key, 6'. The exchange (line 20) then uses 6 as the private number a. Later, 'key' means the hash output of s (line 24). So 'key' names two different things, and the reused value 6 makes them easy to confuse." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'shared secret' (s, line 24) and 'key' (line 24, the hashed output) are kept apart in the text, but the Answer section and the title use 'secret key' loosely." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'hard to reverse' (line 14), 'one-way operation' (line 52) and 'no known fast reverse' (line 52) name the same idea. The first phrase is used before the concept is defined." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 56): the text does not say what makes this enough, or what a smaller count such as 2^80 would mean for security." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'It needs smaller numbers for the same security' (line 80): it does not say how much smaller or why." + }, + { + "n": 6, + "severity": "gap", + "kind": "missing_why", + "text": "Line 49: Euclid's algorithm 'finds this undoing number even for a huge p'. The text gives neither how it does this nor why the undoing number always exists." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 24: it does not say why s is passed through a hash function instead of being used directly as the key." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 50: it does not say why scrambled-looking outputs imply that no fast reverse exists. 'A result near 8 says nothing about where 6 is' is an intuition, not a reason." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 82: it does not say why p must be prime for a good g to reach every non-zero remainder." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 56: there is no hint of why the best known methods are faster than trying exponents one by one." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "Line 5: 'A version of it starts most HTTPS connections' is context that the argument does not use." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "Scope bullets on elliptic curves, AES and quantum computers (lines 80, 81, 83) are detours that no question uses." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "Euclid's algorithm and the modular inverse are used as a black box." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The first-idea paragraph assumes one-way hardness before explaining it." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "No reason is given for hashing s." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "The 2^112 security figure has no reference point." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151044-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151044-quiz-prose.json new file mode 100644 index 0000000..eb0091f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151044-quiz-prose.json @@ -0,0 +1,187 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:10:36", + "model": "default", + "prompt_sha": "97e170ce48d4", + "reply": { + "answers": { + "1": "The main objects are Alice and Bob, Eve (who only reads messages) and Mallory (who can change them). There are two public numbers everyone sees: a prime modulus p and a base g. Each side has a private number (a for Alice, b for Bob) and a public number (A = g^a mod p, B = g^b mod p). Both sides end up with a shared secret s = g^(ab) mod p, which goes through a hash function to make the encryption key. The other objects are the one-way operation g^x mod p, the discrete logarithm (the reverse problem) and authentication (digital signatures).", + "2": "p and g are public. Each side's public number is made from its own private number using p and g. Each side then raises the other side's public number to its own private number, mod p, to get s. Hashing s gives the key. Eve sees p, g, A and B, but not a, b or s. To get s she would in effect have to solve the discrete logarithm for a or b. Mallory can put his own public number in place of A and B. Authentication is what stops him.", + "3": "Each side sends g raised to its private number mod p. Raising the other side's value to your own private number gives g^(ab) mod p for both, because of the exponent rule and because taking remainders along the way doesn't change the result. Eve, who sees only g^a and g^b mod p, would have to reverse the one-way operation, and no fast method for that is known.", + "4": "The key variable is the size of p (and of the range a and b are picked from). With p = 23, Eve can try exponents in order and finds a = 6 on the 6th try. With a 2048-bit p and random a and b, trying them one by one is hopeless. Even the best known methods need about 2^112 operations, roughly 10^14 years at a trillion operations per second. Alice's own work grows only slowly, about 4,000 steps for a 2048-bit exponent, because each squaring doubles the exponent. Another variable is the choice of g. A bad g (22 for p = 23) gives only two possible values for s.", + "5": "Assumption: no one has proven the discrete logarithm is hard, or that s can't be found from A and B some other way. Security rests on decades of failed attempts. Also, Eve is assumed to only read messages, never change them. Established facts: the exponent rule (x^a)^b = x^(ab), and the fact that taking remainders along the way doesn't change the final remainder. The text shows this with (x + 23k)y = xy + 23ky. In the example, 19^6 mod 23 = 8^15 mod 23 = 2.", + "6": "Alice and Bob agree in public on a prime p and a base g. Each secretly picks a random private number and sends g to that power, mod p. Each raises what they receive to their own private number, mod p, and both get g^(ab) mod p, which they hash to make the key. Eve sees only g^a and g^b mod p. Multiplying them only adds the exponents, and getting a back is the discrete logarithm, which has no known fast method when p is huge (an assumption, not a proof). The exchange doesn't stop someone who replaces messages, so real systems add digital signatures for authentication.", + "7": "The hardest part was the claim that powers mod p are hard to reverse. The text says the next section 'shows why no fast method for it is known', but it only lists values that 'jump around' and says 'a result near 8 says nothing about where 6 is'. That is intuition, not a reason. It would have helped to say plainly that this shows why the easy reversals (logarithm, undoing a multiplication) don't work, not that no other method exists. The '2^112 (an estimate)' figure also needed a word on where it comes from.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8 (= 5^6 mod 23, matching the table).", + "9": "With plain powers, A = 5^6 = 15625, and Eve just takes log base 5 to get 6, because plain powers grow in order. Mod p makes the results jump around, so the size of the result says nothing about the exponent, and no fast reverse is known. Plain multiplication mod p doesn't work either, because Euclid's algorithm quickly finds the number that undoes it.", + "10": "No. The exchange stops someone who only reads, but nothing in it tells Alice who sent a public number. Authentication (a digital signature on B) is needed.", + "11": "If Alice sends one-way(key), Bob can't recover the key any more than Eve can, because both receive exactly the same thing. So each side needs its own private number. Each side does half the calculation, and the other finishes it without undoing anything.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2 and Bob computes 8^15 mod 23 = 2, so s = 2. They must be equal because (5^15)^6 = (5^6)^15 = 5^(90) by the exponent rule. Reducing mod 23 along the way removes only multiples of 23 at each multiplication, so the final remainder doesn't change.", + "13": "Multiplying powers adds the exponents: 8 × 19 mod 23 = 14 = 5^(6+15) mod 23. The secret needs the exponents multiplied, 5^(6×15) mod 23 = 2. There is no known way from A and B to s other than, in effect, finding a or b first.", + "14": "No. It is an assumption backed by decades of failed attempts, not a proof. The text also says a large quantum computer could solve the discrete logarithm fast.", + "15": "Mallory can put his own public number (for example 5^3 mod 23 = 10) in place of both A and B. He then shares one secret with Alice (6) and another with Bob (5). He decrypts each message, reads it, and re-encrypts it for the other side, and neither of them notices. Authentication with digital signatures prevents this." + }, + "audit": { + "terms": [ + "Line 14 calls 6 'a key' ('Alice chooses a key, the number 6'). In line 20 the same 6 is 'a private number', and from line 24 'key' means the hash of s. So one word covers two different things.", + "Line 14 says 'hard to reverse' and line 52 says 'one-way operation'. Line 39 calls the reverse problem 'the discrete logarithm'. It's the same idea, but the formal name comes after the informal phrase has been used twice.", + "Line 19 says 'modulus p' and also '(the number after \"mod\")', while line 11 says 'mod 23'. That's fine, but the reader has to connect 23 = p themselves before line 19." + ], + "unexplained": [ + "Line 56 says 'about 2^112 operations (an estimate)' without saying where the estimate comes from, or how it changes for a bigger or smaller p.", + "Line 56 says 'a p of 2048 bits, about 617 decimal digits' without saying why 2048 was chosen, or what a smaller p would give up.", + "Line 80 says ECDH 'needs smaller numbers for the same security' but gives no measure of how much smaller." + ], + "missing_why": [ + "Line 39 says 'the next section shows why no fast method for it is known'. Line 50 ('A result near 8 says nothing about where 6 is') only rules out the logarithm shortcut. It doesn't explain why other methods fail, and the 'best known methods are faster' in line 56 are never described.", + "Line 24 says Alice and Bob 'feed s into the same hash function' but doesn't say why s isn't used as the key directly.", + "Line 19 requires a prime modulus without saying why. The reason (a good g reaches every non-zero remainder) appears only in the Scope list at line 82.", + "Line 58 says '2048 squarings plus at most 2048 multiplications' but doesn't show how the exponent's binary digits decide when to multiply. The 5^6 example skips that step.", + "Line 56 says a and b are picked 'at random from that whole range' without saying why being random matters, beyond making one-by-one trying hopeless." + ], + "excess": [ + "Lines 79–83 cover Scope items (AES, elliptic curves, quantum computers) that the argument and questions don't use.", + "Line 5 ('A version of it starts most HTTPS connections') is context the argument doesn't use.", + "Line 75 (Answer) repeats the summary at line 3 almost word for word." + ] + }, + "gaps": [ + "The explanation doesn't actually show why the discrete logarithm is hard. It only shows that the simple reversals fail, even though it promises to show why no fast method is known.", + "There's no explanation of where the 2^112 figure comes from, or what the 'best known methods' are.", + "The reason for using a prime p is left to the Scope section." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 1, + "11": 2, + "12": 2, + "13": 2, + "14": 2, + "15": 2 + }, + "reasons": { + "10": "The answer correctly says 'No', notes that the exchange only stops a reader, and says authentication is needed. It never states the attack: an active attacker can replace A and B with his own public numbers and run a separate exchange with each side. Saying 'nothing tells Alice who sent a public number' points at the gap but does not say what an attacker does with it. Partial." + }, + "pass": false + }, + "usage": { + "cost": 0.1907202, + "seconds": 43 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "Line 14 calls 6 'a key' ('Alice chooses a key, the number 6'). In line 20 the same 6 is 'a private number', and from line 24 'key' means the hash of s. So one word covers two different things." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Line 14 says 'hard to reverse' and line 52 says 'one-way operation'. Line 39 calls the reverse problem 'the discrete logarithm'. It's the same idea, but the formal name comes after the informal phrase has been used twice." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "Line 19 says 'modulus p' and also '(the number after \"mod\")', while line 11 says 'mod 23'. That's fine, but the reader has to connect 23 = p themselves before line 19." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "Line 56 says 'about 2^112 operations (an estimate)' without saying where the estimate comes from, or how it changes for a bigger or smaller p." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 56 says 'a p of 2048 bits, about 617 decimal digits' without saying why 2048 was chosen, or what a smaller p would give up." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 80 says ECDH 'needs smaller numbers for the same security' but gives no measure of how much smaller." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 39 says 'the next section shows why no fast method for it is known'. Line 50 ('A result near 8 says nothing about where 6 is') only rules out the logarithm shortcut. It doesn't explain why other methods fail, and the 'best known methods are faster' in line 56 are never described." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 24 says Alice and Bob 'feed s into the same hash function' but doesn't say why s isn't used as the key directly." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 19 requires a prime modulus without saying why. The reason (a good g reaches every non-zero remainder) appears only in the Scope list at line 82." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 58 says '2048 squarings plus at most 2048 multiplications' but doesn't show how the exponent's binary digits decide when to multiply. The 5^6 example skips that step." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Line 56 says a and b are picked 'at random from that whole range' without saying why being random matters, beyond making one-by-one trying hopeless." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "Lines 79–83 cover Scope items (AES, elliptic curves, quantum computers) that the argument and questions don't use." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Line 5 ('A version of it starts most HTTPS connections') is context the argument doesn't use." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "Line 75 (Answer) repeats the summary at line 3 almost word for word." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "The explanation doesn't actually show why the discrete logarithm is hard. It only shows that the simple reversals fail, even though it promises to show why no fast method is known." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "There's no explanation of where the 2^112 figure comes from, or what the 'best known methods' are." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The reason for using a prime p is left to the Scope section." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151159-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151159-quiz-prose.json new file mode 100644 index 0000000..b867901 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151159-quiz-prose.json @@ -0,0 +1,194 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:11:47", + "model": "default", + "prompt_sha": "97e170ce48d4", + "reply": { + "answers": { + "1": "Alice and Bob (the two parties), Eve (an eavesdropper who reads but does not change messages), Mallory (an attacker who replaces messages), the public numbers p (a prime modulus) and g (the base), the private numbers a and b, the public numbers A = g^a mod p and B = g^b mod p, the shared secret s = g^(ab) mod p, the hash function that turns s into the encryption key, the one-way operation g^x mod p and its hard reverse (the discrete logarithm), and authentication (digital signatures).", + "2": "p and g are agreed in public. Each party keeps a private number and uses it to compute a public number: A = g^a mod p and B = g^b mod p. These are sent openly, so Eve sees p, g, A and B. Each party raises the other's public number to their own private number mod p. Both get the same s, which is hashed into the key. Eve could only get s by finding a or b from A or B, which is the discrete logarithm. No fast method for it is known. Mallory can swap in her own public number and end up sharing one secret with each side. Authentication prevents this.", + "3": "Each side sends g^private mod p. Raising the other side's value to your own private number gives (g^b)^a = (g^a)^b = g^(ab) mod p, so both get the same secret. Eve sees only g^a and g^b, and multiplying them only adds the exponents. To get s she would have to undo the one-way operation, which no known fast method can do.", + "4": "The most important variable is the size of p, together with the range a and b are drawn from. With a small p (23), Eve tries exponents one by one and finds a = 6 on the 6th try. With a 2048-bit p and random a and b from that whole range, trying one by one is hopeless. Even the best known methods need about 2^112 operations, around 10^14 years at a trillion operations per second. Alice's own work grows only with the number of bits (about 4,000 steps), because each squaring doubles the exponent. Separately, a bad g (e.g. 22) makes s take only two possible values.", + "5": "Assumption: the discrete logarithm is hard, and there is no other way to get s from A and B. The explanation says this is not proven and rests on decades of failed attempts. A simplification is that Eve only reads and never changes messages. Established fact: with p = 23 and g = 5, 19^6 mod 23 = 8^15 mod 23 = 2. Also, (x^a)^b = x^(ab), and taking remainders along the way does not change the final remainder.", + "6": "Alice and Bob publicly agree on a prime p and a base g, and each secretly picks a private number. Each sends g raised to their private number, mod p (the remainder after dividing by p). Each raises the number they receive to their own private number mod p, and both get g^(ab) mod p, which they hash into an encryption key. An eavesdropper who sees only g^a and g^b cannot combine them (multiplying adds the exponents instead of multiplying them). Undoing g^x mod p (the discrete logarithm) has no known fast method for huge p, although this is not proven. The exchange does not stop someone who replaces messages, so real systems add digital signatures.", + "7": "The 'Why real numbers are huge' section was hardest. The 2^112 figure appears without any link to the 2048-bit size. 'Best known methods are faster' is not explained even briefly. The square-and-multiply example (5^2, then 5^3, then 5^6) does not say why the multiply step is needed or how you pick the order of steps from the exponent's bits. One sentence on writing the exponent in binary would help. The remainder argument in 'Why both get 2' is short but works.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8, and Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8, which equals 5^6 mod 23 = 8 in the explanation's table.", + "9": "Plain powers grow in order, so Eve could take the logarithm base 5 of A (e.g. log5 of 15625 = 6) and recover the private number. Taking remainders mod p makes the results jump around (5, 2, 10, 4, 20, 8, ...), so a result near 8 says nothing about where the exponent is. No fast reverse is known: it is a one-way operation.", + "10": "No. Nothing in the exchange tells Alice who sent a public number. Mallory can replace both public numbers. Authentication (digital signatures, and in HTTPS certificates) is needed.", + "11": "Whatever Alice sends, Bob would have to turn it back into the key. If Eve cannot reverse the one-way operation, Bob cannot either, because he has the same information Eve has. So each side needs its own private number. Each does half the calculation, and the other side finishes it without undoing it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2 and Bob computes 8^15 mod 23 = 2, so s = 2. They must be equal because Alice computes (5^15)^6 and Bob computes (5^6)^15, and both equal 5^(6×15) by the exponent rule. Taking remainders along the way only removes multiples of 23: (x + 23k)·y = xy + 23ky, and a power is a chain of products. So the final remainder is the same.", + "13": "Multiplying powers adds the exponents: 8 × 19 mod 23 = 14 = 5^(6+15) mod 23. The secret needs the exponents multiplied, 5^(6×15) mod 23 = 2. No known way from A and B to s avoids, in effect, finding a or b first.", + "14": "No. It is an assumption. No one has proven that the discrete logarithm is hard, or that s cannot be found from A and B some other way. The belief rests on decades of failed attempts. A large quantum computer could solve the discrete logarithm fast.", + "15": "Run a man-in-the-middle attack. Mallory picks her own private number (3) and sends 10 = 5^3 mod 23 to both sides in place of their public numbers. She then shares 6 with Alice and 5 with Bob. She decrypts each message, reads it, and re-encrypts it for the other side, and neither side notices. Only authentication stops this." + }, + "audit": { + "terms": [ + "The same number is called 'key' and later 'private number': line 14 'Alice chooses a key, the number 6' versus line 20 'private number ... Alice picks a = 6'. The reader may think the private number is the key, while line 24 says the key is the hashed output of s.", + "'shared secret' (line 24) versus 'key' (lines 9, 24, 75 'never send the key'). The answer says they 'never send the key', but what they agree on directly is s. The key is hash(s).", + "'hard to reverse' (line 14), 'no known fast method' (line 3), 'no known fast reverse' and 'one-way operation' (line 52) all name the same idea before it is defined at line 52.", + "'eavesdropper' (line 9) versus 'a reader' (line 66) for Eve's role." + ], + "unexplained": [ + "'about 2^112 operations (an estimate)' (line 56): no link to the 2048-bit size and no explanation of why it is below the 2^2048 tries needed by brute force.", + "'a p of 2048 bits, about 617 decimal digits' (line 56): it does not say what security would be lost with fewer bits or gained with more.", + "'about 4,000 steps' (line 58): the claim of at most 2048 multiplications is not tied to the binary digits of the exponent." + ], + "missing_why": [ + "Fast exponentiation (line 58): '5^3 = 2 × 5 = 10, 5^6 = 10^2' does not explain why you multiply by 5 at some steps and not others, or how the steps follow from the exponent.", + "Hashing s (line 24): it does not say why s is fed into a hash function instead of being used directly as the key.", + "'The best known methods are faster' (line 56): it gives no hint why they beat trying exponents one by one.", + "Why p must be prime (line 19, and only partly in Scope line 81): step 1 requires a prime without saying why.", + "'A result near 8 says nothing about where 6 is' (line 50): this shows the scrambling but does not argue why no shortcut exists. The point is only made later as an unproven assumption." + ], + "excess": [ + "Scope bullets on Elliptic-curve Diffie–Hellman, AES and quantum computers (lines 80–82) are detours the argument does not use. Line 80 also has two bullets merged into one line.", + "'A standard fast method (Euclid's algorithm)' (line 49): the name is not needed. Showing the inverse 14 is enough." + ] + }, + "gaps": [ + "It does not explain how 2048 bits leads to the 2^112 estimate, or why the best methods beat brute force.", + "It does not say why the shared secret is hashed before use as the key.", + "The square-and-multiply procedure is only sketched, with no rule for choosing between squaring and multiplying.", + "Scope line 80 merges the 'Elliptic-curve' and 'Using the key' bullets because a line break is missing." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 1, + "14": 2, + "15": 2 + }, + "reasons": { + "13": "Shows correctly that multiplying adds the exponents (14 = 5^(6+15), not s = 2) and that Eve would in effect have to find a or b. But it never says that finding a or b is the discrete logarithm problem, or that this problem has no known fast method. The reason Eve is stuck is left implicit." + }, + "pass": false + }, + "usage": { + "cost": 0.20325159999999998, + "seconds": 52 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same number is called 'key' and later 'private number': line 14 'Alice chooses a key, the number 6' versus line 20 'private number ... Alice picks a = 6'. The reader may think the private number is the key, while line 24 says the key is the hashed output of s." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'shared secret' (line 24) versus 'key' (lines 9, 24, 75 'never send the key'). The answer says they 'never send the key', but what they agree on directly is s. The key is hash(s)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'hard to reverse' (line 14), 'no known fast method' (line 3), 'no known fast reverse' and 'one-way operation' (line 52) all name the same idea before it is defined at line 52." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'eavesdropper' (line 9) versus 'a reader' (line 66) for Eve's role." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 56): no link to the 2048-bit size and no explanation of why it is below the 2^2048 tries needed by brute force." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'a p of 2048 bits, about 617 decimal digits' (line 56): it does not say what security would be lost with fewer bits or gained with more." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 4,000 steps' (line 58): the claim of at most 2048 multiplications is not tied to the binary digits of the exponent." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Fast exponentiation (line 58): '5^3 = 2 × 5 = 10, 5^6 = 10^2' does not explain why you multiply by 5 at some steps and not others, or how the steps follow from the exponent." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Hashing s (line 24): it does not say why s is fed into a hash function instead of being used directly as the key." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'The best known methods are faster' (line 56): it gives no hint why they beat trying exponents one by one." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Why p must be prime (line 19, and only partly in Scope line 81): step 1 requires a prime without saying why." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'A result near 8 says nothing about where 6 is' (line 50): this shows the scrambling but does not argue why no shortcut exists. The point is only made later as an unproven assumption." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Scope bullets on Elliptic-curve Diffie–Hellman, AES and quantum computers (lines 80–82) are detours the argument does not use. Line 80 also has two bullets merged into one line." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "'A standard fast method (Euclid's algorithm)' (line 49): the name is not needed. Showing the inverse 14 is enough." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "It does not explain how 2048 bits leads to the 2^112 estimate, or why the best methods beat brute force." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "It does not say why the shared secret is hashed before use as the key." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The square-and-multiply procedure is only sketched, with no rule for choosing between squaring and multiplying." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "Scope line 80 merges the 'Elliptic-curve' and 'Using the key' bullets because a line break is missing." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151256-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151256-quiz-prose.json new file mode 100644 index 0000000..54f5d2a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151256-quiz-prose.json @@ -0,0 +1,196 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:12:48", + "model": "default", + "prompt_sha": "98dd2856417f", + "reply": { + "answers": { + "1": "Alice and Bob, who want a shared key. Eve, who reads every message but changes none. Mallory, who can replace messages. The public numbers p (a prime modulus, 23) and g (the base, 5). The private numbers a and b. The public numbers A = g^a mod p and B = g^b mod p. The shared secret s, which goes through a hash function to become the encryption key. The one-way operation g^x mod p, whose reverse is the discrete logarithm. Authentication (digital signatures) is the fix for Mallory.", + "2": "p and g are agreed in public. Each side keeps its private number and sends g raised to it, mod p (A or B). Each side then raises the other side's public number to its own private number, mod p, and both get s = g^(ab) mod p. Eve sees p, g, A and B but not a, b or s. To get s she would in effect have to find a or b from A or B, which is the discrete logarithm. Mallory sits in the middle and makes a separate secret with each side. Authentication links a public number to the person who sent it.", + "3": "Each side combines its private exponent with the other side's public power. Since (g^b)^a = (g^a)^b = g^(ab), and taking remainders along the way doesn't change the final remainder, both get the same s. Eve only has g^a and g^b, which give her g^(a+b) when multiplied, so she would need to undo the exponentiation mod p, and no fast method for that is known.", + "4": "The size of p, with a and b chosen at random from its whole range. With p = 23, Eve can try exponents one by one and finds a = 6 on her 6th try. With a 2048-bit p, trying them one by one is hopeless, and even the best known methods need about 2^112 operations (about 10^14 years at a trillion per second). Meanwhile Alice's work only grows with the number of bits: about 4,000 steps, because each squaring doubles the exponent. A bigger p makes Eve's work much harder but adds little to Alice's and Bob's.", + "5": "Assumption: the discrete logarithm is hard, and Eve has no other route from A and B to s. This is not proven; it rests on decades of failed attempts. The setup also assumes Eve only reads and never changes messages. Established facts: 5^6 mod 23 = 8, and both sides compute 2. Also, (x + 23k)·y = xy + 23ky, which is why taking remainders early is safe.", + "6": "Alice and Bob agree in public on a prime p and a base g. Each keeps a random private exponent and sends g raised to it, mod p. Each raises the number it receives to its own exponent, and both get g^(ab) mod p, which they hash into a key. An eavesdropper can only multiply the public numbers, which adds the exponents instead of multiplying them, so she would have to solve the discrete logarithm, and no fast method is known for a large p (this is believed, not proven). It does not stop someone who replaces messages, so real systems add signatures.", + "7": "The hardest part was the claim that a 2048-bit exponent needs only about 4,000 steps. The example (5^2 → 5^3 → 5^6) doesn't show how the exponent's bits decide when to square and when to multiply. A worked example with the exponent written in binary would help. The argument that 'results jump around' doesn't by itself show why no fast reverse exists. The step that taking remainders early is safe is also dense, but its small example helps.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8, which matches 5^6 mod 23 = 8 in the table.", + "9": "Without mod, A = 5^6 = 15625, and Eve just takes the logarithm base 5 to get 6, because plain powers grow in order. Mod p scrambles the order (5, 2, 10, 4, 20, 8, ...), so a result gives no hint of where the exponent is. Plain multiplication mod p is also weak, because Euclid's algorithm quickly finds the number that undoes it.", + "10": "No. Nothing in the exchange tells Alice who sent a public number. Mallory can swap in her own public number and make a separate secret with each side. Authentication, such as digital signatures, is needed.", + "11": "Bob receives the same output Eve does. If Eve can't get the key back from it, Bob can't either. That's why each side needs its own private number, so that each can finish the other's calculation without undoing it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2, and Bob computes 8^15 mod 23 = 2, so s = 2. They must be equal because (5^15)^6 = (5^6)^15 = 5^(90). Taking remainders along the way only removes multiples of 23 from each factor, so the final remainder is the same.", + "13": "Multiplying adds the exponents: 8 × 19 mod 23 = 14 = 5^(6+15) mod 23. The secret needs the exponents multiplied, 5^(6×15). Eve would in effect have to find a or b, for example get 6 from 8, which is the discrete logarithm. That is hard because powers mod p jump around with no order a logarithm could use. For a large p, the only known methods are trying exponents (hopeless) or faster methods that still take about 2^112 operations.", + "14": "No. It is an unproven assumption, supported by decades of failed attempts. A large quantum computer could solve it fast.", + "15": "Mallory can replace each public number with her own (for example 10 = 5^3 mod 23). She then shares one secret with Alice (6) and another with Bob (5), decrypts every message, reads it, and re-encrypts it for the other side, and neither of them notices." + }, + "audit": { + "terms": [ + "Alice's private number 6 is called 'a key' in the first idea ('Alice chooses a key, the number 6', line 14) but 'private number' a = 6 later (line 20). The word 'key' also means the final encryption key made from s (line 24).", + "'shared secret' s (line 24) vs 'the key for ordinary encryption' (line 24) vs 'the key' in the Answer (line 75, 'never send the key'). It's easy to mix up s with the key.", + "'hard to reverse' (line 14) vs 'one-way operation' (line 52) vs 'discrete logarithm' (lines 39, 52): one idea under three names, with the formal name given only later.", + "'modulus' p (line 19) vs 'the number after mod' vs '23' in 'mod 23' (line 11)." + ], + "unexplained": [ + "'about 2^112 operations (an estimate)' (line 56): no source and no way to see how it depends on the 2048-bit size of p, i.e. what a smaller or larger p would give.", + "'2048 bits, about 617 decimal digits' (line 56): it doesn't say why 2048 is enough or what a smaller size would risk.", + "'about 4,000 steps' (line 58): the count is asserted without showing how the exponent's bits lead to it." + ], + "missing_why": [ + "Square-and-multiply (line 58): 'Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications'. It isn't explained how the bits of the exponent choose between squaring and multiplying.", + "Why s is put through a hash function rather than used directly as the key (line 24).", + "'A result near 8 says nothing about where 6 is' (line 50): jumbled-looking output is offered as the reason no fast reverse exists, which is suggestive rather than an explanation.", + "'The best known methods are faster' (line 56): no hint of why they are faster yet still slow.", + "Why p must be prime (lines 19, 81): it is mentioned only in Scope and only partly explained." + ], + "excess": [ + "Scope bullets on elliptic curves, AES and quantum computers (lines 80-82): none of the questions or the main argument uses them.", + "'A version of it starts most HTTPS connections' (line 5): useful context, but the argument doesn't use it.", + "The Answer section (lines 73-75) repeats the summary in line 3 and adds nothing." + ] + }, + "gaps": [ + "The explanation doesn't show how to compute something like 5^15 mod 23 = 19 or 8^15 mod 23 = 2 by hand. The reader has to trust these values or check them against the power table.", + "Line 80 is broken: the 'Using the key' bullet is joined onto the elliptic-curve bullet.", + "The reader isn't told why a fast way to reverse multiplication mod p (Euclid's algorithm) exists while none is known for powers mod p." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2, + "14": 2, + "15": 2 + }, + "reasons": { + "6": "Full marks, but one phrase overstates: 'an eavesdropper can only multiply the public numbers' is not right. Multiplying is the obvious shortcut that fails; Eve is not limited to it. The rest of the answer correctly says she would need the discrete logarithm.", + "14": "Full marks. The quantum-computer remark is correct but out of scope per model.md.", + "7": "No single correct answer. It names specific hard parts: repeated squaring by binary exponent, why scrambling implies no fast reverse, and safety of early remainders." + }, + "pass": true + }, + "usage": { + "cost": 0.1924918, + "seconds": 44 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "Alice's private number 6 is called 'a key' in the first idea ('Alice chooses a key, the number 6', line 14) but 'private number' a = 6 later (line 20). The word 'key' also means the final encryption key made from s (line 24)." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'shared secret' s (line 24) vs 'the key for ordinary encryption' (line 24) vs 'the key' in the Answer (line 75, 'never send the key'). It's easy to mix up s with the key." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'hard to reverse' (line 14) vs 'one-way operation' (line 52) vs 'discrete logarithm' (lines 39, 52): one idea under three names, with the formal name given only later." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'modulus' p (line 19) vs 'the number after mod' vs '23' in 'mod 23' (line 11)." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 56): no source and no way to see how it depends on the 2048-bit size of p, i.e. what a smaller or larger p would give." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'2048 bits, about 617 decimal digits' (line 56): it doesn't say why 2048 is enough or what a smaller size would risk." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 4,000 steps' (line 58): the count is asserted without showing how the exponent's bits lead to it." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Square-and-multiply (line 58): 'Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications'. It isn't explained how the bits of the exponent choose between squaring and multiplying." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Why s is put through a hash function rather than used directly as the key (line 24)." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'A result near 8 says nothing about where 6 is' (line 50): jumbled-looking output is offered as the reason no fast reverse exists, which is suggestive rather than an explanation." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'The best known methods are faster' (line 56): no hint of why they are faster yet still slow." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "Why p must be prime (lines 19, 81): it is mentioned only in Scope and only partly explained." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Scope bullets on elliptic curves, AES and quantum computers (lines 80-82): none of the questions or the main argument uses them." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "'A version of it starts most HTTPS connections' (line 5): useful context, but the argument doesn't use it." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "The Answer section (lines 73-75) repeats the summary in line 3 and adds nothing." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "The explanation doesn't show how to compute something like 5^15 mod 23 = 19 or 8^15 mod 23 = 2 by hand. The reader has to trust these values or check them against the power table." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "Line 80 is broken: the 'Using the key' bullet is joined onto the elliptic-curve bullet." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "The reader isn't told why a fast way to reverse multiplication mod p (Euclid's algorithm) exists while none is known for powers mod p." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/understanding.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/understanding.md new file mode 100644 index 0000000..9a7bd0b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/explainers/key-exchange/review/understanding.md @@ -0,0 +1,24 @@ +# Review record: key-exchange (Published tier, Stage 1 prose) + +## Checks run + +| Check | Result | +|---|---| +| `explainer check key-exchange` | ok (final) | +| Probe (`2026-10-04T145920-probe`) | 16 findings; adopted 1 2 3 6 7 8 10 12 14 15 (two private numbers needed, Eve-shortcut assumption, multiplication case made concrete, remainder rule instance, range of a/b, one name per concept, 2^112 estimate, hash/quantum moved to Scope) | +| Cold read, narrative (`150134`) | 5 blocking (terse beats, remainder rule, unclear multiplication case, placeholder close): all fixed in narrative.md | +| Cold read, prose (`150411`, `150625`, `150725`) | round 2 had 1 blocking (mod used before defined, from a round-1 fix): fixed. Round 3: no blocking | +| Blind test, prose | runs `150452` PASS 15/15; `150823` item 12 = 1 (question asked for numbers, rubric wanted reason: question widened); `150941` item 13 = 1 (prose now links "find a or b" to discrete logarithm); `151044` item 10 = 1 (terse correct answer; grader variance); `151159` item 13 = 1 (question narrower than rubric: widened); final run PASS 15/15 | +| Broken copy (mod "only keeps numbers small"; exchange "proves identity") | FAIL — the test can fail | + +## Declined findings (edge) + +- Why p must be prime, in full: one line in Scope only. +- Where 2^112 comes from; how the faster discrete-log methods work: labeled estimate, out of scope. +- How Euclid's algorithm works: named as a fast standard method; detail not needed for the argument. +- Why scrambled outputs imply no fast reverse: stated as the assumption it is. +- Scope names (AES, elliptic curves, quantum): kept as pointers. + +## Rule learned + +A quiz `ask` must request everything its rubric `expect` requires (the reason, not only the numbers); two failures came from asks narrower than their expects. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/read.json new file mode 100644 index 0000000..9187efd --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/read.json @@ -0,0 +1,49 @@ +{ + "answers": { + "1": { + "answer": "Each person raises the other's public number to their own private number, mod p. This operation is safe because it's a one-way function: no known fast method can reverse it to find the private numbers from the public ones.", + "quote": "Each raises the other's public number to their own private number, mod p. To get further, Eve must find a or b, and no fast method is known for a large p." + }, + "2": { + "answer": "Both compute the same value because when Alice raises g^b to power a and Bob raises g^a to power b, the exponent rule makes both equal g^(ab). The key is never sent because each person computes it locally from the public numbers and their private number.", + "quote": "Alice computed (5^15)^6, and Bob computed (5^6)^15. By the exponent rule, both equal 5^(6×15)." + }, + "3": { + "answer": "The exchange alone doesn't stop a man-in-the-middle attacker who intercepts and replaces the public numbers with their own. Authentication using digital signatures stops this attack.", + "quote": "It does not stop Mallory, an attacker who can replace messages. Real systems add **authentication**: Bob attaches to B a value that only Bob can produce and anyone can check (a digital signature)." + }, + "4": { + "answer": "Alice sends 8 and Bob sends 19. They share the shared secret s = 2.", + "quote": "Alice sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. The result, s = 2, is the **shared secret**." + } + }, + "stops": [ + "Each side then does one half of a calculation, and the other side finishes it without undoing it", + "(x + 23k) × y = x × y + 23ky, for any whole number k", + "Can Eve combine them into 2?", + "Multiplying powers adds the exponents", + "Plain powers grow in order, so a logarithm reverses them", + "Multiplying by 14 undoes multiplying by 5, because 5 × 14 = 70 = 3 × 23 + 1", + "For exponents 1 to 22, 5^x mod 23 gives: 5, 2, 10, 4, 20, 8, 17, 16, 11, 9, 22, 18, 21, 13, 19, 3, 15, 6, 7, 12, 14, 1", + "No one has proven that the discrete logarithm is hard", + "Nothing in the exchange tells Alice who sent a public number", + "hash function, a fixed scrambling recipe" + ], + "excess": [ + "A version of it starts most HTTPS connections", + "A first idea fails", + "*Predict before you read on:*", + "A standard fast method (Euclid's algorithm) finds this undoing number", + "At a trillion operations per second, that is about 10^14 years", + "Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings", + "Example: 5^3 = 125, and 125 mod 23 = 10. Reducing first gives (25 mod 23) × 5 = 2 × 5 = 10", + "A bad g fails: for example, with g = 22 the powers mod 23 are only 1 and 22", + "The calculations below use one piece of notation" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/run.json new file mode 100644 index 0000000..0fe357d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/key-exchange/explanation.md`, as prose only, because the answer is a short chain of steps worked with small numbers (p = 23, g = 5) that a diagram or page would only repeat; it passed the probe and cold read and scored 15/15 on the blind test (a deliberately broken copy failed), and two scratch files that rm needed approval to delete, `calc.py` and `explainers/key-exchange-broken/`, are still there to remove.", + "error": null, + "cost": 2.4703193999999997, + "seconds": 1162, + "turns": 112, + "files": [ + "explainers/key-exchange/explanation.md" + ], + "review_cost": 2.0136, + "reviews": 12 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/explanation.md new file mode 100644 index 0000000..9e8bd97 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/explanation.md @@ -0,0 +1,50 @@ +# How can two strangers agree on a secret key in public? + +**Each person keeps one number private and sends a value computed from it. Each person then combines the other person's value with their own private number. Both get the same shared key, but the shared key is never sent. An eavesdropper sees every message and still cannot compute it, because one step is easy to do and, as far as anyone knows, very hard to undo.** + +The method is the Diffie–Hellman key exchange. Below, Alice and Bob want a shared key. Eve reads every message on the channel. + +"x mod p" means the remainder after you divide x by p. For example, 25 mod 23 = 2. + +## How it works + + + +1. Agree in public on a prime p and a base g. Here, p = 23 and g = 5. Eve knows them too. +2. Each person picks a **private number** at random and never sends it. Alice picks a = 6. Bob picks b = 15. +3. Each person computes a **public value**, g to the power of their private number, mod p, and sends it: + - Alice: A = 5^6 mod 23 = 8. + - Bob: B = 5^15 mod 23 = 19. +4. Each person raises the *other* public value to their *own* private number: + - Alice: 19^6 mod 23 = **2**. + - Bob: 8^15 mod 23 = **2**. + +The two results are equal because 19 = 5^15 and 8 = 5^6, so both people compute 5^(6·15) mod 23. Order does not matter: (g^b)^a = (g^a)^b = g^(ab). The **shared key** is 2. + +## Why Eve fails + + + +Eve sees 23, 5, 8 and 19. To get the key she needs a or b. That means solving 5^a mod 23 = 8 for a. This problem is the **discrete logarithm**. + +At p = 23 Eve can try every exponent: 5^1, 5^2, … mod 23 gives 5, 2, 10, 4, 20, 8. She finds a = 6 on try 6. So p must be large. Real systems use a p of 2048 bits (about 617 digits). The best known methods then need about 2^112 operations, far beyond any computer (NIST estimate). Alice and Bob still compute their values quickly, because repeated squaring needs only about 2 × 2048 multiplications. + +Eve cannot combine 8 and 19 directly either. 8 × 19 gives 5^(6+15), not 5^(6·15). + +## Why this form + + + +**Why powers, not multiplication?** Suppose Alice sent A = g·a mod p = 5·6 mod 23 = 7. Eve multiplies 7 by 14, because 5·14 mod 23 = 1. She gets 7·14 mod 23 = 6, which is a. Division undoes multiplication fast. No similar shortcut is known for powers mod p. + + + +**Why mod p?** Without it, Alice sends 5^6 = 15625. Eve computes log base 5 of 15625 = 6. Plain powers grow smoothly, so an ordinary logarithm undoes them. Mod p scrambles the order: the list 5, 2, 10, 4, 20, 8 above has no pattern to follow. + +## Limits + + + +- **Eve must only read (assumption).** Suppose Mallory can also replace messages. Mallory picks 3 and sends 5^3 mod 23 = 10 to Alice in place of 19. Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6. Alice now shares key 6 with Mallory and believes it is Bob. Mallory does the same with Bob. The fix is to authenticate the public values, for example with signatures and certificates, as HTTPS does. +- **Hardness is assumed, not proven.** No fast method for the discrete logarithm is known on ordinary computers. A large quantum computer would break it (Shor's algorithm), so newer systems add post-quantum key exchange. +- Most systems today use the same idea on elliptic curves (X25519), which gives equal security with much smaller numbers. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/model.md new file mode 100644 index 0000000..ca20fe5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/model.md @@ -0,0 +1,82 @@ +# Semantic model: Diffie–Hellman key exchange + +> Source of truth. Every rendering derives from this file. + +## Central question + +How can two people who have never met agree on a secret key over a channel that anyone can read? + +## Audience and prior knowledge + +A technical reader: school mathematics (powers, remainders), basic programming. Knows what encryption is in a general way. Does not know modular arithmetic as a tool, one-way functions, or Diffie–Hellman. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Alice, Bob | The two people who want a shared key. | — | +| Eve | An eavesdropper who reads every message on the channel but does not change messages. | assumption (passive) | +| Mallory | An attacker who can read and replace messages. | used for the limit case | +| public numbers p, g | A large prime p and a base g, known to everyone. | fact | +| private number | A number each person picks at random and never sends (a for Alice, b for Bob). | fact | +| public value | g raised to the private number, mod p (A = g^a mod p, B = g^b mod p); sent in the open. | fact | +| shared key | g^(ab) mod p, which both compute and Eve cannot. | fact | +| mod p | The remainder after division by p. | definition | + +## Causal chain + +```text +pick private number → send public value → raise the other's public value to your private number → same shared key +(B^a = g^(ba) = g^(ab) = A^b) +Eve has g, p, A, B but needs a or b → finding a from A = g^a mod p (discrete logarithm) is infeasible for large p +``` + +## Quantities + +Worked example: p = 23, g = 5, a = 6, b = 15. +- A = 5^6 mod 23 = 8. B = 5^15 mod 23 = 19. +- Alice: 19^6 mod 23 = 2. Bob: 8^15 mod 23 = 2. Shared key = 2. +- Size of p: at p = 23 Eve tries all 22 exponents at once. At 2048 bits (about 617 digits) the best known methods need about 2^112 operations (NIST estimate). + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| exponentiation | multiply: A = g·a mod p | A = 5·6 mod 23 = 7. Eve multiplies by the inverse of 5 (14): 7·14 mod 23 = 6 = a. Division undoes multiplication fast. | +| mod p | plain powers, no mod | 5^6 = 15625; Eve computes log base 5 of 15625 = 6. Plain powers grow smoothly, so a logarithm undoes them. Mod p scrambles the order: 5^1..5^6 mod 23 = 5, 2, 10, 4, 20, 8. | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| Eve cannot get the key | Eve sees 23, 5, 8, 19; to get 2 she needs a or b. For a 2048-bit p no known method finds them. | Small p: at p = 23 Eve tries 5^1, 5^2, … until she gets 8, and finds a = 6 within 22 tries. | +| The key is shared only with the intended person | Passive Eve only reads. | Active Mallory replaces B with M = 5^3 mod 23 = 10. Alice computes 10^6 mod 23 = 6; Mallory computes 8^3 mod 23 = 6. Alice shares key 6 with Mallory and thinks it is Bob. Fix: authenticate public values (signatures, certificates). | + +## Terms + +- mod p: remainder after division by p. +- private number: never sent. public value: sent. shared key: computed, never sent. +- discrete logarithm: finding a from g^a mod p. + +## Epistemic status + +- **Mathematical consequence:** B^a = A^b mod p. +- **Assumed:** the discrete logarithm is hard for large p. Not proven; no fast classical method is known. A large quantum computer would break it (Shor's algorithm). +- **Estimate:** 2^112 operations for 2048-bit p. + +## Confusion points + +- "The key is sent encrypted" → the shared key is never sent at all; each side computes it. +- "Eve can multiply A and B" → A·B = g^(a+b), not g^(ab). +- "This proves who you are talking to" → no; it needs authentication (Mallory case). + +## Scope + +- **Out of scope:** elliptic-curve Diffie–Hellman (X25519) — same idea in a different group, smaller numbers; used in TLS today. +- **Out of scope:** how the shared key then encrypts data (AES); post-quantum key exchange (ML-KEM). + +## Representation decision + +- **Stage:** 1 +- **Reason:** a short sequence of steps with one worked example; no topology or parameter space to explore. +- **Levels:** L1 the answer / L2 the steps with numbers / L3 why Eve fails and why this form / L4 limits. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/model.yaml new file mode 100644 index 0000000..629be53 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/explainers/key-exchange/model.yaml @@ -0,0 +1,68 @@ +values: + p: 23 + g: 5 + a: 6 + b: 15 + A: 8 + B: 19 + key: 2 + mallory_m: 3 + mallory_M: 10 + mallory_key: 6 + mult_A: 7 + inverse_g: 14 + plain_power: 15625 + real_bits: 2048 + real_digits: 617 + real_cost_exp: 112 + +allow: [1, 2, 3, 4, 10, 20, 22, 25] + +accept_unjustified: + log: "log appears only as Eve's attack on the no-mod alternative (why-mod), not as an operation of the method" + +require: + - path: p + in: [prose] + - path: g + in: [prose] + - path: A + in: [prose] + - path: B + in: [prose] + - path: key + in: [prose] + +claims: + - id: mech-exchange + kind: mechanism + statement: "Each side raises the other's public value to its own private number mod p; both get g^(ab) mod p." + example: "p=23, g=5, a=6, b=15: A=8, B=19; 19^6 mod 23 = 2 and 8^15 mod 23 = 2." + - id: why-exp + kind: why + about: [exponentiation] + statement: "Exponentiation mod p is easy forward and hard to reverse; multiplication is easy to reverse." + alternative: "A = g*a mod p" + counterexample: "A = 5*6 mod 23 = 7; Eve multiplies by 14 (inverse of 5) and gets 6 = a." + - id: why-mod + kind: why + about: [mod] + statement: "Mod p scrambles the powers so no smooth logarithm undoes them." + alternative: "plain powers g^a" + counterexample: "5^6 = 15625; log base 5 of 15625 = 6." + - id: guarantee-eve + kind: guarantee + statement: "A passive eavesdropper cannot compute the shared key if p is large." + example: "Eve sees 23, 5, 8, 19 and needs a or b; for 2048-bit p no known method finds them." + counterexample: "At p = 23 Eve tries all 22 exponents and finds a = 6." + ask: "Eve sees p, g, A and B. Why can she not compute the shared key?" + - id: limit-mitm + kind: limit + statement: "Without authentication, an active attacker can sit in the middle." + counterexample: "Mallory sends 10 instead of 19; Alice and Mallory both compute key 6." + ask: "What can an attacker who can change messages do, and what stops it?" + +terms: + - term: shared key + means: "g^(ab) mod p, computed by both sides, never sent" + avoid: [sends the key, send the key] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/read.json new file mode 100644 index 0000000..9593ef1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "Modular exponentiation (g^a mod p) makes this possible. What makes it safe is that the reverse operation—computing discrete logarithm—is extremely hard.", + "quote": "one step is easy to do and, as far as anyone knows, very hard to undo." + }, + "2": { + "answer": "Both compute the same value g^(ab) mod p because exponent order doesn't matter: (g^b)^a = (g^a)^b. The key itself is never transmitted.", + "quote": "Order does not matter: (g^b)^a = (g^a)^b = g^(ab)." + }, + "3": { + "answer": "A man-in-the-middle attack where Mallory replaces messages. The fix is to authenticate the public values with signatures and certificates.", + "quote": "The fix is to authenticate the public values, for example with signatures and certificates, as HTTPS does." + }, + "4": { + "answer": "Alice sends 8, Bob sends 19. Their shared key is 2.", + "quote": "Alice: A = 5^6 mod 23 = 8. Bob: B = 5^15 mod 23 = 19. The **shared key** is 2." + } + }, + "stops": [ + "Agree in public on a prime p and a base g.", + "Order does not matter: (g^b)^a = (g^a)^b = g^(ab).", + "Eve multiplies 7 by 14, because 5·14 mod 23 = 1.", + "Mod p scrambles the order: the list 5, 2, 10, 4, 20, 8 above has no pattern.", + "The best known methods then need about 2^112 operations, far beyond any computer.", + "repeated squaring needs only about 2 × 2048 multiplications." + ], + "excess": [ + "For example, 25 mod 23 = 2.", + "Eve knows them too.", + "She finds a = 6 on try 6.", + "At p = 23 Eve can try every exponent: 5^1, 5^2, … mod 23 gives 5, 2, 10, 4, 20, 8.", + "Eve cannot combine 8 and 19 directly either. 8 × 19 gives 5^(6+15), not 5^(6·15).", + "Most systems today use the same idea on elliptic curves (X25519), which gives equal security with much smaller numbers.", + "A large quantum computer would break it (Shor's algorithm), so newer systems add post-quantum key exchange." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/run.json new file mode 100644 index 0000000..40f984a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/key-exchange/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/key-exchange/explanation.md`, as prose (with its `model.md` and `model.yaml` source files), because Diffie–Hellman key exchange is a short sequence of steps that a single worked example with small numbers (p = 23, g = 5, shared key 2) carries completely; a diagram or interactive page would add nothing. It passes `explainer check` and covers why Eve fails, why powers and mod p are used, and the man-in-the-middle limit; I checked the arithmetic by hand, but the attempt to confirm it with Python wasn't approved.", + "error": null, + "cost": 0.36011959999999993, + "seconds": 80, + "turns": 14, + "files": [ + "explainers/key-exchange/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/explainers/quorum/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/explainers/quorum/explanation.md new file mode 100644 index 0000000..be4b4b1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/explainers/quorum/explanation.md @@ -0,0 +1,95 @@ +# Why 5 servers survive 2 failures, but 4 servers survive only 1 + +At first this looks wrong. Four servers are more than three, so surely they should survive more failures. They don't. A 4-server cluster tolerates exactly as many failures as a 3-server cluster: one. To see why, you need one rule that these clusters follow and the reason the rule exists. + +## The setting + +Many systems keep several copies of the same data on different machines. Databases, configuration stores (etcd, ZooKeeper, Consul) and coordination services all do this. The copies are there so that if a machine dies, the data and the service carry on. + +Copies create a new problem, though: **the servers have to agree**. Suppose a client says "set the balance to £50" and, a moment later, another client says "set the balance to £80". Every server has to end up with the same answer, applied in the same order. If some servers think the balance is £50 and others think it is £80, the system is broken, and broken in a quiet way that nobody notices until later. + +Protocols such as Raft and Paxos solve this. They differ in detail, but they share one rule: + +> **A decision only counts once a majority of the servers have accepted it.** + +A majority means *more than half of all the servers in the cluster*, counting the ones that are currently down. That set of servers is called a **quorum**. + +## Why a majority, and not "whoever is still up"? + +When a server stops responding, the others **cannot tell whether it has crashed or whether the network between them has broken**. Both cases look the same from the outside: messages go out and nothing comes back. + +That matters because of the dangerous case, a **network partition**. All the servers are running fine, but the network splits them into groups that can't talk to each other. Each group sees the other group as "dead". + +Suppose the rule were "the servers that are still reachable carry on". After a partition, *both* groups would carry on. One side accepts "balance = £50", the other accepts "balance = £80", and each side believes it is the real cluster. This is called **split brain**, and it is exactly the disagreement the system exists to prevent. + +The majority rule prevents it with a simple fact about counting: + +> **Two majorities of the same group always share at least one member.** + +If each of two groups contained more than half the servers, together they would contain more than all of them, which is impossible unless some server is in both. So at most one side of any partition can hold a majority. That side keeps working and the other side stops and waits. Because any two quorums overlap, every decision is seen by at least one server that also took part in the previous decision, and that server is what makes later decisions consistent with earlier ones. + +So the majority rule is not an arbitrary choice. It is the smallest group size that guarantees two groups can never both act independently. + +## Now count + +A cluster keeps working as long as a majority of its servers are still up and can reach each other. So: + +- **quorum size** = the smallest number that is more than half of *n*, which is ⌊n/2⌋ + 1 +- **failures tolerated** = n − quorum size + +**5 servers.** More than half of 5 (2.5) is **3**. You can lose 5 − 3 = **2** servers and still have 3 left, which is a majority. ✅ + +**4 servers.** More than half of 4 (2) is **3**. Exactly half is *not* a majority, because two groups of 2 could both claim to be "half". You can lose 4 − 3 = **1** server. Lose 2 and only 2 remain, which is not more than half, so the cluster stops accepting writes. ❌ + +| Servers (n) | Quorum (majority) | Failures tolerated | +|:-----------:|:-----------------:|:------------------:| +| 1 | 1 | 0 | +| 2 | 2 | 0 | +| 3 | 2 | **1** | +| 4 | 3 | **1** | +| 5 | 3 | **2** | +| 6 | 4 | **2** | +| 7 | 4 | **3** | + +The pattern is that the number of failures tolerated only goes up when you go from an even number of servers to an odd one. Adding a 4th server to a 3-server cluster raises the quorum from 2 to 3, so the extra server is used up by the larger quorum. + +## "But with 4, can't the 2 survivors just carry on?" + +This is the natural objection, so here it is in concrete form. Four servers, A, B, C and D. The network splits into {A, B} and {C, D}. + +``` + [A] [B] ║ [C] [D] + ║ + "C and D ║ "A and B + are dead"║ are dead" +``` + +If 2 out of 4 were enough to continue, both halves would continue. Both would accept writes and their data would diverge. Nobody in either half can tell that the other half is alive, so neither half can safely decide that it is the one to keep going. The only safe behaviour is for **both halves to stop**. + +Now try 5 servers split 3–2: + +``` + [A] [B] [C] ║ [D] [E] + 3 of 5: ║ 2 of 5: + majority, ║ not a majority, + keeps going ║ waits +``` + +However you split 5, one side always has at least 3, so exactly one side carries on. An odd number can never be divided into two equal halves, and that is the whole difference. + +A crash is handled the same way as a partition, because from the inside the two look the same. With 4 servers and 2 crashed, the 2 survivors are in exactly the position of the {A, B} group above. They cannot know the other two aren't alive somewhere, so they must stop. + +## What this means in practice + +- **Clusters are usually built with odd sizes: 3, 5, sometimes 7.** A 4th or 6th server costs money and adds one more machine that has to acknowledge each write, but it adds no extra failure tolerance. +- **To survive *f* failures you need 2f + 1 servers.** That is 3 servers for 1 failure, 5 for 2 and 7 for 3. +- **Even sizes aren't useless.** The extra server can spread out read load or act as a warm spare. It just doesn't help the cluster stay up. +- **Bigger isn't automatically better.** Every write has to wait for a majority, so larger clusters mean more servers to wait for on each write. Most production systems stop at 5. + +## One caveat + +All of this assumes that a failed server simply **stops**: it crashes or becomes unreachable, but it never lies. If servers might send wrong or malicious messages (called *Byzantine* failures), the arithmetic gets stricter. Surviving *f* such servers needs 3f + 1 machines. The reasoning is still based on overlapping groups. The overlap just has to be large enough to outvote the liars. + +## Summary + +A cluster may only act when **more than half** of its servers agree, because any two such groups overlap and so two halves of a broken network can never both make decisions. With 5 servers, more than half is 3, so 2 can fail. With 4 servers, more than half is still 3, so only 1 can fail. Exactly half doesn't count, since the other half could make the same claim. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/read.json new file mode 100644 index 0000000..19b26ed --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/read.json @@ -0,0 +1,40 @@ +{ + "answers": { + "1": { + "answer": "A decision only counts once a majority of the servers have accepted it.", + "quote": "A decision only counts once a majority of the servers have accepted it." + }, + "2": { + "answer": "A majority ensures two groups can never both act independently during a partition, because any two majorities of the same group must share at least one server.", + "quote": "Two majorities of the same group always share at least one member." + }, + "3": { + "answer": "A 6-server cluster survives 2 failures, as shown in the table where a quorum of 4 is needed, and 6 minus 4 equals 2.", + "quote": "6 | 4 | **2**" + }, + "4": { + "answer": "The group of 3 can still commit writes because 3 is a majority of 5 servers, while the group of 2 cannot because 2 is not more than half.", + "quote": "3 of 5: majority, keeps going ║ 2 of 5: not a majority, waits" + } + }, + "stops": [ + "\"counting the ones that are currently down\" - why count failed servers when computing majority?", + "\"Two majorities of the same group always share at least one member\" - needs proof of why this is mathematically true", + "The floor notation ⌊n/2⌋ + 1 in line 37 may be unfamiliar to some readers", + "Why exactly half is not a majority: \"two groups of 2 could both claim to be 'half'\" needs clearer explanation", + "The connection between network partition and crash scenarios being \"the same from the inside\" could be explained more" + ], + "excess": [ + "The £50/£80 balance example is illustrative but not essential to understanding the core rule", + "\"Clusters are usually built with odd sizes: 3, 5, sometimes 7\" is practice context not needed for the questions", + "The formula \"To survive f failures you need 2f + 1 servers\" is redundant with the table above it", + "The discussion that \"Even sizes aren't useless\" and can spread read load is contextual but off-topic", + "The entire Byzantine failures caveat section (3f + 1) is a separate case not relevant to the main explanation" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": false + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/run.json new file mode 100644 index 0000000..2254900 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/quorum/explanation.md`, a prose explanation, because the answer is a short chain of reasoning (why a majority is needed, then counting) that reads best as text, with a table and two small partition sketches doing the job diagrams would.", + "error": null, + "cost": 0.0993212, + "seconds": 32, + "turns": 2, + "files": [ + "explainers/quorum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/explanation.md new file mode 100644 index 0000000..0d92e3a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/explanation.md @@ -0,0 +1,88 @@ +# Why 5 servers survive 2 failures, but 4 servers survive only 1 + +**Short answer.** A cluster accepts a decision only when a strict majority of *all* its servers agree. A majority of 5 is 3, so 2 servers can fail and 3 remain. A majority of 4 is also 3, so only 1 server can fail. The 4th server raises the number of votes you need, but not the number of failures you can absorb. + +The rest of this page explains why the rule must be "a majority of all servers", and not something cheaper. + +## The problem: one server cannot see why another went silent + +A **cluster** is a group of servers that keep identical copies of some data, for example a bank balance. Copies stay identical only if all servers apply the same writes in the same order. So the cluster must *agree* on each write before it accepts it. + +Suppose server A stops getting replies from server B. Two different situations produce exactly this observation: + +1. B has crashed. +2. B is healthy, but the network between A and B is broken. This is a **partition**: the network splits the servers into groups that cannot reach each other. + +A cannot tell these cases apart. In case 2, B may be accepting writes in its own group right now. So any safe rule must assume that silent servers might be alive and making decisions. + +## The simplest rule fails + +**Rule 1: "Servers that can reach each other decide together."** + +Take 5 servers and a balance of $100. The network splits them into a group of 3 and a group of 2. + +- The group of 3 accepts "withdraw $80". Its balance is $20. +- The group of 2 accepts "withdraw $90". Its balance is $10. + +Each group thinks it is the whole surviving cluster. The two copies now disagree, and $170 left an account that held $100. This failure is called **split brain**. + +## The fix: any two deciding groups must share a server + +Split brain needs two groups that decide independently. Prevent it like this: require every deciding group to be large enough that **any two such groups share at least one server**. That shared server votes for at most one of two conflicting writes, so at most one of them can pass. + +Call the required group size the **quorum**, q. Two groups of size q, inside n servers, must overlap when their sizes add up to more than n: + + q + q > n, so q > n/2 + +The smallest whole number above n/2 is + + q = floor(n/2) + 1 (a strict majority) + +Check with 5 servers: q = 3. Two groups of 3 need 6 seats but only 5 exist, so they share at least one server. In the 3|2 split above, only the group of 3 can decide. The group of 2 refuses the $90 withdrawal. + +## Why "half" is not enough + +**Rule 2: "Half the servers is enough" (q = n/2).** + +Take 4 servers split 2|2. Each half has 2 servers, which equals n/2. Both halves decide, and split brain returns. Two groups of 2 fit inside 4 servers with no overlap. So 4 servers need q = 3, the same as 5 servers. + +## From quorum to failure tolerance + +The cluster keeps working while at least q servers are alive and connected. So it tolerates + + f = n − q failures. + +> **Predict before you read the table:** how many failures does a 6-server cluster tolerate? + +| Servers n | Quorum q | Failures tolerated f | +|---|---|---| +| 3 | 2 | 1 | +| **4** | **3** | **1** | +| **5** | **3** | **2** | +| 6 | 4 | 2 | +| 7 | 4 | 3 | + +Each step from an odd n to the next even n adds 1 to q and 0 to f. This is the answer to the original question: going from 4 to 5 servers adds a server without raising q, so f rises from 1 to 2. + +## What "survive" means, and where it stops + +- **Holds:** 5 servers, 2 crash. 3 remain, which equals q. The cluster keeps accepting writes. +- **Breaks:** 5 servers, 3 crash. 2 remain, fewer than q. The cluster stops accepting writes until a server returns. + +In the second case the cluster is *unavailable*, but it is still *correct*. It refuses to decide rather than risk split brain. A majority quorum trades availability for consistency on purpose. + +## What the even server costs you + +An even server does not just fail to help. It adds one more machine that can fail, and the tolerance stays the same. Assume each server is down 1% of the time, independently (an illustrative assumption). The chance that the cluster lacks a quorum is then: + +| Servers | Down when | P(no quorum) | +|---|---|---| +| 3 | 2 or more down | ≈ 0.030% | +| 4 | 2 or more down | ≈ 0.059% | +| 5 | 3 or more down | ≈ 0.001% | + +A 4-server cluster loses its quorum about twice as often as a 3-server cluster. This is why production clusters (etcd, ZooKeeper, Consul) normally use 3, 5, or 7 servers. + +## Scope + +This page assumes servers fail by stopping. Servers that send false messages (**Byzantine faults**) need a larger cluster: n ≥ 3f + 1, so 4 servers to survive 1 such server. The algorithms that run the votes, such as Raft and Paxos, are also out of scope; they all rest on the majority-overlap rule above. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/model.md new file mode 100644 index 0000000..084b796 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/model.md @@ -0,0 +1,43 @@ +# Model: majority quorum and failure tolerance + +## Central question +Why does a 5-server cluster survive 2 failures, but a 4-server cluster survives only 1? + +## Reader +Technical, new to distributed systems. Knows integer division, basic probability, what a server and a network are. + +## Entities and definitions +- Cluster: n servers that must agree on one sequence of decisions (for example, writes to a database). +- Failure (crash): a server stops responding. It does not send wrong messages. +- Partition: the network splits the servers into groups that cannot reach each other. +- Quorum: the minimum number of servers that must agree before the cluster accepts a decision. +- Majority quorum: q = floor(n/2) + 1. +- Tolerated failures: f = n - q = floor((n-1)/2). + +## Central causal chain +1. A server cannot tell a crashed peer from a peer that is cut off by a partition. +2. So the rule must stay safe if the "missing" servers are alive and deciding on their own. +3. Safety requires that two groups which can each decide must share at least one server. That server refuses to approve conflicting decisions. +4. Two groups of size q always share a server only if q + q > n, so q > n/2. The smallest such q is floor(n/2) + 1. +5. The cluster keeps working while at least q servers are alive, so it survives n - q failures. +6. n = 5: q = 3, f = 2. n = 4: q = 3, f = 1. The 4th server raises q but not f. + +## Alternative (fails) +- "Any connected group decides": 5 servers split 3|2; both sides accept conflicting writes (split brain). +- "Half is enough" (q = n/2): 4 servers split 2|2; both halves have 2 = q; both decide. + +## Guarantee, two cases +- Holds: 5 servers, 2 crash; 3 remain = q; cluster continues. +- Breaks: 5 servers, 3 crash; 2 remain < q; cluster stops accepting decisions. It stays consistent but unavailable. + +## Quantity meaning +Independent failure probability p = 0.01 per server; P(cluster lacks quorum): +- n=3: ~0.00030; n=4: ~0.00059 (worse than 3); n=5: ~0.0000099. + +## Epistemic status +- Steps 3–6: mathematical consequence. +- Crash-only failures: assumption. +- Independent failures with p = 0.01: assumption for illustration. + +## Scope +Not covered: servers that lie (Byzantine faults, need n >= 3f + 1); leader election details (Raft, Paxos); weighted or flexible quorums. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/model.yaml new file mode 100644 index 0000000..6939ec3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/explainers/quorum/model.yaml @@ -0,0 +1,20 @@ +formula: + quorum: "floor(n/2) + 1" + tolerated: "floor((n-1)/2)" +clusters: + - {n: 1, quorum: 1, tolerated: 0} + - {n: 2, quorum: 2, tolerated: 0} + - {n: 3, quorum: 2, tolerated: 1} + - {n: 4, quorum: 3, tolerated: 1} + - {n: 5, quorum: 3, tolerated: 2} + - {n: 6, quorum: 4, tolerated: 2} + - {n: 7, quorum: 4, tolerated: 3} +probability_example: + p_server_down: 0.01 + p_no_quorum: + n3: 0.00030 + n4: 0.00059 + n5: 0.0000099 +assumptions: + - crash failures only (no lying servers) + - independent failures (probability example only) diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/read.json new file mode 100644 index 0000000..204e615 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "A strict majority of all servers must agree to accept a write.", + "quote": "A cluster accepts a decision only when a strict majority of *all* its servers agree." + }, + "2": { + "answer": "To prevent split brain, where two groups independently accept conflicting writes. A majority ensures any two deciding groups share at least one server, so that server can vote for at most one conflicting write.", + "quote": "any two such groups share at least one server. That shared server votes for at most one of two conflicting writes, so at most one can pass." + }, + "3": { + "answer": "A 6-server cluster survives 2 failures. With n=6, the quorum q=4, so f = n−q = 2.", + "quote": "The cluster keeps working while at least q servers are alive and connected. So it tolerates f = n − q failures." + }, + "4": { + "answer": "The group of 3 can commit writes, because it contains the quorum size for 5 servers. The group of 2 cannot commit because it lacks q=3.", + "quote": "Check with 5 servers: q = 3. In the 3|2 split above, only the group of 3 can decide. The group of 2 refuses the $90 withdrawal." + } + }, + "stops": [ + "any two such groups share at least one server — Why all pairs, not just neighboring groups?", + "q + q > n, so q > n/2 — Why does adding group sizes determine overlap?", + "The smallest whole number above n/2 is q = floor(n/2) + 1 — Unexplained floor function reasoning", + "So it tolerates f = n − q failures — Formula stated without derivation or proof", + "That shared server votes for at most one of two conflicting writes — Voting mechanism undefined", + "So any safe rule must assume that silent servers might be alive — Logical jump from observation", + "A majority quorum trades availability for consistency on purpose — What availability means here?", + "The network splits the servers into groups that cannot reach each other — Why is partition relevant to quorum rule?" + ], + "excess": [ + "Short answer at lines 2-3 — Repeated in the detailed explanation that follows", + "The rest of this page explains why — Meta-framing that doesn't add content", + "Specific $100 balance and withdrawal amounts — Numbers are illustrative but not essential", + "Probability calculations in table at lines 76-82 — Beyond scope of core principle", + "Scope section mentioning Byzantine faults and Raft/Paxos algorithms — Extensions not needed" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/run.json new file mode 100644 index 0000000..e39c93b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/quorum/explanation.md`, plus `model.md` and `model.yaml` as its source of truth. I chose prose because the argument is a short chain (a quiet server might be cut off rather than crashed → any two deciding groups must share a server → you need a strict majority → failures you can survive = n − majority), and a table plus one numbered example of two clusters splitting carries it with no diagram or interactive page needed. I didn't run `explainer check` because I had no shell in this session.", + "error": null, + "cost": 0.18852040000000003, + "seconds": 52, + "turns": 4, + "files": [ + "explainers/quorum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/explanation.md new file mode 100644 index 0000000..fc32017 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/explanation.md @@ -0,0 +1,79 @@ +# Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1? + +**A cluster decides a change only when more than half of all its servers accept the change. For 5 servers, more than half is 3, so 5 − 3 = 2 servers can fail. For 4 servers, more than half is also 3, so only 4 − 3 = 1 server can fail.** + +The rest of this page explains why the rule must be "more than half". + +## The rule + +A *cluster* is a group of n servers that keep copies of the same data. Take 5 servers, A to E. A client proposes a change, for example x = 1. + +- A server *accepts* the change when it stores the change on disk and replies "yes". +- When enough servers accept, the change is *decided*: it is final. +- The number of accepts needed is the *quorum size*. A group of that many servers, for example {A, B, C}, is a *quorum*. +- A server *fails* when it stops answering: it crashed, or the network cannot reach it. + +The cluster decides changes one at a time, in order: change 1, change 2, and so on. For each position in that order, a server accepts only one change. The examples below use change 7, an arbitrary position. + +If fewer than a quorum of servers answer, the cluster *stops*. It keeps every decided change, but it decides no new change until enough servers return. + +## Why not let the survivors go on? + +Take a cluster of 4 servers, A to D. C and D fail. Why can A and B not go on alone? + +A sends a message to D and gets no reply. Maybe D crashed. Or maybe the network between them is cut, and D still runs and talks to C. This split of the network into groups that cannot reach each other is a *partition*. A sees the same silence in both cases, however long it waits, so A cannot know which case is true. + +Letting A and B go on means quorum size 2. Now suppose the cause is a partition, {A, B} on one side and {C, D} on the other: + +| Group | Change it accepts as change 7 | Number of accepts | Quorum size | Result | +|---|---|---|---|---| +| A, B | x = 1 | 2 | 2 | decided | +| C, D | x = 2 | 2 | 2 | decided | + +Both groups decide, and the copies now disagree for good. This is *split brain*. It happens because two groups of 2 among 4 servers need not share a server. So the quorum size must be more than half of n: then two groups cannot both decide. + +## Why more than half works + +More than half of all n servers is the *majority size*: floor(n/2) + 1. For 5 servers it is 3; for 4 servers it is also 3. Clusters set the quorum size to the majority size: a larger quorum is also safe, but it lets fewer servers fail. + +Two quorums each hold more than half of the servers, so together they list more than n servers. Some server must therefore be in both. For 5 servers, {A, B, C} and {C, D, E} list 3 + 3 = 6 > 5, and C is in both. For 4 servers, two groups of 3 list 6 > 4. + +The shared server blocks split brain. Suppose A, B, C accepted x = 1 for change 7, so x = 1 is decided. Later, C, D, E are asked to accept x = 2 for change 7. C already accepted x = 1 for that position, so C refuses. Then x = 2 gets only 2 accepts, from D and E, and it needs 3. + +This depends on one assumption: a server remembers what it accepted, even after a restart. If C restarts with an empty disk and accepts x = 2, then C, D, E decide x = 2 too, and split brain returns. + +**Predict:** the network splits 5 servers into {A, B, C} and {D, E}. What does each side do? + +A, B, C have 3 accepts, and 3 ≥ 3, so they keep deciding. D and E have 2, and 2 < 3, so they stop. The copies cannot disagree. + +Count more than half of *all* configured servers, failed or not. If D and E counted "more than half of the 2 servers we can reach", that would be 2, and they would decide x = 2 while A, B, C decide x = 1. + +## Counting failures + +*Tolerance* is how many servers can fail while the cluster still decides changes: + +tolerance = n − majority size + +| Servers n | Majority size | Tolerance | +|---|---|---| +| 3 | 2 | 1 | +| 4 | 3 | 1 | +| 5 | 3 | 2 | +| 6 | 4 | 2 | +| 7 | 4 | 3 | + +With 4 servers and 2 failed, only 2 servers answer, and 2 < 3: the cluster stops. + +A tolerance of 1 means a second failure stops the cluster. A tolerance of 2 means one server can be off for an upgrade while another one fails. + +Going from an even n to the next odd n keeps the majority size (4 → 5: both 3), so tolerance grows by 1. Going from an odd n to the next even n raises the majority size by 1 (5 → 6: 3 → 4), so tolerance stays the same. The extra server only makes each change wait for one more accept. That is why clusters usually have an odd number of servers. + +## Answer + +Both clusters need 3 accepts to decide a change, because 3 is more than half of 4 and of 5. Any two groups of more than half share a server, and that server stops them from deciding different changes. 5 servers have 2 to spare. 4 servers have only 1. + +## Limits + +- **Assumed:** servers crash or become unreachable, but never send a false "yes". Servers that lie need larger clusters. +- **Simplified:** in real protocols (Raft, Paxos), the shared server does more than refuse. It reports x = 1 to the new group, and the new group keeps x = 1. +- **Not covered:** how servers detect a failure, how the stopped side catches up when the network heals, and how reads work. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/model.md new file mode 100644 index 0000000..e5578c3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/model.md @@ -0,0 +1,139 @@ +# Semantic model: quorum (majority voting in a replicated cluster) + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +Why does a cluster of 5 servers keep working after 2 servers fail, but a cluster of 4 servers keeps working after only 1? + +## Audience and prior knowledge + +A technical reader: school mathematics and science, basic programming. Knows what a server and a network are, and that a database can keep copies on several machines. New to distributed consensus: does not know the words quorum, partition, or split brain. Does not know Raft or Paxos. Floor and integer division are known. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| cluster | n servers that each keep a copy of the same data and must agree on every change to it. | definition | +| accept | a server stores a proposed change on disk and replies "yes". | definition | +| position | the place of a change in the order the cluster decides changes (change 1, change 2, …); a server accepts only one change per position, e.g. change 7. | definition | +| decide | a change becomes final because a quorum accepted it. | definition | +| failure | a server stops answering: it crashed, or the network cannot reach it. | assumption (crash failures only; no server lies; disks survive a restart) | +| quorum size | the number of servers that must accept a change before it is decided. Here always the majority size of n. | definition | +| quorum | any group of at least quorum-size servers, e.g. {A, B, C} in a 5-server cluster. | definition | +| majority size | more than half of all n configured servers: floor(n/2) + 1. | definition | +| partition | the network splits the servers into groups that cannot reach each other, while every server keeps running. | definition | +| split brain | two groups each decide a different change, so the copies disagree. | definition | +| tolerance | the number of failed servers the cluster can lose and still decide new changes. | definition | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| quorum size | is set to | majority size | so any two quorums share at least one server | +| shared server | blocks | split brain | it already accepted the first change and refuses a conflicting one | +| failure | removes a vote from | quorum | cluster stops deciding if fewer than quorum size servers answer | +| n | sets | majority size, tolerance | tolerance = n − majority size | + +## Dependencies (learning order) + +1. A change is decided only when a quorum accepts it. +2. A server cannot tell a crashed server from a cut-off one. +3. So the quorum size must be a majority; two majorities always share a server, and that server blocks a second, conflicting change. +4. Tolerance = n − majority size. Count for 5 and for 4. +5. Generalize: the table for n = 3..7; the even server adds nothing to tolerance. + +## Causal chain + +```text +a server cannot tell a crash from a partition + → each group must assume the missing servers may still run and decide + → quorum size must be a majority, so two groups cannot both decide + → majority size of 5 is 3, and majority size of 4 is also 3 + → cluster of 5 can lose 5 − 3 = 2; cluster of 4 can lose 4 − 3 = 1 +``` + +## Temporal sequence + +1. A client sends change x = 1 to a 5-server cluster. +2. A accepts (1 vote), B accepts (2), C accepts (3 = quorum size). x = 1 is decided. D and E can catch up later. +3. In a 4-server cluster with C and D down, only A and B accept: 2 < 3, so x = 1 waits. No decided change is lost; no new change is decided until C or D returns. + +## Quantities + +- majority size(n) = floor(n/2) + 1. n = 4 → 3; n = 5 → 3. +- tolerance(n) = n − majority size(n). n = 4 → 1; n = 5 → 2. +- Table: n = 3, 4, 5, 6, 7 → majority size 2, 3, 3, 4, 4 → tolerance 1, 1, 2, 2, 3. +- Overlap of two quorums is at least 2 × majority size − n: n = 5 → 3 + 3 − 5 = 1. +- Meaning of tolerance: at 1, a second failure stops the cluster from deciding; at 2, the cluster can lose one server while another is down for an upgrade. +- Cost of the even server: n = 6 waits for 4 acceptances per change instead of 3, and tolerance stays 2. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| 5 servers, 2 failed | 3 answer = majority size 3 → keeps deciding | 3 ≥ 3 | +| 4 servers, 1 failed | 3 answer → keeps deciding | 3 ≥ 3 | +| 4 servers, 2 failed | 2 answer < 3 → stops deciding | 2 < 3 | +| 5 servers, partition {A,B,C} / {D,E} | left side decides, right side stops; copies cannot disagree | 3 ≥ 3, 2 < 3 | +| 4 servers, partition {A,B} / {C,D} | neither side decides | 2 < 3 on both sides | +| 4 servers, quorum size lowered to 2, partition {A,B} / {C,D} | both sides decide different changes → split brain | 2 + 2 = 4, no shared server | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| quorum size = majority size (more than half) | quorum size = half (2 of 4), so 2 survivors can go on | partition {A,B} / {C,D}: A, B decide x = 1, C, D decide x = 2. Both groups have 2 = quorum size. Copies disagree: split brain. | +| count the majority of all n configured servers | count the majority of the servers each side can reach | 5 servers split {A,B,C} / {D,E}: D, E reach 2 servers, majority of 2 is 2, so they decide x = 2 while A, B, C decide x = 1: split brain. | +| quorum size = majority, not all n | quorum size = 5 of 5 | one failure stops the cluster: tolerance 0. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| Two quorums of majority size always share a server (guarantee) | n = 5: {A,B,C} and {C,D,E}: 3 + 3 = 6 > 5, so at least 1 server (C) is in both. | Size 2 of 5 or 4: {A,B} and {C,D} share nothing (2 + 2 = 4). | +| The shared server blocks split brain (mechanism) | A, B, C accept x = 1. Then C, D, E are asked to accept x = 2. C already accepted x = 1 and refuses; x = 2 has 2 votes < 3. | If C restarts with an empty disk and accepts x = 2, then C, D, E decide x = 2 too: split brain. So the assumption "a server keeps what it accepted" matters. | +| A server cannot tell crash from partition (mechanism) | A sends to D and gets no reply, however long it waits. Case 1: D crashed. Case 2: the link A–D is cut and D still accepts changes from E. In both cases A sees the same thing: no reply. | — | +| 5 servers tolerate 2 failures (mechanism) | 5 − 2 = 3 answering ≥ 3. | Third failure: 2 answering < 3 → stops. | +| 4 servers tolerate 1 failure (mechanism) | 4 − 1 = 3 ≥ 3. | 4 − 2 = 2 < 3 → stops. | + +## Terms + +- **cluster:** n servers keeping copies of the same data. +- **accept:** a server stores a proposed change on disk and replies "yes". +- **decide:** a change becomes final because a quorum accepted it. "Keeps working" means "can still decide new changes". Use "decide", not "write" or "commit". +- **stops:** the cluster still holds every decided change but decides no new change until enough servers return. +- **failure:** a server stops answering (crash or unreachable). +- **quorum size:** a number, e.g. 3. **quorum:** a group of servers, e.g. {A, B, C}. Never swap them. +- **majority size:** more than half of all n configured servers, counting failed servers too: floor(n/2) + 1. +- **partition:** the network splits the servers into groups that cannot reach each other. +- **split brain:** two groups decide different changes. +- **tolerance:** how many failed servers the cluster can lose and still decide. At 1, a second failure stops it; at 2, it can lose one server while another is down for an upgrade. + +## Epistemic status + +- **Mathematical consequence:** two majorities overlap; tolerance = n − (floor(n/2) + 1). +- **Assumed:** crash failures only (a server stops or is unreachable; it never sends a false "yes"). A server keeps what it accepted across a restart. Quorum size is fixed by the configured n. +- **Simplification:** "the shared server refuses x = 2" stands in for the real protocol step, in which the shared server reports x = 1 to the new group and the group keeps x = 1 (Raft, Paxos). + +## Confusion points + +- "With 2 of 4 alive, why not just continue?" → the 2 cannot know whether the other 2 crashed or are cut off and still deciding. +- "Majority of the servers still running" → no: majority of all n configured servers. Otherwise each side of a partition counts itself a majority. +- "More servers is always safer" → going from 5 to 6 raises quorum size to 4; tolerance stays 2. +- "Survive" does not mean "lose no data": a stopped cluster keeps its decided changes; it only stops deciding new ones. + +## Scope + +- **Out of scope:** servers that lie (Byzantine failures) need more servers: 4 servers to tolerate 1 liar. See PBFT. +- **Out of scope:** how a leader is elected, how a server detects a timeout, and how a minority side catches up after the partition heals. See the Raft paper ("In Search of an Understandable Consensus Algorithm"). +- **Out of scope:** reasons to use even n anyway (read load, a tie-breaker "witness" server), and reads that skip the quorum. +- **Out of scope:** weighted votes and flexible quorums (different sizes for reads and changes). +- **Practice:** Raft and Paxos systems such as etcd and ZooKeeper use majority quorums and recommend odd n. + +## Representation decision + +- **Stage:** 1 +- **Reason:** the argument is a short chain of counting; one table (n, majority size, tolerance) carries the comparison. Nothing to explore or animate. +- **Levels:** L1 the answer / L2 the counting / L3 why majority (partition) / L4 general table. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/model.yaml new file mode 100644 index 0000000..2b68006 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/model.yaml @@ -0,0 +1,94 @@ +# Machine-readable half of model.md. `explainer check quorum` holds every rendering to it. + +values: + n_big: 5 + n_small: 4 + majority_5: 3 + majority_4: 3 + tolerance_5: 2 + tolerance_4: 1 + overlap_5: 1 + table: + n: [3, 4, 5, 6, 7] + majority: [2, 3, 3, 4, 4] + tolerance: [1, 1, 2, 2, 3] + +allow: [0, 1, 2, 3, 4, 5, 6, 7] + +require: + - path: majority_5 + in: [prose] + - path: tolerance_5 + in: [prose] + - path: tolerance_4 + in: [prose] + +claims: + - id: quorum-rule + kind: definition + statement: "A change is decided only when a quorum (a group of at least quorum-size servers) has accepted it; quorum size is the majority size of all n configured servers, floor(n/2) + 1." + example: "n = 5: quorum size 3. A, B, C accept x = 1, so x = 1 is decided." + + - id: cannot-tell + kind: mechanism + statement: "A server cannot tell a crashed server from a cut-off one: both look like no reply." + example: "A sends to D and gets no reply, however long it waits. D crashed, or the A–D link is cut and D still accepts changes from E. A sees no reply in both cases." + ask: "Two of the four servers in a cluster stop answering. Why can the other two not simply go on deciding changes?" + + - id: why-majority + kind: why + about: [majority] + statement: "Quorum size must be more than half of n, so that two groups cannot both decide." + alternative: "Quorum size = half: 2 of 4, so 2 survivors can go on." + counterexample: "Partition {A,B} / {C,D}: A, B decide x = 1 and C, D decide x = 2. Both groups have 2 = quorum size. Split brain." + ask: "In a 4-server cluster, why not set the quorum size to 2 so the cluster survives 2 failures?" + + - id: why-all-n + kind: why + about: [count] + statement: "Count the majority of all n configured servers, not of the servers a side can reach." + alternative: "Each side counts the majority of the servers it can reach." + counterexample: "5 servers split {A,B,C} / {D,E}: D, E reach 2, majority of 2 is 2, so D, E decide x = 2 while A, B, C decide x = 1." + + - id: overlap + kind: guarantee + statement: "Two quorums of majority size each hold more than half of n, so together more than n: they always share a server, and that server refuses a conflicting change for the same position." + example: "n = 5: {A,B,C} and {C,D,E}: 3 + 3 = 6 > 5, so C is in both. C accepted x = 1 and refuses x = 2; x = 2 gets 2 votes < 3." + counterexample: "If C restarts with an empty disk and accepts x = 2, C, D, E decide x = 2 too: split brain." + + - id: count-5-4 + kind: mechanism + statement: "Tolerance = n − majority size. Majority size of 5 and of 4 is 3, so 5 servers tolerate 2 failures and 4 tolerate 1." + example: "5 − 2 = 3 ≥ 3 keeps deciding; 4 − 2 = 2 < 3 stops." + ask: "Why does a 5-server cluster tolerate 2 failures but a 4-server cluster only 1?" + + - id: even-adds-nothing + kind: mechanism + statement: "Adding a server to an odd cluster raises the majority size by 1, so tolerance does not grow." + example: "n = 5 → 6: majority size 3 → 4, tolerance 2 → 2, and each change waits for 4 acceptances instead of 3." + + - id: stop-not-loss + kind: definition + statement: "A cluster that stops keeps every decided change; it only decides no new change until enough servers return." + example: "4 servers with C, D down: x = 1 gets 2 votes < 3 and waits until C or D returns." + +terms: + - term: quorum size + means: "a number of servers (e.g. 3); 'quorum' is a group of servers (e.g. {A, B, C})" + avoid: [quorum of 3, a quorum is 3] + - term: decide + means: "a change becomes final because a quorum accepted it" + avoid: [commit, writes] + +budget: {prose: 900, reason: "two why-claims each need a partition case with servers and values, plus the crash-vs-partition case and the n table"} + +quiz: + - q: "A cluster of 6 servers: how many failed servers can it lose and still decide new changes?" + expect: "2. Majority size is 4, so 6 − 4 = 2; the sixth server adds nothing over 5." + misconception: "3, because half of 6 is 3" + - q: "In a 4-server cluster, why not let 2 servers be enough to decide a change?" + expect: "A partition {A,B}/{C,D} would let both halves decide different changes (split brain); two groups of 2 out of 4 need not share a server." + misconception: "Because 2 servers are too slow or too few to store the data" + - q: "A 5-server cluster splits into a group of 3 and a group of 2. What does each group do?" + expect: "The 3-server group keeps deciding (3 ≥ 3); the 2-server group stops (2 < 3) and decides nothing, so the copies cannot disagree." + misconception: "The 2-server group keeps deciding with a majority of the 2 servers it can reach" diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/narrative.md new file mode 100644 index 0000000..d6bab69 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/narrative.md @@ -0,0 +1,54 @@ +# Narrative: quorum + +> Pass 3. `model.md` says what is true. This file says the path a first-time reader takes to it. + +## 1. Reader before and after + +- **Before:** a technical reader (school maths and science, basic programming). Knows servers, networks, and that data can be copied to several machines. Does not know quorum, partition, split brain, Raft or Paxos. +- **After:** "A cluster decides a change only when more than half of all its servers accept it, because any two such groups share a server. 5 needs 3, so it can lose 2. 4 also needs 3, so it can lose only 1." + +## 2. Question and motive + +- **Question:** Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1? Result in words: both clusters need 3 servers to agree, because the rule is "more than half of all servers", and more than half of 4 is 3, the same as for 5. +- **Why care:** it is surprising: the 4th server costs money but buys nothing. It decides how you size etcd, ZooKeeper, or any Raft cluster. +- **Why this approach:** ask "why not let the 2 survivors go on?" The answer (they cannot tell a crash from a cut cable) forces the majority rule; the counting then follows. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| cluster | n servers with copies of the same data that must agree on each change | 5 servers A–E | 1 | +| accept | a server stores a proposed change on disk and replies "yes" | A, B, C accept x = 1 | 2 | +| decide | a change becomes final because a quorum accepted it | x = 1 decided after 3 accepts | 2 | +| quorum size / quorum | the number needed (3) / a group that big ({A,B,C}) | n = 5 | 2 | +| majority size | floor(n/2) + 1 of all configured servers | 5 → 3, 4 → 3 | 4 | +| partition | network splits servers into groups that cannot reach each other | {A,B} / {C,D} | 3 | +| split brain | two groups decide different changes | x = 1 vs x = 2 | 3 | +| tolerance | failed servers the cluster can lose and still decide | 5 → 2, 4 → 1 | 6 | +| x = 1, x = 2 | two conflicting changes to one value | — | 2 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 only 1? | — | question as title; answer in bold | "A cluster of n servers keeps copies of the same data. It decides a change only when more than half of all n servers accept it. For 5 servers that is 3, so 5 − 3 = 2 servers can fail. For 4 servers it is also 3, so only 4 − 3 = 1 can fail. The rest explains why the rule is 'more than half'." | the question and the result | +| 2 | name | What does "accept" and "decide" mean? | so | 5 servers A–E; change x = 1 accepted by A, B, C | "A client proposes the next change, for example x = 1. A server accepts it when it stores it on disk and replies yes. Once 3 servers (A, B, C) accept, the change is decided: final. The number needed (here 3) is the quorum size; a group that big ({A, B, C}) is a quorum. If too few servers answer, the cluster stops: it keeps every decided change but decides no new one. The cluster decides changes one at a time, in order: change 1, change 2, and so on. For each position in that order, a server accepts only one change." | accept, decide, quorum size, quorum, stop, one-per-position rule | +| 3 | objection | In a 4-server cluster with 2 down, why can't the other 2 go on? | but | 4 servers; A waits 500 ms for D; partition {A,B}/{C,D} | "Now take a cluster of 4 servers, A–D, where C and D stop answering. Why can A and B not go on? A sends to D and gets no reply in 500 ms. D may have crashed. Or the network between them is cut (a partition) and D still runs, talking to C. A sees the same silence in both cases. Letting A and B go on means quorum size 2. Then A, B decide x = 1 for change 7 while C, D decide x = 2 for change 7. The copies now disagree for good: split brain." | why a quorum of half breaks | +| 4 | mechanism | What quorum size is safe? | therefore | 5 servers: {A,B,C} and {C,D,E}; C refuses x = 2 | "Make the quorum size more than half of all n servers: the majority size, floor(n/2) + 1. Back to 5 servers. Two groups of 3 hold 6 memberships, but there are only 5 servers, so at least one server is in both groups, here C. C already accepted x = 1 for change 7, so it refuses x = 2 for change 7, and x = 2 gets only 2 of the 3 votes it needs." | the majority rule and why it works | +| 5 | test | (predict) 5 servers split into {A,B,C} and {D,E}: what does each side do? | so | prediction prompt, then answer | "A, B, C have 3 ≥ 3 and keep deciding. D, E have 2 < 3 and stop. The copies cannot disagree. Note: count more than half of all 5 configured servers. If D and E counted 'more than half of the 2 servers we can reach', they would decide too." | the rule applied | +| 6 | generalize | So how many failures can a cluster of n take? | therefore | table n = 3..7 with majority size and tolerance | "Tolerance is how many servers can fail while the cluster still decides: n − majority size. Read the table: 4 and 5 both need 3. Adding a 4th server to 3, or a 6th to 5, raises the majority size by one, so tolerance stays the same; each change must also wait for one more accept." | the general pattern | +| 7 | close | — | so | — | "5 survives 2 and 4 survives 1 because both need 3 servers, and 3 is the smallest group that must share a server with any other group. Use odd cluster sizes. This assumes servers crash but never lie." | the answer and its limits | + +## 5. Concrete to symbol + +floor(n/2) + 1 appears after the 5 → 3 and 4 → 3 instances; checked against 6 → 4, 7 → 4. + +## 6. Links between representations + +The table rows for n = 4 and n = 5 repeat the numbers from beats 1 and 4. + +## 7. Close the loop + +- **Answer:** both clusters need 3 servers to decide; 5 has 2 spare, 4 has 1 spare. +- **General argument:** more than half is the smallest quorum where any two quorums share a server; the shared server stops two groups deciding different changes. +- **Payoff:** choose odd cluster sizes. Scope: lying servers, leader election, reads. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/decisions.yaml new file mode 100644 index 0000000..c1bb6a5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/decisions.yaml @@ -0,0 +1,233 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T151129-probe#1: + decision: adopted +2026-10-04T151129-probe#10: + decision: adopted +2026-10-04T151129-probe#11: + decision: adopted +2026-10-04T151129-probe#12: + decision: adopted +2026-10-04T151129-probe#13: + decision: adopted +2026-10-04T151129-probe#14: + decision: adopted +2026-10-04T151129-probe#2: + decision: adopted +2026-10-04T151129-probe#3: + decision: adopted +2026-10-04T151129-probe#4: + decision: adopted +2026-10-04T151129-probe#5: + decision: adopted +2026-10-04T151129-probe#6: + decision: adopted +2026-10-04T151129-probe#7: + decision: declined +2026-10-04T151129-probe#8: + decision: declined +2026-10-04T151129-probe#9: + decision: declined +2026-10-04T151328-coldread-narrative#1: + decision: adopted +2026-10-04T151328-coldread-narrative#10: + decision: declined +2026-10-04T151328-coldread-narrative#11: + decision: declined +2026-10-04T151328-coldread-narrative#12: + decision: declined +2026-10-04T151328-coldread-narrative#13: + decision: declined +2026-10-04T151328-coldread-narrative#14: + decision: declined +2026-10-04T151328-coldread-narrative#15: + decision: declined +2026-10-04T151328-coldread-narrative#16: + decision: declined +2026-10-04T151328-coldread-narrative#17: + decision: declined +2026-10-04T151328-coldread-narrative#18: + decision: declined +2026-10-04T151328-coldread-narrative#19: + decision: declined +2026-10-04T151328-coldread-narrative#2: + decision: adopted +2026-10-04T151328-coldread-narrative#20: + decision: declined +2026-10-04T151328-coldread-narrative#21: + decision: declined +2026-10-04T151328-coldread-narrative#22: + decision: declined +2026-10-04T151328-coldread-narrative#23: + decision: declined +2026-10-04T151328-coldread-narrative#24: + decision: declined +2026-10-04T151328-coldread-narrative#25: + decision: declined +2026-10-04T151328-coldread-narrative#26: + decision: declined +2026-10-04T151328-coldread-narrative#27: + decision: declined +2026-10-04T151328-coldread-narrative#28: + decision: declined +2026-10-04T151328-coldread-narrative#29: + decision: declined +2026-10-04T151328-coldread-narrative#3: + decision: adopted +2026-10-04T151328-coldread-narrative#30: + decision: declined +2026-10-04T151328-coldread-narrative#31: + decision: declined +2026-10-04T151328-coldread-narrative#32: + decision: declined +2026-10-04T151328-coldread-narrative#33: + decision: declined +2026-10-04T151328-coldread-narrative#34: + decision: declined +2026-10-04T151328-coldread-narrative#35: + decision: declined +2026-10-04T151328-coldread-narrative#36: + decision: declined +2026-10-04T151328-coldread-narrative#37: + decision: declined +2026-10-04T151328-coldread-narrative#38: + decision: declined +2026-10-04T151328-coldread-narrative#39: + decision: declined +2026-10-04T151328-coldread-narrative#4: + decision: adopted +2026-10-04T151328-coldread-narrative#40: + decision: declined +2026-10-04T151328-coldread-narrative#5: + decision: adopted +2026-10-04T151328-coldread-narrative#6: + decision: adopted +2026-10-04T151328-coldread-narrative#7: + decision: adopted +2026-10-04T151328-coldread-narrative#8: + decision: declined +2026-10-04T151328-coldread-narrative#9: + decision: declined +2026-10-04T151438-coldread-narrative#1: + decision: adopted +2026-10-04T151438-coldread-narrative#10: + decision: declined +2026-10-04T151438-coldread-narrative#11: + decision: adopted +2026-10-04T151438-coldread-narrative#12: + decision: declined +2026-10-04T151438-coldread-narrative#13: + decision: adopted +2026-10-04T151438-coldread-narrative#14: + decision: adopted +2026-10-04T151438-coldread-narrative#15: + decision: adopted +2026-10-04T151438-coldread-narrative#2: + decision: adopted +2026-10-04T151438-coldread-narrative#3: + decision: adopted +2026-10-04T151438-coldread-narrative#4: + decision: adopted +2026-10-04T151438-coldread-narrative#5: + decision: adopted +2026-10-04T151438-coldread-narrative#6: + decision: adopted +2026-10-04T151438-coldread-narrative#7: + decision: adopted +2026-10-04T151438-coldread-narrative#8: + decision: adopted +2026-10-04T151438-coldread-narrative#9: + decision: adopted +2026-10-04T151602-coldread-prose#1: + decision: adopted +2026-10-04T151602-coldread-prose#10: + decision: adopted +2026-10-04T151602-coldread-prose#11: + decision: adopted +2026-10-04T151602-coldread-prose#12: + decision: declined +2026-10-04T151602-coldread-prose#13: + decision: declined +2026-10-04T151602-coldread-prose#14: + decision: declined +2026-10-04T151602-coldread-prose#2: + decision: declined +2026-10-04T151602-coldread-prose#3: + decision: adopted +2026-10-04T151602-coldread-prose#4: + decision: adopted +2026-10-04T151602-coldread-prose#5: + decision: adopted +2026-10-04T151602-coldread-prose#6: + decision: adopted +2026-10-04T151602-coldread-prose#7: + decision: adopted +2026-10-04T151602-coldread-prose#8: + decision: adopted +2026-10-04T151602-coldread-prose#9: + decision: adopted +2026-10-04T151643-quiz-prose#1: + decision: adopted +2026-10-04T151643-quiz-prose#10: + decision: adopted +2026-10-04T151643-quiz-prose#11: + decision: adopted +2026-10-04T151643-quiz-prose#12: + decision: declined +2026-10-04T151643-quiz-prose#13: + decision: adopted +2026-10-04T151643-quiz-prose#14: + decision: declined +2026-10-04T151643-quiz-prose#15: + decision: declined +2026-10-04T151643-quiz-prose#2: + decision: adopted +2026-10-04T151643-quiz-prose#3: + decision: adopted +2026-10-04T151643-quiz-prose#4: + decision: adopted +2026-10-04T151643-quiz-prose#5: + decision: adopted +2026-10-04T151643-quiz-prose#6: + decision: adopted +2026-10-04T151643-quiz-prose#7: + decision: adopted +2026-10-04T151643-quiz-prose#8: + decision: adopted +2026-10-04T151643-quiz-prose#9: + decision: adopted +2026-10-04T151817-quiz-prose#1: + decision: declined +2026-10-04T151817-quiz-prose#10: + decision: declined +2026-10-04T151817-quiz-prose#11: + decision: declined +2026-10-04T151817-quiz-prose#2: + decision: declined +2026-10-04T151817-quiz-prose#3: + decision: adopted +2026-10-04T151817-quiz-prose#4: + decision: declined +2026-10-04T151817-quiz-prose#5: + decision: declined +2026-10-04T151817-quiz-prose#6: + decision: declined +2026-10-04T151817-quiz-prose#7: + decision: declined +2026-10-04T151817-quiz-prose#8: + decision: declined +2026-10-04T151817-quiz-prose#9: + decision: declined +2026-10-04T151859-coldread-prose#1: + decision: declined +2026-10-04T151859-coldread-prose#2: + decision: adopted +2026-10-04T151859-coldread-prose#3: + decision: declined +2026-10-04T151859-coldread-prose#4: + decision: declined +2026-10-04T151859-coldread-prose#5: + decision: declined +2026-10-04T151859-coldread-prose#6: + decision: adopted +2026-10-04T151859-coldread-prose#7: + decision: adopted diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151129-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151129-probe.json new file mode 100644 index 0000000..4c14ad4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151129-probe.json @@ -0,0 +1,211 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "quorum", + "at": "2026-10-04T15:11:29", + "model": "default", + "prompt_sha": "7d2607e2a66b", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "The overlap guarantee is argued as a size count (3+3=6>5). The step showing how the shared server actually stops split brain is missing, and so is the assumption it depends on.", + "why_it_matters": "The Relationships table says the shared server 'accepts only one of two conflicting changes', but no example with servers and values shows it. This needs an unstated assumption: a server remembers what it accepted and does not accept a conflicting change. Without that example the reader sees why the groups overlap but not why the overlap is enough.", + "suggested_addition": "Holds: n=5, A,B,C accept x=1, so x=1 is decided. D,E,C are then asked to accept x=2. C already holds x=1 and refuses, so x=2 has only 2 votes, which is less than 3, and is not decided. Breaks: C crashes, restarts with its memory wiped, and accepts x=2. Now C,D,E is 3 votes and x=2 is decided too: split brain. Add 'a server keeps what it accepted across a crash' to the Assumed list." + }, + { + "rule": 4, + "severity": "main", + "about": "No worked example of a partition where majority quorum works. The partition example is only the failing one (4 servers, quorum 2).", + "why_it_matters": "L3 ('why majority') shows only what goes wrong. The reader never sees the majority side carry on while the minority side safely stops. The confusion point 'majority of running servers' has no numbers either.", + "suggested_addition": "n=5, partition {A,B,C} / {D,E}. The left side has 3 ≥ 3 and keeps deciding. The right side has 2 < 3 and stops, so the copies cannot diverge. If quorum were counted from reachable servers, D,E would see 2 reachable, take majority(2)=2, and decide x=2 while A,B,C decide x=1: split brain." + }, + { + "rule": 4, + "severity": "main", + "about": "Dependency 2 ('a server cannot tell a crashed server from a cut-off one') has no worked example.", + "why_it_matters": "The causal chain starts from this claim. A reader new to distributed systems needs to see why the two cases look the same from inside, or the requirement for a majority looks arbitrary.", + "suggested_addition": "A sends to D and waits 500 ms. Case 1: D has crashed. Case 2: the link to D is cut and D is still accepting changes from C and E. In both cases A gets the same thing: no reply after 500 ms. A has nothing else to go on." + }, + { + "rule": 3, + "severity": "main", + "about": "'Quorum' is used for two things: a number (Entities: 'the number of servers that must accept') and a group of servers (Relationships: 'any two quorums share a server').", + "why_it_matters": "The overlap argument is about groups, and the tolerance count is about a number. Using one word for both blurs the central step.", + "suggested_addition": "Define 'quorum size' = majority(n), e.g. 3 for n=5, and 'quorum' = any group of at least that many servers, e.g. {A,B,C}. Then two quorums of 5 servers share at least 3+3−5 = 1 server." + }, + { + "rule": 3, + "severity": "edge", + "about": "Deciding new changes goes by several names: 'keeps working', 'survive', 'keeps deciding', 'stops writes', 'decide new changes'. 'Write' is never defined, and reads are never mentioned.", + "why_it_matters": "A reader may think 'stops writes' is a different state from 'stops deciding', or may think reads stop too.", + "suggested_addition": "Pick 'decide' as the single term and define 'keeps working = can still decide new changes'. Replace 'writes' in the tolerance meaning: with tolerance 1, a 2nd failure means no new change can be decided." + }, + { + "rule": 3, + "severity": "edge", + "about": "Terms used without a definition: 'accept' (what a server does), 'shared server' (a node in Relationships), and 'f' in the Scope line (n ≥ 3f+1).", + "why_it_matters": "'Accept' is the action every mechanism step depends on, and 'f' appears nowhere else in the file.", + "suggested_addition": "accept: a server stores the proposed change and replies yes, e.g. C stores x=1 and replies to the client. Either define f = number of lying servers tolerated (f=1 → n=4), or drop the formula." + }, + { + "rule": 2, + "severity": "edge", + "about": "The general overlap claim is shown only for n=5, and the general tolerance formula is never derived.", + "why_it_matters": "L4 states tolerance = floor((n−1)/2) for every n, but the reader cannot check where it comes from.", + "suggested_addition": "Two majorities always overlap because 2·(floor(n/2)+1) ≥ n+1 > n. For n=4 that gives 3+3=6, so they share at least 2 servers. Odd n=2k+1: majority k+1, tolerance k (n=7: 4, 3). Even n=2k: majority k+1, tolerance k−1 (n=6: 4, 2)." + }, + { + "rule": 2, + "severity": "edge", + "about": "The claim 'majority is the smallest size where any two groups overlap' has no breaking instance at n=5. The 'no data is lost' claim when the cluster stops has no instance at all.", + "why_it_matters": "Both are guarantees stated without numbers.", + "suggested_addition": "At n=5, size 2 breaks it: {A,B} and {C,D} share nothing (2+2=4<5). No data lost: x=1 is decided on A,B,C. Any later quorum of 3 out of 5 contains at least one of A,B,C, so x=1 is always found. This breaks if failed servers lose their disks: if A,B,C all lose storage, x=1 is gone." + }, + { + "rule": 4, + "severity": "edge", + "about": "The Temporal sequence steps have no numbers.", + "why_it_matters": "Step 2 (decided once a quorum accepts) and step 3 (the change waits) are mechanism steps described only in words.", + "suggested_addition": "n=5: the client sends x=1. A accepts (1 vote), B accepts (2), C accepts (3 = quorum), and x=1 is decided. D and E can reply later. n=4 with C,D down: A,B accept, 2 < 3, so x=1 waits until C or D returns." + }, + { + "rule": 4, + "severity": "edge", + "about": "Missing case: 4 servers split 2/2 under majority quorum.", + "why_it_matters": "No server fails, yet the whole cluster stops. This shows the cost of even n directly and connects partition to tolerance.", + "suggested_addition": "n=4, partition {A,B} / {C,D}: each side has 2 < 3, so neither decides. n=5 split {A,B,C} / {D,E}: one side has 3 and keeps working." + }, + { + "rule": 5, + "severity": "edge", + "about": "The model never says what a larger majority costs the reader.", + "why_it_matters": "The claim that 'the even server adds nothing' leaves out that it adds a cost: every change has to wait for more acceptances.", + "suggested_addition": "n=5 waits for 3 acceptances per change, and n=6 waits for 4. That is one more server, one more acceptance per change, and still tolerance 2." + } + ], + "excess": [ + "Why-this-form row about floor (3.5 servers): the audience already knows floor and integer division.", + "Scope detail 'n ≥ 3f + 1, e.g. 4 servers for 1 liar; see PBFT': introduces an undefined f and is not needed for the central question.", + "Established practice list naming ZooKeeper and etcd: not needed to answer the central question." + ], + "next_questions": [ + { + "q": "In a 5-server cluster split 3/2, what does the 2-server side do, and what happens to it when the network heals?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How does a server decide that another server has failed? Is it a timeout, and how long?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "What if the server in the overlap crashes and restarts? Does it remember what it accepted?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If 4 servers is no better than 3, why would anyone run an even number?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Do reads also need a majority, or can any one server answer a read?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.15055179999999996, + "seconds": 63 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "The overlap guarantee is argued as a size count (3+3=6>5). The step showing how the shared server actually stops split brain is missing, and so is the assumption it depends on. → Holds: n=5, A,B,C accept x=1, so x=1 is decided. D,E,C are then asked to accept x=2. C already holds x=1 and refuses, so x=2 has only 2 votes, which is less than 3, and is not decided. Breaks: C crashes, restarts with its memory wiped, and accepts x=2. Now C,D,E is 3 votes and x=2 is decided too: split brain. Add 'a server keeps what it accepted across a crash' to the Assumed list." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 4", + "text": "No worked example of a partition where majority quorum works. The partition example is only the failing one (4 servers, quorum 2). → n=5, partition {A,B,C} / {D,E}. The left side has 3 ≥ 3 and keeps deciding. The right side has 2 < 3 and stops, so the copies cannot diverge. If quorum were counted from reachable servers, D,E would see 2 reachable, take majority(2)=2, and decide x=2 while A,B,C decide x=1: split brain." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 4", + "text": "Dependency 2 ('a server cannot tell a crashed server from a cut-off one') has no worked example. → A sends to D and waits 500 ms. Case 1: D has crashed. Case 2: the link to D is cut and D is still accepting changes from C and E. In both cases A gets the same thing: no reply after 500 ms. A has nothing else to go on." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 3", + "text": "'Quorum' is used for two things: a number (Entities: 'the number of servers that must accept') and a group of servers (Relationships: 'any two quorums share a server'). → Define 'quorum size' = majority(n), e.g. 3 for n=5, and 'quorum' = any group of at least that many servers, e.g. {A,B,C}. Then two quorums of 5 servers share at least 3+3−5 = 1 server." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 3", + "text": "Deciding new changes goes by several names: 'keeps working', 'survive', 'keeps deciding', 'stops writes', 'decide new changes'. 'Write' is never defined, and reads are never mentioned. → Pick 'decide' as the single term and define 'keeps working = can still decide new changes'. Replace 'writes' in the tolerance meaning: with tolerance 1, a 2nd failure means no new change can be decided." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 3", + "text": "Terms used without a definition: 'accept' (what a server does), 'shared server' (a node in Relationships), and 'f' in the Scope line (n ≥ 3f+1). → accept: a server stores the proposed change and replies yes, e.g. C stores x=1 and replies to the client. Either define f = number of lying servers tolerated (f=1 → n=4), or drop the formula." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 2", + "text": "The general overlap claim is shown only for n=5, and the general tolerance formula is never derived. → Two majorities always overlap because 2·(floor(n/2)+1) ≥ n+1 > n. For n=4 that gives 3+3=6, so they share at least 2 servers. Odd n=2k+1: majority k+1, tolerance k (n=7: 4, 3). Even n=2k: majority k+1, tolerance k−1 (n=6: 4, 2)." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 2", + "text": "The claim 'majority is the smallest size where any two groups overlap' has no breaking instance at n=5. The 'no data is lost' claim when the cluster stops has no instance at all. → At n=5, size 2 breaks it: {A,B} and {C,D} share nothing (2+2=4<5). No data lost: x=1 is decided on A,B,C. Any later quorum of 3 out of 5 contains at least one of A,B,C, so x=1 is always found. This breaks if failed servers lose their disks: if A,B,C all lose storage, x=1 is gone." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 4", + "text": "The Temporal sequence steps have no numbers. → n=5: the client sends x=1. A accepts (1 vote), B accepts (2), C accepts (3 = quorum), and x=1 is decided. D and E can reply later. n=4 with C,D down: A,B accept, 2 < 3, so x=1 waits until C or D returns." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 4", + "text": "Missing case: 4 servers split 2/2 under majority quorum. → n=4, partition {A,B} / {C,D}: each side has 2 < 3, so neither decides. n=5 split {A,B,C} / {D,E}: one side has 3 and keeps working." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 5", + "text": "The model never says what a larger majority costs the reader. → n=5 waits for 3 acceptances per change, and n=6 waits for 4. That is one more server, one more acceptance per change, and still tolerance 2." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "Why-this-form row about floor (3.5 servers): the audience already knows floor and integer division." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Scope detail 'n ≥ 3f + 1, e.g. 4 servers for 1 liar; see PBFT': introduces an undefined f and is not needed for the central question." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Established practice list naming ZooKeeper and etcd: not needed to answer the central question." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151328-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151328-coldread-narrative.json new file mode 100644 index 0000000..d247516 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151328-coldread-narrative.json @@ -0,0 +1,399 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "quorum", + "at": "2026-10-04T15:13:28", + "model": "default", + "prompt_sha": "46d121ad7ad5", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: answer in bold. Said: \"Both clusters need 3 servers to agree; 5 − 3 = 2, 4 − 3 = 1.\"", + "unresolved": [ + "\"Both clusters\": no cluster has been introduced, and nothing says one has 5 servers and the other has 4", + "\"agree\": agree on what?", + "\"5 − 3 = 2, 4 − 3 = 1\": nothing says what the 2 and the 1 count (failures the cluster can survive)" + ], + "unsaid": [ + "The question itself (why 5 survives 2 failures but 4 survives only 1) appears only in the author's 'Reader's question' column. The reader hears an answer with no question before it." + ], + "leap": [ + "Why 3 for both? No reason is given yet. That is fine for a hook, but it is not marked as something to be explained later." + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 2 (name)", + "text": "Shown: x = 1 accepted by A, B, C. Said: \"A cluster decides a change only when a quorum accepts it. Define accept, decide, quorum size, quorum.\"", + "unresolved": [ + "\"quorum\": the script says \"Define accept, decide, quorum size, quorum\" but never gives the definitions, so the reader still does not know what a quorum is", + "\"accept\"", + "\"decide\"", + "\"quorum size\"" + ], + "unsaid": [ + "\"x = 1\": what x is and that it stands for a proposed change", + "\"A, B, C\": these letters are never said to be servers, and nothing says how many servers the cluster has", + "that the quorum size here is 3" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "Shown: partition {A,B}/{C,D}; 500 ms timeout case. Said: \"A cannot tell crash from cut-off. With quorum size 2, both halves decide: split brain.\"", + "unresolved": [ + "\"split brain\": named but never defined", + "\"partition\" (shown only)", + "\"cut-off\": cut off from what? The network split is never described in words" + ], + "unsaid": [ + "\"500 ms timeout case\"", + "the {A,B}/{C,D} split, and that we have moved from 5 servers to a 4-server cluster", + "that the two halves decide *different* changes (no x = 2 is said here)" + ], + "leap": [ + "Why both halves deciding is harmful. The conflict (x = 1 on one side, x = 2 on the other) is not stated", + "Why \"A cannot tell crash from cut-off\" means it must not simply carry on" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Shown: {A,B,C} and {C,D,E}; C refuses x = 2. Said: \"More than half of all n: two such groups share a server, and that server refuses the conflict. Count all n, not reachable ones.\"", + "unresolved": [ + "\"n\": never said to mean the number of servers", + "\"the conflict\": x = 2 has never been said" + ], + "unsaid": [ + "\"C refuses x = 2\": the visual introduces x = 2 without the words mentioning it", + "E appears, so we are back to 5 servers, but nothing says so" + ], + "leap": [ + "Why the shared server refuses. No rule has been stated that a server accepts only one value per change", + "\"Count all n, not reachable ones\": no reason is given here, and the link back to beat 3 (survivors cannot tell crash from cut-off) is not made" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (test)", + "text": "Shown: predict, then answer. Said: \"3 side decides, 2 side stops.\"", + "unresolved": [ + "\"stops\": its meaning (no new changes, no data loss) is only given in beat 7" + ], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (generalize)", + "text": "Shown: table n = 3..7. Said: \"Tolerance = n − majority size. Even n adds nothing, and costs one more accept per change.\"", + "unresolved": [ + "\"majority size\": beat 4 said \"more than half\" but never named it or gave its value per n", + "\"Tolerance\": only defined by the equation" + ], + "unsaid": [ + "the table's columns and any formula in it (for example floor(n/2)+1) are not read out" + ], + "leap": [ + "\"Even n adds nothing\": the reason (more than half of 2k and of 2k−1 is the same number) is left for the reader to work out from the table" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (close)", + "text": "Said: \"Answer again; 'stop' means no new changes, not data loss; scope pointers.\"", + "unresolved": [ + "\"scope pointers\": no actual content is given" + ], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 1: the question is never asked. The reader hears \"Both clusters need 3 servers to agree; 5 − 3 = 2, 4 − 3 = 1\" without knowing what the 2 and 1 count.", + "Beat 2: \"Define accept, decide, quorum size, quorum.\" The definitions are not in the script, so the central term \"quorum\" stays undefined.", + "Beat 3: \"both halves decide: split brain\". Nothing says the halves decide *different* changes, so the harm is never stated.", + "Beat 4: \"More than half of all n\". n is never defined.", + "Beat 4: \"that server refuses the conflict\". The rule that makes it refuse is never given, and this is the core of the argument." + ], + "cuts": [ + "Beat 3: the shown \"500 ms timeout case\". The words never use it, so it is noise.", + "Beat 7: \"scope pointers\" with no content. Either fill it in or cut it.", + "Beat 6: \"and costs one more accept per change\". This is minor, and the main line does not need it." + ], + "question": { + "text": "Why does a 5-server cluster survive 2 failures while a 4-server cluster survives only 1?", + "known_at": "The script never states it outright. The reader can only infer it from beat 1's \"5 − 3 = 2, 4 − 3 = 1\", and fully only by beat 6, where \"Tolerance\" is introduced.", + "result_stated": "Yes, at beat 1: \"Both clusters need 3 servers to agree; 5 − 3 = 2, 4 − 3 = 1.\" It is restated at beat 6: \"Tolerance = n − majority size. Even n adds nothing\"." + }, + "motive": "There is no reason to care in the spoken script. The cost point (a 4th server buys nothing) appears only in the plan. Beat 6's \"Even n adds nothing, and costs one more accept per change\" comes closest, but it arrives late. The reason for the approach is partly there in beat 3: \"A cannot tell crash from cut-off.\"", + "close": "Only partly. Beat 7 says \"Answer again\" but the script gives no wording, and the general argument (any two majorities share a server) is not restated in words at the close. It was said only in beat 4: \"two such groups share a server, and that server refuses the conflict\".", + "recap": "A cluster only accepts a change if more than half of all its servers agree, so two groups can't decide conflicting things. 5 and 4 both need 3, so 5 can lose 2 but 4 can lose only 1.", + "strongest": "Beat 4: \"two such groups share a server, and that server refuses the conflict\". The visual {A,B,C} and {C,D,E} grounds it with a concrete overlap.", + "weakest": "Beat 2: \"Define accept, decide, quorum size, quorum.\" This is an instruction to the author rather than an explanation, so the key vocabulary never reaches the reader.", + "suggestions": [ + "Open beat 1 by asking the question out loud before giving the answer.", + "In beat 2, write out the definitions, for example: accept means a server stores the change and replies yes; a quorum is a group of that many servers. Say that A to E are servers.", + "In beat 3, say that one half sets x = 1 and the other sets x = 2, and explain why that is bad.", + "In beat 4, define n as the total number of configured servers. State that each server accepts only one value for a given change, which is why C refuses.", + "Read out floor(n/2)+1 and walk through the n = 4 and n = 5 rows of the table in words.", + "In beat 7, restate the overlap argument in one sentence and list the actual scope limits." + ] + }, + "usage": { + "cost": 0.12129479999999998, + "seconds": 43 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 1: the question is never asked. The reader hears \"Both clusters need 3 servers to agree; 5 − 3 = 2, 4 − 3 = 1\" without knowing what the 2 and 1 count." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: \"Define accept, decide, quorum size, quorum.\" The definitions are not in the script, so the central term \"quorum\" stays undefined." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 3: \"both halves decide: split brain\". Nothing says the halves decide *different* changes, so the harm is never stated." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: \"More than half of all n\". n is never defined." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: \"that server refuses the conflict\". The rule that makes it refuse is never given, and this is the core of the argument." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 3: the shown \"500 ms timeout case\". The words never use it, so it is noise." + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 7: \"scope pointers\" with no content. Either fill it in or cut it." + }, + { + "n": 8, + "severity": "excess", + "kind": "cut", + "text": "Beat 6: \"and costs one more accept per change\". This is minor, and the main line does not need it." + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): \"Both clusters\": no cluster has been introduced, and nothing says one has 5 servers and the other has 4" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): \"agree\": agree on what?" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): \"5 − 3 = 2, 4 − 3 = 1\": nothing says what the 2 and the 1 count (failures the cluster can survive)" + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): The question itself (why 5 survives 2 failures but 4 survives only 1) appears only in the author's 'Reader's question' column. The reader hears an answer with no question before it." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): Why 3 for both? No reason is given yet. That is fine for a hook, but it is not marked as something to be explained later." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (name): \"quorum\": the script says \"Define accept, decide, quorum size, quorum\" but never gives the definitions, so the reader still does not know what a quorum is" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (name): \"accept\"" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (name): \"decide\"" + }, + { + "n": 17, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (name): \"quorum size\"" + }, + { + "n": 18, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (name): \"x = 1\": what x is and that it stands for a proposed change" + }, + { + "n": 19, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (name): \"A, B, C\": these letters are never said to be servers, and nothing says how many servers the cluster has" + }, + { + "n": 20, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (name): that the quorum size here is 3" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): \"split brain\": named but never defined" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): \"partition\" (shown only)" + }, + { + "n": 23, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): \"cut-off\": cut off from what? The network split is never described in words" + }, + { + "n": 24, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): \"500 ms timeout case\"" + }, + { + "n": 25, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): the {A,B}/{C,D} split, and that we have moved from 5 servers to a 4-server cluster" + }, + { + "n": 26, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): that the two halves decide *different* changes (no x = 2 is said here)" + }, + { + "n": 27, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): Why both halves deciding is harmful. The conflict (x = 1 on one side, x = 2 on the other) is not stated" + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): Why \"A cannot tell crash from cut-off\" means it must not simply carry on" + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): \"n\": never said to mean the number of servers" + }, + { + "n": 30, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): \"the conflict\": x = 2 has never been said" + }, + { + "n": 31, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): \"C refuses x = 2\": the visual introduces x = 2 without the words mentioning it" + }, + { + "n": 32, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): E appears, so we are back to 5 servers, but nothing says so" + }, + { + "n": 33, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): Why the shared server refuses. No rule has been stated that a server accepts only one value per change" + }, + { + "n": 34, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): \"Count all n, not reachable ones\": no reason is given here, and the link back to beat 3 (survivors cannot tell crash from cut-off) is not made" + }, + { + "n": 35, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (test): \"stops\": its meaning (no new changes, no data loss) is only given in beat 7" + }, + { + "n": 36, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (generalize): \"majority size\": beat 4 said \"more than half\" but never named it or gave its value per n" + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (generalize): \"Tolerance\": only defined by the equation" + }, + { + "n": 38, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (generalize): the table's columns and any formula in it (for example floor(n/2)+1) are not read out" + }, + { + "n": 39, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (generalize): \"Even n adds nothing\": the reason (more than half of 2k and of 2k−1 is the same number) is left for the reader to work out from the table" + }, + { + "n": 40, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (close): \"scope pointers\": no actual content is given" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151438-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151438-coldread-narrative.json new file mode 100644 index 0000000..83d3122 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151438-coldread-narrative.json @@ -0,0 +1,218 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "quorum", + "at": "2026-10-04T15:14:38", + "model": "default", + "prompt_sha": "46d121ad7ad5", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "A cluster of n servers keeps copies of the same data. It decides a change only when more than half of all n servers accept it. 5 needs 3 so 2 can fail; 4 also needs 3 so only 1 can fail.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (name)", + "text": "Defines accept, decide, quorum size, quorum, and stop, using x = 1 accepted by A, B, C.", + "unresolved": [], + "unsaid": [], + "leap": [ + "\"the next change\" assumes the changes come in a fixed order, one slot at a time, which is never said. Beat 4's rule (\"Each server accepts only one change as the next change\") depends on this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "A gets no reply from D in 500 ms: crash or partition, A can't tell. With quorum size 2, {A,B} decide x = 1 and {C,D} decide x = 2: split brain.", + "unresolved": [], + "unsaid": [ + "The Shown column switches to 4 servers, after beat 2's 5 servers A–E. The Said column never says the cluster is now 4 servers. The reader has to work this out from the names A–D.", + "The question this beat answers, \"why can't the other 2 go on?\", is never asked in the words. The text jumps straight into \"A sends to D\"." + ], + "leap": [ + "\"So suppose quorum size were 2\": the words never say that 2 is exactly what 'let the 2 survivors go on' means in a 4-server cluster, or that it is half of 4." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Quorum size = majority size floor(n/2)+1. Two groups of 3 among 5 total 6 > 5 places, so they share C. C has already accepted x = 1, so it refuses x = 2. Count all configured servers, not the reachable ones.", + "unresolved": [ + "\"6 > 5 places\": 'places' is an odd word for this counting step. It is only shown for n = 5, and the general version ('two majorities together have more than n members') is never said." + ], + "unsaid": [ + "The Shown column has the overlapping groups {A,B,C} and {C,D,E}. The words do not say how C could be in contact with both groups, given that beat 3 framed the problem as a partition." + ], + "leap": [ + "\"Each server accepts only one change as the next change\" is a new rule, and the whole argument depends on it. It appears here for the first time, with no reason given." + ], + "excess": [ + "\"D and E reach only each other, and 'more than half of 2' would let them decide\" gives away the scenario that beat 5 asks the reader to predict." + ], + "purpose_unclear": false + }, + { + "at": "Beat 5 (test)", + "text": "Predict: {A,B,C} | {D,E}. A, B, C have 3 ≥ 3 and keep deciding. D, E have 2 < 3 and stop.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [ + "The prediction was already answered at the end of beat 4 (\"D and E reach only each other...\"), so the test gives the reader nothing new to work out." + ], + "purpose_unclear": false + }, + { + "at": "Beat 6 (generalize)", + "text": "Tolerance = n − majority size. A table for n = 3..7 shows that 4 and 5 both need 3. Adding a server to an odd-sized cluster raises the majority size, so tolerance stays the same and each change waits for one more accept.", + "unresolved": [], + "unsaid": [ + "The table rows for n = 3, 6 and 7 are shown but not read aloud. Minor." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (close)", + "text": "5 survives 2 and 4 survives 1 because both need 3, and 3 is the smallest group that must share a server with any other. Use odd sizes. Assumes servers crash but never lie; see the Raft and PBFT papers.", + "unresolved": [ + "\"PBFT\"", + "\"Raft\"", + "\"leader election\"" + ], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 4: \"Each server accepts only one change as the next change\". This rule makes the shared server C refuse x = 2, and it is introduced here without warning or reason. Without it, overlapping groups prove nothing.", + "Beat 3: the switch from 5 servers to 4 is not said in words (it appears only in Shown), so the reader rereads to find out which cluster they are in." + ], + "cuts": [ + "Beat 4: cut the D/E sentence (\"Count all n configured servers... would let them decide\"), or move it into beat 5's answer. It currently spoils the prediction.", + "Beat 7: cut \"leader election, and reads are covered in the Raft and PBFT papers\". It brings in unknown names right at the close; \"servers crash but never lie\" is enough as the scope." + ], + "question": { + "text": "Why can a 5-server cluster lose 2 servers while a 4-server cluster can lose only 1? Known from Beat 1, where it is the title.", + "known_at": "Beat 1", + "result_stated": "Yes, Beat 1: \"For 5 servers that is 3, so 5 − 3 = 2 servers can fail. For 4 servers it is also 3, so only 4 − 3 = 1 can fail.\"" + }, + "motive": "There is a reason for the approach in Beat 1, \"The rest explains why the rule is 'more than half'\", and in Beat 3's objection, \"A sees the same silence in both cases\". A reason to care about the question is missing from the words. The plan's point that \"the 4th server costs money but buys nothing\" is never said; only Beat 6's \"tolerance stays the same\" and Beat 7's \"Use odd cluster sizes\" hint at it.", + "close": "Yes, Beat 7: \"5 survives 2 and 4 survives 1 because both need 3 servers, and 3 is the smallest group that must share a server with any other group.\" The general argument (overlap means a shared server, and that server refuses a conflicting change) is said in words, though the refusal rule itself comes late, in Beat 4.", + "recap": "A cluster only decides a change when more than half of all its servers agree, so two groups can never both decide different things; that means 4 servers need 3 just like 5 do, so the 4th server buys you no extra failure tolerance.", + "strongest": "Beat 3: \"A sees the same silence in both cases\" makes it concrete why the survivors cannot simply carry on, and leads straight into split brain.", + "weakest": "Beat 4: it packs the formula, the overlap count (\"6 > 5 places\"), a brand-new one-vote-per-slot rule, and the counting caveat that gives away Beat 5 into one beat.", + "suggestions": [ + "Introduce the rule \"a server accepts only one change for each slot\" in Beat 2, alongside accept.", + "Say the general overlap argument in words: two groups that are each more than half of n together have more than n members, so they must share a server.", + "Say in Beat 3's words that the cluster is now 4 servers and that a quorum of 2 is exactly half.", + "Add the cost motive to Beat 1: a 4th server costs money but buys no extra tolerance." + ] + }, + "usage": { + "cost": 0.11751439999999999, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: \"Each server accepts only one change as the next change\". This rule makes the shared server C refuse x = 2, and it is introduced here without warning or reason. Without it, overlapping groups prove nothing." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 3: the switch from 5 servers to 4 is not said in words (it appears only in Shown), so the reader rereads to find out which cluster they are in." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Beat 4: cut the D/E sentence (\"Count all n configured servers... would let them decide\"), or move it into beat 5's answer. It currently spoils the prediction." + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "Beat 7: cut \"leader election, and reads are covered in the Raft and PBFT papers\". It brings in unknown names right at the close; \"servers crash but never lie\" is enough as the scope." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (name): \"the next change\" assumes the changes come in a fixed order, one slot at a time, which is never said. Beat 4's rule (\"Each server accepts only one change as the next change\") depends on this." + }, + { + "n": 6, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): The Shown column switches to 4 servers, after beat 2's 5 servers A–E. The Said column never says the cluster is now 4 servers. The reader has to work this out from the names A–D." + }, + { + "n": 7, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): The question this beat answers, \"why can't the other 2 go on?\", is never asked in the words. The text jumps straight into \"A sends to D\"." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): \"So suppose quorum size were 2\": the words never say that 2 is exactly what 'let the 2 survivors go on' means in a 4-server cluster, or that it is half of 4." + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): \"6 > 5 places\": 'places' is an odd word for this counting step. It is only shown for n = 5, and the general version ('two majorities together have more than n members') is never said." + }, + { + "n": 10, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The Shown column has the overlapping groups {A,B,C} and {C,D,E}. The words do not say how C could be in contact with both groups, given that beat 3 framed the problem as a partition." + }, + { + "n": 11, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): \"Each server accepts only one change as the next change\" is a new rule, and the whole argument depends on it. It appears here for the first time, with no reason given." + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (generalize): The table rows for n = 3, 6 and 7 are shown but not read aloud. Minor." + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (close): \"PBFT\"" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (close): \"Raft\"" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (close): \"leader election\"" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151602-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151602-coldread-prose.json new file mode 100644 index 0000000..1733bd5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151602-coldread-prose.json @@ -0,0 +1,370 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "quorum", + "at": "2026-10-04T15:16:02", + "model": "default", + "prompt_sha": "95ac16e43338", + "reply": { + "paragraphs": [ + { + "at": "Title (line 1)", + "text": "Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold summary (line 5)", + "text": "A cluster decides a change only when more than half of all its servers accept the change...", + "unresolved": [ + "decides a change", + "accept the change" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 7", + "text": "The rest of this page explains why the rule must be \"more than half\".", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 11", + "text": "A cluster is a group of n servers that keep copies of the same data...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, list (lines 13-15)", + "text": "A server accepts the change when... decided... quorum size... quorum", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 17", + "text": "The cluster decides changes one at a time, in order... For each position in that order, a server accepts only one change.", + "unresolved": [], + "leap": [ + "The rule that a server accepts only one change per position is stated with no reason. Its role only becomes clear at line 42." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 19", + "text": "If fewer than a quorum of servers answer, the cluster stops...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why not let the survivors go on?, line 23", + "text": "Take a cluster of 4 servers, A to D. C and D stop answering...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 25", + "text": "A sends a message to D and gets no reply in 500 ms... partition...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 27", + "text": "Letting A and B go on means quorum size 2...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Table (lines 29-32)", + "text": "Group | Accepts for change 7 | Votes | Quorum size | Result", + "unresolved": [], + "leap": [ + "\"change 7\": the number 7 comes with no setup. The reader has to infer that it is just an arbitrary position in the order." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 34", + "text": "Both groups decide, and the copies now disagree for good. This is split brain...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why more than half works, line 38", + "text": "Set the quorum size to more than half... majority size: floor(n/2) + 1...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 40", + "text": "Take any two quorums of 3... Together they list 6 servers, but the cluster has only 5...", + "unresolved": [], + "leap": [ + "The overlap argument is shown only for n = 5. The general version (two groups that are each more than n/2 add up to more than n) is never said in words." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 42", + "text": "C already accepted x = 1 for change 7, so C refuses x = 2 for change 7...", + "unresolved": [], + "leap": [ + "\"C already accepted x = 1 for change 7\": the example has moved from 4 servers to 5, and no 5-server scenario was set up in which A, B, C accepted x = 1 first. The reader has to rebuild the scenario." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 44", + "text": "This depends on one assumption: a server remembers what it accepted, even after a restart...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 46", + "text": "Predict: the network splits 5 servers into {A, B, C} and {D, E}...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 48", + "text": "A, B, C have 3 votes, and 3 ≥ 3, so they keep deciding...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 50", + "text": "Count more than half of all configured servers, failed or not...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Counting failures, lines 54-56", + "text": "Tolerance is how many servers can fail... tolerance = n − majority size", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Table (lines 58-64)", + "text": "Servers n | Majority size | Tolerance", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 66", + "text": "With 5 servers and 2 failed, 3 servers answer, and 3 ≥ 3... With 4 servers and 2 failed... the cluster stops.", + "unresolved": [], + "leap": [], + "excess": [ + "This mostly restates the opening bold paragraph and the table rows for 4 and 5, adding only the ≥/< comparison." + ], + "purpose_unclear": false + }, + { + "at": "Line 68", + "text": "A tolerance of 1 means a second failure stops the cluster...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 70", + "text": "Adding a server to an odd-sized cluster also raises the majority size by 1...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer, line 74", + "text": "Both clusters need 3 servers to decide a change, because 3 is the smallest group that must share a server with every other group...", + "unresolved": [], + "leap": [ + "\"every other group\" means every other group of that size (every other quorum). The wording could be read as any group at all." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, line 78", + "text": "Assumed: servers crash or become unreachable, but never send a false \"yes\"...", + "unresolved": [ + "PBFT algorithm" + ], + "leap": [ + "\"4 to tolerate 1 liar\" is given without a reason. That is acceptable because it is marked as out of scope." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, line 79", + "text": "Simplified: in real protocols such as Raft and Paxos...", + "unresolved": [ + "Raft", + "Paxos" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, line 80", + "text": "Not covered: how servers detect a failure...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Line 42, \"C already accepted x = 1 for change 7\": this is the core step showing why overlap prevents split brain, but the 5-server scenario it depends on (A, B, C accepting x = 1 first) is never set up. The only earlier change-7 example used 4 servers." + ], + "cuts": [ + "Line 66: shorten or merge it into the table. It repeats the bold opening paragraph.", + "Line 25: \"in 500 ms\" is a concrete detail the argument does not need. It is small and harmless." + ], + "question": { + "text": "Why can a 5-server cluster keep working with 2 servers down while a 4-server cluster can only lose 1? Behind that: why must the decision rule be \"more than half of all servers\"?", + "known_at": "Title (line 1), sharpened by line 7", + "result_stated": "Yes, in words, at line 5: \"For 5 servers, more than half is 3, so 5 − 3 = 2 servers can fail. For 4 servers, more than half is also 3, so only 4 − 3 = 1 server can fail.\"" + }, + "motive": "Reason to care about the question: the split-brain table and line 34, \"Both groups decide, and the copies now disagree for good.\" Reason for the approach: line 25, \"A sees the same silence in both cases, so A cannot know which case is true\", which explains why survivors cannot simply go on. Line 70 adds a practical payoff: \"That is why clusters usually have an odd number of servers.\"", + "close": "Yes. Line 74: \"Both clusters need 3 servers to decide a change, because 3 is the smallest group that must share a server with every other group. 5 servers have 2 to spare. 4 servers have only 1.\" This states the general overlap argument in words, though only loosely.", + "recap": "A cluster needs a majority of all its servers to agree, so that any two deciding groups share a server that blocks conflicting decisions; 4 and 5 servers both need 3, so 5 has one more server to spare.", + "strongest": "Lines 25-34: the indistinguishable-silence point leads straight into the partition table, which shows split brain concretely.", + "weakest": "Line 42: the switch from 4 to 5 servers leaves out the setup in which C first accepted x = 1.", + "suggestions": [ + "Before line 42, add one sentence such as \"Suppose A, B, C have already accepted x = 1 for change 7, and D, E now receive x = 2.\"", + "After the 5-server overlap example at line 40, add the general sentence: two groups that each hold more than half of n servers together count more than n, so they must share a server.", + "Mention once at line 17 why a server accepts only one change per position (so that a shared server can block a conflict), or point ahead to line 42." + ] + }, + "usage": { + "cost": 0.1212488, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Line 42, \"C already accepted x = 1 for change 7\": this is the core step showing why overlap prevents split brain, but the 5-server scenario it depends on (A, B, C accepting x = 1 first) is never set up. The only earlier change-7 example used 4 servers." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Line 66: shorten or merge it into the table. It repeats the bold opening paragraph." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Line 25: \"in 500 ms\" is a concrete detail the argument does not need. It is small and harmless." + }, + { + "n": 4, + "severity": "edge", + "kind": "unresolved", + "text": "Bold summary (line 5): decides a change" + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Bold summary (line 5): accept the change" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "The rule, line 17: The rule that a server accepts only one change per position is stated with no reason. Its role only becomes clear at line 42." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Table (lines 29-32): \"change 7\": the number 7 comes with no setup. The reader has to infer that it is just an arbitrary position in the order." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Line 40: The overlap argument is shown only for n = 5. The general version (two groups that are each more than n/2 add up to more than n) is never said in words." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Line 42: \"C already accepted x = 1 for change 7\": the example has moved from 4 servers to 5, and no 5-server scenario was set up in which A, B, C accepted x = 1 first. The reader has to rebuild the scenario." + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "Answer, line 74: \"every other group\" means every other group of that size (every other quorum). The wording could be read as any group at all." + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Limits, line 78: PBFT algorithm" + }, + { + "n": 12, + "severity": "edge", + "kind": "leap", + "text": "Limits, line 78: \"4 to tolerate 1 liar\" is given without a reason. That is acceptable because it is marked as out of scope." + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Limits, line 79: Raft" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Limits, line 79: Paxos" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151643-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151643-quiz-prose.json new file mode 100644 index 0000000..ab664ca --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151643-quiz-prose.json @@ -0,0 +1,171 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "quorum", + "at": "2026-10-04T15:16:29", + "model": "default", + "prompt_sha": "fda2e8264e06", + "reply": { + "answers": { + "1": "A cluster of n servers holding copies of the same data; the client that proposes a change (e.g. x = 1); the change and its position in the order (change 7); a server's acceptance (\"yes\", stored on disk); a quorum, meaning a group of servers of quorum size; the majority size floor(n/2)+1; a partition, meaning network groups that cannot reach each other; split brain; tolerance.", + "2": "A client proposes a change to the cluster's servers. Each server accepts at most one change per position. A change is decided when a quorum of servers accept it. The quorum size is set to the majority of all n configured servers, so any two quorums share at least one server. A server in both quorums refuses a second, conflicting change, so two sides of a partition cannot both decide. Tolerance is n minus the majority size.", + "3": "Any two majorities of the same n servers must share a server. That server accepts only one change per position, so only one change can win each position, and split brain cannot happen. This holds even when a partition looks the same as a crash. The price is that the cluster stops once fewer than a majority of servers answer.", + "4": "The key variable is n, the cluster size. Going from odd n to n+1 raises the majority size by 1, so tolerance stays the same and each change needs one more accept. Going from even n to n+1 leaves the majority size unchanged, so tolerance rises by 1. Separately, if the quorum size drops to half or less, two disjoint groups can both decide and split brain becomes possible.", + "5": "Assumption: a server remembers what it accepted, even after a restart. A second one: servers never send a false \"yes\". Observation or fact: two quorums of 3 among 5 servers list 6 members, so they must share at least one server. Also: a server cannot tell a crashed peer from a cut network link, because both look like the same silence.", + "6": "A cluster decides a change only when more than half of all its configured servers accept it, and each server accepts only one change per position. A silent server may have crashed, or the network may be partitioned. Letting a smaller group continue could let two sides decide different values (split brain). Any two majorities overlap, and the shared server refuses the second value, so only one value can win. Tolerance is n minus the majority size: 5 servers need 3 and can lose 2, while 4 servers also need 3 and can lose only 1, which is why odd sizes are preferred.", + "7": "The hardest part was the jump from the 5-server overlap example to the general formula floor(n/2)+1. The page never shows that two quorums of that size always add up to more than n, for even n as well as odd. A one-line count for n=4 (3+3=6>4) and n=6 (4+4=8>6) would have closed that gap. A smaller point: the table uses both 'Accepts' and 'Votes'.", + "8": "2. The majority size is 4, and 6 − 4 = 2.", + "9": "Two groups of 2 among 4 servers need not share a server. In a partition {A,B} | {C,D}, both groups could decide different values for the same position, which is split brain.", + "10": "The group of 3 has 3 ≥ 3 votes, so it keeps deciding. The group of 2 has 2 < 3, so it stops: it keeps the changes already decided but decides no new ones until enough servers return.", + "11": "They cannot tell whether C and D crashed or whether the network is cut and C and D are still running and deciding together. If it is a partition, both pairs could decide different values (split brain). The majority of 4 is 3, so 2 servers must stop.", + "12": "A quorum of 2 among 4 servers lets two disjoint pairs each decide during a partition. The result is split brain, where the copies disagree for good. Surviving 2 failures would cost correctness.", + "13": "It tolerates 2 failures. The majority size is floor(6/2)+1 = 4, and 6 − 4 = 2. The 6th server raises the quorum to 4 without adding tolerance compared with 5 servers." + }, + "audit": { + "terms": [ + "'Accepts' and 'Votes' name the same thing in the table header (line 29), and 'votes' is used in lines 42 and 48. Line 13 defines only 'accepts'.", + "'quorum size' (line 15) and 'majority size' (line 38) are used as near-synonyms. The answer section says 'need 3 servers' without either term.", + "'change 7' (line 29) and 'position in that order' (line 17) refer to the same slot.", + "'fail' (lines 5, 54), 'stop answering' (line 23), 'crashed' (line 25) and 'down' (line 68) are used loosely for the same failure." + ], + "unexplained": [ + "'500 ms' (line 25) is given with no meaning. Would a longer or shorter wait change anything? The text implies it would not, but never says so.", + "'4 to tolerate 1 liar' (line 78) gives a number with no reason behind it." + ], + "missing_why": [ + "'floor(n/2) + 1' (line 38): the overlap proof is only shown for n=5 (line 40). It never states that two quorums of this size always total more than n, for even n too.", + "'3 is the smallest group that must share a server with every other group' (line 74): 'every other group' presumably means every other group of quorum size, but this is not stated, and why 2 fails for n=5 is not shown.", + "'Adding a server to an odd-sized cluster also raises the majority size by 1' (line 70): the page does not say why adding to an even-sized cluster doesn't." + ], + "excess": [ + "The HTML comment on line 3 ('Stage 1, rendered from model.md in STE-80') means nothing to this reader.", + "The PBFT bullet (line 78) is a detour that no question or step of the argument uses.", + "The Raft/Paxos 'Simplified' bullet (line 79) names protocols that are never explained. It is harmless but unused." + ] + }, + "gaps": [ + "There is no general argument that majorities overlap for every n, only the 5-server example.", + "It does not say what happens to a change that was accepted by some servers but never decided, for example A and B having accepted x = 1 during a partition.", + "It does not say whether the client learns that a change failed or stalled when the cluster stops." + ] + }, + "grade": { + "scores": { + "1": 1, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 1, + "13": 2 + }, + "reasons": { + "1": "Covers most entities, but 'failure' and 'decide' are not listed as entities, and 'quorum size' is only implied inside the quorum definition. It also adds 'position in the order (change 7)', which model.md does not have.", + "12": "Explains that two disjoint pairs could each decide (split brain), but never states the rule the item expects: the quorum size must be more than half of n, so two groups cannot both decide. It shows why 2 fails, not the general requirement.", + "13": "Correct for n = 6 (majority size 4, 6 − 4 = 2, no gain over 5 servers). The rubric's expect line describes n = 5 and n = 4 rather than 6; the answer was scored against the question as asked." + }, + "pass": false + }, + "usage": { + "cost": 0.1574386, + "seconds": 40 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "'Accepts' and 'Votes' name the same thing in the table header (line 29), and 'votes' is used in lines 42 and 48. Line 13 defines only 'accepts'." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'quorum size' (line 15) and 'majority size' (line 38) are used as near-synonyms. The answer section says 'need 3 servers' without either term." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'change 7' (line 29) and 'position in that order' (line 17) refer to the same slot." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'fail' (lines 5, 54), 'stop answering' (line 23), 'crashed' (line 25) and 'down' (line 68) are used loosely for the same failure." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'500 ms' (line 25) is given with no meaning. Would a longer or shorter wait change anything? The text implies it would not, but never says so." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'4 to tolerate 1 liar' (line 78) gives a number with no reason behind it." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "'floor(n/2) + 1' (line 38): the overlap proof is only shown for n=5 (line 40). It never states that two quorums of this size always total more than n, for even n too." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "'3 is the smallest group that must share a server with every other group' (line 74): 'every other group' presumably means every other group of quorum size, but this is not stated, and why 2 fails for n=5 is not shown." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "'Adding a server to an odd-sized cluster also raises the majority size by 1' (line 70): the page does not say why adding to an even-sized cluster doesn't." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "The HTML comment on line 3 ('Stage 1, rendered from model.md in STE-80') means nothing to this reader." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "The PBFT bullet (line 78) is a detour that no question or step of the argument uses." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "The Raft/Paxos 'Simplified' bullet (line 79) names protocols that are never explained. It is harmless but unused." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "There is no general argument that majorities overlap for every n, only the 5-server example." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "It does not say what happens to a change that was accepted by some servers but never decided, for example A and B having accepted x = 1 during a partition." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "It does not say whether the client learns that a change failed or stalled when the cluster stops." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151817-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151817-quiz-prose.json new file mode 100644 index 0000000..8ec8ed0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151817-quiz-prose.json @@ -0,0 +1,141 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "quorum", + "at": "2026-10-04T15:18:10", + "model": "default", + "prompt_sha": "45bb3c67ca87", + "reply": { + "answers": { + "1": "Servers (n of them) that form a cluster, each holding a copy of the same data. Clients, which propose changes (e.g. x = 1). Changes, which are decided one at a time at numbered positions (change 1, 2, … 7). Accepts: a server stores the change on disk and replies 'yes'. The quorum size and a quorum, which is a group of that many servers. The majority size, floor(n/2)+1. Failures: a crash or an unreachable server. A partition: the network splits into groups that cannot reach each other. Split brain: copies that disagree for good. Tolerance: how many servers can fail while the cluster still decides changes.", + "2": "A client proposes a change to the cluster. Each server accepts at most one change per position. A change is decided when a quorum of servers has accepted it. The quorum size is set to the majority size of all n configured servers. Any two quorums then share at least one server, and that server refuses a second, different value for the same position, so two conflicting decisions cannot happen. Failures and partitions reduce how many servers answer. If fewer than a quorum answer, the cluster stops deciding but keeps what it has already decided. Tolerance = n − majority size.", + "3": "Because a quorum is more than half of all n servers, any two quorums overlap. The shared server has already accepted one value for a position and refuses another, so only one value can be decided and split brain cannot happen. The cost is that the cluster stops when fewer than a majority of servers answer, so it can lose only n − (floor(n/2)+1) servers.", + "4": "The key variable is n, the number of servers. Going from an even n to the next odd n keeps the majority size the same (4→5: both 3), so tolerance rises by 1. Going from an odd n to the next even n raises the majority size by 1 (5→6: 3→4), so tolerance stays the same, and each change just has to wait for one more accept. A second variable is the quorum size. If it drops to half or less, two disjoint groups can both decide, which gives split brain. If it is raised, tolerance drops.", + "5": "Assumption: a server remembers what it accepted even after a restart (its disk is not wiped), and servers never send a false 'yes'. Simplification: in real protocols the shared server reports x = 1 to the new group instead of just refusing. Fact: a server that hears silence cannot tell a crashed peer from a network cut, however long it waits. Also, two groups of more than half of n must share a server, because together they list more than n servers.", + "6": "A cluster decides a change only when a quorum of servers has accepted it, and each server accepts only one change per position. A server cannot tell whether a silent peer crashed or was cut off by the network. So if a quorum could be half or less, both sides of a partition could decide different values (split brain). Requiring more than half of all configured servers, floor(n/2)+1, means any two quorums share a server, and that server refuses the conflicting value. The cluster therefore survives n − majority failures: 2 for n=5, but only 1 for n=4, because 4 and 5 both need 3 accepts.", + "7": "The overlap argument is the hardest part, especially the step 'C refuses'. It depends on the rule from much earlier that a server accepts only one change per position, and the example does not point back to that rule. It also leaves open what happens if a quorum never finished deciding x = 1; the Limits note on Raft/Paxos hints at this but does not resolve it. A one-line reminder of the one-change-per-position rule at the refusal step would help. So would a short note that floor(n/2)+1 is just 'the smallest whole number above n/2'.", + "8": "Majority size for 6 is floor(6/2)+1 = 4, so tolerance is 6 − 4 = 2 failed servers.", + "9": "Two groups of 2 among 4 servers need not share a server. In a partition {A,B} | {C,D}, both sides could decide different values for change 7 (x=1 and x=2), and the copies would disagree for good. That is split brain.", + "10": "The group of 3 has 3 ≥ 3 (the majority size), so it keeps deciding changes. The group of 2 has 2 < 3, so it stops: it keeps the changes already decided but decides nothing new until it can reach enough servers again. The copies cannot disagree.", + "11": "A and B cannot tell whether C and D crashed or are just cut off by a network partition and still running. If C and D are alive on the other side, letting A and B decide with 2 servers would also let C and D decide with 2, and the two sides could decide different values: split brain. 2 is not more than half of 4, so the two groups need not overlap.", + "12": "A quorum of 2 is not more than half of 4. Two disjoint pairs could each decide a different value for the same position during a partition, which produces split brain. Surviving 2 failures this way would cost correctness. The cluster must use quorum 3, which gives tolerance 1.", + "13": "Both need a majority of 3 accepts, because floor(4/2)+1 = floor(5/2)+1 = 3. With 5 servers, 3 still answer after 2 failures, so the cluster keeps deciding. With 4 servers, only 2 answer after 2 failures, and 2 < 3, so it stops. Tolerance is 5−3=2 versus 4−3=1." + }, + "audit": { + "terms": [ + "The required number of accepts is called both 'quorum size' (line 13: 'The number of accepts needed is the *quorum size*') and 'majority size' (line 37: 'the *majority size*: floor(n/2) + 1 ... The quorum size is always the majority size'), and the tolerance formula then uses only 'majority size' (line 55). The reader has to keep two names for one number.", + "Failure is described as 'fails' (line 14), 'stops answering' (line 14), 'become unreachable' (line 77), and 'crash' (line 77). Line 14 does tie these together, so the cost to the reader is small.", + "'Accepts' is used both as a value accepted and as a count in the same table (line 28: column 'Accepts for change 7' holds x = 1, while column 'Accepts received' holds 2)." + ], + "unexplained": [ + "'floor(n/2) + 1' (line 37) is given without saying what floor does, i.e. round down, so for odd n it is the next whole number above n/2. A school-maths reader may not know the notation." + ], + "missing_why": [ + "Line 41, 'C already accepted x = 1 for that position, so C refuses': the refusal depends on the rule in line 16 ('a server accepts only one change' per position), but the text does not refer back to it, and line 16 does not say why servers follow that rule.", + "Line 78, 'the new group keeps x = 1': no reason is given for why the new group should adopt the old value. This is the step that makes the protocol make progress rather than just block.", + "Line 77, 'Servers that lie need larger clusters': stated with no reason and no indication of how much larger." + ], + "excess": [ + "The bold opening (line 3) and the 'Answer' section (line 73) say almost the same thing. The repeat adds only the overlap reason." + ] + }, + "gaps": [ + "The explanation does not say what happens when a change has been accepted by fewer than a quorum before a partition or crash (a partial accept), or how a value stuck at position 7 gets resolved. It is only gestured at in Limits.", + "The single-server example (C refuses) assumes the first decision finished before the second attempt. It does not cover two clients proposing different values at the same moment.", + "It does not say how many more servers are needed when servers can lie (line 77)." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "8": "Full marks: correct count (2) from majority size 4. It omits the expected remark that the sixth server adds nothing over 5. The question does not require that remark." + }, + "pass": true + }, + "usage": { + "cost": 0.1621684, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The required number of accepts is called both 'quorum size' (line 13: 'The number of accepts needed is the *quorum size*') and 'majority size' (line 37: 'the *majority size*: floor(n/2) + 1 ... The quorum size is always the majority size'), and the tolerance formula then uses only 'majority size' (line 55). The reader has to keep two names for one number." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Failure is described as 'fails' (line 14), 'stops answering' (line 14), 'become unreachable' (line 77), and 'crash' (line 77). Line 14 does tie these together, so the cost to the reader is small." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'Accepts' is used both as a value accepted and as a count in the same table (line 28: column 'Accepts for change 7' holds x = 1, while column 'Accepts received' holds 2)." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'floor(n/2) + 1' (line 37) is given without saying what floor does, i.e. round down, so for odd n it is the next whole number above n/2. A school-maths reader may not know the notation." + }, + { + "n": 5, + "severity": "gap", + "kind": "missing_why", + "text": "Line 41, 'C already accepted x = 1 for that position, so C refuses': the refusal depends on the rule in line 16 ('a server accepts only one change' per position), but the text does not refer back to it, and line 16 does not say why servers follow that rule." + }, + { + "n": 6, + "severity": "gap", + "kind": "missing_why", + "text": "Line 78, 'the new group keeps x = 1': no reason is given for why the new group should adopt the old value. This is the step that makes the protocol make progress rather than just block." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 77, 'Servers that lie need larger clusters': stated with no reason and no indication of how much larger." + }, + { + "n": 8, + "severity": "excess", + "kind": "excess", + "text": "The bold opening (line 3) and the 'Answer' section (line 73) say almost the same thing. The repeat adds only the overlap reason." + }, + { + "n": 9, + "severity": "gap", + "kind": "gap", + "text": "The explanation does not say what happens when a change has been accepted by fewer than a quorum before a partition or crash (a partial accept), or how a value stuck at position 7 gets resolved. It is only gestured at in Limits." + }, + { + "n": 10, + "severity": "gap", + "kind": "gap", + "text": "The single-server example (C refuses) assumes the first decision finished before the second attempt. It does not cover two clients proposing different values at the same moment." + }, + { + "n": 11, + "severity": "gap", + "kind": "gap", + "text": "It does not say how many more servers are needed when servers can lie (line 77)." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151859-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151859-coldread-prose.json new file mode 100644 index 0000000..9f11f36 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151859-coldread-prose.json @@ -0,0 +1,317 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "quorum", + "at": "2026-10-04T15:18:59", + "model": "default", + "prompt_sha": "95ac16e43338", + "reply": { + "paragraphs": [ + { + "at": "Title (line 1)", + "text": "Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold lead (line 3)", + "text": "A cluster decides a change only when more than half of all its servers accept the change...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 5", + "text": "The rest of this page explains why the rule must be \"more than half\".", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 9", + "text": "A *cluster* is a group of n servers that keep copies of the same data...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, definition list (lines 11-14)", + "text": "A server *accepts*... *decided*... *quorum size*... *quorum*... *fails*...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 16", + "text": "The cluster decides changes one at a time, in order... For each position in that order, a server accepts only one change...", + "unresolved": [], + "leap": [ + "\"For each position in that order, a server accepts only one change\": this is given as a rule with no reason. The reason only becomes clear at line 41, where this rule is what makes C refuse. A small stumble, not a block." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 18", + "text": "If fewer than a quorum of servers answer, the cluster *stops*...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why not let the survivors go on?, line 22", + "text": "Take a cluster of 4 servers, A to D. C and D fail. Why can A and B not go on alone?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 24", + "text": "A sends a message to D and gets no reply... *partition*... A cannot know which case is true.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 26", + "text": "Letting A and B go on means quorum size 2. Now suppose the cause is a partition...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Partition table (lines 28-31)", + "text": "Group | Accepts for change 7 | Accepts received | Quorum size | Result", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 33", + "text": "Both groups decide... *split brain*... So the quorum size must be more than half of n: then two groups cannot both decide.", + "unresolved": [], + "leap": [ + "\"So the quorum size must be more than half of n\": the example only shows that 2 out of 4 (exactly half) fails. It does not say here why 'more than half' is the fix, or why that fix is enough. That comes in the next section, so this works as a forward pointer, but the 'So' moves faster than the reasoning." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why more than half works, line 37", + "text": "More than half of all n servers is the *majority size*: floor(n/2) + 1... The quorum size is always the majority size.", + "unresolved": [ + "floor(n/2)" + ], + "leap": [ + "\"The quorum size is always the majority size\": the text never says why the quorum isn't set larger than a majority. A larger quorum would also overlap. The real reason, that the smallest safe quorum gives the most tolerance, is never stated." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 39", + "text": "Two quorums each hold more than half of the servers, so together they list more than n servers. Some server must therefore be in both...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 41", + "text": "The shared server blocks split brain. Suppose A, B, C accepted x = 1 for change 7... C refuses...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 43", + "text": "This depends on one assumption: a server remembers what it accepted, even after a restart...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Predict (line 45)", + "text": "**Predict:** the network splits 5 servers into {A, B, C} and {D, E}. What does each side do?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 47", + "text": "A, B, C have 3 accepts, and 3 ≥ 3, so they keep deciding. D and E have 2, and 2 < 3, so they stop. The copies cannot disagree.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 49", + "text": "Count more than half of *all* configured servers, failed or not...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Counting failures, line 53-55", + "text": "*Tolerance* is how many servers can fail while the cluster still decides changes: tolerance = n − majority size", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Tolerance table (lines 57-62)", + "text": "Servers n | Majority size | Tolerance; 3..7", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 65", + "text": "With 5 servers and 2 failed, 3 servers answer, and 3 ≥ 3: the cluster keeps deciding. With 4 servers and 2 failed... the cluster stops.", + "unresolved": [], + "leap": [], + "excess": [ + "The whole paragraph restates the bold lead (line 3) and the n = 4 and n = 5 rows of the table just above it, adding only the ≥/< comparison." + ], + "purpose_unclear": false + }, + { + "at": "line 67", + "text": "A tolerance of 1 means a second failure stops the cluster. A tolerance of 2 means one server can be off for an upgrade while another one fails.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 69", + "text": "Going from an even n to the next odd n keeps the majority size... That is why clusters usually have an odd number of servers.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer (line 73)", + "text": "Both clusters need 3 accepts to decide a change, because 3 is more than half of 4 and of 5, and only groups of more than half must share a server with every other quorum...", + "unresolved": [], + "leap": [ + "\"only groups of more than half must share a server with every other quorum\": the wording is tangled. The reader has to reread it to see that it means 'only quorums of more than half are guaranteed to overlap'. 'Must' can be read as an obligation rather than a guarantee." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, item 1", + "text": "**Assumed:** servers crash or become unreachable, but never send a false \"yes\". Servers that lie need larger clusters.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, item 2", + "text": "**Simplified:** in real protocols (Raft, Paxos), the shared server does more than refuse...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, item 3", + "text": "**Not covered:** how servers detect a failure, how the stopped side catches up..., and how reads work.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Line 65 ('With 5 servers and 2 failed, 3 servers answer...'): it repeats the bold lead and the table. Cut it, or fold the ≥/< check into the table.", + "Line 37 'The quorum size is always the majority size.': cut it or replace it with the reason (it is the smallest size that overlaps). As written it only restates the rule from line 3." + ], + "question": { + "text": "Why can 5 servers survive 2 failures while 4 servers survive only 1? Under that sits a second question: why must the decision rule be 'more than half'?", + "known_at": "The title (line 1); the second question is stated at line 5.", + "result_stated": "Yes, at once and in words in the bold lead: 'For 5 servers, more than half is 3, so 5 − 3 = 2 servers can fail. For 4 servers, more than half is also 3, so only 4 − 3 = 1 server can fail.'" + }, + "motive": "Reason to care about the question: this is weak at the start and only appears late, at line 67 ('one server can be off for an upgrade while another one fails') and line 69 ('That is why clusters usually have an odd number of servers'). Reason for the approach: line 5 ('explains why the rule must be \"more than half\"'), backed by the partition and split-brain example at lines 24-33, which shows that a server cannot tell a crash from a cut network.", + "close": "Yes. Line 73 ('Both clusters need 3 accepts to decide a change, because 3 is more than half of 4 and of 5, and only groups of more than half must share a server with every other quorum. 5 servers have 2 to spare. 4 servers have only 1.') answers the title question and states the overlap argument in words, though the overlap clause is awkwardly phrased.", + "recap": "A cluster can only safely decide with more than half of all its servers, because only then do any two deciding groups share a server that refuses to agree twice. 4 and 5 servers both need 3, so 5 has two to spare and 4 has one.", + "strongest": "Lines 24-33: a server cannot tell a crashed peer from a cut network, and the partition table shows two halves of 4 both deciding. That makes split brain concrete.", + "weakest": "Line 37 ('The quorum size is always the majority size') together with the Answer clause at line 73 ('only groups of more than half must share a server'). The text never says why the quorum is the smallest majority rather than any larger size, and the final wording of the overlap argument is tangled.", + "suggestions": [ + "At line 37, add one sentence: a larger quorum would also be safe, but it would lower tolerance, so the smallest majority is chosen.", + "Rephrase line 73 as 'groups of more than half always share a server, so two of them can never decide different values'.", + "Put a one-line motive near the start, e.g. that choosing the cluster size decides how many simultaneous failures or maintenance outages you can survive.", + "At line 16, briefly say why a server accepts only one change per position, or point ahead to line 41." + ] + }, + "usage": { + "cost": 0.1264728, + "seconds": 41 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Line 65 ('With 5 servers and 2 failed, 3 servers answer...'): it repeats the bold lead and the table. Cut it, or fold the ≥/< check into the table." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Line 37 'The quorum size is always the majority size.': cut it or replace it with the reason (it is the smallest size that overlaps). As written it only restates the rule from line 3." + }, + { + "n": 3, + "severity": "edge", + "kind": "leap", + "text": "The rule, line 16: \"For each position in that order, a server accepts only one change\": this is given as a rule with no reason. The reason only becomes clear at line 41, where this rule is what makes C refuse. A small stumble, not a block." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "line 33: \"So the quorum size must be more than half of n\": the example only shows that 2 out of 4 (exactly half) fails. It does not say here why 'more than half' is the fix, or why that fix is enough. That comes in the next section, so this works as a forward pointer, but the 'So' moves faster than the reasoning." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Why more than half works, line 37: floor(n/2)" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Why more than half works, line 37: \"The quorum size is always the majority size\": the text never says why the quorum isn't set larger than a majority. A larger quorum would also overlap. The real reason, that the smallest safe quorum gives the most tolerance, is never stated." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Answer (line 73): \"only groups of more than half must share a server with every other quorum\": the wording is tangled. The reader has to reread it to see that it means 'only quorums of more than half are guaranteed to overlap'. 'Must' can be read as an obligation rather than a guarantee." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/read.json new file mode 100644 index 0000000..cb4711b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "A cluster commits a write when more than half of all its servers accept it.", + "quote": "A cluster decides a change only when more than half of all its servers accept the change." + }, + "2": { + "answer": "Because with a smaller quorum, two groups in a network partition could both achieve quorum and decide different changes (split brain). A majority quorum ensures any two groups must overlap at some server.", + "quote": "It happens because two groups of 2 among 4 servers need not share a server. So the quorum size must be more than half of n: then two groups cannot both decide." + }, + "3": { + "answer": "A 6-server cluster survives 2 failures.", + "quote": "tolerance = n − majority size" + }, + "4": { + "answer": "The group of 3 can still commit writes because 3 equals the majority requirement. The group of 2 cannot because 2 is less than the required 3.", + "quote": "A, B, C have 3 accepts, and 3 ≥ 3, so they keep deciding. D and E have 2, and 2 < 3, so they stop." + } + }, + "stops": [ + "More than half of all n servers is the *majority size*: floor(n/2) + 1.", + "Two quorums each hold more than half of the servers, so together they list more than n servers. Some server must therefore be in both.", + "This depends on one assumption: a server remembers what it accepted, even after a restart.", + "Count more than half of *all* configured servers, failed or not.", + "This split of the network into groups that cannot reach each other is a *partition*.", + "For each position in that order, a server accepts only one change." + ], + "excess": [ + "Going from an even n to the next odd n keeps the majority size (4 → 5: both 3), so tolerance grows by 1.", + "That is why clusters usually have an odd number of servers.", + "Servers that lie need larger clusters.", + "A tolerance of 1 means a second failure stops the cluster.", + "A tolerance of 2 means one server can be off for an upgrade while another one fails.", + "The rest of this page explains why the rule must be \"more than half\".", + "The examples below use change 7, an arbitrary position.", + "The extra server only makes each change wait for one more accept." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/run.json new file mode 100644 index 0000000..77eadae --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/quorum/explanation.md` (prose, plus the `model.md`, `model.yaml` and `narrative.md` it is built from), because the answer is a short chain of counting that one table can carry, so a diagram or interactive page would add little. The blind test scored 13/13, the last cold read had no blocking findings, and `explainer check` passes; after the blind test I made three small wording edits and cut one repeated sentence, and did not rerun the blind test.", + "error": null, + "cost": 1.4309656, + "seconds": 589, + "turns": 59, + "files": [ + "explainers/quorum/explanation.md" + ], + "review_cost": 0.9567, + "reviews": 7 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/explanation.md new file mode 100644 index 0000000..2697985 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/explanation.md @@ -0,0 +1,48 @@ +# Why do 5 servers survive 2 failures, but 4 servers survive only 1? + +**Both clusters need 3 servers to agree before they make a decision. 5 − 3 = 2 servers can fail. 4 − 3 = 1 server can fail.** The number 3 comes from one rule: the agreeing group must be a majority. Without that rule, two halves of a cluster can decide different things at the same time. + +## The setup + +A **cluster** is a group of servers that keep copies of the same data. Each change, such as "x = 7", is a **decision**. All servers must end up with the same decisions in the same order. + +A **quorum** is the minimum number of servers that must agree before the cluster makes a decision. A server that does not reply is a **failure**. + + +The cluster continues while at least a quorum of servers can reply. So the number of failures it tolerates is *servers − quorum*: + +- 5 servers, quorum 3. Two servers fail. Three reply. 3 ≥ 3, so the cluster continues. +- 4 servers, quorum 3. Two servers fail. Two reply. 2 < 3, so the cluster stops. + +The question is now: why is the quorum 3 for 4 servers, and not 2? + +## Why the quorum must be a majority + + +A server cannot tell a crashed server from a server that it cannot reach. Both look the same: no reply. A **partition** is a network fault that splits the servers into groups that cannot reach each other. + + +Try quorum 2 with 4 servers. A partition splits them into group A (2 servers) and group B (2 servers). Each group sees two silent servers. Each group has 2 servers, which is a quorum. Group A decides x = 7. Group B decides x = 9. When the network heals, the cluster holds two conflicting values and has no correct way to choose. This is **split brain**. + +The fix is to make two quorums impossible at the same time. Two groups of size *q* fit into *n* servers without a shared server only if 2*q* ≤ *n*. So choose 2*q* > *n*: the smallest such *q* is a majority, ⌊*n*/2⌋ + 1. Any two majorities then share at least one server, and that server refuses to agree to two conflicting decisions. + +- 5 servers: 3 + 3 = 6 > 5. A split of 3 + 2 lets only the side with 3 decide. +- 4 servers: 2 + 2 = 4, not > 4. The quorum must rise to 3. A 2 + 2 split now stops both sides. That is safe, but it is the "2 failures stop the cluster" case from above. + +## The pattern + +| Servers | Quorum (majority) | Failures tolerated | +|---|---|---| +| 3 | 2 | 1 | +| 4 | 3 | 1 | +| 5 | 3 | 2 | +| 6 | 4 | 2 | +| 7 | 4 | 3 | + + +Adding a server to an odd cluster raises the quorum by one, so the tolerance stays the same. Going from an even to an odd count adds one tolerated failure. To tolerate *f* failures, use 2*f* + 1 servers. The fourth server is not useless: it holds another copy and can serve reads. But it adds no failure tolerance, and it is one more server that can fail. This is why clusters (etcd, ZooKeeper, Consul) normally run 3, 5, or 7 servers. + +## Limits + +- **Assumption:** a failed server stops or goes silent. If servers can send false messages (Byzantine faults), the rule becomes 3*f* + 1 servers. +- This text does not cover how protocols such as Raft and Paxos use quorums to elect a leader and copy the log. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/model.md new file mode 100644 index 0000000..eed640d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/model.md @@ -0,0 +1,97 @@ +# Semantic model: majority quorums and failure tolerance + +## Central question + +Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1? + +## Audience and prior knowledge + +A technical reader: school mathematics, basic programming. Knows what a server and a network are. Does not know consensus, quorum, network partition, or split brain. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| cluster | n servers that keep copies of the same data and must agree on each change. | fact | +| decision | a change that the cluster accepts, such as "x = 7". | fact | +| quorum | the minimum number of servers that must agree before the cluster makes a decision; here a majority, floor(n/2) + 1. | fact (design rule) | +| failure | a server that does not reply: crashed, or cut off by the network. | fact | +| partition | a network fault that splits the servers into groups that cannot reach each other. | fact | +| split brain | two groups each make a decision, and the decisions conflict. | fact | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| quorum | prevents | split brain | two majorities always share a server | +| failure | reduces | servers that can reply | cluster stops when replies < quorum | +| n | sets | quorum | q = floor(n/2) + 1 | + +## Dependencies (learning order) + +1. A silent server can be dead or only cut off; the others cannot tell which. +2. So a group must be sure that no other group can decide at the same time. +3. Majority guarantees this; it sets the quorum. +4. Failures tolerated = n − quorum. + +## Causal chain + +```text +crash looks like partition → each group must assume the others are alive +→ only one group may decide → quorum must be a majority (two majorities overlap) +→ tolerated failures = n − majority → 5 gives 2, 4 gives 1 +``` + +## Quantities + +- quorum q = floor(n/2) + 1. n=3 → 2, n=4 → 3, n=5 → 3, n=6 → 4, n=7 → 4. +- failures tolerated f = n − q. n=3 → 1, n=4 → 1, n=5 → 2, n=6 → 2, n=7 → 3. +- servers needed to tolerate f failures: n = 2f + 1. +- Overlap test: two groups of size q both fit without sharing a server only if 2q ≤ n. Majority makes 2q > n. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| 4 servers, quorum 2 (half) | partition 2 + 2: both sides decide | 2 + 2 = 4 is not > 4; no shared server | +| 4 servers, quorum 3 | partition 2 + 2: neither side decides; cluster stops | safe, but 2 failures stop it | +| 5 servers, quorum 3 | partition 3 + 2: only the 3 side decides | 3 + 3 = 6 > 5 | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| quorum = majority | quorum = half (2 of 4) | split 2 + 2: side A writes x = 7, side B writes x = 9; both have a quorum; split brain | +| quorum = majority | "count the servers still alive" | a crash and a partition look the same; each side of a 2 + 2 split sees 2 alive and decides | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| tolerance = n − majority | 5 servers, 2 fail: 3 reply ≥ 3, cluster continues | 4 servers, 2 fail: 2 reply < 3, cluster stops | +| even n adds no tolerance | 3 → 1, 4 → 1 | — | + +## Terms + +- cluster, decision, quorum, failure, partition, split brain: as in Entities. + +## Epistemic status + +- **Mathematical consequence:** overlap of majorities; f = n − q. +- **Assumed:** servers fail by stopping or going silent, not by lying. + +## Confusion points + +- "With 4 servers, 3 are left after 1 failure, so 2 more could go" → quorum is 3, not "any survivors". +- "The 4th server is useless" → it adds read capacity and a spare copy; it adds no failure tolerance, and it adds one more server that can fail. + +## Scope + +- **Out of scope:** servers that lie (Byzantine faults) need n = 3f + 1 — see PBFT. +- **Out of scope:** how Raft and Paxos use quorums for leader election and log replication. + +## Representation decision + +- **Stage:** 1 +- **Reason:** the argument is a short chain plus one comparison table; no topology the reader must hold. +- **Levels:** L1 the rule / L2 failures vs quorum / L3 why majority / L4 n = 2f + 1. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/model.yaml new file mode 100644 index 0000000..54a482a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/explainers/quorum/model.yaml @@ -0,0 +1,51 @@ +values: + n5: + servers: 5 + quorum: 3 + tolerated: 2 + n4: + servers: 4 + quorum: 3 + tolerated: 1 + n3: {servers: 3, quorum: 2, tolerated: 1} + n6: {servers: 6, quorum: 4, tolerated: 2} + n7: {servers: 7, quorum: 4, tolerated: 3} + bad_quorum_4: 2 + conflict: {a: 7, b: 9} + +allow: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + +require: + - path: n5/tolerated + in: [prose] + - path: n4/quorum + in: [prose] + +claims: + - id: tolerance + kind: mechanism + statement: "Failures tolerated = servers − quorum; the cluster stops when fewer than a quorum reply." + example: "5 servers, quorum 3: 2 fail, 3 reply, cluster continues. 4 servers, quorum 3: 2 fail, 2 reply, cluster stops." + - id: why-majority + kind: why + statement: "The quorum must be a majority so that two groups cannot both decide." + alternative: "quorum = half of the servers (2 of 4)" + counterexample: "A partition splits 4 servers 2 + 2; both sides reach quorum 2; one writes x = 7, the other x = 9." + ask: "With 4 servers, why can the quorum not be 2?" + - id: silent + kind: definition + statement: "A crashed server and a cut-off server look the same: no reply." + - id: even-n + kind: guarantee + statement: "Adding a server to an odd cluster raises the quorum but not the tolerance." + example: "3 servers tolerate 1; 4 servers tolerate 1; 5 tolerate 2." + counterexample: "Going from 4 to 5 (even to odd) adds tolerance: 1 to 2." + +accept_unjustified: {log: "'log' is the Raft replicated log named in Scope, not a logarithm"} + +terms: + - term: quorum + means: "the minimum number of servers that must agree before a decision" + avoid: [majority vote count] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/read.json new file mode 100644 index 0000000..dea3fd8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/read.json @@ -0,0 +1,40 @@ +{ + "answers": { + "1": { + "answer": "A quorum (minimum number of agreeing servers, which must be a majority) must reach consensus before a write is committed as a decision.", + "quote": "A quorum is the minimum number of servers that must agree before the cluster makes a decision." + }, + "2": { + "answer": "A majority ensures only one quorum can exist after a partition, preventing split brain where two separate groups each commit conflicting decisions.", + "quote": "Without that rule, two halves of a cluster can decide different things at the same time." + }, + "3": { + "answer": "A 6-server cluster survives 2 failures, since it requires quorum of 4, leaving 6 − 4 = 2 servers that can fail.", + "quote": "| 6 | 4 | 2 |" + }, + "4": { + "answer": "The group of 3 can commit writes because it has the majority (quorum) for 5 servers. The group of 2 lacks majority and cannot.", + "quote": "A split of 3 + 2 lets only the side with 3 decide." + } + }, + "stops": [ + "⌊*n*/2⌋ + 1", + "Any two majorities then share at least one server, and that server refuses to agree to two conflicting decisions.", + "Byzantine faults", + "Two groups of size *q* fit into *n* servers without a shared server only if 2*q* ≤ *n*.", + "Raft and Paxos", + "A server cannot tell a crashed server from a server that it cannot reach." + ], + "excess": [ + "The fourth server is not useless: it holds another copy and can serve reads.", + "To tolerate *f* failures, use 2*f* + 1 servers.", + "This is why clusters (etcd, ZooKeeper, Consul) normally run 3, 5, or 7 servers.", + "This text does not cover how protocols such as Raft and Paxos use quorums to elect a leader and copy the log." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/run.json new file mode 100644 index 0000000..a29b461 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/quorum/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/quorum/explanation.md`, a short prose explanation with one table. I chose prose because the answer is a short chain of reasoning: a silent server can't be told apart from a cut-off one, so the group that decides must be a majority, and the number of failures a cluster survives is servers minus that majority. Worked numbers and the table cover it without needing a diagram or an interactive page. The supporting model files, `explainers/quorum/model.md` and `model.yaml`, are in the same folder, and `explainer check quorum` passes.", + "error": null, + "cost": 0.41601699999999997, + "seconds": 71, + "turns": 21, + "files": [ + "explainers/quorum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/explainers/two-tides/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/explainers/two-tides/explanation.md new file mode 100644 index 0000000..cf7dbc9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/explainers/two-tides/explanation.md @@ -0,0 +1,143 @@ +# Why are there two high tides a day, not one? + +If the Moon's gravity pulls the ocean toward it, you would expect one heap of water on the side facing the Moon. As the Earth turns, each place would pass under that heap once a day, so you would get one high tide a day. + +In fact most coasts get two high tides a day, about 12 hours 25 minutes apart. So there must be a second heap of water on the side of the Earth facing *away* from the Moon. This explainer shows where that second heap comes from. + +The short answer is this: **tides are not caused by the Moon's pull. They are caused by the *difference* in the Moon's pull between one part of the Earth and another.** That difference points toward the Moon on the near side and away from the Moon on the far side. So the ocean is stretched in both directions. + +There is an interactive companion, [`index.html`](index.html), where you can switch between "the pull" and "the difference in pull" and watch the two bulges form. It's worth opening once you reach section 2. + +--- + +## 1. The whole Earth is falling toward the Moon + +The Moon's gravity acts on all of the Earth, not just the oceans. It pulls on the rock, the core, you, and the water. As a result the Earth as a whole accelerates toward the Moon. (That acceleration is what keeps the Earth and Moon orbiting their common centre of mass. If neither was moving sideways, they would simply fall together.) + +The Earth is in free fall toward the Moon, in the same way an astronaut in orbit is in free fall toward the Earth. An astronaut in the space station doesn't feel the Earth's gravity as a force, because they and the station are falling together. In the same way, the Earth's surface does not "feel" the Moon's average pull. Everything on and in the Earth gets that average pull, so it moves along with the planet and nothing shifts relative to anything else. + +What *can* be felt is any part of the Moon's pull that is **different from the average**. The average here means the pull at the Earth's centre. + +## 2. Gravity weakens with distance, so the pull is uneven + +Gravity falls off as 1/r². The Earth is about 12,700 km across, and the Moon is about 384,000 km away. So different parts of the Earth are at noticeably different distances from the Moon: + +| Location | Distance to Moon | Moon's pull compared with the centre | +|---|---|---| +| Near side (Moon overhead) | d − R | a bit **stronger** | +| Earth's centre | d | the reference | +| Far side (Moon underfoot) | d + R | a bit **weaker** | + +Here *d* is the Earth–Moon distance and *R* is the Earth's radius. + +Now subtract the pull at the centre from the pull at each point. What is left over is called the **tidal acceleration**: + +- **Near side:** the pull is stronger than at the centre, so the leftover points *toward* the Moon. Water there is pulled toward the Moon a bit more than the Earth is, and it rises. +- **Far side:** the pull is weaker than at the centre, so the leftover points *away* from the Moon. The Earth is pulled toward the Moon a bit more than the water on the far side is. The water is left behind, and it rises too. +- **The two sides at 90°:** these points are about as far from the Moon as the centre is. But the pull there is aimed at the Moon, which is not quite parallel to the pull at the centre. The pulls at those points tilt slightly inward toward the Earth–Moon line, so the leftover points *inward*. Water there is squeezed down. + +The result is a stretch along the Earth–Moon line and a squeeze across it. The ocean takes on a slightly rugby-ball shape, with **two bulges: one under the Moon and one opposite it.** + +``` + squeeze + ↓ + ← ( · · · · · ) → ● Moon + bulge ↑ bulge + squeeze + (leftover pull points AWAY (leftover pull points TOWARD + from the Moon on the far side) the Moon on the near side) +``` + +The far-side bulge is often explained as a "centrifugal effect" of the Earth–Moon orbit. You can get the right answer that way if you are careful about reference frames, but you don't need it. The two bulges would still appear if the Earth and Moon fell straight toward each other with no orbit at all. Uneven gravity combined with free fall is enough. + +## 3. A little algebra + +Write *GM* for the Moon's mass times the gravitational constant. + +At the near-side point, the leftover pull is: + +``` +GM/(d−R)² − GM/d² +``` + +Factor out GM/d², and use the approximation (1 − x)⁻² ≈ 1 + 2x for small x (here x = R/d ≈ 1/60): + +``` +GM/d² · [ (1 − R/d)⁻² − 1 ] ≈ GM/d² · 2R/d = 2GMR/d³ (toward the Moon) +``` + +On the far side the same steps give about 2GMR/d³ *away* from the Moon. That is the same size in the opposite direction, so the two bulges are nearly equal. + +On the sides, the pull tilts inward by an angle of about R/d. That gives an inward leftover of about GMR/d³, half as large. + +Two things stand out: + +1. **The tidal effect goes as 1/d³, not 1/d².** It is the *gradient* of gravity, and taking a gradient adds another power of distance. This will matter for the Sun in section 6. +2. **It is tiny.** With the real numbers (GM_Moon ≈ 4.9×10¹² m³/s², R ≈ 6.37×10⁶ m, d ≈ 3.84×10⁸ m), GMR/d³ ≈ 5.5×10⁻⁷ m/s². That is about one ten-millionth of the Earth's surface gravity. + +If you'd like to check the vector picture yourself, this short Python script computes the leftover pull at points around the Earth: + +```python +import math + +GM = 4.9e12 # Moon, m^3/s^2 +R = 6.37e6 # Earth radius, m +d = 3.84e8 # Earth–Moon distance, m + +def pull(x, y): + """Moon's gravitational acceleration at (x, y); Moon sits at (d, 0).""" + dx, dy = d - x, -y + r = math.hypot(dx, dy) + return GM * dx / r**3, GM * dy / r**3 + +cx, cy = pull(0, 0) # pull on Earth's centre +for deg in range(0, 360, 45): + a = math.radians(deg) + px, py = pull(R * math.cos(a), R * math.sin(a)) + tx, ty = px - cx, py - cy # tidal = local pull − centre pull + radial = tx * math.cos(a) + ty * math.sin(a) # + = outward (up) + print(f"{deg:3d}° tidal {radial:+.2e} m/s² ({'up' if radial > 0 else 'down'})") +``` + +The output shows "up" at 0° (facing the Moon) and at 180° (facing away), and "down" at 90° and 270°. That is two bulges. + +## 4. How can such a tiny force move an ocean? + +At the near and far points, the tidal acceleration points straight up, against gravity that is ten million times stronger. Up there it does almost nothing. + +What actually moves the water is the **sideways (horizontal) part** of the tidal acceleration. This part is largest about 45° away from the Moon's direction. Nothing opposes it directly. Gravity is vertical, so it cannot hold water in place sideways. Across thousands of kilometres of ocean, this small, steady sideways push herds water toward the two points on the Earth–Moon line. The water piles up there until the slope of the sea surface balances the push. + +If the Earth were covered in a deep ocean that could keep up with the Moon instantly, the result is called the **equilibrium tide**. At its high points the water would be about 0.36 m above average. Halfway between them it would be about 0.18 m below average. The range from high to low would be roughly half a metre. + +## 5. The Earth turns under the two bulges + +The bulges stay lined up with the Moon while the Earth rotates beneath them. A spot on the coast therefore passes: + +> under the near bulge (high) → the side (low) → under the far bulge (high) → the other side (low) → back to the start + +That is **two highs and two lows for each turn of the Earth relative to the Moon.** + +One turn relative to the Moon takes slightly more than 24 hours. While the Earth spins, the Moon moves about 13° along its orbit in the same direction, so the Earth needs about 50 extra minutes to catch up. This "lunar day" lasts about **24 h 50 min**. That is why high tides come about **12 h 25 min** apart, and why high tide is about 50 minutes later each day. + +## 6. The Sun does the same thing, more weakly + +The Sun pulls on the Earth about 180 times harder than the Moon does. But the tidal effect depends on mass/distance³, not mass/distance². The Sun is about 390 times farther away than the Moon, and that wipes out most of its advantage. **The Sun's tidal effect is about 46% of the Moon's.** + +The Sun makes its own pair of bulges, which line up with the Sun. When the Sun, Earth and Moon are in a line (at new and full moon), the two sets of bulges add together and you get larger **spring tides**. When the Sun and Moon are at right angles (at the quarter moons), the Sun's bulges partly fill in the Moon's lows, and you get smaller **neap tides**. This happens twice a month. + +## 7. What real coasts add + +The two-bulge picture explains *why* the basic rhythm is twice a day. Real tides are shaped by much more: + +- **Continents get in the way.** Water can't flow freely around the planet. Instead, each ocean basin sloshes like water in a bathtub, driven twice a day by the tidal push. The size of the tide depends on whether a basin's natural sloshing period is close to 12.4 hours. That is why the Bay of Fundy has ranges over 15 m while the Mediterranean has only a few tens of centimetres. +- **High tide is usually not when the Moon is overhead.** Water takes time to respond, so high tide can lag the Moon by hours. In some basins the tide circulates around a point where the height hardly changes at all. +- **Some places get one tide a day.** The Moon is usually north or south of the equator, which tilts the bulges. A location off the equator then passes through one bulge more squarely than the other, so its two daily highs are unequal. In some basins, such as parts of the Gulf of Mexico, the basin's response strengthens that once-a-day part until it dominates. + +These are complications layered on top of the basic mechanism. The basic mechanism is still a stretch in two opposite directions, caused by gravity that varies across the Earth. + +## Summary + +- The Earth as a whole falls toward the Moon, so the Moon's *average* pull has no effect on the oceans. +- What remains is the *difference* between the pull at each point and the pull at the centre. It points toward the Moon on the near side and away from it on the far side. +- So the ocean bulges on both sides. Each place rotates through both bulges, which gives two high tides per 24 h 50 min. +- The effect goes as 1/d³, which is why the nearby Moon beats the much more massive Sun, about 2 to 1. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/explainers/two-tides/index.html b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/explainers/two-tides/index.html new file mode 100644 index 0000000..7a2eb3d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/explainers/two-tides/index.html @@ -0,0 +1,245 @@ + + + + +Two tides a day — interactive + + + +

Why two high tides? An interactive view

+

Companion to explanation.md. The Earth is shown from above the North Pole. The Moon is kept fixed on the right, so the Earth appears to turn once every 24 h 50 min, which is one lunar day.

+ +
+
+ + +
+
+ +
+
+ 1. Which arrows to show + + + +
+ +
+ 2. Moon distance (arrows only) + +
+ +
+ 3. Time + + +
+ +
+ 4. Add the Sun + + +
+ +
+ Things to try +
    +
  1. Choose the first arrow option and set the distance to 60. The arrows all look almost identical, because the whole Earth is pulled almost uniformly. Now drag the distance down to 3. The near-side arrows get longer and the far-side arrows get shorter.
    2. +
  2. Switch to "minus pull on the centre". Arrows point out toward the Moon, out away from the Moon, and in at the sides. That pattern is what makes the two bulges. It looks the same at every distance, and only its strength changes.
    3. +
  3. Press Play and watch the white dot ("you"). It passes under a bulge twice each lunar day, and the graph shows two peaks.
    4. +
  4. Turn on the Sun. Set days to 0 (new moon) or about 14.8 (full moon) for spring tides, or about 7.4 or 22 (quarter moons) for neap tides.
    5. +
+

The water bulge is exaggerated about 100,000 times. The heights are for an idealised ocean covering the whole planet (the "equilibrium tide"), not for any real coast. Arrows show the Moon's effect only.

+
+
+
+ + + + diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/read.json new file mode 100644 index 0000000..5d022b0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "The far side bulge is caused by the difference in the Moon's pull: the Earth's center is pulled toward the Moon more strongly than the water on the far side is, so the water is left behind and rises.", + "quote": "The Earth is pulled toward the Moon a bit more than the water on the far side is. The water is left behind, and it rises too." + }, + "2": { + "answer": "High tides come every 12 hours 25 minutes because the lunar day (one rotation of Earth relative to the Moon) is about 24 hours 50 minutes, not 24 hours. While Earth rotates, the Moon moves about 13° in its orbit, requiring an extra 50 minutes for Earth to catch up.", + "quote": "That is why high tides come about 12 h 25 min apart, and why high tide is about 50 minutes later each day." + }, + "3": { + "answer": "Tides are largest at new and full moon, when the Sun, Earth, and Moon are aligned. At these times the Sun's bulges add to the Moon's bulges, creating spring tides.", + "quote": "When the Sun, Earth and Moon are in a line (at new and full moon), the two sets of bulges add together and you get larger spring tides." + }, + "4": { + "answer": "High tides would come every 12 hours. Since there are two high tides per rotation of Earth relative to the bulges, and a fixed Moon would require just one 24-hour rotation (not the current 24-hour-50-minute lunar day), high tides would occur every 12 hours.", + "quote": "That is two highs and two lows for each turn of the Earth relative to the Moon. One turn relative to the Moon takes slightly more than 24 hours. While the Earth spins, the Moon moves about 13° along its orbit in the same direction, so the Earth needs about 50 extra minutes to catch up." + } + }, + "stops": [ + "gradient: \"It is the gradient of gravity, and taking a gradient adds another power of distance.\" (undefined term)", + "free fall concept: \"The Earth is in free fall toward the Moon, in the same way an astronaut in orbit is in free fall toward the Earth.\" (needs careful understanding)", + "tidal acceleration: \"What is left over is called the tidal acceleration\" (brief definition of complex concept)", + "Why being left behind causes rising: \"The water is left behind, and it rises too.\" (mechanism unclear)", + "equilibrium tide: \"If the Earth were covered in a deep ocean that could keep up with the Moon instantly, the result is called the equilibrium tide.\" (specialized term, unclear significance)", + "sloshing period: \"The size of the tide depends on whether a basin's natural sloshing period is close to 12.4 hours.\" (undefined)", + "How sideways force moves water: \"Nothing opposes it directly. Gravity is vertical, so it cannot hold water in place sideways.\" (logic needs work)", + "Why water takes time to respond: \"Water takes time to respond, so high tide can lag the Moon by hours.\" (no mechanism given)", + "Power law significance: \"The tidal effect goes as 1/d³, not 1/d².\" (why this matters for Sun vs Moon unexplained)", + "\"that difference\" referent: \"They are caused by the difference in the Moon's pull between one part of the Earth and another. That difference points toward the Moon\" (needs clearer antecedent)" + ], + "excess": [ + "Python script: \"If you'd like to check the vector picture yourself, this short Python script computes the leftover pull\"", + "Centrifugal effect: \"The far-side bulge is often explained as a 'centrifugal effect'... but you don't need it.\"", + "Specific basin examples: \"That is why the Bay of Fundy has ranges over 15 m while the Mediterranean has only a few tens of centimetres.\"", + "Detailed numbers: \"With the real numbers (GM_Moon ≈ 4.9×10¹² m³/s²...)\"", + "Magnitude comparison: \"That is about one ten-millionth of the Earth's surface gravity.\"", + "Specific equilibrium heights: \"At its high points the water would be about 0.36 m above average.\"", + "Sun's bulge obvious consequence: \"The Sun makes its own pair of bulges, which line up with the Sun.\"", + "Tide circulation detail: \"In some basins the tide circulates around a point where the height hardly changes at all.\"", + "Once-a-day tide specifics: \"In some basins, such as parts of the Gulf of Mexico, the basin's response strengthens that once-a-day part\"", + "Calculation intermediate result: \"That gives an inward leftover of about GMR/d³, half as large.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/run.json new file mode 100644 index 0000000..cfe593f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/none/run.json @@ -0,0 +1,13 @@ +{ + "result": "I made `explainers/two-tides/explanation.md` and `explainers/two-tides/index.html`: the prose walks through the argument, with a little algebra and a runnable Python check, and the interactive page lets the reader flip from \"the pull\" to \"the pull minus the pull on the centre\" and watch the two bulges form, which a still image shows poorly (not tested in a browser).", + "error": null, + "cost": 0.4116578, + "seconds": 131, + "turns": 3, + "files": [ + "explainers/two-tides/explanation.md", + "explainers/two-tides/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/explainers/two-tides/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/explainers/two-tides/explanation.md new file mode 100644 index 0000000..e13b407 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/explainers/two-tides/explanation.md @@ -0,0 +1,77 @@ +# Why are there two high tides a day, not one? + +**Short answer.** The Moon does not just pull the ocean toward itself. It pulls the near side of the Earth harder than the centre, and the centre harder than the far side. That *difference* in pull stretches the Earth's water along the Earth–Moon line. The result is two bulges: one facing the Moon and one facing away. The Earth turns under both bulges each day, so most coastlines get two high tides. + +## The obvious model fails + +The simplest idea is "the Moon's gravity pulls the water toward the Moon." That gives one bulge, on the side that faces the Moon. A place on the Earth would pass under it once per rotation: one high tide a day. + +Most coasts have two high tides a day, about 12 h 25 min apart. So the simple model is missing something. The missing part is that the Moon pulls on the *whole Earth*, not only on the water. + +## Step 1: the whole Earth falls toward the Moon + +The Earth and the Moon orbit their common centre of mass. In practice, the Earth is in free fall toward the Moon. The Moon's gravity accelerates the Earth's centre toward the Moon all the time. + +We live on the Earth and move with it. So what we see is not the Moon's pull itself. We see how the pull at each point **differs from** the pull at the centre. This difference is the **tidal acceleration**. + +> Before you read on: the far side is pulled toward the Moon less than the centre. Relative to the centre, which way does far-side water move? + +## Step 2: compute the pull at three points + +Gravity weakens with distance: acceleration = GM / d². Use the Moon's values: + +- GM of the Moon = 4.90 × 10¹² m³/s² +- Earth–Moon distance d = 384,400 km +- Earth radius R = 6,371 km + +| Point | Distance to Moon | Moon's pull (m/s²) | Minus pull at centre | +|---|---|---|---| +| Near side | d − R = 378,029 km | 3.429 × 10⁻⁵ | **+1.13 × 10⁻⁶** (toward Moon) | +| Centre | d = 384,400 km | 3.316 × 10⁻⁵ | 0 | +| Far side | d + R = 390,771 km | 3.209 × 10⁻⁵ | **−1.07 × 10⁻⁶** (away from Moon) | + +```text +Moon's pull on each point (longer arrow = stronger pull): + + far side centre near side Moon + •──> •───> •────> ● + +Subtract the pull at the centre (the whole Earth already has this motion): + + <• • •> + 1.07e-6 away zero 1.13e-6 toward +``` + +The near-side water moves toward the Moon faster than the Earth does, so it bulges toward the Moon. The far-side water moves toward the Moon *slower* than the Earth does. The Earth falls away from it, so it bulges away from the Moon. The two values are almost equal. Both are close to 2GMR/d³ = 1.10 × 10⁻⁶ m/s². + +At the two sides (90° from the Moon), the Moon's pull points slightly inward, toward the Earth–Moon line. This squeezes the water there by about GMR/d³ = 0.55 × 10⁻⁶ m/s². Water flows from the squeezed sides toward the stretched ends: + +```text + squeeze + ↓ + bulge ← ( Earth ) → bulge Moon ● + ↑ + squeeze +``` + +**Why the far side is not pushed by "centrifugal force from the Earth's spin":** the spin is symmetric about the Earth's axis. It does not know where the Moon is, so it cannot make a bulge that points away from the Moon. The far bulge comes only from the difference in the Moon's pull. + +## Step 3: what the number means + +1.1 × 10⁻⁶ m/s² is about one ten-millionth of the Earth's gravity (9.8 m/s²). It cannot lift water vertically in any useful way. It works because part of it points *along* the surface, and nothing opposes horizontal flow except the slope of the water itself. The ocean flows until the slope of its surface balances the horizontal tidal pull, over a distance of thousands of kilometres. + +For an ideal ocean that covers the whole Earth, the result is a difference of about half a metre between bulge and squeeze (model output, "equilibrium tide"). A larger tidal acceleration would give taller bulges. Because the effect goes as 1/d³, a Moon at half the distance would give 8 times the tidal acceleration. + +## Step 4: the Earth turns under the bulges + +The two bulges stay aligned with the Moon. The Earth turns under them. A point on the coast passes through bulge, squeeze, bulge, squeeze in one turn. That is two high tides and two low tides. + +The cycle is 24 h 50 min, not 24 h. In one day the Moon moves about 13° along its orbit, in the same direction as the Earth turns. The Earth needs about 50 more minutes to face the Moon again. So high tides come every 12 h 25 min, and they are about 50 minutes later each day. + +## The Sun does the same thing + +The Sun produces tidal bulges by the same mechanism. Its pull is much stronger than the Moon's, but tides depend on the *difference* across the Earth, which goes as M/d³. For the Sun, the result is about 46% of the Moon's effect. When the Sun and Moon line up (new and full moon), their bulges add: large **spring tides**. When they are at 90° (quarter moons), they partly cancel: small **neap tides**. + +## What this model leaves out + +This is the idealized model: an ocean that covers the Earth and responds at once. Real tides are also shaped by continents, ocean depth, and resonance of ocean basins. These change the timing and the height a lot (for example, more than 15 m in the Bay of Fundy). In some places, such as parts of the Gulf of Mexico, basin shape and the Moon's tilt relative to the equator make one of the two daily tides dominate, so the coast sees one high tide a day. The two bulges are still the cause; local geography decides how they show up. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/read.json new file mode 100644 index 0000000..dcafcd5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "The far-side water is pulled toward the Moon less strongly than Earth's center, so the Earth falls away from it, causing it to bulge away from the Moon.", + "quote": "The far-side water moves toward the Moon *slower* than the Earth does. The Earth falls away from it, so it bulges away from the Moon." + }, + "2": { + "answer": "The Moon moves along its orbit by about 13 degrees per day, requiring the Earth to rotate an extra 50 minutes to face the Moon again.", + "quote": "In one day the Moon moves about 13° along its orbit, in the same direction as the Earth turns. The Earth needs about 50 more minutes to face the Moon again." + }, + "3": { + "answer": "Tides are largest at new and full moons when the Sun and Moon line up, because their tidal bulges add together to create large spring tides.", + "quote": "When the Sun and Moon line up (new and full moon), their bulges add: large **spring tides**." + }, + "4": { + "answer": "Every 12 hours, because Earth rotates once in 24 hours passing through 2 bulges, and without the Moon's orbital motion, no extra 50 minutes would be needed.", + "quote": "The cycle is 24 h 50 min, not 24 h. In one day the Moon moves about 13° along its orbit, in the same direction as the Earth turns. The Earth needs about 50 more minutes to face the Moon again." + } + }, + "stops": [ + "how the pull at each point **differs from** the pull at the centre", + "Gravity weakens with distance: acceleration = GM / d²", + "Both are close to 2GMR/d³ = 1.10 × 10⁻⁶ m/s²", + "At the two sides (90° from the Moon), the Moon's pull points slightly inward", + "nothing opposes horizontal flow except the slope of the water itself", + "difference of about half a metre between bulge and squeeze (model output, \"equilibrium tide\")", + "The cycle is 24 h 50 min, not 24 h", + "Before you read on: the far side is pulled toward the Moon less than the centre. Relative to the centre, which way does far-side water move?", + "Subtract the pull at the centre (the whole Earth already has this motion)", + "the spin is symmetric about the Earth's axis. It does not know where the Moon is" + ], + "excess": [ + "The Earth and the Moon orbit their common centre of mass", + "Why the far side is not pushed by \"centrifugal force from the Earth's spin\"", + "Because the effect goes as 1/d³, a Moon at half the distance would give 8 times the tidal acceleration", + "This is the idealized model: an ocean that covers the Earth and responds at once", + "more than 15 m in the Bay of Fundy", + "The two values are almost equal. Both are close to 2GMR/d³", + "basin shape and the Moon's tilt relative to the equator make one of the two daily tides dominate", + "Real tides are also shaped by continents, ocean depth, and resonance of ocean basins", + "The result is two bulges: one facing the Moon and one facing away", + "When they are at 90° (quarter moons), they partly cancel: small **neap tides**" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/run.json new file mode 100644 index 0000000..d6485d4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made one file, `explainers/two-tides/explanation.md`: prose with two small inline text diagrams, because the explanation is a short chain of steps (pull at three points, minus the pull at the centre, Earth turning under two bulges) that a worked numeric table and an arrow sketch cover without needing an interactive page.", + "error": null, + "cost": 0.1563002, + "seconds": 42, + "turns": 2, + "files": [ + "explainers/two-tides/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/explanation.md new file mode 100644 index 0000000..8115d92 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/explanation.md @@ -0,0 +1,74 @@ +# Why are there two high tides a day, not one? + +Most coasts get two high tides a day. One comes when the Moon is high in the sky. The other comes when the Moon is on the opposite side of the Earth. If the Moon simply pulls water toward itself, the second high tide makes no sense. + +**The Moon does not lift the ocean toward itself. It stretches the ocean along the Earth–Moon line, and a stretch has two ends. The Earth turns through both ends, so a place gets two high tides.** + +## The pull at three points + +The Moon's pull is the acceleration that the Moon's gravity gives at a point. Take three points on the Earth–Moon line: the **near side** (the surface point that faces the Moon), Earth's **centre**, and the **far side** (the surface point opposite the Moon). The unit is µm/s², one millionth of a m/s². + +| Point | Distance to Moon | Moon's pull (toward the Moon) | +|---|---|---| +| near side | 378,029 km | 34.32 µm/s² | +| centre | 384,400 km | 33.19 µm/s² | +| far side | 390,771 km | 32.12 µm/s² | + +All three pulls point toward the Moon. So why does the far-side water not move toward the Moon? + +## Only the part that differs from the centre's pull acts + + +The Earth and the Moon orbit their shared centre of mass. An orbit is free fall: the whole Earth falls toward the Moon, and Earth's centre accelerates at 33.19 µm/s². An acceleration that every part shares does not change the shape of a falling body. + +So subtract the centre's pull. The remainder is the **tidal acceleration**: the Moon's pull at a point minus the Moon's pull at Earth's centre. Here + means toward the Moon. + +- Near side: 34.32 − 33.19 = +1.13. The near-side water falls toward the Moon faster than the centre, so it moves ahead of the centre, toward the Moon. +- Far side: 32.12 − 33.19 = −1.07. The far-side water falls toward the Moon slower than the centre, so it is left behind, away from the Moon. + + +Both results point away from Earth's centre. The Moon does not push the far side away. It pulls the far side less than it pulls the centre, and the centre's fall sets the reference. The near value is about 5% larger because the pull changes faster closer to the Moon. + +Now take a **side point**, 90° from the near side. Its pull has almost the same size as the centre's, 33.18 µm/s², so along the Earth–Moon line the two cancel. But the pull points at the Moon, from 6371 km off the line, so it tilts by R/d = 6371 / 384,400 = 0.0166 rad (0.95°) toward Earth's centre. That tilt leaves 33.19 × 0.0166 = 0.55 µm/s², pointing toward Earth's centre. So the tidal acceleration stretches the Earth along the Earth–Moon line and squeezes it across that line. + +## How a tiny acceleration moves an ocean + +1.1 µm/s² is about one ten-millionth of g (9.81 m/s²). Up or down, the tidal acceleration changes the weight of the water by 1 part in 10 million, which does nothing. But Earth's gravity has no part along a level surface, so the tidal acceleration is the only sideways push. It is largest 45° from the near side, at 0.82 µm/s², and points toward the nearer end of the line. Water flows until the sea surface slopes just enough that gravity along the slope balances this push. The water gathers into two **bulges**: one under the Moon, one opposite it. A **high tide** is the time when a place passes through a bulge. + +```text + side point: 0.55 µm/s² down, low water + | + ↙ v ↘ + . - ~~~~~~~~~~~ - . + far-side bulge <--( Earth )--> near-side bulge -----> Moon + (-1.07 µm/s²) ' - ~~~~~~~~~~~ - ' (+1.13 µm/s²) + ↖ ^ ↗ + | + side point + + ↙ ↘ ↖ ↗ : sideways push along the surface, 0.82 µm/s² at 45°, toward the nearer bulge +``` + +## Why subtract: two tests + + +The simplest idea, that the Moon's pull itself raises the tide, fails two tests. + + +First, imagine that something held the Earth still, so that it did not fall toward the Moon. Then nothing subtracts the centre's 33.19. The far side's 32.12 µm/s² points toward the Moon, which on the far side is down into the Earth. That Earth has one bulge, not two. The far bulge exists because the Earth falls. + +Second, the Sun. Its pull on the Earth is 179 times the Moon's, so the pull alone predicts solar tides 179 times larger. But a pull GM/r² changes by about 2R/d of itself across one Earth radius R, at distance d. So the tidal acceleration is about the pull times 2R/d. For the Moon: 33.19 × 2 × 6371 / 384,400 = 1.10, between 1.13 and 1.07. The Sun is 389 times farther than the Moon, so its tidal acceleration is about 179 / 389 = 0.46 times the Moon's. Observed solar tides are about half the Moon's. + +## Two high tides per turn + + +The bulges stay on the Earth–Moon line, and the Earth turns under them. The Moon also moves about 12° a day along its orbit, in the same direction that the Earth turns. The Earth turns about 15° an hour, so it needs about 50 minutes more than 24 hours to face the Moon again: a **lunar day** of 24 h 50 min. A place passes through both bulges in each lunar day, so high tides come every half lunar day: 12 h 25 min. A high tide at 8 in the morning gives the next at about 8:25 in the evening. + +## The answer + +The Moon pulls the near side more than Earth's centre, and the centre more than the far side. The Earth falls with its centre, so these differences stretch the ocean into two bulges, and the Earth turns through both each lunar day. No centrifugal force is needed. The pulls depend only on distance, so an Earth that fell straight at the Moon, with no orbit, would get the same +1.13 and −1.07 and the same two bulges. + +## Scope + +- **Height:** in an ideal ocean, the bulges rise about 0.36 m above the mean (an estimate). Look up "equilibrium tide". +- **Real oceans:** continents and basin shapes make tides lag the Moon, reach metres on some coasts, and come once a day on others. Look up "dynamic theory of tides". diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/model.md new file mode 100644 index 0000000..f6ab5a9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/model.md @@ -0,0 +1,144 @@ +# Semantic model: two high tides a day + +> Source of truth. Every rendering derives from this file. + +## Central question + +Why does a place on the coast get two high tides a day, when there is only one Moon pulling on one side of the Earth? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (Newton's law of gravity, inverse-square law, acceleration, vectors, free fall), basic programming. New to tides. Knows the Moon causes tides and that the Earth spins once a day. Does not know the idea of a tidal acceleration, or why the far side of the Earth has a bulge. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Earth | A ball of rock of radius 6371 km, covered mostly by ocean. | fact | +| Moon | The body that causes most of the tide, 384,400 km from Earth's centre on average. | fact | +| Moon's pull | The acceleration the Moon's gravity gives an object at a point; it points at the Moon and falls with the square of distance. | established fact (Newton) | +| near side | The point on Earth's surface that faces the Moon, 6371 km closer than the centre. | definition | +| far side | The point on Earth's surface opposite the Moon, 6371 km farther than the centre. | definition | +| side point | A surface point 90° from the near side; 384,453 km from the Moon. | definition | +| tidal acceleration | The Moon's pull at a point minus the Moon's pull at Earth's centre (a vector subtraction). | definition | +| bulge | Water gathered on the Earth–Moon line, because there the tidal acceleration points outward and along the surface it points toward the line. There are two: under the Moon and opposite the Moon. | model output (ideal ocean) | +| high tide | The time when a place passes through a bulge and its water level is highest. | observation | +| lunar day | 24 h 50 min: the time for the Moon to come back to the same place in the sky. | fact | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Moon's pull | accelerates | the whole Earth | Earth falls freely toward the Moon (as part of its orbit around the shared centre of mass). | +| Earth's free fall | cancels | the Moon's pull at the centre (33.19 µm/s²) | Earth's centre accelerates at 33.19 µm/s²; relative to the centre, each point feels only its own pull minus 33.19. | +| tidal acceleration | points away from Earth's centre at | near side and far side | Stretches Earth along the Earth–Moon line. | +| tidal acceleration | points toward Earth's centre at | the sides (90° from the Moon) | Squeezes in the other direction. Half the size. | +| tidal acceleration | moves water toward | the two bulges | Its sideways part along the surface moves the water. | +| Earth's rotation | carries a place through | both bulges each lunar day | Two high tides. | + +## Dependencies (learning order) + +1. The Moon's pull is weaker farther from the Moon (inverse square). +2. The whole Earth falls toward the Moon, so the pull it shares with every part does nothing to its shape. +3. Tidal acceleration = pull at a point − pull at the centre. +4. The tidal acceleration points away from the centre on both the near side and the far side → two bulges. +5. The Earth turns under the two bulges → two high tides per lunar day. + +## Causal chain + +```text +pull falls with distance → near side pulled more than centre, centre more than far side +→ Earth falls as a whole, so only the tidal acceleration acts +→ near side pulled ahead of centre; far side left behind centre +→ water stretched into two bulges on the Earth–Moon line +→ Earth turns under both bulges → two high tides each 24 h 50 min +``` + +## Temporal sequence + +1. A place passes under the Moon: high tide (near-side bulge). +2. About 6 h 12 min later the place is 90° away: low tide. +3. About 12 h 25 min after the first, the place is on the far side: second high tide (far-side bulge). +4. About 24 h 50 min after the first, the place is under the Moon again. + +## Quantities + +Constants: GM of Moon = 4.905 × 10¹² m³/s²; Earth–Moon distance d = 384,400 km; Earth radius R = 6371 km; GM of Sun = 1.327 × 10²⁰ m³/s²; Earth–Sun distance 1.496 × 10⁸ km. + +Moon's pull, a = GM/r², in µm/s² (1 µm/s² = 10⁻⁶ m/s²): + +| Point | Distance to Moon (km) | Moon's pull (µm/s²) | Pull minus centre's pull (µm/s²) | +|---|---|---|---| +| near side | 378,029 | 34.32 | +1.13 (toward Moon) | +| centre | 384,400 | 33.19 | 0 | +| far side | 390,771 | 32.12 | 1.07 away from Moon (−1.07) | +| side point | 384,453 | 33.18, tilted 0.95° toward the centre | 0.55 toward Earth's centre | + +- Side point: its pull has almost the same size as the centre's, but it points at the Moon, so it is tilted by R/d = 0.0166 rad toward Earth's centre. The vector difference is 33.19 × 0.0166 = 0.55 µm/s², pointing down. +- On the Earth–Moon line, GM/(d−R)² − GM/d² ≈ (GM/d²)(2R/d) = 2GMR/d³ = 1.10 µm/s². The exact values 1.13 and 1.07 differ from this by about 3%. The factor 1/d³ means the tidal acceleration falls with the cube of distance. +- Sideways part (along the surface): (3/2)(GMR/d³) sin 2θ, θ the angle from the near side. 0 at 0° and 90°; largest at 45°, 0.82 µm/s², pointing toward the nearer bulge. This part moves water. +- Meaning: 1.1 µm/s² is about one ten-millionth (1.1 × 10⁻⁷) of g = 9.81 m/s². Up or down, it changes weight by a negligible amount. Sideways, nothing opposes it except the slope of the sea, so over a whole ocean it gathers water into the bulges. Low value: zero at the centre. High value: near and far sides (up), 45° points (sideways). +- Sun: pull on Earth 5930 µm/s², about 179 times the Moon's pull. Sun's tidal acceleration ≈ 0.51 µm/s², about 0.46 of the Moon's, because the Sun is 389 times farther and the tidal acceleration falls as 1/d³. Check: tidal acceleration ≈ pull × 2R/d, so Sun/Moon ≈ 179 / 389 = 0.46. +- Why the vertical part does nothing and the sideways part acts: vertically the tidal acceleration adds to g = 9.81 m/s² and changes weight by 1 part in 10 million. Along a level surface gravity has no component, so the tidal acceleration is the only sideways push; water flows until the slope of the sea balances it. +- Near/far asymmetry: 1.13 vs 1.07, about 5% apart, because the pull changes faster closer to the Moon. Too small to matter here. +- Lunar day: the Earth turns about 15° an hour; the Moon moves about 12° per day along its orbit, in the same direction Earth turns, so Earth needs about 50 min more than 24 h to face the Moon again: 24 h 50 min. Two high tides per lunar day → one each 12 h 25 min. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Earth held fixed (not falling toward the Moon) | One bulge only, under the Moon. The far side pull (32.12 µm/s²) points toward the Moon, that is down into the Earth. | Nothing subtracts the centre's pull; every point is pulled Moonward. | +| Uniform pull (same at every point) | No tide at all. | Every part falls together; 33.19 − 33.19 = 0 at every point. | +| Sun instead of Moon | Smaller tide (0.46×) despite 179× stronger pull. | Tide follows the tidal acceleration, which falls as 1/d³. | +| Earth falls straight at the Moon (no orbit) | Same two bulges: +1.13 and −1.07. | The pulls at the three points depend only on distance, not on orbital motion. | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Subtract the centre's pull | Use the Moon's pull itself as the cause of the tide | Predicts one bulge (all three points pulled Moonward: 34.32, 33.19, 32.12 µm/s²), and predicts the Sun's tide 179× the Moon's. Observed: two bulges, Sun's tide about half the Moon's. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| The far side is stretched away from the Moon (assumption: Earth falls freely toward the Moon) | Centre pulled 33.19, far side 32.12 µm/s²: the far side is left behind by 1.07 µm/s². | Earth held fixed: far side pulled 32.12 µm/s² toward the Moon and into the Earth; no far bulge. | +| Two bulges give two high tides a day (assumption: Earth turns under the Moon) | Earth turns once per 24 h 50 min relative to the Moon → high tide every 12 h 25 min. | — (the rotation is an observation; no alternative needed) | +| Mechanism: subtraction | 34.32 − 33.19 = +1.13; 32.12 − 33.19 = −1.07; side 0.55 inward. | — | + +## Terms + +- **Moon's pull**: the acceleration toward the Moon at a point, from Newton's law. Example 33.19 µm/s² at Earth's centre. +- **tidal acceleration**: Moon's pull at a point minus Moon's pull at the centre. Zero at the centre; +1.13 µm/s² at the near side. +- **near side / far side**: the surface points nearest to / farthest from the Moon. +- **bulge**: raised ocean on the Earth–Moon line; two of them. +- **high tide**: the time a place passes through a bulge. +- **side point**: a surface point 90° from the near side; tidal acceleration 0.55 µm/s² toward the centre. +- **lunar day**: 24 h 50 min; the time for the Moon to come back to the same place in the sky. + +## Epistemic status + +- **Observed / established:** Most coasts have two high tides per 24 h 50 min. Newton's law of gravity. +- **Mathematical consequence:** The tidal acceleration points outward on both near and far sides, with nearly equal size (1.13 vs 1.07). +- **Assumed:** Equilibrium tide: ocean covers the Earth and responds at once to the tidal acceleration. Real oceans do not; see Scope. +- **Estimated:** Equilibrium bulge height about 0.36 m above the mean at its peak (range about 0.5 m). + +## Confusion points + +- "The Moon pushes the far side away" → No. The Moon pulls the far side, but less than it pulls the centre. The far side is left behind. +- "The far bulge comes from centrifugal force of the Earth–Moon spin" → An unnecessary and misleading story. The tidal acceleration alone gives both bulges; it would exist even if Earth fell straight at the Moon with no orbit. +- "Stronger pull means bigger tide" → No; the Sun pulls 179× harder but raises a tide 0.46× the Moon's. The tidal acceleration matters, not the pull. +- "High tides come every 12 h" → Every 12 h 25 min, because the Moon moves on in its orbit. + +## Scope + +- **Out of scope:** Why real tides are often metres high, lag the Moon, or happen once a day at some places (for example the Gulf of Mexico) — the ocean basins respond as resonating bodies; continents block the bulges; the Moon's tilt makes the two bulges unequal at a given latitude. Look up "dynamic theory of tides" and "amphidromic points". +- **Out of scope:** How high the bulges are — in an ideal ocean, about 0.36 m above the mean at the peak (an estimate); real coasts amplify this. Look up "equilibrium tide". +- **Out of scope:** Spring and neap tides — the Sun's bulges add to or partly cancel the Moon's. +- **Out of scope:** The solid Earth also stretches, by a few tens of centimetres; the ocean tide is measured relative to the land. Look up "Earth tide". + +## Representation decision + +- **Stage:** 1 (prose) +- **Reason:** The argument is a short chain: three numbers, one subtraction, one rotation. A table of the three points carries the comparison; a small text figure carries the geometry. Nothing needs exploration with a parameter. +- **Levels:** L1 the answer / L2 the subtraction at three points / L3 why subtract (free fall; Sun case) / L4 not needed. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/model.yaml new file mode 100644 index 0000000..1d3e1c3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/model.yaml @@ -0,0 +1,87 @@ +# Machine-readable half of model.md. `explainer check two-tides` holds every rendering to it. + +values: + earth_radius_km: 6371 + moon_distance_km: 384400 + near_distance_km: 378029 + far_distance_km: 390771 + pull_near: 34.32 # µm/s² + pull_centre: 33.19 + pull_far: 32.12 + pull_side: 33.18 + tidal_near: 1.13 + tidal_far: 1.07 + tidal_far_signed: -1.07 + tidal_side: 0.55 + near_far_diff_pct: 5 + earth_turn_deg_per_h: 15 + side_tilt_deg: 0.95 + bulge_height_m: 0.36 + example_high_tide_h: 8 + example_next_high_tide_min: 25 + tidal_sideways_max: 0.82 + sun_pull_ratio: 179 + sun_tide_ratio: 0.46 + sun_distance_ratio: 389 + lunar_day: "24 h 50 min" + tide_interval: "12 h 25 min" + +allow: [2, 3, 45, 90, 24, 50, 12, 9.81, 10, 7, 0.0166, 1.1, 6] + +require: + - path: pull_centre + in: [prose] + - path: tidal_near + in: [prose] + - path: tidal_far + in: [prose] + - path: tide_interval + in: [prose] + +claims: + - id: mech-subtract + kind: mechanism + statement: "The tidal acceleration at a point is the Moon's pull there minus the Moon's pull at Earth's centre, because the whole Earth falls toward the Moon with the centre's acceleration." + example: "Near side 34.32 − 33.19 = +1.13 µm/s² (toward the Moon); far side 32.12 − 33.19 = −1.07 µm/s² (away from the Moon); side point 0.55 µm/s² toward Earth's centre." + ask: "On the far side of the Earth, which way does the tidal acceleration point, and why?" + - id: why-subtract + kind: why + statement: "Subtract the centre's pull; the Moon's pull itself does not predict the tides." + alternative: "Take the Moon's pull itself as the cause of the tide." + counterexample: "All points are pulled toward the Moon (34.32, 33.19, 32.12 µm/s²), so this predicts one bulge. It also predicts a solar tide 179 times the Moon's; the observed solar tide is 0.46 times the Moon's." + ask: "The Sun pulls on the Earth about 179 times harder than the Moon does. Why are its tides smaller?" + - id: guar-far-bulge + kind: guarantee + statement: "The far side has a bulge because the Earth falls freely toward the Moon." + example: "Centre pulled 33.19, far side 32.12 µm/s²: the far side is left behind by 1.07 µm/s²." + counterexample: "If the Earth were held fixed, the far side's 32.12 µm/s² would point toward the Moon, down into the Earth: no far bulge, one bulge only." + - id: mech-rotation + kind: mechanism + statement: "The Earth turns under the two bulges, so a place passes through a bulge twice per lunar day." + example: "Lunar day 24 h 50 min, so a high tide every 12 h 25 min." + ask: "Why are high tides about 12 h 25 min apart, not 12 h?" + - id: misc-push + kind: misconception + statement: "The Moon pushes the far-side water away." + correction: "The Moon pulls the far side less than it pulls the centre, so the far-side water is left behind as the Earth falls toward the Moon." + +terms: + - term: tidal acceleration + means: "the Moon's pull at a point minus the Moon's pull at Earth's centre" + avoid: [tide-raising force, differential acceleration, tidal force] + - term: Moon's pull + means: "the acceleration the Moon's gravity gives at a point" + avoid: [average pull] + +budget: {prose: 950, reason: "the subtraction at four points, why the sideways part acts, the why with the Sun counterexample, the held-Earth break case, the rotation timing, and a text figure"} + +quiz: + - q: "Why is there a high tide on the side of the Earth facing away from the Moon?" + expect: "The Moon pulls the far side less than it pulls Earth's centre. The whole Earth falls toward the Moon with the centre's acceleration, so the far-side water is left behind: its tidal acceleration points away from the Moon." + misconception: "The Moon pushes the far side away, or centrifugal force throws water out." + - q: "If the Earth were held fixed in space and did not fall toward the Moon, how many bulges would there be?" + expect: "One, under the Moon, because every point would be pulled toward the Moon and nothing subtracts the centre's pull." + misconception: "Still two." + - q: "A place has a high tide at 08:00. When is the next high tide, roughly, and why not exactly 12 hours later?" + expect: "About 20:25, because the Moon moves along its orbit, so the Earth needs 24 h 50 min to face it again, and there are two bulges in that time." + misconception: "20:00 exactly." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/narrative.md new file mode 100644 index 0000000..46e8fd9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/narrative.md @@ -0,0 +1,52 @@ +# Narrative: two-tides + +## 1. Reader before and after + +- **Before:** A technical reader who knows Newton's inverse-square gravity, acceleration as a vector, free fall, and that the Earth turns once a day. Knows "the Moon causes tides". Has never asked why the side away from the Moon also has a high tide. +- **After:** "The Moon pulls the near side more than Earth's centre, and the centre more than the far side; because the whole Earth falls toward the Moon, only these differences act, and they stretch the ocean into two bulges that the Earth turns through each 24 h 50 min." + +## 2. Question and motive + +- **Question:** The Moon is on one side of the Earth. Why is the water high on both sides? Result in words: the Moon does not lift the water toward itself; it stretches the Earth's ocean along the Earth–Moon line, and a stretch has two ends. +- **Why care:** The obvious picture (the Moon pulls water toward it) predicts one high tide a day. Most coasts have two. The obvious picture is wrong in a way that teaches something about gravity. +- **Why this approach:** Ask what the pull is at three points, near side, centre, far side, with real numbers. Then ask which of those pulls can change the Earth's shape. + +## 3. Introduction ledger + +| Reference | Means | Grounded by | Beat | +|---|---|---|---| +| Moon's pull | acceleration from the Moon's gravity at a point | 33.19 µm/s² at Earth's centre | 2 | +| µm/s² | 10⁻⁶ m/s² | g is 9.81 m/s² = 9,810,000 µm/s² | 2 | +| near side / far side | surface points nearest / farthest from the Moon | 378,029 km / 390,771 km from the Moon | 2 | +| tidal acceleration | Moon's pull at a point minus the pull at the centre | +1.13 at the near side | 4 | +| side point | surface point 90° from the near side | 0.55 µm/s² toward the centre | 5 | +| bulge | water gathered on the Earth–Moon line | the two ends of the stretch | 6 | +| lunar day | 24 h 50 min, time for the Moon to come back to the same place in the sky | Moon moves about 12° a day | 8 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | Why two high tides? | — | question, then the answer in bold | Most coasts get two high tides a day, one when the Moon is overhead and one when it is on the other side of the Earth. Answer: the Moon stretches the ocean along the Earth–Moon line; a stretch has two ends; the Earth turns through both. | the result | +| 2 | instance | How strong is the pull at different points? | So compute it | table: near side 378,029 km, centre 384,400 km, far side 390,771 km | near side 34.32, centre 33.19, far side 32.12 µm/s², all toward the Moon | all three point toward the Moon, unequal | +| 3 | objection | Then why is the far water not pulled toward the Moon? | But | — | The Earth and Moon orbit their shared centre of mass; an orbit is free fall, so Earth's centre accelerates toward the Moon at 33.19. A shared acceleration changes nothing inside a falling body (astronauts float in an orbiting station). | only the part that differs from the centre's pull acts | +| 4 | name | What is left? | Therefore subtract | difference column; sign: + toward the Moon | tidal acceleration = pull − 33.19: near +1.13 (ahead of the centre, toward the Moon), far −1.07 (left behind, away from the Moon). Both point away from Earth's centre. | both ends pulled outward relative to the centre | +| 5 | mechanism | What about the sides? | And | side row | A side point is 90° from the near side. Its pull points at the Moon, so it tilts 0.95° toward Earth's centre; subtracting the centre's pull leaves 0.55 inward. | stretch along the line, squeeze across | +| 6 | mechanism | 1.1 µm/s² is about 10⁻⁷ g. How does it move water? | But it is tiny | ASCII figure | Up/down: it competes with g, so it changes weight by 1 part in 10 million; no effect. Along the surface: gravity has no part along a level surface, so the tidal acceleration (up to 0.82 at 45° from the near side) is the only sideways push; water flows until the sea slopes enough to balance it. Water gathers at two bulges. | two bulges | +| 7 | test | Does the subtraction matter? | So check it | — | If something held the Earth still, nothing would subtract 33.19; the far side's 32.12 points toward the Moon, into the Earth: one bulge. Sun: pull 179× the Moon's but 389× farther; the difference across Earth is about pull × 2R/d, so 179/389 = 0.46× tide. | why subtract | +| 8 | mechanism | When do high tides come? | Therefore | timeline | Earth turns under bulges. The Moon moves about 12° a day in the same direction, so Earth takes 24 h 50 min (a lunar day) to face it again; high tides 12 h 25 min apart. | timing | +| 9 | close | — | So | — | Two high tides because the Moon pulls the near side more and the far side less than Earth's centre, and the falling Earth feels only these differences. The far bulge needs no centrifugal force (a common textbook story). Scope: real heights, once-a-day coasts, spring/neap. | the answer | + +## 5. Concrete to symbol + +Pull = GM/r² first with numbers; then the approximation 2GMR/d³ only to explain the Sun's cube law. + +## 6. Links between representations + +The table and the text use the same four point names and the same numbers. + +## 7. Close the loop + +- **Answer:** Two high tides because the tidal acceleration stretches the ocean along the Earth–Moon line, toward the Moon on the near side and away on the far side. +- **General argument:** Any body falling in a non-uniform gravity field stretches along the line to the source. +- **Payoff:** Predict the next high tide (12 h 25 min later), why the Sun's tide is smaller. Scope: real tide heights, once-a-day tides, spring/neap. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/decisions.yaml new file mode 100644 index 0000000..7836770 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/decisions.yaml @@ -0,0 +1,173 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T150203-probe#1: + decision: adopted +2026-10-04T150203-probe#10: + decision: adopted +2026-10-04T150203-probe#11: + decision: declined +2026-10-04T150203-probe#12: + decision: declined +2026-10-04T150203-probe#13: + decision: adopted +2026-10-04T150203-probe#14: + decision: adopted +2026-10-04T150203-probe#15: + decision: adopted +2026-10-04T150203-probe#2: + decision: adopted +2026-10-04T150203-probe#3: + decision: adopted +2026-10-04T150203-probe#4: + decision: adopted +2026-10-04T150203-probe#5: + decision: declined +2026-10-04T150203-probe#6: + decision: adopted +2026-10-04T150203-probe#7: + decision: adopted +2026-10-04T150203-probe#8: + decision: adopted +2026-10-04T150203-probe#9: + decision: adopted +2026-10-04T150440-coldread-narrative#1: + decision: adopted +2026-10-04T150440-coldread-narrative#10: + decision: adopted +2026-10-04T150440-coldread-narrative#11: + decision: adopted +2026-10-04T150440-coldread-narrative#12: + decision: adopted +2026-10-04T150440-coldread-narrative#13: + decision: adopted +2026-10-04T150440-coldread-narrative#14: + decision: adopted +2026-10-04T150440-coldread-narrative#15: + decision: declined +2026-10-04T150440-coldread-narrative#16: + decision: adopted +2026-10-04T150440-coldread-narrative#17: + decision: adopted +2026-10-04T150440-coldread-narrative#18: + decision: adopted +2026-10-04T150440-coldread-narrative#19: + decision: adopted +2026-10-04T150440-coldread-narrative#2: + decision: adopted +2026-10-04T150440-coldread-narrative#20: + decision: adopted +2026-10-04T150440-coldread-narrative#21: + decision: adopted +2026-10-04T150440-coldread-narrative#22: + decision: adopted +2026-10-04T150440-coldread-narrative#23: + decision: adopted +2026-10-04T150440-coldread-narrative#24: + decision: adopted +2026-10-04T150440-coldread-narrative#25: + decision: adopted +2026-10-04T150440-coldread-narrative#26: + decision: adopted +2026-10-04T150440-coldread-narrative#27: + decision: adopted +2026-10-04T150440-coldread-narrative#28: + decision: adopted +2026-10-04T150440-coldread-narrative#3: + decision: adopted +2026-10-04T150440-coldread-narrative#4: + decision: adopted +2026-10-04T150440-coldread-narrative#5: + decision: adopted +2026-10-04T150440-coldread-narrative#6: + decision: adopted +2026-10-04T150440-coldread-narrative#7: + decision: adopted +2026-10-04T150440-coldread-narrative#8: + decision: adopted +2026-10-04T150440-coldread-narrative#9: + decision: adopted +2026-10-04T150700-coldread-prose#1: + decision: adopted +2026-10-04T150700-coldread-prose#2: + decision: adopted +2026-10-04T150700-coldread-prose#3: + decision: adopted +2026-10-04T150700-coldread-prose#4: + decision: adopted +2026-10-04T150700-coldread-prose#5: + decision: declined +2026-10-04T150700-coldread-prose#6: + decision: adopted +2026-10-04T150700-coldread-prose#7: + decision: adopted +2026-10-04T150700-coldread-prose#8: + decision: adopted +2026-10-04T150821-quiz-prose#1: + decision: adopted +2026-10-04T150821-quiz-prose#10: + decision: declined +2026-10-04T150821-quiz-prose#11: + decision: adopted +2026-10-04T150821-quiz-prose#12: + decision: declined +2026-10-04T150821-quiz-prose#13: + decision: declined +2026-10-04T150821-quiz-prose#14: + decision: declined +2026-10-04T150821-quiz-prose#15: + decision: declined +2026-10-04T150821-quiz-prose#16: + decision: declined +2026-10-04T150821-quiz-prose#17: + decision: declined +2026-10-04T150821-quiz-prose#18: + decision: declined +2026-10-04T150821-quiz-prose#19: + decision: declined +2026-10-04T150821-quiz-prose#2: + decision: adopted +2026-10-04T150821-quiz-prose#20: + decision: adopted +2026-10-04T150821-quiz-prose#3: + decision: adopted +2026-10-04T150821-quiz-prose#4: + decision: adopted +2026-10-04T150821-quiz-prose#5: + decision: adopted +2026-10-04T150821-quiz-prose#6: + decision: declined +2026-10-04T150821-quiz-prose#7: + decision: adopted +2026-10-04T150821-quiz-prose#8: + decision: declined +2026-10-04T150821-quiz-prose#9: + decision: declined +2026-10-04T150959-quiz-prose#1: + decision: declined +2026-10-04T150959-quiz-prose#10: + decision: declined +2026-10-04T150959-quiz-prose#11: + decision: declined +2026-10-04T150959-quiz-prose#12: + decision: declined +2026-10-04T150959-quiz-prose#13: + decision: adopted +2026-10-04T150959-quiz-prose#14: + decision: adopted +2026-10-04T150959-quiz-prose#15: + decision: declined +2026-10-04T150959-quiz-prose#2: + decision: adopted +2026-10-04T150959-quiz-prose#3: + decision: declined +2026-10-04T150959-quiz-prose#4: + decision: declined +2026-10-04T150959-quiz-prose#5: + decision: declined +2026-10-04T150959-quiz-prose#6: + decision: declined +2026-10-04T150959-quiz-prose#7: + decision: adopted +2026-10-04T150959-quiz-prose#8: + decision: declined +2026-10-04T150959-quiz-prose#9: + decision: adopted diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150203-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150203-probe.json new file mode 100644 index 0000000..3269c81 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150203-probe.json @@ -0,0 +1,224 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "two-tides", + "at": "2026-10-04T15:02:03", + "model": "default", + "prompt_sha": "4e0747130ed6", + "reply": { + "gaps": [ + { + "rule": 4, + "severity": "main", + "about": "The step where the 'sideways part' of the tidal acceleration moves water into the bulges has no numbers and no worked example.", + "why_it_matters": "The model says the 1.1 µm/s² vertical part 'cannot lift water against gravity directly' and that the sideways part does the work. But it never gives that sideways part's size or where on Earth it peaks. Without that, the reader is told the vertical push is too weak and is never shown what actually piles the water up.", + "suggested_addition": "Horizontal tidal acceleration = (3/2)(GM/d³)R·sin2θ, where θ is the angle from the sub-Moon point. It is 0 at θ = 0° and 90°, and largest at θ = 45°: 1.5 × 0.55 = 0.82 µm/s², pointing toward the nearer bulge. A table at θ = 0°, 45°, 90°, 135°, 180° with values 0, +0.82, 0, −0.82, 0 µm/s² would show water being pushed from the sides toward both bulges." + }, + { + "rule": 4, + "severity": "main", + "about": "The inward squeeze at the sides (0.55 µm/s², 'half the size') is stated, but never calculated or explained.", + "why_it_matters": "The low tide and the 'two bulges, not one' shape depend on the sides being pulled inward. But the three-point table only covers points on the Earth–Moon line. The reader sees no subtraction for the side points, so the squeeze looks asserted rather than derived.", + "suggested_addition": "At a side point, the distance to the Moon is ≈ √(384,400² + 6371²) ≈ 384,453 km, so the pull is ≈ 33.18 µm/s², almost the same as at the centre. But it points at the Moon, which is tilted by R/d = 0.01657 rad from the centre's pull direction. Subtracting the vectors leaves ≈ 33.19 × 0.01657 = 0.55 µm/s², pointing toward Earth's centre. Add a fourth row, 'side', to the table: pull 33.18, difference 0.55 inward." + }, + { + "rule": 1, + "severity": "edge", + "about": "The approximation 2GMR/d³ (and the 1/d³ law) is given without saying why it has this form instead of the direct difference.", + "why_it_matters": "The Sun-versus-Moon argument depends on 'differences fall as 1/d³'. The reader can do the direct subtraction but is never shown where the cube comes from, or why it gives 1.10 when the exact values are 1.13 and 1.07.", + "suggested_addition": "GM/(d−R)² − GM/d² ≈ GM/d² × 2R/d = 2GMR/d³ = 33.19 × 2 × 0.01657 = 1.10 µm/s². The exact near and far values (1.13, 1.07) differ by about ±3%, because of higher terms of order 3R/d ≈ 0.05. Then the Sun: d is 389× larger and GM is 2.71×10⁷× larger, so 2.71×10⁷ / 389³ ≈ 0.46." + }, + { + "rule": 2, + "severity": "main", + "about": "The argument names the wrong object: the Relationships row says free fall 'cancels the average Moon's pull', but the operation is subtracting the pull at the centre.", + "why_it_matters": "The pull at the centre (33.19) is not the average of the near and far values ((34.32 + 32.12)/2 = 33.22). Giving the same idea two names here also leaves implicit the key step: the water is measured relative to an Earth whose centre accelerates at exactly 33.19 µm/s².", + "suggested_addition": "Change the row to: 'Earth's free fall cancels the Moon's pull at the centre (33.19 µm/s²)'. Add one sentence: Earth's centre and the oceans all accelerate toward the Moon, and the centre's acceleration is 33.19 µm/s². Relative to the centre, each point only feels its own pull minus 33.19." + }, + { + "rule": 2, + "severity": "edge", + "about": "The 'uniform pull gives no tide' claim and the 'tidally locked' break case have no numbers.", + "why_it_matters": "Each guarantee should have a numeric case where it holds and one where it breaks. These two rows are bare statements.", + "suggested_addition": "Uniform pull: 33.19 at every point, so 33.19 − 33.19 = 0 everywhere, so no bulge. Tidally locked: the Moon's rotation period equals its orbital period (27.3 days), so relative to Earth the 'lunar day' is infinite and the number of high tides per day is 0. Contrast Earth: 1 rotation per 24 h 50 min relative to the Moon gives 2 per 24 h 50 min." + }, + { + "rule": 2, + "severity": "edge", + "about": "The confusion point 'the bulge would exist even if Earth fell straight at the Moon with no orbit' has no worked case.", + "why_it_matters": "This is the model's main rebuttal of the centrifugal story, and the reader is asked to accept it without numbers.", + "suggested_addition": "With no orbit (straight fall from rest at d = 384,400 km), the centre still accelerates at 33.19, the near side at 34.32 and the far side at 32.12. The differences are still +1.13 and −1.07, the same as in orbit. So both bulges appear without any rotation of the Earth–Moon pair." + }, + { + "rule": 4, + "severity": "edge", + "about": "The lunar day of 24 h 50 min is asserted without a worked calculation.", + "why_it_matters": "The 12 h 25 min spacing is one of the model's observable predictions. The reader is told only that 'the Moon moves along its orbit', with no numbers.", + "suggested_addition": "The Moon goes once around relative to the Sun every 29.53 days, about 12.2° per day, in the same direction Earth spins. Earth must turn an extra 12.2°, which at 15° per hour takes about 0.81 h ≈ 49–50 min. So the lunar day is ≈ 24 h 50 min, and half of it is 12 h 25 min." + }, + { + "rule": 3, + "severity": "edge", + "about": "One concept has many names: 'tidal acceleration', 'pull minus centre's pull', 'difference', 'tide-raising force', 'differential acceleration' (Audience section), 'the differences'. The centre's pull is also called 'the average Moon's pull'.", + "why_it_matters": "A new reader may think 'tide-raising force' or 'the difference' is a different quantity from 'tidal acceleration'.", + "suggested_addition": "Use only 'tidal acceleration' after it is defined, and only 'the Moon's pull at the centre (33.19 µm/s²)' for the subtracted term." + }, + { + "rule": 3, + "severity": "edge", + "about": "Terms used before they are defined, or never defined: 'equilibrium tide' (in the Entities table, defined only in Epistemic status), 'lunar day' (used in Relationships, missing from Entities), 'sides (90° from the Moon)', 'tidally locked', 'spring tides'.", + "why_it_matters": "These are new to the stated audience.", + "suggested_addition": "Add Entities rows: 'side point: a surface point 90° from the near side, ≈ 384,453 km from the Moon'; 'lunar day: 24 h 50 min'; 'equilibrium tide: the ocean shape that would balance the tidal acceleration instantly'. Define 'tidally locked' (the same face always points at the other body, period 27.3 days for the Moon) or remove it." + }, + { + "rule": 3, + "severity": "edge", + "about": "The definition of 'bulge' says the tidal acceleration 'pushed' water, which contradicts the confusion point 'the Moon pushes the far side away → No'.", + "why_it_matters": "It plants the very misconception the model is trying to correct.", + "suggested_addition": "Bulge: water gathered on the Earth–Moon line because the tidal acceleration there points outward (+1.13 / −1.07 µm/s² relative to the centre), and along the surface it points toward the line (up to 0.82 µm/s²)." + }, + { + "rule": 5, + "severity": "edge", + "about": "The bulge height (0.36 m) and the tidal acceleration (1.1 µm/s²) are never linked, and the near/far asymmetry (1.13 vs 1.07) is never given a meaning.", + "why_it_matters": "The reader sees a tiny acceleration and a height of 0.36 m but not how one leads to the other, or whether the near-side bulge is noticeably higher.", + "suggested_addition": "Equilibrium: the ocean surface tilts until Earth's gravity along the slope balances the horizontal tidal acceleration, a slope of about 0.82e-6 / 9.81 ≈ 8×10⁻⁸. Over about 6,000 km from bulge to side, that gives a rise and fall of order 0.5 m (range 0.54 m, peak 0.36 m above the mean). The near-side tidal acceleration is about 5% larger than the far side's, so the near bulge is about 5% taller, roughly 2 cm. That is too small to notice." + }, + { + "rule": 5, + "severity": "edge", + "about": "The Sun/Moon tidal ratio of 0.46 is not turned into something the reader can observe.", + "why_it_matters": "A ratio is more meaningful when it predicts something measurable.", + "suggested_addition": "Aligned (spring tide): 1 + 0.46 = 1.46× the Moon-only range. At right angles (neap tide): 1 − 0.46 = 0.54×. So spring ranges are about 2.7× neap ranges in the equilibrium model." + } + ], + "excess": [ + "The 'Sun and Moon in line (spring tides)' row in Alternative states: Scope declares spring tides out of scope.", + "Scope's 'Tidal friction and the Moon moving away' line is harmless but adds nothing to the central question.", + "The tidally locked Moon example, unless 'tidally locked' is defined; as it stands it adds a new idea just to show a break case." + ], + "next_questions": [ + { + "q": "If the tidal acceleration is only 1 ten-millionth of g, how does it move the ocean at all, and how high does each bulge get?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why are the two high tides in a day often different heights, and why do some places get only one?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Why are real tides metres high, and why does high tide come hours after the Moon is overhead?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Does the solid Earth also stretch into two bulges, and if so, why do we see the ocean rise relative to the land?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Isn't the far bulge really caused by centrifugal force from the Earth–Moon orbit, as I was taught?", + "answered_by_model": true, + "declared_out_of_scope": false + } + ] + }, + "usage": { + "cost": 0.17102900000000001, + "seconds": 65 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 4", + "text": "The step where the 'sideways part' of the tidal acceleration moves water into the bulges has no numbers and no worked example. → Horizontal tidal acceleration = (3/2)(GM/d³)R·sin2θ, where θ is the angle from the sub-Moon point. It is 0 at θ = 0° and 90°, and largest at θ = 45°: 1.5 × 0.55 = 0.82 µm/s², pointing toward the nearer bulge. A table at θ = 0°, 45°, 90°, 135°, 180° with values 0, +0.82, 0, −0.82, 0 µm/s² would show water being pushed from the sides toward both bulges." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 4", + "text": "The inward squeeze at the sides (0.55 µm/s², 'half the size') is stated, but never calculated or explained. → At a side point, the distance to the Moon is ≈ √(384,400² + 6371²) ≈ 384,453 km, so the pull is ≈ 33.18 µm/s², almost the same as at the centre. But it points at the Moon, which is tilted by R/d = 0.01657 rad from the centre's pull direction. Subtracting the vectors leaves ≈ 33.19 × 0.01657 = 0.55 µm/s², pointing toward Earth's centre. Add a fourth row, 'side', to the table: pull 33.18, difference 0.55 inward." + }, + { + "n": 3, + "severity": "edge", + "kind": "rule 1", + "text": "The approximation 2GMR/d³ (and the 1/d³ law) is given without saying why it has this form instead of the direct difference. → GM/(d−R)² − GM/d² ≈ GM/d² × 2R/d = 2GMR/d³ = 33.19 × 2 × 0.01657 = 1.10 µm/s². The exact near and far values (1.13, 1.07) differ by about ±3%, because of higher terms of order 3R/d ≈ 0.05. Then the Sun: d is 389× larger and GM is 2.71×10⁷× larger, so 2.71×10⁷ / 389³ ≈ 0.46." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 2", + "text": "The argument names the wrong object: the Relationships row says free fall 'cancels the average Moon's pull', but the operation is subtracting the pull at the centre. → Change the row to: 'Earth's free fall cancels the Moon's pull at the centre (33.19 µm/s²)'. Add one sentence: Earth's centre and the oceans all accelerate toward the Moon, and the centre's acceleration is 33.19 µm/s². Relative to the centre, each point only feels its own pull minus 33.19." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 2", + "text": "The 'uniform pull gives no tide' claim and the 'tidally locked' break case have no numbers. → Uniform pull: 33.19 at every point, so 33.19 − 33.19 = 0 everywhere, so no bulge. Tidally locked: the Moon's rotation period equals its orbital period (27.3 days), so relative to Earth the 'lunar day' is infinite and the number of high tides per day is 0. Contrast Earth: 1 rotation per 24 h 50 min relative to the Moon gives 2 per 24 h 50 min." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 2", + "text": "The confusion point 'the bulge would exist even if Earth fell straight at the Moon with no orbit' has no worked case. → With no orbit (straight fall from rest at d = 384,400 km), the centre still accelerates at 33.19, the near side at 34.32 and the far side at 32.12. The differences are still +1.13 and −1.07, the same as in orbit. So both bulges appear without any rotation of the Earth–Moon pair." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 4", + "text": "The lunar day of 24 h 50 min is asserted without a worked calculation. → The Moon goes once around relative to the Sun every 29.53 days, about 12.2° per day, in the same direction Earth spins. Earth must turn an extra 12.2°, which at 15° per hour takes about 0.81 h ≈ 49–50 min. So the lunar day is ≈ 24 h 50 min, and half of it is 12 h 25 min." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 3", + "text": "One concept has many names: 'tidal acceleration', 'pull minus centre's pull', 'difference', 'tide-raising force', 'differential acceleration' (Audience section), 'the differences'. The centre's pull is also called 'the average Moon's pull'. → Use only 'tidal acceleration' after it is defined, and only 'the Moon's pull at the centre (33.19 µm/s²)' for the subtracted term." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 3", + "text": "Terms used before they are defined, or never defined: 'equilibrium tide' (in the Entities table, defined only in Epistemic status), 'lunar day' (used in Relationships, missing from Entities), 'sides (90° from the Moon)', 'tidally locked', 'spring tides'. → Add Entities rows: 'side point: a surface point 90° from the near side, ≈ 384,453 km from the Moon'; 'lunar day: 24 h 50 min'; 'equilibrium tide: the ocean shape that would balance the tidal acceleration instantly'. Define 'tidally locked' (the same face always points at the other body, period 27.3 days for the Moon) or remove it." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 3", + "text": "The definition of 'bulge' says the tidal acceleration 'pushed' water, which contradicts the confusion point 'the Moon pushes the far side away → No'. → Bulge: water gathered on the Earth–Moon line because the tidal acceleration there points outward (+1.13 / −1.07 µm/s² relative to the centre), and along the surface it points toward the line (up to 0.82 µm/s²)." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 5", + "text": "The bulge height (0.36 m) and the tidal acceleration (1.1 µm/s²) are never linked, and the near/far asymmetry (1.13 vs 1.07) is never given a meaning. → Equilibrium: the ocean surface tilts until Earth's gravity along the slope balances the horizontal tidal acceleration, a slope of about 0.82e-6 / 9.81 ≈ 8×10⁻⁸. Over about 6,000 km from bulge to side, that gives a rise and fall of order 0.5 m (range 0.54 m, peak 0.36 m above the mean). The near-side tidal acceleration is about 5% larger than the far side's, so the near bulge is about 5% taller, roughly 2 cm. That is too small to notice." + }, + { + "n": 12, + "severity": "edge", + "kind": "rule 5", + "text": "The Sun/Moon tidal ratio of 0.46 is not turned into something the reader can observe. → Aligned (spring tide): 1 + 0.46 = 1.46× the Moon-only range. At right angles (neap tide): 1 − 0.46 = 0.54×. So spring ranges are about 2.7× neap ranges in the equilibrium model." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "The 'Sun and Moon in line (spring tides)' row in Alternative states: Scope declares spring tides out of scope." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Scope's 'Tidal friction and the Moon moving away' line is harmless but adds nothing to the central question." + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "The tidally locked Moon example, unless 'tidally locked' is defined; as it stands it adds a new idea just to show a break case." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150440-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150440-coldread-narrative.json new file mode 100644 index 0000000..8223fa9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150440-coldread-narrative.json @@ -0,0 +1,339 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "two-tides", + "at": "2026-10-04T15:04:40", + "model": "default", + "prompt_sha": "6fd08b11fa54", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: the answer in bold. Said: \"The Moon stretches the ocean; a stretch has two ends; the Earth turns through both.\"", + "unresolved": [], + "unsaid": [], + "leap": [ + "The question is never asked in words. The beat gives the answer (\"a stretch has two ends\") before the reader hears that there are two high tides a day, or that the far side has one. The reader has to rebuild the puzzle from the answer." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (instance)", + "text": "Shown: table of three points. Said: \"Pull falls with distance squared: 34.32, 33.19, 32.12 µm/s².\"", + "unresolved": [], + "unsaid": [ + "The table's point names (near side, centre, far side) and their distances from the Moon are not said. The words list three numbers without saying which point each belongs to, or that all three point toward the Moon." + ], + "leap": [], + "excess": [ + "\"Pull falls with distance squared\": this reader knows the inverse-square law." + ], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "Said: \"The whole Earth falls toward the Moon at 33.19. A shared acceleration changes nothing inside a falling body (as in an orbiting space station).\"", + "unresolved": [], + "unsaid": [], + "leap": [ + "\"The whole Earth falls toward the Moon\": the reader thinks of the Moon orbiting the Earth, not the Earth falling toward the Moon. The beat does not say why the Earth is in free fall toward the Moon. The space-station comparison helps once the reader accepts that." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (name)", + "text": "Shown: difference column. Said: \"tidal acceleration: +1.13 near, −1.07 far\"", + "unresolved": [ + "the + and − signs in \"+1.13\" and \"−1.07\": nothing says which direction is positive" + ], + "unsaid": [ + "The column is centre-subtracted (34.32−33.19 and 32.12−33.19), but the words do not define tidal acceleration as 'pull minus pull at the centre'.", + "The key conclusion, that both ends are pulled outward relative to the centre, is not said. It is only in the plan's 'Reader now knows' column." + ], + "leap": [ + "The step from −1.07 to 'the far side is pushed away from the Moon' is left for the reader to work out." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: side row. Said: \"pull tilted toward centre, 0.55 inward\"", + "unresolved": [ + "which point \"side\" means (90° from the near side) is not said" + ], + "unsaid": [ + "The conclusion 'stretch along the line, squeeze across' is not said." + ], + "leap": [ + "Why the pull at a side point tilts toward the centre (the lines to the Moon converge) is not given." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Shown: ASCII figure. Said: \"The sideways part, up to 0.82 at 45°, is unopposed; water flows to two bulges.\"", + "unresolved": [ + "\"sideways part\": sideways relative to what (along the surface?)", + "\"45°\": measured from what" + ], + "unsaid": [ + "The beat's own problem, that these accelerations are tiny (about 10⁻⁷ g), is never said, so the reader does not know what this beat is answering." + ], + "leap": [ + "\"is unopposed\": no reason is given for why the horizontal component is unopposed when the vertical one is swamped by g. This is the step that explains how a tiny force moves the ocean." + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 7 (test)", + "text": "Said: \"Held Earth: one bulge. Sun: 179× pull but 0.46× tide, because 1/d³.\"", + "unresolved": [ + "\"Held Earth\": a hypothetical Earth held fixed so it does not fall toward the Moon; never explained", + "\"1/d³\": where a cube law comes from is never said (2GMR/d³ is in the plan only)" + ], + "unsaid": [], + "leap": [ + "Why a held Earth would have one bulge is not argued.", + "How a difference of pulls turns into 1/d³ is not shown." + ], + "excess": [ + "The Sun comparison is useful but is a side test, off the main question of two bulges." + ], + "purpose_unclear": false + }, + { + "at": "Beat 8 (mechanism)", + "text": "Shown: timeline. Said: \"Earth turns under bulges; 24 h 50 min lunar day; 12 h 25 min.\"", + "unresolved": [ + "\"lunar day\"" + ], + "unsaid": [], + "leap": [ + "The reader knows the Earth turns in 24 h. Nothing says why the period here is 24 h 50 min (the Moon moves about 12° a day)." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (close)", + "text": "Said: \"Restate; misconception about centrifugal force; scope.\"", + "unresolved": [ + "\"misconception about centrifugal force\": centrifugal force never came up before, so the reader does not know what misconception is meant" + ], + "unsaid": [], + "leap": [ + "The 'Said' is only a stage direction. No actual restatement, general argument or scope is given to the reader." + ], + "excess": [], + "purpose_unclear": true + } + ], + "blocking": [ + "Beat 4, \"+1.13 near, −1.07 far\": the sign convention and the meaning 'both ends pulled outward from the centre' are not said, and this is the core of the two-bulge answer.", + "Beat 6, \"The sideways part, up to 0.82 at 45°, is unopposed\": no reason is given for why the horizontal part acts while the vertical part does not. Without it the reader cannot see how a force of about 10⁻⁷ g moves water.", + "Beat 9, \"Restate; misconception about centrifugal force; scope.\": there is no closing content, so the answer and the general argument are never actually said." + ], + "cuts": [ + "Beat 9: either drop \"misconception about centrifugal force\" or set it up earlier. As it stands it brings in a topic the reader has never seen.", + "Beat 2: drop \"Pull falls with distance squared\" (already known) and use the space to name which number belongs to which point.", + "Beat 7: the Sun comparison (\"179× pull but 0.46× tide, because 1/d³\") could be shortened or moved after the close, because its 1/d³ step is never explained." + ], + "question": { + "text": "The Moon is on one side of the Earth, so why is there a high tide on the far side too, giving two high tides a day?", + "known_at": "Never asked outright in 'Said'. It can be inferred from Beat 1's answer (\"a stretch has two ends\").", + "result_stated": "Yes, in Beat 1: \"The Moon stretches the ocean; a stretch has two ends; the Earth turns through both.\"" + }, + "motive": "Missing from the script. The reason to care (the naive picture predicts one tide but coasts have two) and the reason for the approach (compare the pull at three points, then ask which differences change the Earth's shape) appear only in the plan's 'Question and motive' section. Neither is in Said or Shown.", + "close": "Not delivered. Beat 9 says only \"Restate; misconception about centrifugal force; scope.\" The general argument ('any body falling in a non-uniform field stretches along the line to the source') appears only in plan section 7.", + "recap": "The whole Earth falls toward the Moon, so only the differences in pull matter: the near side is pulled a bit more and the far side a bit less, which stretches the ocean into two bulges.", + "strongest": "Beats 2–3: real numbers at three points, then the free-fall argument (\"A shared acceleration changes nothing inside a falling body (as in an orbiting space station)\"). This takes away the naive picture.", + "weakest": "Beat 9: the close is a placeholder with no content, and it brings in an unexplained centrifugal misconception. A close second is Beat 6's unexplained \"unopposed\".", + "suggestions": [ + "Open with the puzzle in words: 'The Moon pulls water toward it, so why is there also a high tide on the side facing away?'", + "In Beat 4, say it plainly: 'positive means toward the Moon; so the near water is pulled ahead of the Earth and the far water is left behind, and both move away from the centre.'", + "In Beat 6, explain 'unopposed': straight up or down, 1 µm/s² is nothing against g = 9.81 m/s², but along the surface nothing resists it, so the water slides.", + "Define 'held Earth' as 'if the Earth were held still instead of falling', and give a one-line reason for 1/d³ (a difference of 1/d² terms across the Earth's width).", + "Say why the lunar day is 24 h 50 min: the Moon moves about 12° along its orbit each day.", + "Write the close out in full: the answer, then the general rule that anything falling in a non-uniform field is stretched along the line to the source." + ] + }, + "usage": { + "cost": 0.1251516, + "seconds": 46 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4, \"+1.13 near, −1.07 far\": the sign convention and the meaning 'both ends pulled outward from the centre' are not said, and this is the core of the two-bulge answer." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 6, \"The sideways part, up to 0.82 at 45°, is unopposed\": no reason is given for why the horizontal part acts while the vertical part does not. Without it the reader cannot see how a force of about 10⁻⁷ g moves water." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 9, \"Restate; misconception about centrifugal force; scope.\": there is no closing content, so the answer and the general argument are never actually said." + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "Beat 9: either drop \"misconception about centrifugal force\" or set it up earlier. As it stands it brings in a topic the reader has never seen." + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 2: drop \"Pull falls with distance squared\" (already known) and use the space to name which number belongs to which point." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 7: the Sun comparison (\"179× pull but 0.46× tide, because 1/d³\") could be shortened or moved after the close, because its 1/d³ step is never explained." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): The question is never asked in words. The beat gives the answer (\"a stretch has two ends\") before the reader hears that there are two high tides a day, or that the far side has one. The reader has to rebuild the puzzle from the answer." + }, + { + "n": 8, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): The table's point names (near side, centre, far side) and their distances from the Moon are not said. The words list three numbers without saying which point each belongs to, or that all three point toward the Moon." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): \"The whole Earth falls toward the Moon\": the reader thinks of the Moon orbiting the Earth, not the Earth falling toward the Moon. The beat does not say why the Earth is in free fall toward the Moon. The space-station comparison helps once the reader accepts that." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (name): the + and − signs in \"+1.13\" and \"−1.07\": nothing says which direction is positive" + }, + { + "n": 11, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name): The column is centre-subtracted (34.32−33.19 and 32.12−33.19), but the words do not define tidal acceleration as 'pull minus pull at the centre'." + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name): The key conclusion, that both ends are pulled outward relative to the centre, is not said. It is only in the plan's 'Reader now knows' column." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (name): The step from −1.07 to 'the far side is pushed away from the Moon' is left for the reader to work out." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): which point \"side\" means (90° from the near side) is not said" + }, + { + "n": 15, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The conclusion 'stretch along the line, squeeze across' is not said." + }, + { + "n": 16, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): Why the pull at a side point tilts toward the centre (the lines to the Moon converge) is not given." + }, + { + "n": 17, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): \"sideways part\": sideways relative to what (along the surface?)" + }, + { + "n": 18, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): \"45°\": measured from what" + }, + { + "n": 19, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): The beat's own problem, that these accelerations are tiny (about 10⁻⁷ g), is never said, so the reader does not know what this beat is answering." + }, + { + "n": 20, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): \"is unopposed\": no reason is given for why the horizontal component is unopposed when the vertical one is swamped by g. This is the step that explains how a tiny force moves the ocean." + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): \"Held Earth\": a hypothetical Earth held fixed so it does not fall toward the Moon; never explained" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): \"1/d³\": where a cube law comes from is never said (2GMR/d³ is in the plan only)" + }, + { + "n": 23, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test): Why a held Earth would have one bulge is not argued." + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test): How a difference of pulls turns into 1/d³ is not shown." + }, + { + "n": 25, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (mechanism): \"lunar day\"" + }, + { + "n": 26, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (mechanism): The reader knows the Earth turns in 24 h. Nothing says why the period here is 24 h 50 min (the Moon moves about 12° a day)." + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (close): \"misconception about centrifugal force\": centrifugal force never came up before, so the reader does not know what misconception is meant" + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (close): The 'Said' is only a stage direction. No actual restatement, general argument or scope is given to the reader." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150700-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150700-coldread-prose.json new file mode 100644 index 0000000..31ffd7c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150700-coldread-prose.json @@ -0,0 +1,270 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "two-tides", + "at": "2026-10-04T15:07:00", + "model": "default", + "prompt_sha": "fb797624d4da", + "reply": { + "paragraphs": [ + { + "at": "Title + line 3 (opening)", + "text": "Most coasts get two high tides a day... If the Moon simply pulls water toward itself, the second high tide makes no sense.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 5 (bold summary)", + "text": "The Moon does not lift the ocean toward itself. It stretches the ocean along the Earth–Moon line...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 9 (The pull at three points)", + "text": "The Moon's pull is the acceleration that the Moon's gravity gives at a point...", + "unresolved": [], + "leap": [], + "excess": [ + "It points at the Moon and gets weaker with the square of the distance." + ], + "purpose_unclear": false + }, + { + "at": "table, lines 11-15", + "text": "| Point | Distance to Moon | Moon's pull | Tidal acceleration |", + "unresolved": [ + "Tidal acceleration (the column header and its values +1.13, 0, −1.07 µm/s² come before the term or its sign convention is defined; both are only given at line 24)" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 17", + "text": "All three pulls point toward the Moon. So why does the far-side water not move toward the Moon?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 22 (Only the part that differs...)", + "text": "The Earth and the Moon orbit their shared centre of mass. An orbit is free fall...", + "unresolved": [], + "leap": [], + "excess": [ + "Astronauts in an orbiting station float for the same reason." + ], + "purpose_unclear": false + }, + { + "at": "line 24", + "text": "So subtract the centre's pull. The remainder is the tidal acceleration...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 26 (near side bullet)", + "text": "Near side: 34.32 − 33.19 = +1.13...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 27 (far side bullet)", + "text": "Far side: 32.12 − 33.19 = −1.07...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 30", + "text": "Both results point away from Earth's centre. The Moon does not push the far side away...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 32 (side point)", + "text": "Now take a side point, 90° from the near side...", + "unresolved": [], + "leap": [ + "\"Subtract the centre's pull, and 0.55 µm/s² is left, pointing down\": it doesn't say why the along-line part cancels (33.18 vs 33.19), leaving only the 0.95° tilt component." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 36 (How a tiny acceleration moves an ocean)", + "text": "1.1 µm/s² is about one ten-millionth of g... The water gathers into two bulges", + "unresolved": [], + "leap": [ + "\"It is largest 45° from the near side, at 0.82 µm/s²\": stated without a reason or derivation, since only the 0°, 90° and 180° points were worked out." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "diagram, lines 38-47", + "text": "far side ←( ↘ Earth ↙ )→ near side ... (↘ ↙ : the sideways push, 0.82 at 45°, toward the nearer bulge)", + "unresolved": [ + "\"0.55 down\" and \"0.82 at 45°\" have no units in the diagram" + ], + "leap": [ + "The arrows ↘ ↙ both point inward toward the centre. The legend and line 36 say the push points 'toward the nearer bulge', which is outward toward the ends of the line. The reader has to stop and reconcile the picture with the text." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 52 (Why subtract: two tests)", + "text": "The simplest idea is that the Moon's pull itself raises the tide. It fails two tests.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": true + }, + { + "at": "line 55 (first test)", + "text": "First, imagine that something held the Earth still... The far bulge exists because the Earth falls.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 57 (Sun test)", + "text": "Second, the Sun. Its pull on the Earth is 179 times the Moon's...", + "unresolved": [], + "leap": [ + "\"the tidal acceleration is about the pull times 2R/d\": the factor is stated, not derived. The reader can check it against the table (33.19 × 2·6371/384400 ≈ 1.10), but the text doesn't do this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 62 (Two high tides per turn)", + "text": "The bulges stay on the Earth–Moon line, and the Earth turns under them...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 66 (The answer)", + "text": "The Moon pulls the near side more than Earth's centre... An Earth that fell straight at the Moon, with no orbit, would have the same two bulges.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 70 (Scope: Height)", + "text": "in an ideal ocean, the bulges rise about 0.36 m above the mean (an estimate)...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 71 (Scope: Real oceans)", + "text": "continents block the bulges and basins slosh with their own timing...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 72 (Scope: Spring and neap)", + "text": "the Sun's bulges add to the Moon's or partly cancel them.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Line 22: drop \"Astronauts in an orbiting station float for the same reason.\" The sentence before it already makes the point.", + "Line 9: drop \"It points at the Moon and gets weaker with the square of the distance.\" This reader already knows inverse-square gravity." + ], + "question": { + "text": "Why is there a high tide on the side of the Earth away from the Moon, giving two high tides a day instead of one?", + "known_at": "Title and line 3: \"If the Moon simply pulls water toward itself, the second high tide makes no sense.\"", + "result_stated": "Yes, at line 5: \"It stretches the ocean along the Earth–Moon line, and a stretch has two ends. The Earth turns through both ends, so a place gets two high tides.\"" + }, + "motive": "Reason to care about the question (line 3): \"If the Moon simply pulls water toward itself, the second high tide makes no sense.\" Reason for the approach of subtracting the centre's pull (line 22): \"An orbit is free fall... An acceleration that every part shares does not change the shape of a falling body.\" It is backed up at line 52 by \"It fails two tests\" (the held-still Earth and the Sun ratio).", + "close": "Yes, at line 66: \"The Moon pulls the near side more than Earth's centre, and the centre more than the far side. The falling Earth feels only these differences, which stretch the ocean into two bulges. The Earth turns through both every 24 h 50 min.\"", + "recap": "The whole Earth is falling toward the Moon, so only differences in the Moon's pull count: the near side is pulled more than the centre and the far side less, which stretches the ocean into two bulges, and the Earth turns through both.", + "strongest": "Line 55: the held-still-Earth test. \"That Earth has one bulge, not two. The far bulge exists because the Earth falls.\" It isolates the cause of the far bulge cleanly.", + "weakest": "The diagram at lines 38-47. The ↘ ↙ arrows point inward, but the legend says the push is 'toward the nearer bulge', and the values have no units. A smaller weak spot is the table at lines 11-15, whose 'Tidal acceleration' column appears before the term is defined at line 24.", + "suggestions": [ + "Redraw the sideways-push arrows so they point outward toward the near and far bulges, and add units (µm/s²) inside the diagram.", + "Either leave the 'Tidal acceleration' column out of the first table, or add one line saying it is defined in the next section.", + "Give a one-line reason for the 45° maximum, or for the 2R/d factor, e.g. from differentiating GM/d²." + ] + }, + "usage": { + "cost": 0.12958219999999998, + "seconds": 43 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Line 22: drop \"Astronauts in an orbiting station float for the same reason.\" The sentence before it already makes the point." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Line 9: drop \"It points at the Moon and gets weaker with the square of the distance.\" This reader already knows inverse-square gravity." + }, + { + "n": 3, + "severity": "edge", + "kind": "unresolved", + "text": "table, lines 11-15: Tidal acceleration (the column header and its values +1.13, 0, −1.07 µm/s² come before the term or its sign convention is defined; both are only given at line 24)" + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "line 32 (side point): \"Subtract the centre's pull, and 0.55 µm/s² is left, pointing down\": it doesn't say why the along-line part cancels (33.18 vs 33.19), leaving only the 0.95° tilt component." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "line 36 (How a tiny acceleration moves an ocean): \"It is largest 45° from the near side, at 0.82 µm/s²\": stated without a reason or derivation, since only the 0°, 90° and 180° points were worked out." + }, + { + "n": 6, + "severity": "edge", + "kind": "unresolved", + "text": "diagram, lines 38-47: \"0.55 down\" and \"0.82 at 45°\" have no units in the diagram" + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "diagram, lines 38-47: The arrows ↘ ↙ both point inward toward the centre. The legend and line 36 say the push points 'toward the nearer bulge', which is outward toward the ends of the line. The reader has to stop and reconcile the picture with the text." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "line 57 (Sun test): \"the tidal acceleration is about the pull times 2R/d\": the factor is stated, not derived. The reader can check it against the table (33.19 × 2·6371/384400 ≈ 1.10), but the text doesn't do this." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150821-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150821-quiz-prose.json new file mode 100644 index 0000000..73e8b89 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150821-quiz-prose.json @@ -0,0 +1,205 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "two-tides", + "at": "2026-10-04T15:08:08", + "model": "default", + "prompt_sha": "94be2bc7b92f", + "reply": { + "answers": { + "1": "The Moon, the Earth (its centre plus three surface points: the near side, the far side and a side point 90° away), the ocean water, the Moon's pull (the acceleration its gravity gives at a point), the tidal acceleration, the two bulges, the Earth's rotation and the lunar day. The Sun is also used as a comparison.", + "2": "The Moon's pull gets weaker with distance: 34.32 µm/s² at the near side, 33.19 at the centre and 32.12 at the far side. Earth and Moon orbit their shared centre of mass, so the whole Earth is in free fall toward the Moon. Only the difference from the centre's pull (the tidal acceleration) changes the Earth's shape. That difference points outward at the near and far sides and inward at the side points. Along the surface it pushes water toward the two ends of the Earth–Moon line, which forms two bulges. The Earth turns under these bulges.", + "3": "Because the Earth falls freely toward the Moon, only the difference between the Moon's pull at a point and its pull at the centre acts on the water. That difference stretches the ocean into two bulges along the Earth–Moon line, and the turning Earth carries each place through both bulges.", + "4": "The key variable is distance d to the tide-raising body. The tidal acceleration is roughly the pull times 2R/d. The pull itself falls as 1/d², so the tide falls faster than the pull, roughly as 1/d³ (the explanation only implies this). A farther body gives smaller tides. That is why the Sun's pull is 179 times the Moon's but its tide is only about 0.46 times the Moon's. A larger Earth radius R would give larger tides.", + "5": "Assumption: an ideal ocean in equilibrium with no continents (the 0.36 m bulge height is called an estimate), and the Earth treated as falling freely. Observation: solar tides are about half the size of the Moon's. Also, high tides come about every 12 h 25 min.", + "6": "The Moon's pull is a little stronger on the near side of the Earth than at the centre, and a little weaker on the far side. The whole Earth is in free fall toward the Moon, so only the differences from the centre's pull matter. Those differences point outward along the Earth–Moon line and inward across it. Along the surface they push water toward the two ends of the line, which makes two bulges. The Earth turns under both bulges once per lunar day of 24 h 50 min, so high tides come every 12 h 25 min.", + "7": "The hardest parts were the side-point tilt (0.95°, 0.0166 rad) and the 2R/d formula. Both are stated, not derived. It would help to say that the tilt angle is R/d, and to show where the factor 2 comes from. It was also unclear how a sideways push of 0.82 µm/s² at 45° turns into a sea slope.", + "8": "The Moon pulls the far side less than it pulls the Earth's centre (32.12 vs 33.19 µm/s²). The Earth as a whole falls toward the Moon, so the far-side water is left behind, away from the Moon. Its tidal acceleration is −1.07 µm/s², which points outward from the Earth's centre.", + "9": "One. Nothing would subtract the centre's pull, so on the far side the Moon's pull would point down into the Earth. Only the near-side bulge would remain.", + "10": "At about 20:25. The Moon moves about 12° a day along its orbit in the same direction the Earth turns. The Earth therefore needs 24 h 50 min to face the Moon again, and half of that is 12 h 25 min.", + "11": "It points away from the Moon, which on the far side is outward from the Earth's centre. The far side is pulled less (32.12) than the centre (33.19), so the difference is −1.07 µm/s².", + "12": "Tides depend on the difference in pull across the Earth, roughly the pull times 2R/d, not on the pull itself. The Sun is 389 times farther away than the Moon, so its tide is about 179/389 ≈ 0.46 times the Moon's.", + "13": "There are two bulges, and a place passes through both in each lunar day. The lunar day is 24 h 50 min, not 24 h, because the Moon moves about 12° a day in the direction the Earth turns. Half of 24 h 50 min is 12 h 25 min." + }, + "audit": { + "terms": [ + "'tide' means two things in the Sun paragraph (line 60). In 'its tide is about 0.46 times the Moon's' it means tidal acceleration. In 'Observed solar tides are about half' it means water height. The explanation treats the two as the same without saying so.", + "'bulge' (line 36 onward) and 'high tide' (lines 3, 65) both name the same feature: one is the water shape, the other is what a coastal place sees. The link is implied rather than stated.", + "'stretches the ocean' (line 5), 'tidal acceleration' (line 24) and 'these differences' (line 69) all name the same quantity." + ], + "unexplained": [ + "Line 13 vs line 15: near side +1.13 vs far side −1.07. Why the two ends are unequal is never said. The reader may wonder whether the near bulge is larger.", + "Line 32, '0.95° (0.0166 rad)': where the angle comes from is not stated (it is R/d).", + "Line 36, '0.82 µm/s² at 45°': given with no derivation and no comparison to the 1.1 values along the line.", + "Line 65, '12° a day' leading to '50 minutes': the conversion is not shown.", + "Line 73, '0.36 m above the mean': not linked to the accelerations, and no hint of how it relates to observed metre-scale tides beyond the Scope note." + ], + "missing_why": [ + "Line 60: 'tidal acceleration is about the pull times 2R/d'. Neither the factor 2 nor the R/d form is justified.", + "Line 36: why the sideways push is largest at 45°.", + "Line 36: 'Water flows until the sea slopes just enough to balance this push'. How a slope balances a sideways force is not explained.", + "Line 32: why the side-point tilt equals the inward component (pull × angle)." + ], + "excess": [ + "Line 9: defining µm/s² as one millionth of a m/s² is something this reader already knows.", + "Line 22: 'An orbit is free fall' restates something the reader already knows (free fall), though it does link that idea to the orbit.", + "Line 69: the centrifugal-force aside adds a claim ('An Earth that fell straight... same two bulges') that the questions do not use.", + "Lines 74–75: the 'Real oceans' and spring/neap bullets are detours the argument and questions do not use. They are short and clearly labelled." + ] + }, + "gaps": [ + "No derivation of the 2R/d tidal formula, so the Sun comparison has to be taken on trust.", + "The near/far asymmetry (1.13 vs 1.07) is not explained.", + "The 45° maximum and the slope-balance mechanism are asserted, not shown.", + "How 12° a day becomes about 50 minutes is not shown." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 1, + "12": 2, + "13": 2 + }, + "reasons": { + "10": "Kept at 2, borderline. It explains why the gap is not exactly 12 h (the Moon moves about 12° a day, so the lunar day is 24 h 50 min) and gives 20:25. It halves the lunar day without saying this is because there are two bulges.", + "11": "It gives the direction (away from the Moon, outward) and the subtraction 32.12 − 33.19 = −1.07. It does not say why the centre's pull is subtracted: the whole Earth falls toward the Moon with the centre's acceleration. The expected answer requires this reason." + }, + "pass": false + }, + "usage": { + "cost": 0.1745564, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "'tide' means two things in the Sun paragraph (line 60). In 'its tide is about 0.46 times the Moon's' it means tidal acceleration. In 'Observed solar tides are about half' it means water height. The explanation treats the two as the same without saying so." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'bulge' (line 36 onward) and 'high tide' (lines 3, 65) both name the same feature: one is the water shape, the other is what a coastal place sees. The link is implied rather than stated." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'stretches the ocean' (line 5), 'tidal acceleration' (line 24) and 'these differences' (line 69) all name the same quantity." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "Line 13 vs line 15: near side +1.13 vs far side −1.07. Why the two ends are unequal is never said. The reader may wonder whether the near bulge is larger." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 32, '0.95° (0.0166 rad)': where the angle comes from is not stated (it is R/d)." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 36, '0.82 µm/s² at 45°': given with no derivation and no comparison to the 1.1 values along the line." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "Line 65, '12° a day' leading to '50 minutes': the conversion is not shown." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "Line 73, '0.36 m above the mean': not linked to the accelerations, and no hint of how it relates to observed metre-scale tides beyond the Scope note." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 60: 'tidal acceleration is about the pull times 2R/d'. Neither the factor 2 nor the R/d form is justified." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 36: why the sideways push is largest at 45°." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Line 36: 'Water flows until the sea slopes just enough to balance this push'. How a slope balances a sideways force is not explained." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "Line 32: why the side-point tilt equals the inward component (pull × angle)." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Line 9: defining µm/s² as one millionth of a m/s² is something this reader already knows." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "Line 22: 'An orbit is free fall' restates something the reader already knows (free fall), though it does link that idea to the orbit." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "Line 69: the centrifugal-force aside adds a claim ('An Earth that fell straight... same two bulges') that the questions do not use." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "Lines 74–75: the 'Real oceans' and spring/neap bullets are detours the argument and questions do not use. They are short and clearly labelled." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "No derivation of the 2R/d tidal formula, so the Sun comparison has to be taken on trust." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "The near/far asymmetry (1.13 vs 1.07) is not explained." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "The 45° maximum and the slope-balance mechanism are asserted, not shown." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "How 12° a day becomes about 50 minutes is not shown." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150959-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150959-quiz-prose.json new file mode 100644 index 0000000..8022c21 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150959-quiz-prose.json @@ -0,0 +1,167 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "two-tides", + "at": "2026-10-04T15:09:53", + "model": "default", + "prompt_sha": "94be2bc7b92f", + "reply": { + "answers": { + "1": "The Moon, the Earth (its near side, centre, far side and side points), the ocean water, Earth's gravity g, the Sun, and the Earth's rotation, which brings in the lunar day.", + "2": "The Moon's gravity pulls every point of the Earth toward the Moon. The pull is weaker farther away: 34.32, 33.19 and 32.12 µm/s² at the near side, centre and far side. The Earth and Moon orbit their shared centre of mass, so the Earth is in free fall toward the Moon at the centre's acceleration. Only the difference from the centre's pull (the tidal acceleration) changes the ocean's shape. It points outward at the near and far sides and inward at the side points. Its sideways part, along the surface, is not resisted by Earth's gravity, so water flows into two bulges on the Earth–Moon line. The Earth turns under the bulges, and the Moon moves along its orbit, so each place passes through both bulges once per lunar day.", + "3": "The Moon pulls the near side more than the centre and the centre more than the far side, while the whole Earth falls with its centre. What is left is a stretch along the Earth–Moon line with two ends, and the turning Earth carries each place through both ends.", + "4": "The key variable is the distance d to the tide-raising body. The tidal acceleration is about the pull × 2R/d, and the pull falls as 1/d². So the tidal effect falls faster than the pull. A closer Moon would give much bigger bulges and a farther one much smaller bulges. That is why the Sun, which pulls 179 times harder but is 389 times farther, gives only about 0.46 of the Moon's tide. If the Earth stopped falling (held fixed), the subtraction would vanish and only one bulge would remain.", + "5": "Assumption or simplification: an ideal ocean in equilibrium with no continents; the 0.36 m bulge height is called an estimate. Observation: solar tides are seen to be about half the height of lunar tides. Most coasts get two high tides a day, about 12 h 25 min apart.", + "6": "The Moon's pull weakens with distance, so it is stronger at the Earth's near side than at its centre, and stronger at the centre than at the far side. The Earth is in free fall toward the Moon, so a shared acceleration does nothing. Only the leftover, the tidal acceleration, acts: about +1.13 µm/s² at the near side, −1.07 at the far side (both pointing away from the centre) and inward at the sides. This is far too small to lift water against g, but it is the only sideways push along a level surface, so water flows into two bulges on the Earth–Moon line. The Earth turns through both bulges once per lunar day of 24 h 50 min (longer than 24 h because the Moon moves along its orbit), so high tides come about every 12 h 25 min.", + "7": "Two parts were hardest. First, the side-point tilt calculation and the claim that the sideways push peaks at 0.82 µm/s² at 45°: no reason is given for 45° or for that value. A short line on how the stretch along the line and the squeeze across it combine would help. Second, the closing remark that 'no centrifugal force is needed', with an Earth falling straight at the Moon. Centrifugal force was never introduced, so this comes out of nowhere.", + "8": "Because the Earth as a whole is falling toward the Moon at the centre's acceleration (33.19 µm/s²). The far side is pulled less (32.12), so it falls more slowly than the centre and is left behind, away from the Moon. Relative to the centre that is an outward tidal acceleration of −1.07 µm/s², and water gathers there into a second bulge.", + "9": "One. With no fall, nothing subtracts the centre's pull. The far side's pull points toward the Moon, which on the far side is down into the Earth, so only the near side would bulge.", + "10": "At about 20:25. The bulges stay on the Earth–Moon line, and the Moon moves about 12° a day in the same direction the Earth turns. So the Earth needs about 50 minutes more than 24 h to face the Moon again (a lunar day of 24 h 50 min), and half of that is 12 h 25 min.", + "11": "Away from the Moon, which on the far side is outward from Earth's centre. The far side's pull (32.12) is smaller than the centre's (33.19), so far side minus centre is −1.07 µm/s²: it lags behind the falling centre. The Moon does not push it; it just pulls it less.", + "12": "Tides come from the difference in pull across the Earth, about the pull × 2R/d, not from the pull itself. The Sun is 389 times farther away than the Moon, so its tidal acceleration is about 179/389 ≈ 0.46 of the Moon's. That matches observed solar tides of about half the lunar height.", + "13": "The Moon advances about 12° a day along its orbit in the direction of Earth's spin. At 15°/h the Earth needs about 50 extra minutes to face the Moon again, so a lunar day is 24 h 50 min. Two bulges per lunar day gives about 12 h 25 min between high tides." + }, + "audit": { + "terms": [ + "The same horizontal component is called 'the tidal acceleration is the only sideways push' (section 'How a tiny acceleration moves an ocean') and 'sideways push along the surface' (diagram legend). This is fine, but 'push' sits awkwardly next to 'The Moon does not push the far side away' (section 'Only the part that differs').", + "Two words for the same direction: 'pointing down' (side point paragraph) and 'toward Earth's centre' (same paragraph), with 'down into the Earth' in the first test. Minor." + ], + "unexplained": [ + "'It is largest 45° from the near side, at 0.82 µm/s²' (section 'How a tiny acceleration moves an ocean'): no reason is given for why the peak is at 45° or how 0.82 is obtained.", + "'the bulges rise about 0.36 m above the mean (an estimate)' (Scope): there is no hint of how this follows from the 1.1 µm/s² slope balance.", + "'the two bulges differ by about 5%' (section 'Only the part that differs'): it says nothing about whether this matters for observed tide heights.", + "'The Moon also moves about 12° a day' (section 'Two high tides per turn'): presented as a given. It is fine to take it as data, but no source or reasoning is offered." + ], + "missing_why": [ + "'the tidal acceleration is about the pull times 2R/d' (section 'Why subtract: two tests'): nothing explains why it has this form, i.e. where the factor 2 comes from in the inverse-square fall-off. The numeric check shows it works but not why.", + "'Water flows until the sea surface slopes just enough that gravity along the slope balances this push': the step from this balance to two bulges of a given height is not shown.", + "'No centrifugal force is needed: an Earth that fell straight at the Moon, with no orbit, would have the same two bulges' (The answer): stated with no argument, and it answers a question the text never raised." + ], + "excess": [ + "'The answer' section mostly repeats the bold summary at the top and the subtraction section. Its only new content is the centrifugal remark, which is itself unexplained.", + "The Scope bullet on real oceans (dynamic theory, once-a-day tides) is not used by the argument. It is short and acceptable as a pointer." + ] + }, + "gaps": [ + "Why the sideways push peaks at 45° and why its value there is 0.82 µm/s².", + "Where the 2R/d factor comes from (the derivative of an inverse-square pull).", + "The centrifugal-force remark has no setup, so a reader cannot judge it.", + "How 0.36 m follows from the acceleration balance is not shown." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": {}, + "pass": true + }, + "usage": { + "cost": 0.17281239999999998, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same horizontal component is called 'the tidal acceleration is the only sideways push' (section 'How a tiny acceleration moves an ocean') and 'sideways push along the surface' (diagram legend). This is fine, but 'push' sits awkwardly next to 'The Moon does not push the far side away' (section 'Only the part that differs')." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Two words for the same direction: 'pointing down' (side point paragraph) and 'toward Earth's centre' (same paragraph), with 'down into the Earth' in the first test. Minor." + }, + { + "n": 3, + "severity": "gap", + "kind": "unexplained", + "text": "'It is largest 45° from the near side, at 0.82 µm/s²' (section 'How a tiny acceleration moves an ocean'): no reason is given for why the peak is at 45° or how 0.82 is obtained." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'the bulges rise about 0.36 m above the mean (an estimate)' (Scope): there is no hint of how this follows from the 1.1 µm/s² slope balance." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'the two bulges differ by about 5%' (section 'Only the part that differs'): it says nothing about whether this matters for observed tide heights." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'The Moon also moves about 12° a day' (section 'Two high tides per turn'): presented as a given. It is fine to take it as data, but no source or reasoning is offered." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "'the tidal acceleration is about the pull times 2R/d' (section 'Why subtract: two tests'): nothing explains why it has this form, i.e. where the factor 2 comes from in the inverse-square fall-off. The numeric check shows it works but not why." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "'Water flows until the sea surface slopes just enough that gravity along the slope balances this push': the step from this balance to two bulges of a given height is not shown." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "'No centrifugal force is needed: an Earth that fell straight at the Moon, with no orbit, would have the same two bulges' (The answer): stated with no argument, and it answers a question the text never raised." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "'The answer' section mostly repeats the bold summary at the top and the subtraction section. Its only new content is the centrifugal remark, which is itself unexplained." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "The Scope bullet on real oceans (dynamic theory, once-a-day tides) is not used by the argument. It is short and acceptable as a pointer." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "Why the sideways push peaks at 45° and why its value there is 0.82 µm/s²." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "Where the 2R/d factor comes from (the derivative of an inverse-square pull)." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The centrifugal-force remark has no setup, so a reader cannot judge it." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "How 0.36 m follows from the acceleration balance is not shown." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/read.json new file mode 100644 index 0000000..417d767 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/read.json @@ -0,0 +1,40 @@ +{ + "answers": { + "1": { + "answer": "The Moon pulls the far side less strongly than Earth's center. As the Earth falls toward the Moon, the far-side water is left behind and bulges away from the Moon.", + "quote": "The Moon does not push the far side away. It pulls the far side less than it pulls the centre, and the centre's fall sets the reference." + }, + "2": { + "answer": "The Moon moves about 12° per day along its orbit in the same direction as Earth's rotation. Earth therefore needs about 50 minutes more than 24 hours to face the Moon again (a lunar day of 24 h 50 min). Since a place passes through both bulges each lunar day, high tides come every half lunar day: 12 h 25 min.", + "quote": "The Moon also moves about 12° a day along its orbit, in the same direction that the Earth turns. The Earth turns about 15° an hour, so it needs about 50 minutes more than 24 hours to face the Moon again: a lunar day of 24 h 50 min." + }, + "3": { + "answer": "not covered", + "quote": "" + }, + "4": { + "answer": "Every 12 hours. If the Moon stayed fixed, Earth would need only 24 hours (not 24 h 50 min) to rotate and face it again. Since there are two bulges on the Earth-Moon line, a place passes through both bulges once per 24-hour rotation, giving high tides every 12 hours.", + "quote": "A place passes through both bulges in each lunar day, so high tides come every half lunar day: 12 h 25 min." + } + }, + "stops": [ + "An orbit is free fall: the whole Earth falls toward the Moon, and Earth's centre accelerates at 33.19 µm/s².", + "An acceleration that every part shares does not change the shape of a falling body.", + "so it tilts by R/d = 6371 / 384,400 = 0.0166 rad (0.95°) toward Earth's centre.", + "Water flows until the sea surface slopes just enough that gravity along the slope balances this push.", + "The bulges stay on the Earth–Moon line, and the Earth turns under them." + ], + "excess": [ + "A high tide at 8 in the morning gives the next at about 8:25 in the evening.", + "The near value is about 5% larger because the pull changes faster closer to the Moon.", + "Observed solar tides are about half the Moon's.", + "in an ideal ocean, the bulges rise about 0.36 m above the mean (an estimate).", + "Look up 'equilibrium tide'." + ], + "grounded": { + "1": true, + "2": true, + "3": false, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/run.json new file mode 100644 index 0000000..954e8b9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/two-tides/explanation.md`, a prose explanation (plus its `model.md`, `model.yaml` and `narrative.md`). Prose fits because the argument is a short chain — the Moon's pull at three points, one subtraction, then the Earth's rotation — and a table and a small text figure carry it, so a diagram or interactive page would add nothing; the probe, cold reads, final `explainer check` and blind test all passed, though I made small wording fixes after the blind test and didn't re-run it.", + "error": null, + "cost": 2.064919, + "seconds": 705, + "turns": 101, + "files": [ + "explainers/two-tides/explanation.md" + ], + "review_cost": 0.7731, + "reviews": 5 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/explanation.md new file mode 100644 index 0000000..8da97e9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/explanation.md @@ -0,0 +1,60 @@ +# Why are there two high tides a day, not one? + + + +**The Moon raises two bulges of ocean on opposite sides of Earth: one facing the Moon and one facing away. Earth turns under both bulges, so a point on the coast passes through two high tides in each turn.** + +## Predict first + +The Moon pulls water toward it. If that were the whole story, how many bulges would you expect, and where? Decide before you read on. + +## The Moon pulls every part of Earth by a different amount + +The Moon's gravity gives an acceleration, here called the **pull**, of GM/r². The pull gets weaker with distance r. Earth is 12,742 km wide, so its near side is closer to the Moon than its far side. With the Moon at 384,400 km and Earth's radius 6,371 km: + +| Point on Earth | Distance to Moon | Pull (m/s²) | Pull minus centre's pull (m/s²) | +|---|---|---|---| +| Near side | 378,029 km | 3.433e-5 | 1.13e-6 toward the Moon | +| Centre | 384,400 km | 3.320e-5 | 0 | +| Far side | 390,771 km | 3.213e-5 | 1.07e-6 away from the Moon | + +All three pulls point toward the Moon. So why does the far water move away from the Moon? + +## Only the difference in pull moves water + + +Earth is not held in place. The whole Earth falls toward the Moon with the pull at its centre, 3.320e-5 m/s². (Earth and Moon orbit each other, so this fall never ends in a collision.) The common part of the pull moves rock and water together, so it changes nothing on the surface. +What moves water relative to Earth is the **tidal acceleration**: the pull at a point minus the pull at Earth's centre. That is the last column of the table. + + +- On the near side, the water is pulled more than Earth's centre. It falls toward the Moon a little faster than Earth does: a bulge toward the Moon. +- On the far side, the water is pulled less than Earth's centre. Earth falls toward the Moon a little faster than this water. The water falls behind: a bulge away from the Moon. + + +Nothing pushes the far water. It is pulled toward the Moon, but less than the rest of Earth. + +```text + far bulge near bulge + ___ ___ + (relative) <- / \ EARTH / \ -> ( MOON ) + \___/ centre \___/ + pulled less than centre pulled more than centre +``` + +The tidal acceleration is about 1.1e-6 m/s², about one ten-millionth of g (9.81 m/s²). In an ideal ocean, it raises the water by about 0.5 m from low to high. + +## Evidence that the difference is what counts: the Sun + +The Sun's pull on Earth is 179 times the Moon's. If the raw pull made tides, solar tides would dwarf lunar tides. They do not: solar tides are 0.46 of lunar tides. A difference across a fixed width falls off as 1/d³, not 1/d², and that punishes the distant Sun. + +## Why two highs, and why not exactly every 12 hours + + +The two bulges stay lined up with the Moon. Earth turns under them. A point on the coast passes the near bulge, then the far bulge half a turn later. + +The Moon also moves along its orbit, about 12° per day, in the same direction as Earth turns. Earth must turn a little extra to bring the point back under the Moon. So the cycle is a **lunar day** of 24 h 50 min, and high tides come about every 12 h 25 min. High tide comes about 50 minutes later each day. + +## Limits + +- **Assumption:** this is the idealized "equilibrium tide": one ocean covers Earth and adjusts at once. Real oceans have continents and basins that resonate. Some coasts, such as parts of the Gulf of Mexico, get only one high tide a day. +- **Not covered:** spring and neap tides, where the Sun's smaller pair of bulges adds to or partly cancels the Moon's. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/model.md new file mode 100644 index 0000000..3180289 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/model.md @@ -0,0 +1,100 @@ +# Semantic model: two high tides a day + +> Source of truth. Every rendering derives from this file. + +## Central question + +Why does a point on the coast get two high tides a day, not one, when the Moon is on only one side of Earth? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (Newton's law of gravity, inverse-square, acceleration), basic programming. New to tides. Knows that the Moon causes tides. Does not know that only the *difference* in pull matters, or why there is a bulge on the side away from the Moon. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Moon's pull | Gravitational acceleration toward the Moon, GM/r², at a given point. | fact | +| Earth's centre | The point whose acceleration describes how Earth as a whole falls toward the Moon. | fact | +| Tidal acceleration | Moon's pull at a point minus Moon's pull at Earth's centre. | definition | +| Near bulge | Ocean bulge on the side facing the Moon. | fact (idealized) | +| Far bulge | Ocean bulge on the side away from the Moon. | fact (idealized) | +| Lunar day | Time for a point on Earth to return under the Moon: 24 h 50 min. | fact | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| Moon's pull | falls off with distance (1/r²) | near side > centre > far side | | +| Earth's centre | sets the pull that the whole Earth follows | tidal acceleration | uniform part gives no tide | +| Tidal acceleration | points away from Earth's centre on both sides | near bulge, far bulge | | +| Earth's rotation | carries a point through both bulges | two high tides per lunar day | | + +## Causal chain + +```text +pull falls with distance → near side pulled more than centre, far side less +→ relative to Earth's centre, near water moves toward Moon, far water lags behind (away) +→ two bulges on opposite sides → Earth rotates under both → two high tides per 24 h 50 min +``` + +## Quantities + +Constants: G = 6.674e-11, Moon mass 7.35e22 kg, distance 384,400 km, Earth radius 6,371 km. + +| Point | Distance to Moon | Moon's pull (m/s²) | Minus centre (m/s²) | +|---|---|---|---| +| Near side | 378,029 km | 3.433e-5 | +1.13e-6 (toward Moon) | +| Centre | 384,400 km | 3.320e-5 | 0 | +| Far side | 390,771 km | 3.213e-5 | −1.07e-6 (away from Moon, relative) | + +Approximation: tidal acceleration ≈ 2GMR/d³ = 1.10e-6 m/s², about one ten-millionth of g (9.81). + +Sun check: Sun's pull on Earth 5.93e-3 m/s², 179× the Moon's; Sun's tidal acceleration 5.05e-7 m/s², 0.46 of the Moon's. Tide follows 1/d³, the difference, not 1/d², the pull. + +Timing: Moon moves about 12.2° per day along its orbit, so a lunar day is 24 h 50 min, and high tides come every 12 h 25 min. + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Subtract the pull at Earth's centre | Use the raw pull: water flows toward the Moon | Raw pull is toward the Moon everywhere (3.2e-5 to 3.4e-5), predicting one bulge and one high tide a day. Observation: two. Also predicts Sun tides 179× the Moon's; observed is about 0.46×. | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| Uniform pull makes no tide | Whole Earth accelerates at 3.320e-5 toward Moon; that part moves water and rock together. | If pull were the same at all points (no 1/r² fall-off), the difference would be 0: no bulges. | +| Two bulges → two highs | Rotation of 360° relative to the Moon in 24 h 50 min passes two bulges 180° apart: highs 12 h 25 min apart. | Real basins: some coasts (parts of the Gulf of Mexico) get one high tide a day, because basin shape and Moon's tilt change the response. | + +## Terms + +- **Pull**: the Moon's gravitational acceleration at one point. +- **Tidal acceleration**: pull at a point minus pull at Earth's centre. Small value (1e-6 m/s²) yet enough to raise water about half a metre in the idealized ocean. +- **Bulge**: region where water is higher than average. +- **Lunar day**: 24 h 50 min. + +## Epistemic status + +- **Established:** two bulges in the equilibrium model; lunar day 24 h 50 min; Sun/Moon ratio 0.46. +- **Mathematical consequence:** near/far tidal accelerations from Newton's law. +- **Assumed:** idealized ocean covering Earth, responding instantly (equilibrium tide). +- **Estimate:** idealized lunar tide range about 0.5 m. + +## Confusion points + +- "The Moon pulls water toward it, so one bulge." → Earth is pulled too; only the difference shapes the water. +- "Far bulge is from centrifugal force of Earth spinning." → Earth's daily spin is the same everywhere around the equator; it shapes Earth's equator, not the tides. The cause is the pull difference. +- "Far side is pushed away." → Nothing pushes. The far side is pulled toward the Moon, but less than Earth's centre, so it falls behind. + +## Scope + +- **Out of scope:** spring and neap tides (Sun and Moon aligned or at right angles) — pointer: add the Sun's bulges, 0.46 of the Moon's. +- **Out of scope:** real tide heights and timing (resonance, continents, Bay of Fundy) — pointer: dynamic tide theory, harmonic tide prediction. +- **Out of scope:** unequal pairs of highs (diurnal inequality) from the Moon's tilt. + +## Representation decision + +- **Stage:** 1 +- **Reason:** A short causal chain with one table of numbers; the geometry fits in one small text figure. +- **Levels:** L1 two bulges, Earth turns under them / L2 pull differs across Earth / L3 subtract the centre's pull, Sun check. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/model.yaml new file mode 100644 index 0000000..b6b7104 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/explainers/two-tides/model.yaml @@ -0,0 +1,63 @@ +# Machine-readable half of model.md. `explainer check two-tides` holds every rendering to it. + +values: + moon_distance_km: 384400 + earth_radius_km: 6371 + earth_diameter_km: 12742 + near_distance_km: 378029 + far_distance_km: 390771 + moon_deg_per_day: 12 + daily_delay_min: 50 + interval_hours: 12 + interval_extra_min: 25 + pull_near: 3.433e-5 + pull_centre: 3.320e-5 + pull_far: 3.213e-5 + tidal_near: 1.13e-6 + tidal_far: 1.07e-6 + g: 9.81 + sun_pull_ratio: 179 + sun_tidal_ratio: 0.46 + lunar_day: "24 h 50 min" + high_tide_interval: "12 h 25 min" + idealized_range_m: 0.5 + +allow: [1, 2, 3, 180, 360, 24] + +require: + - path: pull_near + in: [prose] + - path: pull_far + in: [prose] + - path: lunar_day + in: [prose] + - path: sun_tidal_ratio + in: [prose] + +claims: + - id: why-subtract + kind: why + statement: "Only the pull at a point minus the pull at Earth's centre moves water relative to Earth." + alternative: "Use the raw pull: water flows toward the Moon." + counterexample: "Raw pull points toward the Moon everywhere (3.213e-5 to 3.433e-5 m/s²): one bulge, one high tide a day, and Sun tides 179x the Moon's. Observed: two highs, Sun 0.46x." + ask: "Why is there a bulge on the side of Earth away from the Moon?" + - id: mech-bulges + kind: mechanism + statement: "The near side is pulled more than the centre and the far side less, so relative to the centre both sides move outward." + example: "Near +1.13e-6 m/s² toward the Moon; far 1.07e-6 m/s² away from the Moon, relative to the centre." + - id: mech-rotation + kind: mechanism + statement: "Earth rotates under the two bulges, so a point passes two highs per lunar day." + example: "Lunar day 24 h 50 min; highs every 12 h 25 min." + ask: "How long between two high tides, and why not 12 h exactly?" + - id: misc-push + kind: misconception + statement: "Nothing pushes the far side away; it is pulled less than Earth's centre and falls behind." + correction: "The far side is pulled toward the Moon by 3.213e-5 m/s², less than the centre's 3.320e-5 m/s², so it lags behind Earth's fall." + +accept_unjustified: {} + +terms: + - term: tidal acceleration + means: "pull at a point minus pull at Earth's centre" + avoid: [tidal force pushes, pushed away] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/read.json new file mode 100644 index 0000000..9118469 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/read.json @@ -0,0 +1,46 @@ +{ + "answers": { + "1": { + "answer": "Earth is pulled toward the Moon more strongly than the far-side water. Earth's center falls toward the Moon faster than the far water, so the water lags behind relative to Earth's center, creating a bulge away from the Moon.", + "quote": "On the far side, the water is pulled less than Earth's centre. Earth falls toward the Moon a little faster than this water. The water falls behind: a bulge away from the Moon." + }, + "2": { + "answer": "The Moon orbits and moves roughly 12° per day in the same direction Earth rotates. A point on Earth must rotate slightly extra to realign with the Moon, extending the cycle from 24 hours to 24 hours 50 minutes, making high tides about 12 hours 25 minutes apart.", + "quote": "The Moon also moves along its orbit, about 12° per day, in the same direction as Earth turns. Earth must turn a little extra to bring the point back under the Moon. So the cycle is a lunar day of 24 h 50 min, and high tides come about every 12 h 25 min." + }, + "3": { + "answer": "not covered", + "quote": "Not covered: spring and neap tides, where the Sun's smaller pair of bulges adds to or partly cancels the Moon's." + }, + "4": { + "answer": "Every 12 hours. The explanation indicates that two bulges exist and a point passes each bulge in half a turn. If no extra rotation were needed (Moon fixed), a full rotation takes 24 hours, so high tides would come every 12 hours, not the current 12 hours 25 minutes.", + "quote": "The two bulges stay lined up with the Moon. Earth turns under them. A point on the coast passes the near bulge, then the far bulge half a turn later." + } + }, + "stops": [ + "\"Only the difference in pull moves water\" — why subtract center's pull rather than use absolute pulls?", + "\"The water falls behind\" — confusing why falling slower creates a bulge away (needs reference frame clarity)", + "\"A difference across a fixed width falls off as 1/d³, not 1/d²\" — unexplained mathematical step", + "Diagram arrows pointing in opposite directions — hard to parse which direction is 'relative'", + "\"Earth is not held in place\" — what frame of reference makes the center's pull irrelevant?", + "\"The Moon also moves along its orbit, about 12° per day\" — why is this specific angle important?", + "Table with scientific notation (3.433e-5) — magnitude hard to grasp", + "\"A point on the coast passes the near bulge, then the far bulge half a turn later\" — needs mental visualization" + ], + "excess": [ + "\"Predict first\" — pedagogical instruction, not explanation", + "\"The tidal acceleration is about 1.1e-6 m/s² ... raises the water by about 0.5 m\" — specific numbers not needed for mechanism", + "\"Earth and Moon orbit each other, so this fall never ends in a collision\" — orbital mechanics context", + "\"High tide comes about 50 minutes later each day\" — repeats 12h 25min information", + "\"Some coasts, such as parts of the Gulf of Mexico, get only one high tide a day\" — off-topic example", + "\"Real oceans have continents and basins that resonate\" — additional complexity beyond basic mechanism", + "\"with the Moon at 384,400 km and Earth's radius 6,371 km\" — specific distances before table", + "\"The Moon's gravity gives an acceleration, here called the pull, of GM/r²\" — formula not essential to understand mechanism" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/run.json new file mode 100644 index 0000000..3953c0b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/two-tides/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/two-tides/explanation.md`, a prose explanation of under 600 words. Prose fits because the answer is one short chain of cause and effect: the Moon's pull weakens with distance, so only the difference from the pull at Earth's centre moves water, which gives two bulges. A table of worked numbers and one small text sketch cover the geometry, and Mermaid draws force arrows poorly. The supporting `model.md`/`model.yaml` are alongside it, and `explainer check two-tides` passes.", + "error": null, + "cost": 0.5251108, + "seconds": 114, + "turns": 27, + "files": [ + "explainers/two-tides/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/explainers/vaccine-memory/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/explainers/vaccine-memory/explanation.md new file mode 100644 index 0000000..07fed97 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/explainers/vaccine-memory/explanation.md @@ -0,0 +1,179 @@ +# How a vaccine prepares the immune system for a disease it has never met + +## The puzzle + +Your immune system can make antibodies that bind tightly to a virus that evolved last year. Nothing in your genome was written with that virus in mind. So how can the body "know" in advance what to look for? And how can a vaccine, which is not the disease, leave you protected against the disease? + +The short answer is that the immune system doesn't learn the way a student learns a fact. It works more like a search algorithm: + +1. **Generate** a huge random library of detectors before any threat appears. +2. **Select** the few detectors that happen to fit when a threat shows up. +3. **Amplify and refine** those detectors. +4. **Keep copies** of the best ones for next time. + +A vaccine runs steps 2 to 4 on purpose, using a harmless stand-in for the pathogen. When the real pathogen arrives, the body is already at step 4. + +The rest of this document goes through each step. + +--- + +## Step 0: What is actually being recognised + +A virus or bacterium is covered in molecules, mostly proteins. The immune system doesn't recognise "the measles virus" as a whole. It recognises small patches on these molecules, each a few amino acids across. A molecule the immune system can recognise is called an **antigen**, and the specific patch that a detector binds to is called an **epitope**. + +Recognition is physical. Shape and chemistry have to match closely enough that the two molecules stick together. Think of it as a lock and key, except that the fit is graded rather than all-or-nothing. A good fit sticks strongly (**high affinity**). A poor fit sticks weakly and comes off again. + +So the question "how does the body recognise a new virus?" turns into "how does the body come to have a molecule shaped to fit a patch on that virus?" + +--- + +## Step 1: Generate — a random library, built in advance + +The detectors are receptor proteins on the surface of two kinds of white blood cell, **B cells** and **T cells**. Each individual B or T cell carries one kind of receptor, many copies of it, and that receptor is different from almost every other cell's. + +You have only about 20,000 genes, yet you can make billions of different receptors. Here's how. The receptor genes are stored as *parts lists*, not finished genes. As each young B or T cell develops, it cuts and pastes its own DNA. It picks one segment at random from each of several pools (called V, D and J segments) and joins them. At the joins it adds or deletes a few random DNA letters. The receptor is built from two protein chains, each assembled this way, so the combinations multiply. + +In programming terms it's roughly: + +```python +def make_receptor(): + heavy = random.choice(V_heavy) + junk() + random.choice(D) + junk() + random.choice(J_heavy) + light = random.choice(V_light) + junk() + random.choice(J_light) + return (heavy, light) +``` + +The theoretical number of distinct receptors is astronomically large (estimates are often quoted above 10^13). The number actually present in a person at any time is smaller but still huge: billions of different B and T cell receptors. + +Two consequences matter here: + +- **The library exists before any infection.** Somewhere in your blood and lymph nodes there is probably already a handful of cells whose receptor happens to fit a patch on a virus you have never encountered. Nobody designed it. With enough random shapes, some will fit. +- **Each match is rare.** For any particular epitope, only a tiny fraction of cells fit it, perhaps a few in a million or fewer. A handful of cells can't fight an infection. That is the problem the next steps solve. + +There is also a safety filter. While they develop, cells whose receptors bind strongly to the body's *own* molecules are mostly deleted or switched off. If this filter fails, the result is autoimmune disease. + +--- + +## Step 2: Select — the rare matching cells are found and activated + +When an antigen enters the body, whether from an infection or a vaccine, it gets carried to the lymph nodes. Lymph nodes are meeting points where antigen and immune cells concentrate. Billions of B and T cells pass through them, in effect "trying" the antigen against their receptors. + +A cell whose receptor binds is **selected**. But binding alone is not enough to start a full response. The immune system needs a second signal that says "this is dangerous", not just "this is present": + +- The **innate immune system** is the older, built-in part. It doesn't need a library. It has fixed sensors for broad signs of infection, such as bits of bacterial cell wall, viral genetic material in the wrong place, or damaged tissue. When those sensors fire, they put out alarm signals. +- **Helper T cells**, which have recognised the same pathogen, give B cells permission to proceed. + +This two-key system helps prevent accidental attacks on harmless things. It is also why many vaccines include an **adjuvant**, an ingredient whose job is to trip the innate alarm. A purified protein on its own can look harmless and be ignored. The adjuvant supplies the "this is dangerous" signal. + +--- + +## Step 3: Amplify and refine + +### Amplify: clonal expansion + +A selected cell divides repeatedly, roughly once every 6 to 12 hours, for days. One cell becomes thousands, all carrying the same receptor (a **clone**). This is called **clonal selection**: the antigen doesn't create the receptor, it chooses which pre-existing cells get copied. + +Some of the expanded cells go to work right away: + +- **Plasma cells** are B cells turned into antibody factories. An **antibody** is the secreted, free-floating version of the B cell's receptor. Antibodies coat the pathogen. This blocks it from entering cells and marks it for destruction. +- **Killer (cytotoxic) T cells** find and destroy the body's own cells that have been infected. They recognise fragments of viral protein that infected cells display on their surface. + +### Refine: affinity maturation + +The first matches are usually mediocre. They bind, but not tightly. Inside lymph nodes, in structures called **germinal centres**, B cells run what amounts to an evolutionary algorithm: + +1. **Mutate.** The dividing B cells deliberately introduce random point mutations into their receptor genes, at a rate roughly a million times higher than the normal mutation rate. This is called **somatic hypermutation**. +2. **Compete.** The mutants compete for a limited supply of antigen and for help from T cells. Cells that bind more tightly get more of both. +3. **Survive or die.** Winners get survival signals and divide again. Losers die off. +4. **Repeat** over many rounds, for weeks. + +```python +population = selected_clones +for round in range(many): + population = [mutate(cell) for cell in population for _ in range(2)] + population = keep_best(population, by=binding_strength_to(antigen)) +``` + +The result is antibodies that bind the target much more tightly than the starting ones, often by factors of 10 to 1000 or more. B cells can also **class-switch**, changing the antibody's "tail" while keeping the binding end. This lets them make types suited to the blood (IgG) or to mucous surfaces (IgA), for example. + +--- + +## Step 4: Keep copies — memory + +When the antigen is cleared, most of the expanded army dies off. That's deliberate, since keeping it would waste resources. A minority survive as **memory cells**: + +- **Memory B cells** carry the refined, high-affinity receptors. +- **Memory T cells** (both helper and killer types). +- **Long-lived plasma cells** settle in the bone marrow and keep secreting a baseline level of antibody, sometimes for decades. + +Memory cells differ from the original naive cells in three ways that matter: + +| | Naive (before) | Memory (after) | +|---|---|---| +| How many cells fit this antigen | a handful | hundreds to thousands of times more | +| Quality of fit | mediocre | refined by affinity maturation | +| Activation | needs strong signals, slow | responds faster, to less stimulus | + +This is the whole trick. The **first** encounter with an antigen produces a **primary response**. It is slow to start (useful antibody levels take around 1–2 weeks) and modest in size. A **second** encounter produces a **secondary response**. It starts within days, rises much higher, and uses better antibodies. + +``` +antibody +level + │ ╱‾‾‾‾‾╲ + │ ╱ ╲___ secondary + │ ╱ ‾‾‾─── + │ ╱ + │ primary ╱ + │ ╱‾‾╲ │ + │ _____╱ ╲_______ ____________ ╱ + └───┬──────────────────────────────┬────────────────▶ time + first exposure second exposure + (lag ~1-2 weeks) (lag ~days, higher, tighter binding) +``` + +--- + +## So what does a vaccine do? + +Everything above happens during any infection. The danger of a natural infection is that the primary response is **slow**. During those 1–2 weeks the pathogen is multiplying and can kill you, cripple you, or spread to others before the immune system catches up. + +A vaccine gives the immune system its *first encounter* under safe conditions. It shows the body the right antigens, with enough of an alarm signal to trigger a real response, but without a pathogen that can cause the disease. The primary response, with its slow lag, happens while nothing is at stake. When the real pathogen arrives, it meets a **secondary** response: antibodies already circulating, plus memory cells that expand within days. Often the infection is stopped before it causes symptoms, or kept mild. + +So "preparing the immune system for a disease it has never met" means this: the vaccine doesn't install new knowledge. It **selects, enlarges and sharpens** the part of an already-random library that fits this pathogen, and keeps it on standby. + +### Ways to show the antigen + +Different vaccine types are different ways of presenting the same antigens: + +| Type | What is injected | Trade-off | +|---|---|---| +| **Live attenuated** (e.g. measles, MMR) | A weakened strain that still replicates a little | Very realistic, strong long-lasting immunity; not suitable for people with weakened immune systems | +| **Inactivated** (e.g. many flu shots, polio IPV) | Whole pathogen, killed | Can't cause infection; usually needs adjuvant and boosters | +| **Subunit / protein** (e.g. hepatitis B, HPV) | Just one or a few purified proteins, sometimes assembled into virus-like particles | Very safe and precise; relies on adjuvant | +| **Toxoid** (e.g. tetanus, diphtheria) | Inactivated version of the toxin the bacterium makes | Protects against the toxin, which is what actually causes the disease | +| **mRNA / viral vector** (e.g. several COVID-19 vaccines) | Genetic instructions; your own cells briefly make the antigen protein | Fast to design and update; the instructions are temporary and break down within days | + +In every case, the immune system ends up seeing the target's shapes in a context that sets off the alarm, and the four steps run. + +--- + +## Why some details work the way they do + +**Why boosters?** Each additional exposure runs another round of selection and affinity maturation. The memory pool grows and the antibodies get better. Some antigens, especially non-replicating ones, give a weaker first response, so a second or third dose is needed to build a solid memory. + +**Why does protection sometimes fade?** Antibody levels in the blood fall over time. If protection depends on having antibodies *already present* when the pathogen arrives (as with infections of the nose and throat that move fast), it can wane even though memory cells remain. Memory cells usually still make illness less severe, because they shorten the lag. + +**Why do we need new flu vaccines each year?** Memory is specific to the epitopes it was selected on. Influenza mutates its surface proteins quickly (**antigenic drift**), so last year's memory cells fit this year's virus less well. Measles barely changes its key surface protein, which is one reason a childhood measles vaccine can protect for life. + +**Why doesn't it work for everything?** Some pathogens hide their vulnerable patches, change them constantly (HIV), or live inside cells where antibodies can't reach. For those, finding an antigen that produces protective memory is the hard part. + +**Herd immunity.** If enough people have memory, a pathogen can't find enough hosts to keep spreading. That protects people who can't be vaccinated or who respond poorly. + +--- + +## Summary + +- Your body makes billions of B and T cell receptors at random, **before** meeting any pathogen. A few happen to fit almost anything. +- When an antigen appears along with a danger signal, the rare matching cells are **selected** and copied by the thousands. +- In germinal centres, B cells **mutate and compete**, evolving antibodies that bind much more tightly. +- After the threat is gone, **memory cells** and long-lived antibody producers remain: more of them, better at binding, and quicker to respond. +- A vaccine triggers this process with a harmless form of the antigen. The slow first response happens safely, and the real disease meets the fast second response. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/read.json new file mode 100644 index 0000000..81e2378 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "A vaccine must show antigens (the specific shapes that the immune system recognizes) from the pathogen, but without the pathogen itself that causes disease.", + "quote": "It shows the body the right antigens, with enough of an alarm signal to trigger a real response, but without a pathogen that can cause the disease." + }, + "2": { + "answer": "After vaccination, memory cells and antibodies are already present. When the real pathogen arrives, these expanded memory cells respond quickly within days, rather than taking weeks to generate an initial response.", + "quote": "When the real pathogen arrives, it meets a secondary response: antibodies already circulating, plus memory cells that expand within days." + }, + "3": { + "answer": "Memory cells and antibodies are specific to the epitopes (patches) they were selected on. An unrelated virus has completely different epitopes, so the old vaccine memory does not recognize it.", + "quote": "Memory is specific to the epitopes it was selected on." + }, + "4": { + "answer": "Protection weakens or is lost because the memory cells no longer fit the changed epitope well. The shape mismatch prevents the antibodies from binding tightly.", + "quote": "Influenza mutates its surface proteins quickly (antigenic drift), so last year's memory cells fit this year's virus less well." + } + }, + "stops": [ + "It picks one segment at random from each of several pools (called V, D and J segments)", + "If this filter fails, the result is autoimmune disease.", + "The dividing B cells deliberately introduce random point mutations into their receptor genes", + "B cells can also class-switch, changing the antibody's \"tail\" while keeping the binding end.", + "But binding alone is not enough to start a full response.", + "Activation | needs strong signals, slow | responds faster, to less stimulus", + "Helper T cells, which have recognised the same pathogen, give B cells permission", + "Inside lymph nodes, in structures called germinal centres, B cells run what amounts to an evolutionary algorithm", + "The receptor is built from two protein chains, each assembled this way, so the combinations multiply.", + "estimates are often quoted above 10^13" + ], + "excess": [ + "def make_receptor(): heavy = random.choice(V_heavy) + junk() + random.choice(D)", + "population = [mutate(cell) for cell in population for _ in range(2)]", + "ASCII chart showing antibody levels over time with primary and secondary responses", + "Table of vaccine types: Live attenuated, Inactivated, Subunit, Toxoid, mRNA/viral vector", + "Why boosters? Each additional exposure runs another round of selection and affinity maturation.", + "Herd immunity. If enough people have memory, a pathogen can't find enough hosts to keep spreading.", + "Your body makes billions of B and T cell receptors at random, before meeting any pathogen.", + "the vaccine doesn't install new knowledge. It selects, enlarges and sharpens the part of an already-random library", + "Nobody designed it. With enough random shapes, some will fit.", + "In every case, the immune system ends up seeing the target's shapes in a context that sets off the alarm" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/run.json new file mode 100644 index 0000000..e6ea796 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/vaccine-memory/explanation.md`, a prose explainer with short Python sketches, tables and an ASCII graph of antibody levels, because the mechanism is a step-by-step process (random generation, then selection, then refinement through mutation and competition, then memory) that a reader who programs can follow best as a story with algorithm-like code.", + "error": null, + "cost": 0.16533019999999998, + "seconds": 62, + "turns": 2, + "files": [ + "explainers/vaccine-memory/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/explanation.md new file mode 100644 index 0000000..0017182 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/explanation.md @@ -0,0 +1,102 @@ +# How a vaccine prepares the immune system for a disease it has never met + +**Short answer.** The immune system does not design a defense after it meets a pathogen. Before any infection, it already holds a huge library of randomly shaped receptors. A vaccine shows the body harmless copies of a pathogen's shapes. The few cells that happen to match those shapes multiply, and a large group of them stays behind as *memory*. When the real pathogen arrives, the response starts from thousands of matching cells instead of a few, and it wins the race. + +The rest of this file explains each part of that sentence: the library, the selection, the multiplication, the memory, and the race. + +## Why this matters: infection is a race + +A virus copies itself exponentially. As an illustrative model, suppose an infection starts with 100 virus particles that double every 8 hours: + +| Time | Doublings | Virus particles | +|---|---|---| +| Day 0 | 0 | 100 | +| Day 3 | 9 | ~51,000 | +| Day 7 | 21 | ~210,000,000 | + +A first-time immune response takes about 7–14 days to produce large amounts of antibody. By then the virus has had a week of unopposed growth, and the person is sick. A response from memory takes about 2–4 days, and some antibody is already present on day 0. The vaccine's job is to move the defense from the slow column to the fast one. + +## Terms + +- **Antigen**: a molecular shape on a pathogen, for example a region of a surface protein, that a receptor can bind. +- **Lymphocyte**: a white blood cell that carries one receptor shape, in many copies on its surface. There are two main kinds: + - **B cells** make **antibodies**: free-floating copies of their receptor that stick to a pathogen and block it or mark it for destruction. + - **T cells** kill infected cells and help B cells activate. + +## Step 1: the library exists before the disease + +Each new lymphocyte builds its receptor gene by cutting and joining randomly chosen gene segments. The joins also add random letters. The result is that each lymphocyte carries a receptor shape that no one chose. Across the body's hundreds of billions of lymphocytes, the library contains millions of different shapes. + +This is the key to the question. The immune system can "know" a disease it has never met because it never needed to know it. It made a random library in advance, and some shapes in the library fit almost any antigen. + +A random library would also contain receptors that bind the body's own proteins. During development, cells that bind self strongly are mostly removed. This is why the library attacks foreign shapes and mostly ignores the body. + +The cost of a random library: any single antigen matches very few cells. Estimates are about 1 in 100,000 to 1 in 1,000,000 lymphocytes (estimate; varies by antigen and person). A few rare matching cells cannot stop a virus alone. That is why the first response is slow. + +## Step 2: the vaccine selects the matching cells + +A vaccine delivers the antigen without the disease. Common forms: + +- an inactivated (killed) pathogen; +- a weakened (attenuated) pathogen that copies itself poorly; +- a purified pathogen protein (subunit vaccine); +- mRNA or a harmless carrier virus that makes the body's own cells produce one pathogen protein for a short time. + +Antigen alone is not enough. A purified protein injected by itself often gives a weak response, because lymphocytes also need a **danger signal**. Fast, non-specific immune cells give this signal when they detect damage or microbial patterns. Vaccines supply it with an **adjuvant** (an added ingredient that triggers the signal) or through the vaccine's own microbial parts. Without the danger signal, the body treats the antigen as harmless and ignores it. + +With antigen and danger signal together, the rare matching lymphocytes bind the antigen and activate. Non-matching cells do nothing. This is **clonal selection**: the antigen picks its own responders from the library. + +## Step 3: the selected cells multiply, then most die + +An activated lymphocyte divides about every 8 hours (estimate). Its descendants all carry the same receptor. This is **clonal expansion**. + +Worked example (illustrative numbers): + +1. Start: 100 naive cells match the antigen. +2. Expand for 5 days: 15 doublings. 100 × 2¹⁵ ≈ **3.3 million** cells. +3. The antigen is cleared. About 90–95% of the expanded cells die (estimate). Keep 5%: **~164,000** cells. + +Those 164,000 cells are the **memory cells**. The pool is now about 1,600 times larger than the original 100. + +Now compare a second encounter. To reach 3.3 million cells again from 164,000 takes log₂(20) ≈ 4.3 doublings, about **1.4 days**. The first time it took 5 days. Memory cells also activate more easily than naive cells, so the real gap is larger than this arithmetic shows. + +## Step 4: B cells improve their fit + +During the response, activated B cells gather in lymph nodes and mutate their receptor genes at a high rate. Mutants that bind the antigen more tightly receive more survival signals and divide more. Mutants that bind worse die. This is **affinity maturation**: a small evolution process inside the lymph node, with the antigen as the selection test. + +The result is that memory B cells and their antibodies bind much more tightly than the original random match did. A second vaccine dose (a **booster**) runs this process again: it expands the memory pool further and selects even better binders. + +Some B cells also become **long-lived plasma cells** in the bone marrow. They secrete antibody continuously, in some cases for years. This antibody is already in the blood on day 0 of an infection, so it can block the pathogen before any cell has to activate. + +## Summary of the chain + +```text +random receptor library (before any exposure) + │ vaccine: antigen + danger signal + ▼ +rare matching cells activate (selection) + ▼ +matching cells multiply ~8 h/division (expansion) + ▼ +B cells mutate, best binders kept (affinity maturation) + ▼ +most cells die, 5–10% remain (memory pool ~1000× naive pool) + │ real infection + ▼ +large, pre-tuned pool + antibody already present → response in 2–4 days, not 7–14 +``` + +## Where it breaks + +Memory is specific to shapes. It protects only while the pathogen keeps those shapes. + +- **Holds:** measles keeps the same surface shapes. Two doses of measles vaccine give long, strong protection. +- **Breaks:** influenza changes its surface protein a little each year (**antigenic drift**). Last year's antibodies bind this year's virus poorly, so the race starts closer to the slow column. That is why the flu vaccine is updated every year. + +Protection also weakens when antibody levels fall over time. Memory cells often still remain, so a later infection may not be prevented but is usually milder, because the fast response still arrives before the virus reaches its peak. + +**Predict before you read on:** a pathogen keeps most of its surface but changes one region. Which antibodies still help? *Answer: those that bind the unchanged regions. That is why vaccine designers target protein regions that the pathogen cannot change without breaking its own function.* + +## Left out + +How cells display antigen to T cells (MHC molecules), herd immunity, vaccine side effects, and specific schedules. Each is a separate topic. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/model.md new file mode 100644 index 0000000..fe681e6 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/model.md @@ -0,0 +1,53 @@ +# Model: vaccine memory + +## Central question +How does a vaccine prepare the immune system for a disease it has never met? + +## Answer in one line +The immune system does not design a defense after it sees a pathogen. It already holds a huge random library of receptor shapes. A vaccine shows harmless pathogen shapes, and the few cells that match them multiply and leave a large, long-lived memory pool. + +## Reader +Technical, school science, new to immunology. Knows cells divide, proteins have shapes, exponential growth. + +## Entities +- Antigen: a molecular shape on a pathogen that a receptor can bind. +- Lymphocyte: white blood cell with one receptor shape. B cells (make antibodies) and T cells (kill infected cells, help B cells). +- Antibody: soluble copy of a B-cell receptor; binds antigen and blocks or marks the pathogen. +- Memory cell: lymphocyte left after a response; more numerous and faster to activate than the naive precursor. +- Long-lived plasma cell: B-cell descendant in bone marrow; secretes antibody for years. +- Adjuvant / danger signal: innate-immune activation that tells lymphocytes the antigen matters. + +## Causal chain +1. Random gene recombination makes each new lymphocyte carry a random receptor. Self-reactive ones are mostly removed. +2. Vaccine delivers antigen + danger signal. +3. Matching rare cells (~1 per 10^5–10^6, estimate) bind antigen and are activated (clonal selection). +4. Activated cells divide every ~8 h (clonal expansion). +5. B cells mutate receptors and the better binders are selected (affinity maturation). +6. Antigen is gone; ~90–95% of the expanded cells die (contraction); 5–10% stay as memory. +7. Real infection: memory pool is ~1000x larger and pre-tuned; preexisting antibody acts at once. + +## Worked numbers (illustrative) +- 100 naive matching cells, 15 doublings (5 days) → ~3.3 million; keep 5% → ~164,000 memory cells. +- Re-expansion to 3.3 million from 164,000 needs ~4.3 doublings ≈ 1.5 days, vs 5 days. +- Virus race: 100 particles doubling every 8 h → ~51,000 at day 3, ~2×10^8 at day 7. + +## Why each step +- Why a library and not design: receptors are made before exposure; no mechanism builds a receptor from the antigen. +- Why a danger signal: antigen alone is often ignored (tolerance); purified protein without adjuvant gives weak responses. +- Why harmless: inactivated, attenuated, protein subunit, mRNA, or viral vector give the shape without the disease. +- Why boosters: a second exposure expands memory further and adds affinity maturation. + +## Failure case +Memory binds specific shapes. Influenza changes its surface protein (antigenic drift); old antibodies bind poorly; new vaccine each year. Antibody levels also wane over time. + +## Confusion points +- "The vaccine teaches the body": no, it selects from what exists. +- Random library would attack self: developmental deletion (tolerance) removes most self-reactive cells. +- Memory does not always prevent infection; often it prevents severe disease. + +## Epistemic labels +- Precursor frequencies, % surviving contraction, division times: estimates, vary by antigen and person. +- Virus race numbers: illustrative model. + +## Scope (left out) +Antigen presentation (MHC) detail, herd immunity, side effects, specific vaccine schedules. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/model.yaml new file mode 100644 index 0000000..0319068 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/explainers/vaccine-memory/model.yaml @@ -0,0 +1,21 @@ +slug: vaccine-memory +precursor_frequency: "1 in 100,000 to 1 in 1,000,000" # estimate +division_time_hours: 8 # estimate, activated lymphocyte +example: + naive_matching_cells: 100 + expansion_days: 5 + doublings: 15 + peak_cells: 3276800 # 100 * 2^15 + surviving_fraction: 0.05 # estimate, 5-10% range + memory_cells: 163840 + fold_increase_over_naive: 1638 + re_expansion_doublings: 4.32 # log2(20) + re_expansion_days: 1.4 +virus_race: # illustrative + start_particles: 100 + doubling_hours: 8 + day3: 51200 + day7: 209715200 +primary_response_days: "7-14" +memory_response_days: "2-4" +failure_case: influenza antigenic drift diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/read.json new file mode 100644 index 0000000..60895ab --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/read.json @@ -0,0 +1,46 @@ +{ + "answers": { + "1": { + "answer": "A vaccine must contain or produce antigens—shapes from a pathogen—in harmless form (inactivated, weakened, or as a protein the body briefly makes). It doesn't make you ill because these are copies without the disease.", + "quote": "A vaccine delivers the antigen without the disease." + }, + "2": { + "answer": "Vaccination creates thousands of matching memory cells. When real infection arrives, the response starts from these pre-existing cells rather than a few rare ones, so it reaches needed numbers much faster (2–4 days instead of 7–14). Antibody is also already present from day 0.", + "quote": "When the real pathogen arrives, the response starts from thousands of matching cells instead of a few, and it wins the race." + }, + "3": { + "answer": "Memory cells recognize specific shapes. If a new virus has entirely different shapes, the memory cells from vaccination against a different virus won't match those shapes, so no protection exists.", + "quote": "Memory is specific to shapes. It protects only while the pathogen keeps those shapes." + }, + "4": { + "answer": "Protection weakens or is lost. The old antibodies bind poorly to the changed shape, so the immune response becomes slower again, making the race more difficult to win before illness begins.", + "quote": "Last year's antibodies bind this year's virus poorly, so the race starts closer to the slow column." + } + }, + "stops": [ + "\"danger signal\". Fast, non-specific immune cells give this signal when they detect damage or microbial patterns.", + "Each new lymphocyte builds its receptor gene by cutting and joining randomly chosen gene segments.", + "cells that bind self strongly are mostly removed.", + "adjuvant (an added ingredient that triggers the signal)", + "Mutants that bind the antigen more tightly receive more survival signals and divide more.", + "Memory cells also activate more easily than naive cells, so the real gap is larger than this arithmetic shows.", + "log₂(20) ≈ 4.3 doublings, about 1.4 days.", + "non-matching cells do nothing. This is clonal selection." + ], + "excess": [ + "| Day 0 | 0 | 100 | Day 3 | 9 | ~51,000 | Day 7 | 21 | ~210,000,000 |", + "Start: 100 naive cells match the antigen. Expand for 5 days: 15 doublings. 100 × 2¹⁵ ≈ 3.3 million cells.", + "Two doses of measles vaccine give long, strong protection.", + "Predict before you read on: a pathogen keeps most of its surface but changes one region.", + "Each is a separate topic.", + "Protection also weakens when antibody levels fall over time.", + "Measles keeps the same surface shapes.", + "estimates vary by antigen and person" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/run.json new file mode 100644 index 0000000..47211e1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/vaccine-memory/explanation.md`, plus `model.md` and `model.yaml` as the source facts behind it. I chose prose because the subject is one causal chain: random receptor library → selection → expansion → memory → a faster race against the virus. A short text diagram inside the file covers the overview, so an interactive page or separate diagram would not have added anything. I could not run `explainer check` because I have no shell here, so the numbers have not been automatically checked against `model.yaml`.", + "error": null, + "cost": 0.2302082, + "seconds": 68, + "turns": 4, + "files": [ + "explainers/vaccine-memory/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/explanation.md new file mode 100644 index 0000000..92ce5c5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/explanation.md @@ -0,0 +1,75 @@ +# How does a vaccine prepare the immune system for a disease it has never met? + + + +**A vaccine does not teach the body a new shape. The body already has a few cells that bind the virus or bacterium, made at random long before. The vaccine makes those few cells multiply once, without the disease, and many of their descendants stay. So when the real infection arrives, the body controls it in about 3 days instead of about 7.** + +Numbers marked *illustrative* are round numbers that show the mechanism; real values vary by disease and person. + +## Infection is a race + +A **pathogen** is a virus or bacterium that multiplies in the body and causes disease. Illustrative: an infection starts with 100 particles, the pathogen grows 10× per day, and symptoms start above 100 million particles. + +Immune cells also multiply. The body **controls** the infection when its immune cells remove the pathogen faster than the pathogen grows. So ask: what does a vaccine change in this race? + +## The matching cells exist before the pathogen does + + +A **lymphocyte** is a white blood cell that carries many copies of one kind of **receptor**. A receptor is a protein that binds one family of similar molecular shapes. The shape that a receptor binds on a pathogen is an **antigen**; for example, the spike protein on the surface of a coronavirus. + +Each new lymphocyte builds its receptor by joining pieces of genes in a random combination and adding random edits at the joins. So one person carries more than 100 million different receptors (estimate). One gene per receptor is impossible: humans have about 20,000 genes, and no gene list could anticipate a virus that evolved last year. + +Because each receptor binds a family of shapes, and a pathogen shows many different antigens, a few lymphocytes bind almost any new pathogen by chance. + + +This is the central idea. The body does not design a receptor after it sees the pathogen. The pathogen selects from a random library that already exists. + +## The first meeting is slow + + +When an antigen arrives, only the lymphocytes whose receptors bind it activate and divide. This is **clonal selection**. A **clone** is one lymphocyte and all its descendants; they share the same receptor. + +For one antigen, about 1 in 100,000 to 1 in 1,000,000 lymphocytes bind it (estimate). Illustrative: 100 cells. Call this number the **precursor count**: the lymphocytes able to respond at the start. Each activated cell divides about every 8 hours. + + +The clone's cells become **effector cells**, which fight now. Some release **antibodies**: free copies of their receptor that stick to the pathogen and block it. Others kill infected cells. One effector cell can release thousands of antibodies per second, so the response does not need as many cells as the pathogen has particles. Illustrative: 1,000,000 effector cells control the pathogen. For simplicity, both cases below use this one target. + + +How long does the first response take? Let n be the number of divisions. Then 100 × 2^n = 1,000,000, so 2^n = 10,000 and n ≈ 13.3. At 8 hours each, that is 4.4 days. The rare matching cells also need about 2.5 days to meet the antigen and activate. Total: about 7 days. This first response is the **primary response**. In 7 days the pathogen grows from 100 to 100 × 10^7 = 1 billion particles. That is above the symptom level of 100 million, so the person is sick. + +## What stays behind + + +After the pathogen is gone, about 99% of the effector cells die. About 1% stay for years or decades as **memory cells**. Illustrative: 1% of 1,000,000 is 10,000 memory cells, which is 100× the original precursor count of 100. + +At the next meeting with the same antigen, 10,000 × 2^n = 1,000,000 gives 2^n = 100, so n ≈ 6.6 divisions, or 2.2 days. Memory cells already wait in the tissues and need less stimulation, so they activate in under a day. Total: about 3 days. This faster response, which starts from memory cells, is the **secondary response**. + +| | First meeting | Same antigen again | +|---|---|---| +| Precursor count | 100 | 10,000 | +| Divisions to reach 1,000,000 | 13.3 | 6.6 | +| Days to control | about 7 | about 3 | +| Pathogen peak (from 100, ×10 per day) | 1 billion | 100,000 | +| Above 100 million (symptoms)? | yes | no, 1,000× below | + +The rule behind the table: each doubling of the precursor count saves one division, which is 8 hours. + +## What the vaccine does + + +A **vaccine** contains the antigen, or instructions for the body's cells to make it. It contains no pathogen that can multiply to disease. So the vaccine runs the slow first response while nothing is growing, and leaves the memory cells behind. When the real pathogen comes, the race starts with the secondary response, not the primary one. + + +Antigen alone is often not enough. A lymphocyte activates only when it binds its antigen *and* receives a **danger signal** from other immune cells that sense a threat. This rule stops attacks on harmless shapes such as food proteins. A purified protein alone gives no danger signal and often only a weak response. So many vaccines contain an **adjuvant**: an ingredient that triggers the danger signal. + +## When memory does not help + +Memory cells bind only the antigen that made them. Measles keeps the same antigens: two doses of measles vaccine prevent about 97% of cases, for decades. Influenza changes its surface antigens from year to year. The old memory receptors bind poorly, so the response falls back toward the slow first one. That is why flu vaccines prevent only about 40–60% of cases in a typical season and are updated every year. The cause is a changed antigen, not memory cells that wear off. + +A vaccine does not stop the pathogen from entering. It shortens the race. + +## Summary + +A vaccine prepares the body for a disease it has never met because the body does not need to have met it: the matching lymphocytes already exist by chance. The vaccine runs the slow primary response safely, so the real infection meets a secondary response that starts with 100× more cells. + +**Not covered here:** how receptors improve their fit over weeks (affinity maturation), how T cells help B cells, how antibody levels fall over months and why some vaccines need boosters, the types of vaccine (live, inactivated, protein, mRNA), and herd immunity. A standard textbook such as *Janeway's Immunobiology* covers each. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/model.md new file mode 100644 index 0000000..ef96a12 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/model.md @@ -0,0 +1,163 @@ +# Semantic model: how a vaccine prepares the immune system + +> Source of truth. Every rendering derives from this file. + +## Central question + +How does a vaccine prepare the immune system for a disease it has never met? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (cells, DNA and genes, proteins have shapes, exponential growth), basic programming (lookup, cache, random generation). New to immunology. Does not know: antigen, lymphocyte, receptor, clone, antibody, memory cell, adjuvant, primary/secondary response. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| pathogen | A virus or bacterium that multiplies in the body and causes the disease. | fact | +| antigen | A molecular shape on a pathogen (usually part of a surface protein) that a receptor can bind. | fact | +| lymphocyte | A white blood cell (B cell or T cell) that carries one kind of receptor, many copies of it. | fact | +| receptor | The protein on a lymphocyte surface that binds one antigen shape; each lymphocyte has a different, randomly generated receptor. | fact | +| naive lymphocyte | A lymphocyte that has not yet bound its antigen. | fact | +| clone | All lymphocytes descended from one lymphocyte; they share its receptor. | fact | +| clonal selection | Antigen activates only the lymphocytes whose receptors bind it; those multiply. | fact | +| effector cell | An activated lymphocyte that fights now: plasma cells release antibodies; killer T cells kill infected cells. | fact | +| antibody | A free-floating copy of a B cell receptor; it sticks to the antigen and blocks or tags the pathogen. | fact | +| memory cell | A long-lived lymphocyte left after a response, from an expanded clone, that responds fast to the same antigen. | fact | +| danger signal | A second signal from innate (built-in, non-specific) immune cells that says "this is a threat"; a lymphocyte needs it to activate. | fact (simplified) | +| adjuvant | An ingredient in a vaccine that triggers the danger signal. | fact | +| vaccine | A preparation that shows the antigen (and a danger signal) to the immune system without the pathogen's ability to cause disease. | fact | +| primary response | The response to an antigen met for the first time; it starts from naive lymphocytes. | fact | +| secondary response | The response to an antigen met again; it starts from memory cells and existing antibodies. | fact | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| gene-segment shuffling | generates, before any infection | receptor library | random; not guided by any pathogen | +| antigen + danger signal | activates | matching naive lymphocyte | clonal selection | +| activated lymphocyte | divides into | clone of effector cells | and some memory cells | +| memory cells | outnumber | naive precursors | illustrative 100× (estimate range 10–1,000×) | +| vaccine | runs | primary response | without disease | +| pathogen with same antigen | meets | secondary response | fast, large | + +## Dependencies (learning order) + +1. The pathogen grows exponentially; the defence is a race. +2. The receptor library exists before any infection (random generation). +3. Clonal selection: antigen picks the matching lymphocytes; they multiply. +4. The primary response is slow because the matching clone starts small. +5. After the response, memory cells remain, more of them than before. +6. A vaccine runs step 3–5 without the disease. +7. Memory protects only while the antigen stays the same. + +## Causal chain + +```text +random receptor library (before birth onward) + → vaccine shows antigen + danger signal + → matching rare lymphocytes activate and multiply (≈1 week) + → most effector cells die; memory cells + long-lived plasma cells remain + → real pathogen arrives → memory cells + antibodies respond in ≈3 days + → pathogen controlled before it grows enough to cause disease +``` + +## Temporal sequence + +1. Lymphocytes form continuously in bone marrow and thymus, each with a random receptor. +2. Day 0: vaccine. Days 0–3: antigen reaches lymph nodes, rare matching cells find it and activate. +3. Days 3–7: clone divides about every 8 hours. +4. Days 7–14: response peaks. +5. Weeks after: about 99% of effector cells die; memory cells remain for years to decades. +6. Later infection: secondary response in about 2–4 days. + +## Quantities + +All "illustrative" numbers are a round-number model to show the race, labelled as such. + +- Receptor library: more than 100 million different receptors in one person (estimate; true diversity likely higher). Built by randomly joining gene segments and editing the joins. +- Human genes: about 20,000. So one gene per receptor is impossible. +- Coverage: one receptor binds a family of similar shapes, and a pathogen carries many different antigens; so a random library of >10^8 receptors almost always contains some that fit. +- Self-binding: most new lymphocytes whose receptors bind the body's own molecules are removed before release (established; detail out of scope). +- Naive precursors for one antigen: roughly 1 in 100,000 to 1 in 1,000,000 lymphocytes (estimate). Illustrative precursor count: 100 cells. +- Division time of an activated lymphocyte: about 8 hours (3 per day). +- Why fewer cells can beat more virus: the response does not need to outnumber the pathogen. One plasma cell releases thousands of antibodies per second; one killer T cell kills many infected cells. Illustrative: 1,000,000 effector cells clear the pathogen faster than it grows ("control"). +- Primary: 100 × 2^n = 10^6 → n ≈ 13.3 divisions ≈ 4.4 days, + about 2.5 days to find antigen and activate ≈ 7 days. +- Contraction: about 99% of the 10^6 effector cells die; about 1% remain as memory cells (illustrative 10,000 = 100× the naive count; real ratio estimates 10–1,000×). +- Memory: 10,000 → 2^n = 100 → n ≈ 6.6 divisions ≈ 2.2 days, + about 0.5–1 day to activate (memory cells already sit in tissues and lymph nodes and need less stimulation) ≈ 3 days. +- Illustrative pathogen: starts at 100 particles, grows 10× per day; symptoms appear above about 10^8 particles. Primary: 7 days → 100 × 10^7 = 10^9 (sick; symptoms from day 6). Secondary: 3 days → 100 × 10^3 = 10^5, 1,000× below the symptom level. +- Meaning of the precursor count: low (100) → about 7 days to control; high (10,000) → about 3 days. Every 2× more precursors saves one division (8 h). +- Effectiveness: the percentage drop in disease among vaccinated people compared with unvaccinated people (e.g. 97%: 90 of 100 unvaccinated get measles after exposure, about 3 of 100 vaccinated do). +- Measles: two vaccine doses give about 97% protection, for decades (established). +- Influenza: vaccine effectiveness about 40–60% in a typical season, and a new vaccine each year (established range). + +## Alternative states + +| State | What changes | Why | +|---|---|---| +| no vaccine, first infection | primary response, ≈7 days | pathogen grows ≈10^7×; disease likely | +| vaccinated, same antigen | secondary response, ≈3 days, antibodies already present | pathogen grows ≈10^3× at most; often no disease | +| vaccinated, antigen changed (flu drift) | memory receptors bind poorly | response falls back toward primary | +| antigen without adjuvant | weak or no activation | missing danger signal | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| random receptor library | one gene per pathogen receptor | ~20,000 genes vs >100 million receptors needed; genes cannot list unknown future pathogens | +| clonal selection (multiply only the matching clone) | make all lymphocytes multiply | 100 million kinds × growth: the body cannot hold it; and most would attack nothing or self | +| need for danger signal / adjuvant | activate on any binding | would attack harmless food proteins and self; pure protein vaccines often give weak responses | +| vaccine runs the first response | wait for the real infection | real primary response takes ~7 days while pathogen grows 10^7× | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| memory makes the response fast | 10,000 vs 100 starting cells: 6.6 vs 13.3 divisions; ≈3 vs ≈7 days | if the antigen changes (flu drift), memory receptors no longer bind well; effectiveness 40–60% and a yearly new vaccine | +| memory lasts | measles: 2 doses ≈97% protection for decades | influenza: shape change, not memory loss, needs yearly update | +| clonal expansion math | 100 × 2^13.3 ≈ 10^6 | — | + +## Terms + +- pathogen: the thing that causes the disease (virus, bacterium). +- antigen: the molecular shape that a receptor binds (e.g. the SARS-CoV-2 spike protein). +- lymphocyte: B cell or T cell; each carries one kind of receptor. +- receptor: shape-binding protein on a lymphocyte; random; one kind per lymphocyte. +- naive lymphocyte: has not met its antigen. +- clone: a lymphocyte and its descendants, same receptor. +- clonal selection: antigen activates only matching lymphocytes, which multiply. +- effector cell: activated lymphocyte fighting now (plasma cell, killer T cell). +- antibody: a released copy of a B cell receptor. +- memory cell: long-lived lymphocyte that remains after a response. +- danger signal / adjuvant: the "this is a threat" second signal / the vaccine ingredient that provides it. +- primary / secondary response. +- precursor count: number of lymphocytes able to respond at the start (100 naive, illustrative; 10,000 memory). Low → slow response; high → fast. + +## Epistemic status + +- **Observed / established:** clonal selection; V(D)J recombination; memory cells; primary ≈1–2 weeks vs secondary ≈days; measles and flu effectiveness ranges. +- **Mathematical consequence:** division counts from 2^n. +- **Estimated:** library size, precursor frequency, memory-to-naive ratio. +- **Illustrative model:** 100 cells, 10^6 needed, pathogen 10×/day. +- **Simplified:** the danger signal (in reality several signals and T-cell help). + +## Confusion points + +- "The vaccine teaches the body the shape / the body builds an antibody to fit it." → No instruction: the matching receptor already exists by chance; the vaccine selects and multiplies it. +- "A vaccine is a weak infection that the body fights off, and immunity is the antibodies." → Antibodies fade; the lasting part is the larger pool of memory cells (plus long-lived plasma cells). +- "Vaccinated means you cannot be infected." → The pathogen can still enter; memory shortens the race so it is controlled sooner. +- "Flu vaccine fails because memory wears off." → Mainly the antigen changes. + +## Scope + +- **Out of scope:** antibody decay over months; why some vaccines need boosters. +- **Out of scope:** affinity maturation (B-cell receptors improve their fit over weeks) — germinal centres, immunology textbook (Janeway's Immunobiology). +- **Out of scope:** T-cell help, MHC presentation — same textbook. +- **Out of scope:** vaccine types (live attenuated, inactivated, subunit, mRNA) — they differ in how the antigen is delivered, not in the memory mechanism. +- **Out of scope:** herd immunity, boosters' exact schedule, safety. + +## Representation decision + +- **Stage:** 1 +- **Reason:** One causal chain with one before/after comparison; a table holds the two races. No parameter needs exploring beyond the one comparison. +- **Levels:** L1 answer / L2 the steps / L3 why random library, why memory is fast / L4 limits. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/model.yaml new file mode 100644 index 0000000..3ab38ef --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/model.yaml @@ -0,0 +1,111 @@ +# Machine-readable half of model.md. `explainer check vaccine-memory` holds every rendering to it. + +values: + library_size: "100 million" # different receptors in one person (estimate, lower bound) + human_genes: 20000 + naive_precursors: 100 # illustrative precursor count, first meeting + memory_precursors: 10000 # illustrative, 100x naive + memory_ratio: 100 + division_hours: 8 + cells_needed: 1000000 # illustrative effector cells for control + primary_divisions: 13.3 + primary_division_days: 4.4 + activation_days: 2.5 + memory_divisions: 6.6 + memory_division_days: 2.2 + primary_days: 7 + memory_days: 3 + pathogen_start: 100 # particles + pathogen_growth_per_day: 10 + symptom_level: "100 million" # particles, illustrative + primary_peak: "1 billion" + memory_peak: 100000 + precursor_freq_low: 100000 # 1 in 100,000 + precursor_freq_high: 1000000 + contraction_pct: 99 + measles_effectiveness_pct: 97 + flu_effectiveness_low: 40 + flu_effectiveness_high: 60 + +allow: [1, 2, 3, 6, 10, 1000] + +require: + - path: primary_days + in: [prose] + - path: memory_days + in: [prose] + - path: naive_precursors + in: [prose] + - path: memory_precursors + in: [prose] + - path: symptom_level + in: [prose] + +claims: + - id: library-before + kind: why + about: [receptor library] + statement: "The body makes a random library of receptors before any infection, so a matching lymphocyte already exists when a new antigen arrives." + alternative: "One gene for each receptor, written to fit known pathogens." + counterexample: "About 20,000 human genes cannot list more than 100 million receptors, and genes cannot anticipate a pathogen that has not evolved yet." + ask: "Before a person meets a brand-new virus, does their body already have lymphocytes that can bind it? Why?" + - id: clonal-selection + kind: mechanism + statement: "An antigen activates only the few lymphocytes whose receptors bind it; those divide every 8 hours into a clone." + example: "100 matching cells × 2^13.3 ≈ 1,000,000 effector cells after about 4.4 days of division." + - id: primary-slow + kind: mechanism + statement: "The first response is slow because the matching clone starts at about 100 cells; it needs about 7 days to control the pathogen." + example: "100 → 1,000,000 needs 13.3 divisions (4.4 days) plus about 2.5 days to find the antigen and activate: about 7 days. The pathogen grows from 100 to 1 billion particles, above the symptom level of 100 million." + - id: memory-fast + kind: guarantee + statement: "After a response, memory cells remain, 100× more than the naive count; the next response against the same antigen takes about 3 days." + example: "10,000 → 1,000,000 needs 6.6 divisions (2.2 days) plus under a day to activate: about 3 days. The pathogen peaks at 100,000 particles, 1,000× below the symptom level." + counterexample: "Influenza changes its surface antigens each year; old memory receptors bind poorly, so the response falls back toward the slow first response, and flu vaccine effectiveness is about 40–60%." + ask: "Why is the second response to the same antigen faster than the first?" + - id: vaccine-role + kind: mechanism + statement: "A vaccine shows the antigen with a danger signal but without a pathogen that can multiply to disease, so the slow first response runs at no cost and leaves memory cells." + example: "Measles vaccine, two doses: about 97% effectiveness for decades." + ask: "What does a vaccine actually leave behind in the body that protects a person years later?" + - id: why-adjuvant + kind: why + about: [adjuvant] + statement: "A lymphocyte activates only with a danger signal as well as its antigen; a vaccine adjuvant supplies that signal." + alternative: "Activate on any binding." + counterexample: "Lymphocytes would attack harmless food proteins; and a purified protein alone often gives only a weak response." + - id: selection-not-instruction + kind: misconception + statement: "The vaccine teaches the body the germ's shape, and the body builds a receptor to fit it." + correction: "The vaccine does not teach the body a shape; it selects and multiplies lymphocytes whose random receptors already fit." + example: "Of about 100 million receptors, only the about 100 matching cells divide." + - id: fewer-beats-more + kind: mechanism + statement: "The response wins with fewer cells than pathogen particles because each effector cell removes many particles." + example: "One plasma cell releases thousands of antibodies per second." + +accept_unjustified: {} + +terms: + - term: memory cell + means: "a long-lived lymphocyte left after a response; it is not antibody" + avoid: [memory antibodies, antibody memory] + - term: receptor + means: "the binding protein on a lymphocyte; an antibody is a released copy of a B cell receptor" + avoid: [antibody receptor] + - term: precursor count + means: "the number of lymphocytes able to respond at the start: 100 naive, 10,000 memory" + avoid: [head start count, starting cells] + +budget: {prose: 1000, reason: "the answer needs a random library, a race with two sets of numbers, and a case where memory fails"} + +quiz: + - q: "A person gets a vaccine against a virus whose antigen does not change. Two years later the virus enters their body. Roughly how much faster is their immune response than in an unvaccinated person, and why?" + expect: "About 3 days instead of about 7, because memory cells start from about 100× more matching cells (fewer divisions needed) and activate faster; so the virus stays below the symptom level." + misconception: "The antibodies made two years ago are still there and kill the virus instantly; or the vaccine prevents the virus from entering." + - q: "Why do people need a new flu vaccine each year, while a measles vaccine protects for decades?" + expect: "The flu virus changes its surface antigens, so old memory cells' receptors no longer fit well; measles antigens stay the same." + misconception: "Flu vaccine memory wears off quickly." + - q: "How can the immune system have a matching lymphocyte for a virus that appeared only last year?" + expect: "The receptors are made at random, more than 100 million kinds, before any infection; by chance a few fit almost any shape. The virus selects them." + misconception: "The immune system designs a new receptor to fit the virus after seeing it." diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/narrative.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/narrative.md new file mode 100644 index 0000000..6c3e59f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/narrative.md @@ -0,0 +1,56 @@ +# Narrative: vaccine-memory + +## 1. Reader before and after + +- **Before:** A technical reader: school science (cells, DNA, genes, proteins with shapes, exponential growth) and basic programming. Knows "immune system" and "antibody" only as everyday words. Does not know lymphocyte, receptor, antigen, clone, memory cell, adjuvant. +- **After:** "My body already holds, by chance, a few cells that fit almost any germ; a vaccine makes those few multiply once, safely, and leaves many more of them behind, so the real infection is stopped in about 3 days instead of about 7." + +## 2. Question and motive + +- **Question:** How can a vaccine prepare the body for a disease it has never met? Result in words: the vaccine does not teach a shape. The body already has a matching cell by chance. The vaccine multiplies it once, safely, and the extra cells stay. +- **Why care:** It sounds impossible: the body cannot know the shape of a virus that evolved last year. +- **Why this approach:** Treat infection as a race between two exponential growths. Then ask what changes the result of the race. + +## 3. Introduction ledger + +| Reference | Means | Grounded by | Beat | +|---|---|---|---| +| pathogen | virus or bacterium that multiplies and causes disease | 100 particles, ×10 per day | 2 | +| antigen | molecular shape on the pathogen that a receptor binds | the spike protein of SARS-CoV-2 | 3 | +| lymphocyte | white blood cell (B or T cell) carrying one kind of receptor | — one cell, one receptor kind | 3 | +| receptor | binding protein on a lymphocyte, made at random | >100 million kinds in one person | 3 | +| clone | a lymphocyte and its descendants, same receptor | 100 cells → 1,000,000 | 4 | +| clonal selection | antigen makes only matching lymphocytes divide | 100 of 100 million kinds divide | 4 | +| effector cell | activated lymphocyte that fights now | plasma cell, killer T cell | 4 | +| antibody | released copy of a B cell receptor | thousands per second from one plasma cell | 4 | +| precursor count | lymphocytes able to respond at the start | 100 naive vs 10,000 memory | 5 | +| memory cell | long-lived lymphocyte left after a response | 10,000 = 1% of 1,000,000 | 6 | +| danger signal / adjuvant | second signal "this is a threat" / vaccine ingredient giving it | food proteins do not trigger a response | 7 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How can it prepare for something unknown? | — | answer in bold | "The vaccine does not teach the body a new shape. The body already has a few cells that fit it, made at random. The vaccine makes those cells multiply once, without the disease, and many of them stay. So the next time, the body controls the germ in about 3 days instead of about 7." | the result, in words | +| 2 | approach | What decides if I get sick? | so | pathogen numbers | "A pathogen (a virus or bacterium) multiplies in the body. Illustrative: it starts at 100 particles and grows 10× per day; symptoms start above 100 million particles. Immune cells multiply too. Whichever side gets big enough first wins: infection is a race between the pathogen's growth and the immune cells' growth." | the race and both runners | +| 3 | mechanism | Which immune cells, and how can one fit an unknown germ? | but | library numbers | "A lymphocyte is a white blood cell with one kind of receptor: a protein that binds one family of similar molecular shapes. The shape it binds on a pathogen is an antigen (e.g. the spike protein of a coronavirus). Each new lymphocyte assembles its receptor by joining gene pieces at random, so one person has more than 100 million kinds. The ~20,000 genes could not list them one by one. Because each receptor fits a family of shapes, and a pathogen shows many antigens, a few lymphocytes fit almost any new pathogen by chance." | lymphocyte, receptor, antigen; the library exists in advance | +| 4 | mechanism | What happens when the antigen arrives? | so | clone arithmetic | "Only the lymphocytes whose receptors bind the antigen activate (clonal selection). For one antigen, about 1 in 100,000 to 1 in 1,000,000 lymphocytes fit: illustrative, 100 cells. Each divides every 8 hours. A clone is a cell and all its descendants, with the same receptor. The clone's cells become effector cells: plasma cells release antibodies (free copies of the receptor), killer T cells kill infected cells. Each effector cell removes many particles, so about 1,000,000 effector cells (illustrative) clear the pathogen faster than it grows." | clonal selection, clone, effector cell, antibody, target 1,000,000 | +| 5 | mechanism | Why is the first time slow? | but | 100 × 2^n = 1,000,000; 7 days; 1 billion particles | "Here n is the number of divisions: 100 × 2^n = 1,000,000 gives n ≈ 13.3, which is 4.4 days at 8 hours each. Finding the antigen and activating take about 2.5 days more: about 7 days. By then the pathogen is at 100 × 10^7 = 1 billion particles, above the 100-million symptom level: the person is sick." | primary response | +| 6 | mechanism | What is left after? | therefore | 99% die, 1% stay: 10,000 | "After the pathogen is gone, about 99% of the effector cells die. About 1% stay for years as memory cells: 10,000, which is 100× the original 100. Next time, 10,000 × 2^n = 1,000,000 gives n ≈ 6.6 divisions, 2.2 days; memory cells also activate in under a day: about 3 days. The pathogen reaches only 100,000 particles, 1,000× below the symptom level." | memory cell, secondary response | +| 7 | name | So what does the vaccine do? | therefore | vaccine = antigen + adjuvant, no disease | "A vaccine contains the antigen, or instructions to make it, but no pathogen that can multiply to disease. It also needs a danger signal: lymphocytes activate only if other immune cells also signal 'threat'; this stops attacks on harmless food proteins. The vaccine ingredient that gives this signal is an adjuvant. So the vaccine runs the slow first response (beats 4–6) safely, and leaves the memory cells." | the answer | +| 8 | objection | When does it fail? | but | flu vs measles | "Memory works only for the same antigen. Influenza changes its surface antigens each year, so old receptors fit poorly and the response falls back toward the slow first one; flu vaccines prevent about 40–60% of cases and are updated yearly. Measles antigens stay the same: two doses prevent about 97% of cases for decades." | limit | +| 9 | close | — | so | recap | answer in the opening words; each 2× more precursors saves one 8-hour division | after-sentence | + +## 5. Concrete to symbol + +100 × 2^n = 1,000,000 → n ≈ 13.3. Said: "here n is the number of divisions". 10,000 × 2^n = 1,000,000 → n ≈ 6.6. + +## 6. Links between representations + +One table: first meeting vs vaccinated, same rows (precursor count, divisions, days, pathogen peak, sick?). + +## 7. Close the loop + +- **Answer:** A vaccine prepares the body by running the slow first response safely, so the cells that fit are already many when the disease arrives. +- **General argument:** Memory cuts the precursor gap; each 2× more precursors saves one 8-hour division. +- **Payoff:** Predict when a vaccine will fail (antigen change) and why boosters or yearly updates exist. Scope pointers. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/decisions.yaml new file mode 100644 index 0000000..32a2751 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/decisions.yaml @@ -0,0 +1,193 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T145746-probe#1: + decision: adopted +2026-10-04T145746-probe#10: + decision: adopted +2026-10-04T145746-probe#11: + decision: adopted +2026-10-04T145746-probe#12: + decision: adopted +2026-10-04T145746-probe#13: + decision: adopted +2026-10-04T145746-probe#14: + decision: adopted +2026-10-04T145746-probe#15: + decision: declined +2026-10-04T145746-probe#16: + decision: declined +2026-10-04T145746-probe#2: + decision: adopted +2026-10-04T145746-probe#3: + decision: adopted +2026-10-04T145746-probe#4: + decision: adopted +2026-10-04T145746-probe#5: + decision: declined +2026-10-04T145746-probe#6: + decision: adopted +2026-10-04T145746-probe#7: + decision: adopted +2026-10-04T145746-probe#8: + decision: declined +2026-10-04T145746-probe#9: + decision: adopted +2026-10-04T150002-coldread-narrative#1: + decision: adopted +2026-10-04T150002-coldread-narrative#10: + decision: declined +2026-10-04T150002-coldread-narrative#11: + decision: declined +2026-10-04T150002-coldread-narrative#12: + decision: declined +2026-10-04T150002-coldread-narrative#13: + decision: adopted +2026-10-04T150002-coldread-narrative#14: + decision: adopted +2026-10-04T150002-coldread-narrative#15: + decision: declined +2026-10-04T150002-coldread-narrative#16: + decision: declined +2026-10-04T150002-coldread-narrative#17: + decision: adopted +2026-10-04T150002-coldread-narrative#18: + decision: adopted +2026-10-04T150002-coldread-narrative#19: + decision: declined +2026-10-04T150002-coldread-narrative#2: + decision: adopted +2026-10-04T150002-coldread-narrative#20: + decision: declined +2026-10-04T150002-coldread-narrative#21: + decision: declined +2026-10-04T150002-coldread-narrative#22: + decision: declined +2026-10-04T150002-coldread-narrative#23: + decision: declined +2026-10-04T150002-coldread-narrative#24: + decision: adopted +2026-10-04T150002-coldread-narrative#25: + decision: adopted +2026-10-04T150002-coldread-narrative#26: + decision: adopted +2026-10-04T150002-coldread-narrative#27: + decision: declined +2026-10-04T150002-coldread-narrative#28: + decision: declined +2026-10-04T150002-coldread-narrative#29: + decision: declined +2026-10-04T150002-coldread-narrative#3: + decision: adopted +2026-10-04T150002-coldread-narrative#30: + decision: adopted +2026-10-04T150002-coldread-narrative#31: + decision: adopted +2026-10-04T150002-coldread-narrative#32: + decision: adopted +2026-10-04T150002-coldread-narrative#33: + decision: adopted +2026-10-04T150002-coldread-narrative#34: + decision: adopted +2026-10-04T150002-coldread-narrative#35: + decision: adopted +2026-10-04T150002-coldread-narrative#36: + decision: adopted +2026-10-04T150002-coldread-narrative#37: + decision: adopted +2026-10-04T150002-coldread-narrative#38: + decision: declined +2026-10-04T150002-coldread-narrative#39: + decision: adopted +2026-10-04T150002-coldread-narrative#4: + decision: adopted +2026-10-04T150002-coldread-narrative#40: + decision: declined +2026-10-04T150002-coldread-narrative#41: + decision: declined +2026-10-04T150002-coldread-narrative#42: + decision: adopted +2026-10-04T150002-coldread-narrative#43: + decision: adopted +2026-10-04T150002-coldread-narrative#44: + decision: declined +2026-10-04T150002-coldread-narrative#45: + decision: declined +2026-10-04T150002-coldread-narrative#46: + decision: adopted +2026-10-04T150002-coldread-narrative#47: + decision: adopted +2026-10-04T150002-coldread-narrative#48: + decision: adopted +2026-10-04T150002-coldread-narrative#5: + decision: adopted +2026-10-04T150002-coldread-narrative#6: + decision: adopted +2026-10-04T150002-coldread-narrative#7: + decision: adopted +2026-10-04T150002-coldread-narrative#8: + decision: adopted +2026-10-04T150002-coldread-narrative#9: + decision: declined +2026-10-04T150207-coldread-prose#1: + decision: adopted +2026-10-04T150207-coldread-prose#10: + decision: declined +2026-10-04T150207-coldread-prose#2: + decision: adopted +2026-10-04T150207-coldread-prose#3: + decision: adopted +2026-10-04T150207-coldread-prose#4: + decision: adopted +2026-10-04T150207-coldread-prose#5: + decision: adopted +2026-10-04T150207-coldread-prose#6: + decision: adopted +2026-10-04T150207-coldread-prose#7: + decision: adopted +2026-10-04T150207-coldread-prose#8: + decision: declined +2026-10-04T150207-coldread-prose#9: + decision: adopted +2026-10-04T150328-quiz-prose#1: + decision: adopted +2026-10-04T150328-quiz-prose#10: + decision: declined +2026-10-04T150328-quiz-prose#11: + decision: declined +2026-10-04T150328-quiz-prose#12: + decision: declined +2026-10-04T150328-quiz-prose#13: + decision: declined +2026-10-04T150328-quiz-prose#14: + decision: adopted +2026-10-04T150328-quiz-prose#15: + decision: declined +2026-10-04T150328-quiz-prose#16: + decision: adopted +2026-10-04T150328-quiz-prose#17: + decision: declined +2026-10-04T150328-quiz-prose#18: + decision: declined +2026-10-04T150328-quiz-prose#19: + decision: declined +2026-10-04T150328-quiz-prose#2: + decision: adopted +2026-10-04T150328-quiz-prose#20: + decision: declined +2026-10-04T150328-quiz-prose#21: + decision: adopted +2026-10-04T150328-quiz-prose#22: + decision: declined +2026-10-04T150328-quiz-prose#3: + decision: adopted +2026-10-04T150328-quiz-prose#4: + decision: adopted +2026-10-04T150328-quiz-prose#5: + decision: declined +2026-10-04T150328-quiz-prose#6: + decision: declined +2026-10-04T150328-quiz-prose#7: + decision: declined +2026-10-04T150328-quiz-prose#8: + decision: adopted +2026-10-04T150328-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T145746-probe.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T145746-probe.json new file mode 100644 index 0000000..43d8b4b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T145746-probe.json @@ -0,0 +1,231 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "vaccine-memory", + "at": "2026-10-04T14:57:46", + "model": "default", + "prompt_sha": "74ac0a8c8132", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "The race argument: the clone grows more slowly than the pathogen", + "why_it_matters": "In the file's own numbers the clone triples its doublings each day (2^3 = 8× per day) while the pathogen grows 10× per day. With these numbers the clone never catches up. The file never says why the defence wins, so the core 'race' story doesn't hold together as written.", + "suggested_addition": "Say that one effector cell or antibody removes many pathogen units, so the relevant comparison is killing capacity, not cell count. Illustrative example: if each of the 10^6 effector cells clears 100 virus particles per day, they clear 10^8 per day. A pathogen load of 10^7 then shrinks. Also give a stopping threshold: 'control' means clearance per day exceeds growth per day." + }, + { + "rule": 5, + "severity": "main", + "about": "Pathogen growth of 10^7× vs 10^3× has no disease threshold or starting dose", + "why_it_matters": "The reader can't tell what 10^3× means for symptoms. Without a starting dose and a threshold, 'often no disease' is just asserted.", + "suggested_addition": "Illustrative: the infection starts with 100 particles and symptoms begin around 10^8. Primary response: 100 × 10^7 = 10^9, which is above the threshold, so the person gets sick. Secondary response: 100 × 10^3 = 10^5, which is 1,000× below the threshold, so infection happens with no disease." + }, + { + "rule": 2, + "severity": "main", + "about": "The claim that 'the matching receptor already exists by chance' has no coverage argument", + "why_it_matters": "A learner will ask how 10^8 random receptors can cover every possible antigen shape. The missing step is that each receptor binds many similar shapes (cross-reactivity) and each pathogen carries many antigen shapes. There is also no case where the claim fails.", + "suggested_addition": "Example: if one receptor binds about 1 in 10^5 random shapes, then a library of 10^8 receptors has about 1,000 binders for any shape. A pathogen with 10 separate antigen spots is almost certain to be hit. Breaking case: a small library, such as 10^4 receptors (low diversity in the very old or very young), gives about 0.1 expected binders per shape, so responses can be missing." + }, + { + "rule": 2, + "severity": "main", + "about": "The contraction numbers contradict the memory count", + "why_it_matters": "If 90–95% of 10^6 effector cells die, 50,000–100,000 cells are left, not 10,000. The ratio would then be 500–1,000× the naive count, which is outside the stated 10–100× estimate.", + "suggested_addition": "Either say that only about 1% become memory (10^6 → 10^4), or rework the example: 5% of 10^6 = 50,000. Also show the low end of the estimate: at 10× the naive count (1,000 cells), 2^n = 1,000 gives n ≈ 10 divisions ≈ 3.3 days, plus 0.75 day, ≈ 4 days." + }, + { + "rule": 1, + "severity": "edge", + "about": "'Keep all effector cells forever' fails on the file's own numbers", + "why_it_matters": "10^6 cells × 1,000 infections = 10^9 cells, which is about 0.1% of roughly 10^12 lymphocytes. So the stated 'space runs out' argument doesn't work at these numbers.", + "suggested_addition": "Use real expansion sizes: one response can make 10^7–10^9 cells, up to about 10% of all killer T cells. Add that effector cells are short-lived and cause inflammation. Example: 100 infections × 10% each would exceed the whole pool." + }, + { + "rule": 1, + "severity": "edge", + "about": "Why memory cells activate faster (0.5–1 day vs 2.5 days)", + "why_it_matters": "The ≈3-day result depends on this unexplained shortcut as well as on the head start.", + "suggested_addition": "Say that memory cells already sit in tissues and lymph nodes and need less stimulation to activate. Show the time cost of not having this: 2.2 days of division + 2.5 days of activation = 4.7 days instead of about 3." + }, + { + "rule": 5, + "severity": "edge", + "about": "Measles '97% protection' vs influenza '40–60% effectiveness' are undefined and use two different names", + "why_it_matters": "The reader doesn't know what the percentage measures, so they can't compare the two vaccines.", + "suggested_addition": "Define it once as the reduction in risk of disease, vaccinated vs unvaccinated. Example: if 90 of 100 unvaccinated exposed people get measles, 97% means about 2.7 of 100 vaccinated people do. For flu at 50%: if 10 of 100 unvaccinated people get flu, 5 of 100 vaccinated people do. Use one term throughout." + }, + { + "rule": 5, + "severity": "edge", + "about": "Pre-existing antibodies in the secondary response are not quantified", + "why_it_matters": "The 'Alternative states' table says antibodies are already present, but the model gives no number for their effect and never says how fast they fade.", + "suggested_addition": "Example: antibody levels halve about every 3 weeks (illustrative), so after 1 year about 2^-17 of the peak remains. Long-lived plasma cells keep a low baseline that neutralises part of the starting dose, e.g. 100 → 10 particles." + }, + { + "rule": 3, + "severity": "edge", + "about": "Terms used without definition: long-lived plasma cell, heavy/light chain, V/D/J, affinity maturation, precursor", + "why_it_matters": "These appear in the causal chain, Quantities and Relationships before they are defined (or are never defined). 'Naive precursors' appears in Relationships, but 'precursor count' is defined only in the Terms section.", + "suggested_addition": "Add entity rows: long-lived plasma cell (keeps releasing antibodies for years); heavy and light chain (the two parts of a receptor, each assembled from gene segments); precursor (a lymphocyte able to respond at the start: 100 naive, 10,000 memory)." + }, + { + "rule": 3, + "severity": "edge", + "about": "Concepts with more than one name", + "why_it_matters": "Several names for one concept make readers think there are several concepts.", + "suggested_addition": "Use one name for each: 'danger signal' (not also 'second signal' or 'signals'); 'precursor count' (not also 'starting cells' or 'head-start'); 'receptor fit' (not also 'affinity'); 'protection' (not also 'effectiveness')." + }, + { + "rule": 4, + "severity": "edge", + "about": "The clonal selection step has no worked example with numbers", + "why_it_matters": "The step that answers the central question gets no arithmetic.", + "suggested_addition": "Example: a lymph node scans about 10^7 lymphocytes. At a frequency of 1 in 10^5, about 100 bind the vaccine antigen. Only those 100 divide; the other 99.999% stay idle." + }, + { + "rule": 2, + "severity": "edge", + "about": "Self-reactivity of a random library", + "why_it_matters": "A random library must also make receptors that bind the body's own proteins. Only the danger signal is offered as protection, and the removal of self-binding cells during development is never mentioned.", + "suggested_addition": "Add that most newly made lymphocytes that bind self proteins are removed before release. For T cells, roughly 90%+ of developing cells die in the thymus (estimate)." + } + ], + "excess": [ + "Detailed gene-segment arithmetic (40 V × 23 D × 6 J, light chains ≈320, pairs ≈1.8 million): the comparison of ~20,000 genes vs >10^8 receptors is enough", + "The affinity maturation relationship and the 'B-cell receptors improve fit' step: the Scope section declares this out of scope and it is not needed for the central answer", + "Killer T cells as an effector type: the model's arithmetic follows only one generic clone", + "The Representation decision section, which is plan metadata, not reader content" + ], + "next_questions": [ + { + "q": "How can random receptors already match a pathogen nobody has seen, and why don't they attack my own body?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If the clone grows 8× per day and the pathogen 10× per day, how does the immune system ever win?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why does measles immunity last decades while some vaccines (tetanus, pertussis) need boosters?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Why does the flu antigen change while measles doesn't?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How does an mRNA vaccine show the antigen without the pathogen?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.1667154, + "seconds": 62 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "The race argument: the clone grows more slowly than the pathogen → Say that one effector cell or antibody removes many pathogen units, so the relevant comparison is killing capacity, not cell count. Illustrative example: if each of the 10^6 effector cells clears 100 virus particles per day, they clear 10^8 per day. A pathogen load of 10^7 then shrinks. Also give a stopping threshold: 'control' means clearance per day exceeds growth per day." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 5", + "text": "Pathogen growth of 10^7× vs 10^3× has no disease threshold or starting dose → Illustrative: the infection starts with 100 particles and symptoms begin around 10^8. Primary response: 100 × 10^7 = 10^9, which is above the threshold, so the person gets sick. Secondary response: 100 × 10^3 = 10^5, which is 1,000× below the threshold, so infection happens with no disease." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 2", + "text": "The claim that 'the matching receptor already exists by chance' has no coverage argument → Example: if one receptor binds about 1 in 10^5 random shapes, then a library of 10^8 receptors has about 1,000 binders for any shape. A pathogen with 10 separate antigen spots is almost certain to be hit. Breaking case: a small library, such as 10^4 receptors (low diversity in the very old or very young), gives about 0.1 expected binders per shape, so responses can be missing." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 2", + "text": "The contraction numbers contradict the memory count → Either say that only about 1% become memory (10^6 → 10^4), or rework the example: 5% of 10^6 = 50,000. Also show the low end of the estimate: at 10× the naive count (1,000 cells), 2^n = 1,000 gives n ≈ 10 divisions ≈ 3.3 days, plus 0.75 day, ≈ 4 days." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 1", + "text": "'Keep all effector cells forever' fails on the file's own numbers → Use real expansion sizes: one response can make 10^7–10^9 cells, up to about 10% of all killer T cells. Add that effector cells are short-lived and cause inflammation. Example: 100 infections × 10% each would exceed the whole pool." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 1", + "text": "Why memory cells activate faster (0.5–1 day vs 2.5 days) → Say that memory cells already sit in tissues and lymph nodes and need less stimulation to activate. Show the time cost of not having this: 2.2 days of division + 2.5 days of activation = 4.7 days instead of about 3." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 5", + "text": "Measles '97% protection' vs influenza '40–60% effectiveness' are undefined and use two different names → Define it once as the reduction in risk of disease, vaccinated vs unvaccinated. Example: if 90 of 100 unvaccinated exposed people get measles, 97% means about 2.7 of 100 vaccinated people do. For flu at 50%: if 10 of 100 unvaccinated people get flu, 5 of 100 vaccinated people do. Use one term throughout." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 5", + "text": "Pre-existing antibodies in the secondary response are not quantified → Example: antibody levels halve about every 3 weeks (illustrative), so after 1 year about 2^-17 of the peak remains. Long-lived plasma cells keep a low baseline that neutralises part of the starting dose, e.g. 100 → 10 particles." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 3", + "text": "Terms used without definition: long-lived plasma cell, heavy/light chain, V/D/J, affinity maturation, precursor → Add entity rows: long-lived plasma cell (keeps releasing antibodies for years); heavy and light chain (the two parts of a receptor, each assembled from gene segments); precursor (a lymphocyte able to respond at the start: 100 naive, 10,000 memory)." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 3", + "text": "Concepts with more than one name → Use one name for each: 'danger signal' (not also 'second signal' or 'signals'); 'precursor count' (not also 'starting cells' or 'head-start'); 'receptor fit' (not also 'affinity'); 'protection' (not also 'effectiveness')." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 4", + "text": "The clonal selection step has no worked example with numbers → Example: a lymph node scans about 10^7 lymphocytes. At a frequency of 1 in 10^5, about 100 bind the vaccine antigen. Only those 100 divide; the other 99.999% stay idle." + }, + { + "n": 12, + "severity": "edge", + "kind": "rule 2", + "text": "Self-reactivity of a random library → Add that most newly made lymphocytes that bind self proteins are removed before release. For T cells, roughly 90%+ of developing cells die in the thymus (estimate)." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Detailed gene-segment arithmetic (40 V × 23 D × 6 J, light chains ≈320, pairs ≈1.8 million): the comparison of ~20,000 genes vs >10^8 receptors is enough" + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "The affinity maturation relationship and the 'B-cell receptors improve fit' step: the Scope section declares this out of scope and it is not needed for the central answer" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Killer T cells as an effector type: the model's arithmetic follows only one generic clone" + }, + { + "n": 16, + "severity": "excess", + "kind": "model entry", + "text": "The Representation decision section, which is plan metadata, not reader content" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150002-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150002-coldread-narrative.json new file mode 100644 index 0000000..5be48ad --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150002-coldread-narrative.json @@ -0,0 +1,486 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "vaccine-memory", + "at": "2026-10-04T15:00:02", + "model": "default", + "prompt_sha": "7327675ff3e8", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: answer in 3 sentences. Said: selection, not instruction; memory cells; 7 → 3 days", + "unresolved": [ + "selection, not instruction", + "memory cells", + "7 → 3 days" + ], + "unsaid": [ + "'answer in 3 sentences' is shown, but the Said column gives only three fragments. The reader never hears the answer as sentences." + ], + "leap": [ + "It is not said what '7 → 3 days' measures (days until what?), or what is being selected." + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 2 (approach)", + "text": "Shown: pathogen numbers. Said: infection is a race; 100 particles ×10/day; symptoms above 100 million", + "unresolved": [ + "pathogen", + "particles" + ], + "unsaid": [ + "The 'pathogen numbers' are shown, but the words never say what the pathogen is (a virus or bacterium)." + ], + "leap": [ + "'infection is a race': only one runner is named. The reader is not told what the germ is racing against.", + "No reason is given for treating infection as a race." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Shown: library numbers. Said: random receptors, >100 million kinds; 20,000 genes can't list them; loose fit + many antigens; self-binders removed", + "unresolved": [ + "receptors", + "antigens", + "self-binders", + "library" + ], + "unsaid": [ + "The 'library numbers' are shown, but the words never call it a library or say what it holds. The cell that carries the receptor (lymphocyte) is never named or described." + ], + "leap": [ + "'20,000 genes can't list them': it is not said how random receptors are made from a limited number of genes.", + "'loose fit + many antigens': it is not said why this means some cell will fit." + ], + "excess": [ + "'self-binders removed' is not needed for the main argument at this point." + ], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Shown: clone arithmetic. Said: only matching ~100 cells divide every 8 h; effector cells; why fewer can beat more", + "unresolved": [ + "clone", + "effector cells", + "the antigen" + ], + "unsaid": [ + "The 'clone arithmetic' is shown, but the words do not say what a clone is or give the 1,000,000 target." + ], + "leap": [ + "Where '~100 cells' comes from, after '>100 million kinds'.", + "'why fewer can beat more': the reason is named but not given." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: 13.3 divisions; 7 days; 1 billion particles. Said: the clone starts tiny", + "unresolved": [], + "unsaid": [ + "'13.3 divisions' appears with no equation and no target count in the words.", + "'1 billion particles' is shown, but the words do not compare it with the 100 million symptom threshold (so: sick)." + ], + "leap": [ + "13.3 divisions × 8 h is about 4.4 days, not '7 days'. The difference is not explained." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Shown: 99% die, 1% stay: 10,000. Said: memory cells; 6.6 divisions; 3 days; 100,000 particles", + "unresolved": [], + "unsaid": [ + "'99% die, 1% stay' is shown, but the words do not say it.", + "The words never say that 100,000 particles is below the symptom threshold (so: not sick)." + ], + "leap": [ + "Why 1% stay.", + "6.6 × 8 h is about 2.2 days, not '3 days'." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (name)", + "text": "Shown: vaccine = antigen + adjuvant, no disease. Said: runs beat 4–6 safely; why adjuvant", + "unresolved": [ + "adjuvant", + "beat 4–6" + ], + "unsaid": [ + "The 'adjuvant' is shown, but the words say only 'why adjuvant' and never explain it (no danger signal is given)." + ], + "leap": [ + "It is not said why antigen alone causes no disease." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (objection)", + "text": "Shown: flu vs measles. Said: antigen change; effectiveness defined", + "unresolved": [ + "effectiveness" + ], + "unsaid": [ + "The flu vs measles comparison is shown, but no numbers or definition of effectiveness are given in the words." + ], + "leap": [ + "It is not said why a change in antigen breaks memory (no new matching cells)." + ], + "excess": [ + "The 'effectiveness defined' detail is not needed for the main argument." + ], + "purpose_unclear": false + }, + { + "at": "Beat 9 (close)", + "text": "Shown: recap. Said: answer in the opening words; cache analogy and where it breaks", + "unresolved": [ + "cache analogy" + ], + "unsaid": [], + "leap": [ + "A new analogy appears at the very end, and its content and where it breaks are not given." + ], + "excess": [ + "'answer in the opening words' repeats Beat 1." + ], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 2: 'infection is a race': the opposing runner (the immune cells) is never named.", + "Beat 3: 'receptors' and 'antigens' are used undefined, and the lymphocyte is never introduced.", + "Beat 4: 'clone' and '~100 cells' are unexplained.", + "Beat 5: '13.3 divisions' has no stated target or equation, and it does not match '7 days'.", + "Beat 7: 'adjuvant' is never explained." + ], + "cuts": [ + "Beat 3: 'self-binders removed'", + "Beat 9: 'cache analogy and where it breaks'", + "Beat 8: 'effectiveness defined'", + "Beat 3: '20,000 genes can't list them' (or explain it)" + ], + "question": { + "text": "How can a vaccine prepare the body for a germ it has never met?", + "known_at": "Never stated in Said/Shown. Beat 1 only implies it through the answer.", + "result_stated": "Only in compressed form, Beat 1: 'selection, not instruction; memory cells; 7 → 3 days'" + }, + "motive": "Missing from the beats. 'sounds impossible' appears only in the plan. A reason for the approach is also missing: Beat 2 asserts 'infection is a race' with no reason.", + "close": "Beat 9: 'answer in the opening words'. The answer is not quoted, and the general argument (each doubling of starting cells saves one 8 h division) is not stated in the beats.", + "recap": "Your body already has a few cells that fit a germ by chance; a vaccine multiplies them safely once, so next time there are many more and the race is won in about 3 days instead of about 7.", + "strongest": "Beats 5–6: the same arithmetic, 100 vs 10,000 starting cells, gives 13.3 vs 6.6 divisions and 1 billion vs 100,000 particles.", + "weakest": "Beat 3: four undefined terms ('receptors', 'antigens', 'self-binders', 'library') arrive at once, and the cell carrying the receptor is never named.", + "suggestions": [ + "Name the second runner in Beat 2: immune cells that must multiply to fight back.", + "Define lymphocyte, receptor and antigen in plain words in Beat 3.", + "Say the 1,000,000 target and the 100×2^n equation aloud, and explain why 4.4 days becomes 7 (start-up delay).", + "Explain the adjuvant as the 'this is a threat' signal.", + "State the question and why it seems impossible at the start of Beat 1." + ] + }, + "usage": { + "cost": 0.11253679999999999, + "seconds": 36 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'infection is a race': the opposing runner (the immune cells) is never named." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 3: 'receptors' and 'antigens' are used undefined, and the lymphocyte is never introduced." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'clone' and '~100 cells' are unexplained." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: '13.3 divisions' has no stated target or equation, and it does not match '7 days'." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 7: 'adjuvant' is never explained." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 3: 'self-binders removed'" + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 9: 'cache analogy and where it breaks'" + }, + { + "n": 8, + "severity": "excess", + "kind": "cut", + "text": "Beat 8: 'effectiveness defined'" + }, + { + "n": 9, + "severity": "excess", + "kind": "cut", + "text": "Beat 3: '20,000 genes can't list them' (or explain it)" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): selection, not instruction" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): memory cells" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): 7 → 3 days" + }, + { + "n": 13, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): 'answer in 3 sentences' is shown, but the Said column gives only three fragments. The reader never hears the answer as sentences." + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): It is not said what '7 → 3 days' measures (days until what?), or what is being selected." + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (approach): pathogen" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (approach): particles" + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (approach): The 'pathogen numbers' are shown, but the words never say what the pathogen is (a virus or bacterium)." + }, + { + "n": 18, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (approach): 'infection is a race': only one runner is named. The reader is not told what the germ is racing against." + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (approach): No reason is given for treating infection as a race." + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): receptors" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): antigens" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): self-binders" + }, + { + "n": 23, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): library" + }, + { + "n": 24, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): The 'library numbers' are shown, but the words never call it a library or say what it holds. The cell that carries the receptor (lymphocyte) is never named or described." + }, + { + "n": 25, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): '20,000 genes can't list them': it is not said how random receptors are made from a limited number of genes." + }, + { + "n": 26, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): 'loose fit + many antigens': it is not said why this means some cell will fit." + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): clone" + }, + { + "n": 28, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): effector cells" + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): the antigen" + }, + { + "n": 30, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The 'clone arithmetic' is shown, but the words do not say what a clone is or give the 1,000,000 target." + }, + { + "n": 31, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): Where '~100 cells' comes from, after '>100 million kinds'." + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): 'why fewer can beat more': the reason is named but not given." + }, + { + "n": 33, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): '13.3 divisions' appears with no equation and no target count in the words." + }, + { + "n": 34, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): '1 billion particles' is shown, but the words do not compare it with the 100 million symptom threshold (so: sick)." + }, + { + "n": 35, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 13.3 divisions × 8 h is about 4.4 days, not '7 days'. The difference is not explained." + }, + { + "n": 36, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): '99% die, 1% stay' is shown, but the words do not say it." + }, + { + "n": 37, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): The words never say that 100,000 particles is below the symptom threshold (so: not sick)." + }, + { + "n": 38, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): Why 1% stay." + }, + { + "n": 39, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): 6.6 × 8 h is about 2.2 days, not '3 days'." + }, + { + "n": 40, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (name): adjuvant" + }, + { + "n": 41, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (name): beat 4–6" + }, + { + "n": 42, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (name): The 'adjuvant' is shown, but the words say only 'why adjuvant' and never explain it (no danger signal is given)." + }, + { + "n": 43, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (name): It is not said why antigen alone causes no disease." + }, + { + "n": 44, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (objection): effectiveness" + }, + { + "n": 45, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (objection): The flu vs measles comparison is shown, but no numbers or definition of effectiveness are given in the words." + }, + { + "n": 46, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (objection): It is not said why a change in antigen breaks memory (no new matching cells)." + }, + { + "n": 47, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (close): cache analogy" + }, + { + "n": 48, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (close): A new analogy appears at the very end, and its content and where it breaks are not given." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150207-coldread-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150207-coldread-prose.json new file mode 100644 index 0000000..97d3374 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150207-coldread-prose.json @@ -0,0 +1,294 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "vaccine-memory", + "at": "2026-10-04T15:02:07", + "model": "default", + "prompt_sha": "6b72086ee875", + "reply": { + "paragraphs": [ + { + "at": "Title + bold opening (L1, L5)", + "text": "A vaccine does not teach the body a new shape... controls it in about 3 days instead of about 7.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L7", + "text": "Numbers marked illustrative are round numbers...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Infection is a race, L11", + "text": "A pathogen is a virus or bacterium... symptoms start above 100 million particles.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L13", + "text": "Immune cells also multiply. Whichever side gets big enough first wins...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The matching cells exist before the germ does, L18", + "text": "A lymphocyte is a white blood cell... spike protein on the surface of a coronavirus.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L20", + "text": "Each new lymphocyte builds its receptor by joining pieces of genes...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L22", + "text": "Because each receptor fits a family of shapes... mostly removed while they develop.", + "unresolved": [], + "leap": [], + "excess": [ + "Lymphocytes whose receptors bind the body's own molecules are mostly removed while they develop." + ], + "purpose_unclear": false + }, + { + "at": "L25", + "text": "This is the central idea... The germ selects from a random library that already exists.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The first meeting is slow, L30", + "text": "When an antigen arrives, only the lymphocytes whose receptors bind it activate and divide...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L32", + "text": "For one antigen, about 1 in 100,000 to 1 in 1,000,000 lymphocytes fit... divides about every 8 hours.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L35", + "text": "The clone's cells become effector cells... 1,000,000 effector cells clear the pathogen faster than it grows.", + "unresolved": [ + "Plasma cells", + "B-cell receptor", + "Killer T cells" + ], + "leap": [ + "Why 1,000,000 effector cells is the number that wins: the text gives only 'thousands of antibodies per second', not how that compares with the pathogen's growth" + ], + "excess": [ + "Killer T cells kill infected cells." + ], + "purpose_unclear": false + }, + { + "at": "L38", + "text": "How long does the first response take? Let n be the number of divisions...", + "unresolved": [ + "lymph node" + ], + "leap": [ + "Why activation takes about 2.5 days is only partly explained ('meet the antigen in a lymph node')" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What stays behind, L43", + "text": "After the pathogen is gone, about 99% of the effector cells die. About 1% stay...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L45", + "text": "At the next meeting with the same antigen, 10,000 × 2^n = 1,000,000... Some antibodies may also still be in the blood.", + "unresolved": [], + "leap": [], + "excess": [ + "Some antibodies may also still be in the blood." + ], + "purpose_unclear": false + }, + { + "at": "Table L47-53", + "text": "First meeting vs same antigen again", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L55", + "text": "The rule behind the table: each doubling of the precursor count saves one division...", + "unresolved": [], + "leap": [], + "excess": [ + "A 100× larger count saves about 6.6 divisions. (repeats the table's 13.3 vs 6.6)" + ], + "purpose_unclear": false + }, + { + "at": "What the vaccine does, L60", + "text": "A vaccine contains the antigen, or instructions... the race starts with the secondary response", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L63", + "text": "Antigen alone is often not enough... an adjuvant: an ingredient that triggers the danger signal.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "When memory does not help, L67", + "text": "Memory cells fit only the antigen that made them. Measles... Influenza...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L69", + "text": "Also note what a vaccine does not do: it does not stop the pathogen from entering...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Summary L73", + "text": "The body made the matching cells at random, before any infection...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered L75", + "text": "Not covered here: how B-cell receptors improve their fit...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "L35: 'Killer T cells kill infected cells.' and the unexplained 'Plasma cells'/'B-cell' names. Saying 'some clone cells release antibodies, free copies of their receptor' keeps the point.", + "L45: 'Some antibodies may also still be in the blood.' This side point is never used.", + "L22: the sentence about removing self-binding lymphocytes. The argument does not need it here (L63 covers the safety point).", + "L55: 'A 100× larger count saves about 6.6 divisions.' The table already shows this." + ], + "question": { + "text": "How can a vaccine make the body ready for a germ it has never met, when it can't know the germ's shape ahead of time? The title asks this, and the reader knows it from L1. L13 then reframes it as a race: 'what does a vaccine change in this race?'", + "known_at": "L1 (title), reframed at L13", + "result_stated": "Yes, in words at L5: 'The vaccine makes those few cells multiply once, without the disease, and many of their descendants stay. So when the real germ arrives, the body controls it in about 3 days instead of about 7.'" + }, + "motive": "Reason for the approach: L13 'Immune cells also multiply. Whichever side gets big enough first wins. So ask: what does a vaccine change in this race?' This sets up the counting and doubling argument. Reason to care: mostly implied. The puzzle in the title (preparing for something never met) and L20 'no gene list could anticipate a virus that evolved last year' supply it. No separate stakes are given.", + "close": "Yes. L73: 'The body made the matching cells at random, before any infection. The vaccine selects the few that fit, multiplies them safely, and leaves about 100× more behind. So the real infection is controlled in about 3 days instead of about 7.'", + "recap": "Your body already has a few cells that fit any germ by chance, and a vaccine safely multiplies them ahead of time, so the real infection starts with 100× more fighters and is beaten in about 3 days instead of 7, before symptoms appear.", + "strongest": "L38–L53: the worked doubling arithmetic (100×2^n vs 10,000×2^n) and the table. They make 7 vs 3 days and 1 billion vs 100,000 particles concrete, and every figure checks out.", + "weakest": "L35: 'Plasma cells', 'B-cell receptor' and 'Killer T cells' appear without definition. The claim that 1,000,000 effector cells is enough is asserted with little reason.", + "suggestions": [ + "In L35, say that lymphocytes come in two kinds (B cells, whose receptors can be released as antibodies, and T cells), or leave out the names.", + "Give a one-line reason why about 1% of effector cells survive as memory cells, or say plainly that this is an observed fact.", + "Briefly say what a lymph node is (a small hub where immune cells and antigen meet)." + ] + }, + "usage": { + "cost": 0.121891, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "L35: 'Killer T cells kill infected cells.' and the unexplained 'Plasma cells'/'B-cell' names. Saying 'some clone cells release antibodies, free copies of their receptor' keeps the point." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "L45: 'Some antibodies may also still be in the blood.' This side point is never used." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "L22: the sentence about removing self-binding lymphocytes. The argument does not need it here (L63 covers the safety point)." + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "L55: 'A 100× larger count saves about 6.6 divisions.' The table already shows this." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "L35: Plasma cells" + }, + { + "n": 6, + "severity": "edge", + "kind": "unresolved", + "text": "L35: B-cell receptor" + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "L35: Killer T cells" + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "L35: Why 1,000,000 effector cells is the number that wins: the text gives only 'thousands of antibodies per second', not how that compares with the pathogen's growth" + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "L38: lymph node" + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "L38: Why activation takes about 2.5 days is only partly explained ('meet the antigen in a lymph node')" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150328-quiz-prose.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150328-quiz-prose.json new file mode 100644 index 0000000..7c97599 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150328-quiz-prose.json @@ -0,0 +1,230 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "vaccine-memory", + "at": "2026-10-04T15:03:12", + "model": "default", + "prompt_sha": "a0a72909c349", + "reply": { + "answers": { + "1": "The main objects are the pathogen (a virus or bacterium that multiplies) and its antigens (the shapes a receptor binds). On the immune side there are lymphocytes, each carrying many copies of one kind of receptor, and clones (one lymphocyte plus its descendants). Clones produce effector cells, the antibodies some effector cells release, and memory cells. Finally there are the vaccine, the adjuvant and the danger signal.", + "2": "Each lymphocyte's receptor is built at random from gene pieces, so the body has a large library of different receptors. When a pathogen's antigen appears, only the lymphocytes whose receptors bind it activate and divide into a clone. To activate they also need a danger signal. The clone's cells become effector cells, which kill infected cells or release antibodies (free copies of the receptor) that block the pathogen. After the pathogen is cleared, about 99% of effector cells die and about 1% remain as memory cells for that same antigen. A vaccine supplies the antigen without a pathogen that can multiply, plus often an adjuvant to trigger the danger signal, so the first expansion happens safely and memory cells are left behind.", + "3": "A vaccine's antigen selects the few lymphocytes that already fit it and makes them multiply, leaving about 100× more memory cells than the original number. With that larger starting count, a real infection needs fewer doublings to reach an effective force, so it is controlled in about 3 days instead of 7, before the pathogen reaches the symptom level.", + "4": "The key variable is the precursor count (cells able to respond at the start). Raising it cuts the divisions needed to reach about 1,000,000 effector cells: each doubling saves one division, about 8 hours. Going from 100 to 10,000 cuts 13.3 divisions to 6.6, and the pathogen peak falls from 1 billion to 100,000, below the symptom level. Lowering it, or having an antigen that has changed so memory fits poorly, pushes the response back toward 7 days and illness, because the pathogen grows 10× per day while the response grows only by doubling.", + "5": "Assumption: the illustrative numbers. Infection starts with 100 particles growing 10× per day, symptoms start above 100 million particles, 1,000,000 effector cells is enough to win, and a fixed 8-hour division time holds. Established fact: two doses of measles vaccine prevent about 97% of cases for decades. Also: humans have about 20,000 genes.", + "6": "Each lymphocyte builds a random receptor by joining gene pieces, so the body already holds over 100 million receptor types, and a few of them fit almost any pathogen. An antigen selects those few cells, which divide every ~8 hours. Starting from about 100 cells, the first response needs ~7 days, by which time the pathogen has caused symptoms. Afterward about 1% of the expanded clone survives as memory cells (~100× the original count), so a second meeting needs half the divisions and takes ~3 days, keeping the pathogen below symptom level. A vaccine runs that first expansion with antigen but no growing pathogen (plus often an adjuvant for the needed danger signal). This works only while the antigen stays the same: measles yes, flu no.", + "7": "Hardest: the time totals. The 2.5 days for a first activation and 'under a day' for memory cells are stated without a reason, and the 3-day total only works if you add 2.2 + under a day yourself. It is also unclear how '1 in 100,000 to 1,000,000' turns into 100 cells, because the total number of lymphocytes is never given. Also, 'secondary/primary response' appears only in the vaccine section and never in the table. Stating the total lymphocyte count, giving a short reason for each activation delay, and using one name for the first and second responses would help.", + "8": "According to the explanation, about 3 days to control instead of about 7, so roughly 2.3× faster (about 4 days sooner). Memory cells last 'years or decades', so after two years about 10,000 matching cells remain instead of 100. That means 6.6 divisions instead of 13.3, and memory cells activate in under a day instead of ~2.5 days. The pathogen peaks around 100,000 particles instead of 1 billion, so there are no symptoms.", + "9": "Memory cells fit only the antigen that made them. Measles keeps the same antigens, so old memory still fits and gives the fast response for decades. Influenza changes its surface antigens each year, so old memory receptors fit poorly and the response falls back toward the slow first one. The vaccine is therefore updated yearly. The explanation says the cause is the changed antigen, not memory wearing off.", + "10": "Receptors are not stored one per gene, and they are not designed after the virus appears. Each new lymphocyte joins gene pieces at random and adds random edits, giving more than 100 million receptor types. Each receptor binds a family of similar shapes, and a virus shows many antigens. So by chance a few existing lymphocytes fit even a brand-new virus; the virus selects them.", + "11": "Yes, a few: about 1 in 100,000 to 1,000,000 lymphocytes per antigen (illustrative: 100 cells). The reason is that the receptor library is made at random beforehand, is very large, and each receptor fits a family of shapes.", + "12": "There are more starting cells: about 10,000 memory cells versus 100, so fewer doublings (6.6 vs 13.3) are needed to reach ~1,000,000 effector cells. Also, memory cells already wait in the tissues and need less stimulation, so they activate in under a day instead of ~2.5 days.", + "13": "Memory cells: the roughly 1% of the expanded clone's effector cells that survive for years or decades. For the same antigen they form a starting pool about 100× larger than the original precursor count (illustrative: 10,000 vs 100). Antibodies are not the lasting protection here; their decline is listed as not covered." + }, + "audit": { + "terms": [ + "'germ' (lines 5, 13, 24) and 'pathogen' (line 11 onward) name the same thing; 'germ' is never defined as equal to 'pathogen'.", + "'secondary response' / 'primary one' (line 58) mean the same as 'first meeting' / 'next meeting' / 'Same antigen again' (lines 26, 44, table), but 'primary/secondary' are used without being introduced.", + "'controls it' (lines 5, 49), 'clear the pathogen' (line 34) and 'wins' (line 13) all describe the same end point of the race, but it is never said they are one threshold (reaching 1,000,000 effector cells).", + "'fit' (lines 5, 22, 31), 'bind' (lines 18, 29) and 'match' (heading, line 15) are used interchangeably for receptor-antigen binding.", + "'shape' (lines 5, 18) and 'antigen' (line 18) overlap: line 5 says the vaccine does not teach 'a new shape', and only later is it said that the shape on a pathogen is the antigen." + ], + "unexplained": [ + "'about 1 in 100,000 to 1 in 1,000,000 lymphocytes fit' becomes 'Illustrative: 100 cells' (line 31) without the total lymphocyte count, so the reader cannot check the 100 or see what a higher or lower fraction would mean.", + "'about 2.5 days to meet the antigen and activate' (line 37) and 'under a day' for memory cells (line 44): no basis is given, yet these delays make up a large share of both totals.", + "'about 99% of the effector cells die. About 1% stay' (line 42): it is not stated that 10,000 = 1% of 1,000,000, or what a higher or lower survival share would change (it sets the 100× gain).", + "'about 97% of cases' vs 'about 40–60%' (line 65): it is not said whether these are per person or per population, or how they relate to the 3-day vs 7-day model.", + "'1,000,000 effector cells clear the pathogen faster than it grows' (line 34): the threshold is asserted; it is unclear how it depends on pathogen size, given the pathogen is at 1 billion by then in the first response." + ], + "missing_why": [ + "Why only about 1% of effector cells become long-lived memory cells while the rest die (line 42).", + "Why memory cells 'need less stimulation' and activate faster (line 44).", + "Why the first activation takes about 2.5 days: 'rare matching cells also need ... to meet the antigen' (line 37) hints at search time but does not explain it.", + "Why one fixed threshold of 1,000,000 effector cells applies even though the pathogen count at that time differs by 10,000× between the two scenarios (lines 34, 50)." + ], + "excess": [ + "Line 3 HTML comment 'Stage 1, rendered from model.md in STE-80' is production metadata, not content.", + "Line 71 Summary repeats the bolded opening (line 5) almost word for word, adding only '100× more'." + ] + }, + "gaps": [ + "The total number of lymphocytes is never given, so the step from '1 in 100,000–1,000,000' to 100 precursor cells cannot be checked.", + "There is no reason for the activation delays (2.5 days vs under a day), even though they are a large part of the 7-vs-3-day result.", + "Antibody levels and how long they last are excluded, so the reader cannot tell whether protection years later involves anything besides memory cells.", + "No mechanism is given for why ~1% of effector cells survive as memory.", + "'Primary/secondary response' appear once without definition.", + "It is unclear whether a partly changed antigen (flu) gives a partial speed-up or none; 'falls back toward' is not quantified." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "1": "Full marks. Correct and specific, though it leaves out naive lymphocyte, clonal selection and primary/secondary response as named entities.", + "2": "Full marks. The full mechanism is present: random library, clonal selection, danger signal, effector/antibody, 99% contraction, vaccine with adjuvant.", + "3": "Full marks. Selection plus 100× memory, fewer doublings, about 3 vs 7 days, below symptom level.", + "4": "Full marks. Precursor count is correctly identified as the key variable, with numbers (13.3 vs 6.6 divisions, 8 h per doubling) and the antigen-change limit.", + "5": "Full marks. Illustrative assumptions are separated correctly from established facts (measles 97%, about 20,000 genes).", + "6": "Full marks. The summary covers the whole causal chain and its limit.", + "7": "Full marks. Gives specific, plausible points of difficulty with concrete suggested fixes.", + "8": "Full marks. Gives about 3 vs 7 days, 100× more starting cells (6.6 vs 13.3 divisions), faster activation, and the pathogen staying below symptom level. Does not fall into either 'wrong if' trap.", + "9": "Full marks. The changed surface antigen is named as the cause, and the answer explicitly rejects memory wearing off.", + "10": "Full marks. Random generation of more than 100 million receptors before infection, binding to shape families, and selection by the virus; it explicitly denies that receptors are designed after infection.", + "11": "Full marks. Yes, because a random library exists beforehand; precursor frequency is given.", + "12": "Full marks. 100× more starting memory cells with fewer divisions and faster activation. The 3-day total is not stated outright but follows from the components given.", + "13": "Full marks. Memory cells, about 100× the precursor count, are correctly named as the lasting protection. It does not mention the danger-signal and no-disease framing, but it answers what is left behind correctly." + }, + "pass": true + }, + "usage": { + "cost": 0.20738799999999996, + "seconds": 56 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "'germ' (lines 5, 13, 24) and 'pathogen' (line 11 onward) name the same thing; 'germ' is never defined as equal to 'pathogen'." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'secondary response' / 'primary one' (line 58) mean the same as 'first meeting' / 'next meeting' / 'Same antigen again' (lines 26, 44, table), but 'primary/secondary' are used without being introduced." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'controls it' (lines 5, 49), 'clear the pathogen' (line 34) and 'wins' (line 13) all describe the same end point of the race, but it is never said they are one threshold (reaching 1,000,000 effector cells)." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'fit' (lines 5, 22, 31), 'bind' (lines 18, 29) and 'match' (heading, line 15) are used interchangeably for receptor-antigen binding." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "'shape' (lines 5, 18) and 'antigen' (line 18) overlap: line 5 says the vaccine does not teach 'a new shape', and only later is it said that the shape on a pathogen is the antigen." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'about 1 in 100,000 to 1 in 1,000,000 lymphocytes fit' becomes 'Illustrative: 100 cells' (line 31) without the total lymphocyte count, so the reader cannot check the 100 or see what a higher or lower fraction would mean." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2.5 days to meet the antigen and activate' (line 37) and 'under a day' for memory cells (line 44): no basis is given, yet these delays make up a large share of both totals." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'about 99% of the effector cells die. About 1% stay' (line 42): it is not stated that 10,000 = 1% of 1,000,000, or what a higher or lower survival share would change (it sets the 100× gain)." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'about 97% of cases' vs 'about 40–60%' (line 65): it is not said whether these are per person or per population, or how they relate to the 3-day vs 7-day model." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "'1,000,000 effector cells clear the pathogen faster than it grows' (line 34): the threshold is asserted; it is unclear how it depends on pathogen size, given the pathogen is at 1 billion by then in the first response." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Why only about 1% of effector cells become long-lived memory cells while the rest die (line 42)." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "Why memory cells 'need less stimulation' and activate faster (line 44)." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "Why the first activation takes about 2.5 days: 'rare matching cells also need ... to meet the antigen' (line 37) hints at search time but does not explain it." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "Why one fixed threshold of 1,000,000 effector cells applies even though the pathogen count at that time differs by 10,000× between the two scenarios (lines 34, 50)." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "Line 3 HTML comment 'Stage 1, rendered from model.md in STE-80' is production metadata, not content." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "Line 71 Summary repeats the bolded opening (line 5) almost word for word, adding only '100× more'." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The total number of lymphocytes is never given, so the step from '1 in 100,000–1,000,000' to 100 precursor cells cannot be checked." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "There is no reason for the activation delays (2.5 days vs under a day), even though they are a large part of the 7-vs-3-day result." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "Antibody levels and how long they last are excluded, so the reader cannot tell whether protection years later involves anything besides memory cells." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "No mechanism is given for why ~1% of effector cells survive as memory." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "'Primary/secondary response' appear once without definition." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "It is unclear whether a partly changed antigen (flu) gives a partial speed-up or none; 'falls back toward' is not quantified." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/read.json new file mode 100644 index 0000000..245f629 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "A vaccine must contain the antigen or genetic instructions for making it. It doesn't make you ill because it contains no pathogen capable of multiplying to cause disease.", + "quote": "A **vaccine** contains the antigen, or instructions for the body's cells to make it. It contains no pathogen that can multiply to disease." + }, + "2": { + "answer": "Vaccination creates memory cells that are 100 times more numerous than the original matching lymphocytes. These memory cells activate faster because they already wait in body tissues and require less stimulation.", + "quote": "Illustrative: 1% of 1,000,000 is 10,000 memory cells, which is 100× the original precursor count of 100...Memory cells already wait in the tissues and need less stimulation, so they activate in under a day. Total: about 3 days." + }, + "3": { + "answer": "A vaccine against one virus doesn't protect against an unrelated virus because memory cells only recognize the specific antigen that trained them.", + "quote": "Memory cells bind only the antigen that made them." + }, + "4": { + "answer": "When the virus changes the shape of its antigen, the old memory cell receptors bind poorly, so the immune response falls back to the slower primary response, reducing protection.", + "quote": "The old memory receptors bind poorly, so the response falls back toward the slow first one. That is why flu vaccines prevent only about 40–60% of cases in a typical season and are updated every year." + } + }, + "stops": [ + "joining pieces of genes in a random combination and adding random edits", + "Memory cells already wait in the tissues and need less stimulation", + "need about 2.5 days to meet the antigen and activate", + "binds one family of similar molecular shapes", + "does not need as many cells as the pathogen has particles", + "does not design a receptor...selects from a random library", + "release thousands of antibodies per second", + "1 in 100,000 to 1 in 1,000,000...Illustrative: 100 cells", + "100 × 2^n = 1,000,000, so 2^n = 10,000", + "immune cells that sense a threat" + ], + "excess": [ + "Numbers marked *illustrative* are round numbers", + "One gene per receptor is impossible", + "Some release **antibodies**...Others kill infected cells", + "each doubling saves one division", + "For simplicity, use this one target", + "more than 100 million different receptors", + "Call this number the **precursor count**", + "improve their fit over weeks", + "need about 2.5 days to meet", + "thousands of antibodies per second" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/run.json new file mode 100644 index 0000000..3ed63ce --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/vaccine-memory/explanation.md` as prose with one comparison table: the subject is a single causal chain (a random library of cells, the vaccine picking and multiplying the matching ones, the memory cells left behind) plus one before-and-after race, and prose with numbers carries that better than a diagram or page would. The supporting files `model.md`, `model.yaml` and `narrative.md` are in the same folder; the probe, cold reads and blind test (passed, full marks) all ran with `--run`, `explainer check` passes after the post-test edits, and the blind test was not re-run on those small final edits.", + "error": null, + "cost": 1.5137304, + "seconds": 543, + "turns": 65, + "files": [ + "explainers/vaccine-memory/explanation.md" + ], + "review_cost": 0.6085, + "reviews": 4 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/explanation.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/explanation.md new file mode 100644 index 0000000..6d65505 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/explanation.md @@ -0,0 +1,46 @@ +# How does a vaccine prepare the immune system for a disease it has never met? + + + +**The immune system does not design a defence after it sees a disease. Before any infection, it already holds billions of cells with random receptor shapes. A vaccine shows it a harmless piece of the pathogen, the rare matching cells multiply, and some stay as memory cells. When the real pathogen arrives, the response starts from a large, ready population instead of a few cells.** + +## The parts + +- An **antigen** is a molecular shape on a pathogen, for example a protein on the surface of a virus. +- A **lymphocyte** is a white blood cell (a B cell or a T cell). Each lymphocyte carries one **receptor** type, a protein that binds one shape. +- An **antibody** is a free-floating copy of a B cell receptor. It binds the antigen and blocks the pathogen or marks it for destruction. + +## How it works + + +1. **Random receptors, made in advance.** Each new lymphocyte shuffles gene segments at random to build its receptor. The number of possible receptors is more than 10^11. The set of receptors in your body is your **repertoire**. It already contains a few cells that fit almost any shape, including shapes from viruses that did not exist when you were born. +2. **The antigen selects the match.** For a given antigen, only about 1 in 100,000 to 1 in 1,000,000 lymphocytes has a receptor that binds it. The vaccine brings the antigen to the lymph nodes, where those rare cells meet it. +3. **Clonal expansion.** A lymphocyte that binds the antigen, and also gets an alarm signal, starts to divide. Over about a week, it makes a clone of thousands of identical cells. B cells in the clone make antibodies. +4. **Memory cells remain.** After the response, most clone cells die. Some become **memory cells**: long-lived lymphocytes with the matching receptor. There are now about 10–100 times more matching cells than before. Many have a better fit, because the clone mutates its receptors during expansion and the best binders are kept. + +Analogy: the repertoire is a large ring of random keys. The vaccine does not cut a new key. It finds the keys that fit this lock and copies them. (Unlike keys, the copies can also improve their fit.) + +A vaccine supplies the antigen without the disease: an inactivated pathogen, one purified protein, or mRNA that tells your cells to make that one protein briefly. + +## Why it works: a race + + + +Infection is a race between pathogen growth and immune expansion. Assume, for illustration, a pathogen that multiplies 10 times per day. + +| | First meeting (no vaccine) | Second meeting (after vaccine) | +|---|---|---| +| Matching cells at the start | a few rare naive cells | 10–100 times more memory cells | +| Useful antibody after | 7–14 days | 2–4 days | +| Antibody level | baseline | 10–100 times higher | +| Pathogen growth before response (10×/day) | 7 days: 10,000,000-fold | 3 days: 1,000-fold | + +The vaccine moves the slow first response to a safe time. The real infection then meets the fast second response. The pathogen can still enter the body, but the immune system stops it while it is small, so you get no illness or a mild one. + +All day counts and frequencies are typical estimates; they vary by antigen, vaccine, and person. + +## Limits + +- **Changed antigen.** Memory receptors fit the old shape. A variant with a changed antigen (as with influenza each year) binds less well, so protection drops and the vaccine needs an update. +- **Decline over time.** Antibody levels fall, and for some vaccines memory also weakens. Boosters restore them (tetanus: about every 10 years). +- **Weak immune system.** A person who makes few lymphocytes (for example, during chemotherapy) cannot expand a clone well, so the vaccine protects less. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/model.md b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/model.md new file mode 100644 index 0000000..c28d752 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/model.md @@ -0,0 +1,98 @@ +# Semantic model: how a vaccine prepares the immune system + +> Source of truth. Every rendering derives from this file. + +## Central question + +How does a vaccine prepare the immune system for a disease it has never met? + +## Audience and prior knowledge + +A technical reader (school science, basic programming) new to immunology. Knows cells divide, proteins have shapes, exponential growth. Does not know lymphocyte, antigen, receptor diversity, clonal selection, memory cell. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| antigen | A molecular shape on a pathogen (for example, a virus surface protein) that a receptor can bind. | fact | +| lymphocyte | A white blood cell (B cell or T cell) that carries one receptor type on its surface. | fact | +| receptor | The binding protein of one lymphocyte; its shape is made by random gene shuffling before any infection. | fact | +| repertoire | The full set of different receptors across all lymphocytes in one body. | fact | +| clonal expansion | Rapid division of a lymphocyte whose receptor binds an antigen, making a clone of identical cells. | fact | +| memory cell | A long-lived lymphocyte left after a response; it keeps the matching receptor. | fact | +| antibody | A soluble copy of a B cell receptor; it binds the antigen and blocks or marks the pathogen. | fact | +| vaccine | A harmless source of the antigen, plus an alarm signal (adjuvant or the vaccine's own components). | fact | + +## Relationships + +| From | Relationship | To | +|---|---|---| +| random gene shuffling | makes | repertoire | +| antigen | selects | lymphocyte with a matching receptor | +| selected lymphocyte | undergoes | clonal expansion | +| clonal expansion | leaves | memory cells | +| memory cells | shorten | the second response | + +## Causal chain + +```text +random receptors (before exposure) → vaccine antigen binds the rare match → clonal expansion (~1 week) +→ memory cells remain → real pathogen → large, fast response in days → pathogen stopped early +``` + +## Quantities (estimates, order of magnitude) + +- Matching naive lymphocytes: about 1 in 100,000 to 1 in 1,000,000 lymphocytes. +- Repertoire size: more than 10^11 possible receptors (gene shuffling); about 10^8 or more distinct receptors present at once. +- First response: useful antibody after about 7–14 days. +- Second response: useful antibody after about 2–4 days, at 10–100 times higher levels. +- Memory cells: present in about 10–100 times more cells than the naive match; last years to decades. +- Illustrative race: a pathogen that multiplies 10× per day grows 10^7-fold in 7 days, but only 10^3-fold in 3 days. + +## Alternative states + +| Scenario | What changes | Why | +|---|---|---| +| No vaccine, first infection | Response needs ~7–14 days | Few matching cells; expansion must start from them | +| Vaccinated, infection | Response in ~2–4 days | Memory cells start from a larger, better-fitting clone | +| Variant with a changed antigen | Weaker protection | Memory receptors fit the old shape less well | +| Years later, no booster | Weaker protection for some vaccines | Antibody levels and memory decline | + +## Why this form + +| Operation | Simplest alternative | What it breaks | +|---|---|---| +| Make receptors at random, in advance | Design a receptor after seeing the pathogen | The body has no mechanism to design a protein from a shape; it can only select from what exists | +| Use a harmless antigen | Use the real pathogen | The first response takes ~7 days, while a 10×/day pathogen grows 10^7-fold | + +## Concrete cases + +| Claim | Holds | Breaks | +|---|---|---| +| Memory gives a faster response | Vaccinated: response at day ~3, pathogen grew ~10^3-fold | Variant with a changed antigen: memory fits less well; response is slower | + +## Terms + +antigen, lymphocyte, receptor, repertoire, clonal expansion, memory cell, antibody, vaccine — as above. "Learn" is used only with the meaning "select and amplify", never "design". + +## Epistemic status + +- **Established:** clonal selection, random receptor generation, memory, faster secondary response. +- **Estimated:** all frequencies and days are typical orders of magnitude; they vary by antigen and person. +- **Assumed (illustration):** the 10×/day pathogen growth rate. + +## Confusion points + +- The immune system "studies" the pathogen and designs a counter → no; matching receptors already exist by chance; the vaccine selects them. +- A vaccine contains a weak dose of the disease → many contain only one protein, or instructions (mRNA) to make one. +- Antibodies are the memory → antibodies decline; memory cells are the durable record. + +## Scope + +- **Out of scope:** T cell killing, affinity maturation detail, self-tolerance (why receptors that bind your own body are removed), herd immunity — see an immunology text (for example, Janeway's *Immunobiology*). + +## Representation decision + +- **Stage:** 1 +- **Reason:** One sequential causal chain; one comparison table holds the two responses. +- **Levels:** L1 the answer / L2 the steps / L3 why it works (the race) / L4 limits. diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/model.yaml b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/model.yaml new file mode 100644 index 0000000..2ed23e8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/explainers/vaccine-memory/model.yaml @@ -0,0 +1,52 @@ +# Machine-readable half of model.md. `explainer check vaccine-memory` holds every rendering to it. + +values: + match_frequency_low_inverse: 1000000 + match_frequency_high_inverse: 100000 + repertoire_possible_exp: 11 + first_response_days_min: 7 + first_response_days_max: 14 + second_response_days_min: 2 + second_response_days_max: 4 + fold_increase_min: 10 + fold_increase_max: 100 + growth_per_day: 10 + growth_7_days: 10000000 + growth_3_days: 1000 + +allow: [1, 2, 3, 4, 5, 6, 7] + +require: + - path: first_response_days_min + in: [prose] + - path: second_response_days_max + in: [prose] + - path: growth_7_days + in: [prose] + +claims: + - id: selection-not-design + kind: mechanism + statement: "Receptors are made at random before exposure; the vaccine antigen selects the rare matching lymphocyte, which then divides." + example: "About 1 in 100,000 to 1 in 1,000,000 lymphocytes binds a given antigen." + ask: "Does the immune system build a new receptor after it sees the vaccine?" + - id: why-harmless-antigen + kind: why + statement: "The vaccine gives the slow first response without the danger of the real pathogen." + alternative: "Meet the real pathogen first." + counterexample: "A pathogen that grows 10× per day grows 10,000,000-fold during a 7-day first response." + - id: memory-faster + kind: guarantee + statement: "Memory cells make the second response faster and larger." + example: "Second response in 2–4 days at 10–100 times the antibody level; pathogen grows only about 1,000-fold in 3 days." + counterexample: "A variant with a changed antigen fits memory receptors less well, so protection drops." + ask: "Why does a vaccinated person get less sick even though the pathogen still enters the body?" + +accept_unjustified: {} + +terms: + - term: select + means: "the antigen picks an existing matching cell; nothing is designed" + avoid: [designs an antibody, builds a new receptor to fit] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/read.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/read.json new file mode 100644 index 0000000..60fdb47 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/read.json @@ -0,0 +1,50 @@ +{ + "answers": { + "1": { + "answer": "A vaccine must contain or produce an antigen (a molecular shape from the pathogen). It doesn't make you ill because it supplies this antigen without the disease—using an inactivated pathogen, purified protein, or mRNA that briefly produces one protein.", + "quote": "A vaccine supplies the antigen without the disease: an inactivated pathogen, one purified protein, or mRNA that tells your cells to make that one protein briefly." + }, + "2": { + "answer": "After vaccination, there are 10–100 times more memory cells with matching receptors already present. When the real infection arrives, the response starts from this large ready population instead of finding rare cells, so antibody response occurs in 2–4 days instead of 7–14 days.", + "quote": "After the response, most clone cells die. Some become memory cells: long-lived lymphocytes with the matching receptor. There are now about 10–100 times more matching cells than before." + }, + "3": { + "answer": "The vaccine creates memory cells specific to one virus's antigen shape. Since each lymphocyte binds only one shape, and different viruses have different antigens, memory cells from one vaccine won't recognize an unrelated virus.", + "quote": "Each lymphocyte carries one receptor type, a protein that binds one shape." + }, + "4": { + "answer": "Protection drops. The memory receptors were shaped to fit the old antigen, so when the antigen shape changes, they bind less well and provide reduced protection.", + "quote": "Memory receptors fit the old shape. A variant with a changed antigen (as with influenza each year) binds less well, so protection drops and the vaccine needs an update." + } + }, + "stops": [ + "Each new lymphocyte shuffles gene segments at random to build its receptor.", + "The number of possible receptors is more than 10^11.", + "A lymphocyte that binds the antigen, and also gets an alarm signal, starts to divide.", + "the clone mutates its receptors during expansion and the best binders are kept.", + "There are now about 10–100 times more matching cells than before.", + "an inactivated pathogen, one purified protein, or mRNA that tells your cells to make that one protein briefly.", + "The set of receptors in your body is your repertoire.", + "It already contains a few cells that fit almost any shape, including shapes from viruses that did not exist when you were born.", + "only about 1 in 100,000 to 1 in 1,000,000 lymphocytes has a receptor that binds it.", + "Over about a week, it makes a clone of thousands of identical cells." + ], + "excess": [ + "Analogy: the repertoire is a large ring of random keys. The vaccine does not cut a new key.", + "Unlike keys, the copies can also improve their fit.", + "All day counts and frequencies are typical estimates; they vary by antigen, vaccine, and person.", + "as with influenza each year", + "during chemotherapy", + "Weak immune system. A person who makes few lymphocytes (for example, during chemotherapy) cannot expand a clone well.", + "Decline over time. Antibody levels fall, and for some vaccines memory also weakens.", + "Boosters restore them (tetanus: about every 10 years).", + "the vaccine protects less.", + "Pathogen growth before response (10×/day) | 7 days: 10,000,000-fold | 3 days: 1,000-fold" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/run.json b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/run.json new file mode 100644 index 0000000..1fe0b8c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev-haiku-reader/work/vaccine-memory/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/vaccine-memory/explanation.md` as prose, with a supporting `model.md` and `model.yaml`; `explainer check` passes. I chose prose because the subject is one chain of steps, and a single table can hold the comparison between a first and a second response.", + "error": null, + "cost": 0.4694588000000001, + "seconds": 94, + "turns": 25, + "files": [ + "explainers/vaccine-memory/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/conditions.json b/evals/artifact/results/2026-10-04-dev/conditions.json new file mode 100644 index 0000000..963c67d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/conditions.json @@ -0,0 +1,6 @@ +{ + "none": "none", + "principles": "principles", + "quick": "quick", + "published": "published" +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/antibiotic-resistance.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/antibiotic-resistance.json new file mode 100644 index 0000000..f3dbe31 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/antibiotic-resistance.json @@ -0,0 +1,59 @@ +{ + "key": { + "A": "published", + "B": "principles", + "C": "quick", + "D": "none" + }, + "grades": { + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "Change the target | The target protein changes shape, so the drug cannot bind | MRSA makes a cell-wall enzyme that penicillin-type drugs do not bind (MRSA acquires an extra, different enzyme, PBP2a; its existing target does not change shape)" + ] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "published", + "principles", + "quick", + "none" + ], + "why_best": "A builds the full chain (pre-existing variation, mechanisms, selection, Luria–Delbrück evidence and horizontal transfer) in compact, checkable steps, with worked numbers a technical reader can verify and no detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/bloom-filter.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/bloom-filter.json new file mode 100644 index 0000000..9a0f9e8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/bloom-filter.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "published", + "D": "principles" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "quick", + "published", + "principles", + "none" + ], + "why_best": "A uses one small worked example to show adding, a sure 'no', the deletion failure and a false positive, so the reader gets every core idea with the least text, and its numbers check out." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/float-sum.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/float-sum.json new file mode 100644 index 0000000..280fb72 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/float-sum.json @@ -0,0 +1,61 @@ +{ + "key": { + "A": "principles", + "B": "none", + "C": "quick", + "D": "published" + }, + "grades": { + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "The digits after the point (the *significand*) get a fixed budget: **53 significant binary digits**." + ] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "The other 11 bits hold the sign and the power." + ] + } + }, + "rank": [ + "principles", + "quick", + "none", + "published" + ], + "why_best": "A gives the reason 0.1 cannot be stored (the prime-factor rule, anchored by the 1/3 analogy), then the exact stored values, the tie-rounded sum, why it prints as it does and the fixes, all in a compact text without detours, while C asserts that 0.1 repeats in binary without saying why, B adds side topics and D asks the reader to work through heavy unit arithmetic." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/gps-relativity.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/gps-relativity.json new file mode 100644 index 0000000..affdcbf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/gps-relativity.json @@ -0,0 +1,59 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "principles", + "D": "published" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "Three spheres meet at a point." + ] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "principles", + "published", + "none", + "quick" + ], + "why_best": "C states outright that the receiver only listens, then builds each step from the one before (time to distance, distance to position, clock bias, why a shared bias cannot cover per-satellite clock errors, relativity, the fix) with checked numbers and an honest account of what the 11 km figure means, and it does so without B's length and detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/greenhouse.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/greenhouse.json new file mode 100644 index 0000000..b116665 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/greenhouse.json @@ -0,0 +1,59 @@ +{ + "key": { + "A": "none", + "B": "published", + "C": "principles", + "D": "quick" + }, + "grades": { + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 0, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "\"so the infrared escapes to space only from high, cold air\"" + ] + } + }, + "rank": [ + "principles", + "quick", + "none", + "published" + ], + "why_best": "C gives this reader every step the quiz needs in order: the 255 K budget, the visible-versus-infrared split, the dipole reason, the one-layer model and why it fails, then emission height with the lapse rate. It shows its arithmetic, and it avoids A's blank equations and companion-page detours, D's raw flowchart code and inconsistent numbers (3.8 vs 3.7 W/m², 1 K vs 1.2 K), and B's missing molecular explanation." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/heat-pump.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/heat-pump.json new file mode 100644 index 0000000..ed2143a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/heat-pump.json @@ -0,0 +1,59 @@ +{ + "key": { + "A": "published", + "B": "principles", + "C": "quick", + "D": "none" + }, + "grades": { + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 1 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 1 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "absolute zero, where molecular motion stops" + ] + } + }, + "rank": [ + "published", + "none", + "quick", + "principles" + ], + "why_best": "A builds the model in clear layers, from the energy balance to the refrigerant loop (with why each step is needed) to a worked entropy derivation of the kelvin-based ideal limit using numbers that match its example, so the reader can answer every question, including the 5 °C vs −15 °C comparison, without gaps, missing equations or errors." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/key-exchange.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/key-exchange.json new file mode 100644 index 0000000..66977ff --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/key-exchange.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "none", + "B": "quick", + "C": "principles", + "D": "published" + }, + "grades": { + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "quick", + "principles", + "published", + "none" + ], + "why_best": "B covers every needed step (the operation, why the two results agree, why it is hard, the worked numbers and the man-in-the-middle fix) in the fewest words, and its short contrasts with multiplication and with plain powers explain why the method takes this form without detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/quorum.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/quorum.json new file mode 100644 index 0000000..de74172 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/quorum.json @@ -0,0 +1,59 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "published", + "D": "principles" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "However you split 5, one side always has at least 3, so exactly one side carries on. (False for a three-way split such as 2–2–1, where no side has a majority.)" + ] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "quick", + "principles", + "published", + "none" + ], + "why_best": "A defines every term and reaches the counting rule, the overlap argument (2q > n), the 2–2 split counterexample and the odd/even pattern in the fewest words, with no detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/two-tides.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/two-tides.json new file mode 100644 index 0000000..d1f00cf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/two-tides.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "principles", + "B": "none", + "C": "published", + "D": "quick" + }, + "grades": { + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 0, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 0, + 2 + ], + "misconception": "absent", + "errors": [] + } + }, + "rank": [ + "principles", + "none", + "published", + "quick" + ], + "why_best": "A covers every quiz point in a compact sequence: free fall, a worked three-point table, the 1/d³ scaling, the lunar day and spring/neap tides. It also explicitly refutes the Earth-spin centrifugal misconception, without B's broken placeholders (missing equations, missing code, empty link) and appended UI text." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades-stripped-text/vaccine-memory.json b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/vaccine-memory.json new file mode 100644 index 0000000..e0386db --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades-stripped-text/vaccine-memory.json @@ -0,0 +1,63 @@ +{ + "key": { + "A": "none", + "B": "principles", + "C": "published", + "D": "quick" + }, + "grades": { + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "Across the body's hundreds of billions of lymphocytes, the library contains millions of different shapes." + ] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "A vaccine does not stop the pathogen from entering. It shortens the race." + ] + }, + "quick": { + "quiz": [ + 1, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "Many have a better fit, because the clone mutates its receptors during expansion and the best binders are kept." + ] + } + }, + "rank": [ + "published", + "principles", + "none", + "quick" + ], + "why_best": "C builds the whole mechanism as one quantitative race. It defines each term at first use, and its arithmetic shows exactly why memory shortens the response from about 7 to about 3 days. It names what it leaves out instead of detouring into it, so every quiz answer follows from a single coherent model with minimal extra reading." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/antibiotic-resistance.json b/evals/artifact/results/2026-10-04-dev/grades/antibiotic-resistance.json new file mode 100644 index 0000000..0b376dd --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/antibiotic-resistance.json @@ -0,0 +1,60 @@ +{ + "key": { + "A": "published", + "B": "principles", + "C": "quick", + "D": "none" + }, + "grades": { + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "Then every cell would have the same chance to survive. Instead, a rare fraction survives, and those cells were resistant before they met the drug.", + "Change the target | The target protein changes shape, so the drug cannot bind | MRSA makes a cell-wall enzyme that penicillin-type drugs do not bind" + ] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "published", + "principles", + "quick", + "none" + ], + "why_best": "A covers variation, selection, and cross-species plasmid transfer in the fewest words, and its worked numbers and Luria–Delbrück variance argument show directly why the drug selects resistant cells rather than creating them." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/bloom-filter.json b/evals/artifact/results/2026-10-04-dev/grades/bloom-filter.json new file mode 100644 index 0000000..b3a1c74 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/bloom-filter.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "published", + "D": "principles" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "published", + "principles", + "quick", + "none" + ], + "why_best": "C states up front that the filter keeps only bits, and it runs one small example through add, a correct 'no', a false positive and a failed delete. It then derives the fill fraction f and the rate p ≈ f^k and shows overfilling numerically, so the reader can answer even the all-bits-set case straight from the text, without detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/float-sum.json b/evals/artifact/results/2026-10-04-dev/grades/float-sum.json new file mode 100644 index 0000000..a966f29 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/float-sum.json @@ -0,0 +1,61 @@ +{ + "key": { + "A": "principles", + "B": "none", + "C": "quick", + "D": "published" + }, + "grades": { + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "The digits after the point (the *significand*) get a fixed budget: **53 significant binary digits**." + ] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "The other 11 bits hold the sign and the power." + ] + } + }, + "rank": [ + "principles", + "none", + "quick", + "published" + ], + "why_best": "A gives the full causal chain (the prime-factor rule, stored values as integer × power of 2, the tie rounding and the printing behaviour) in a short, exact form with no detours, so the reader gets a correct model with little effort." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/gps-relativity.json b/evals/artifact/results/2026-10-04-dev/grades/gps-relativity.json new file mode 100644 index 0000000..bcba5d2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/gps-relativity.json @@ -0,0 +1,59 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "principles", + "D": "published" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "Three spheres meet at a point. (Three spheres generally meet at two points; the receiver discards one of them.)" + ] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "published", + "principles", + "quick", + "none" + ], + "why_best": "D builds each idea in order with no detours: it says the receiver only listens, makes the clock bias b concrete with a small 1-D worked example, sets a nanosecond budget, and explains why a shared b cannot absorb satellite drift that differs between satellites." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/greenhouse.json b/evals/artifact/results/2026-10-04-dev/grades/greenhouse.json new file mode 100644 index 0000000..5cb275b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/greenhouse.json @@ -0,0 +1,60 @@ +{ + "key": { + "A": "none", + "B": "published", + "C": "principles", + "D": "quick" + }, + "grades": { + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 0, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "\"the infrared escapes to space only from high, cold air, about 5 km up\" (overstated: through the 8–13 µm window much surface infrared escapes directly, and the text never mentions this)", + "\"The new emitting air is 1 K colder... The surface... warms by the same amount: about 1.2 K before feedbacks\" (inconsistent: the 1 K drop it just derived implies about 1 K of warming, and the 1.2 K figure is given without its source)" + ] + } + }, + "rank": [ + "principles", + "none", + "quick", + "published" + ], + "why_best": "C builds every link the reader needs (energy balance, the wavelength split, the dipole reason, a one-layer model and why it fails, then emission height with the lapse rate, its limits, and worked numbers) in a tight order with prediction prompts, without A's extra detours and length or the gaps in D and B." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/heat-pump.json b/evals/artifact/results/2026-10-04-dev/grades/heat-pump.json new file mode 100644 index 0000000..b972b6e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/heat-pump.json @@ -0,0 +1,60 @@ +{ + "key": { + "A": "published", + "B": "principles", + "C": "quick", + "D": "none" + }, + "grades": { + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 1 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 1 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [ + "absolute zero, where molecular motion stops", + "the loop diagram routes the compressor's output down to the expansion valve and the condenser's output to the compressor, which contradicts the text order evaporator → compressor → condenser → valve" + ] + } + }, + "rank": [ + "published", + "none", + "principles", + "quick" + ], + "why_best": "A builds the model in clear steps: the energy balance, then the four-part loop with consistent numbers at every stage, then the kelvin formula with a worked entropy check of an allowed and a forbidden COP. That made all four answers exact, without D's detours (analogy, code, air-conditioning) or the refrigerant-temperature tables in B and C that led their readers to the wrong limit in Q4." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/key-exchange.json b/evals/artifact/results/2026-10-04-dev/grades/key-exchange.json new file mode 100644 index 0000000..cdd3cfe --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/key-exchange.json @@ -0,0 +1,59 @@ +{ + "key": { + "A": "none", + "B": "quick", + "C": "principles", + "D": "published" + }, + "grades": { + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "\"19 = 5^15 and 8 = 5^6\" (these hold only mod 23, not as plain equalities)" + ] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + } + }, + "rank": [ + "quick", + "principles", + "published", + "none" + ], + "why_best": "B covers the operation, why it is hard, why both sides get the same key, the worked p=23 example and the man-in-the-middle fix in the fewest words, with no detours beyond what this reader needs." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/quorum.json b/evals/artifact/results/2026-10-04-dev/grades/quorum.json new file mode 100644 index 0000000..f76e46f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/quorum.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "quick", + "B": "none", + "C": "published", + "D": "principles" + }, + "grades": { + "quick": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + } + }, + "rank": [ + "quick", + "principles", + "published", + "none" + ], + "why_best": "A gives the counting rule up front, then derives the majority from a single split-brain example and the 2q > n overlap condition, and shows that an even-sized cluster adds no tolerance, all in the fewest words and with no detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/two-tides.json b/evals/artifact/results/2026-10-04-dev/grades/two-tides.json new file mode 100644 index 0000000..e4a7839 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/two-tides.json @@ -0,0 +1,57 @@ +{ + "key": { + "A": "principles", + "B": "none", + "C": "published", + "D": "quick" + }, + "grades": { + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "published": { + "quiz": [ + 2, + 2, + 0, + 2 + ], + "misconception": "corrected", + "errors": [] + }, + "quick": { + "quiz": [ + 2, + 2, + 0, + 2 + ], + "misconception": "absent", + "errors": [] + } + }, + "rank": [ + "principles", + "none", + "published", + "quick" + ], + "why_best": "A covers every quiz point and names and refutes the spin-centrifugal misconception, using compact worked numbers, without B's long detours (code, a broken companion-page dump) and without the spring-tide gap in C and D." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/grades/vaccine-memory.json b/evals/artifact/results/2026-10-04-dev/grades/vaccine-memory.json new file mode 100644 index 0000000..551fbf6 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/grades/vaccine-memory.json @@ -0,0 +1,61 @@ +{ + "key": { + "A": "none", + "B": "principles", + "C": "published", + "D": "quick" + }, + "grades": { + "none": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "Inactivated (e.g. many flu shots, polio IPV) | Whole pathogen, killed | Can't cause infection; usually needs adjuvant and boosters: most inactivated flu shots and IPV contain no adjuvant." + ] + }, + "principles": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [ + "Across the body's hundreds of billions of lymphocytes, the library contains millions of different shapes: this understates the repertoire, which is on the order of 10^8 or more distinct receptors." + ] + }, + "published": { + "quiz": [ + 2, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + }, + "quick": { + "quiz": [ + 1, + 2, + 2, + 2 + ], + "misconception": "absent", + "errors": [] + } + }, + "rank": [ + "published", + "principles", + "none", + "quick" + ], + "why_best": "C carries one consistent worked model (rare precursors, division count, memory pool size, the race against a pathogen growing 10× per day) from start to finish, says explicitly that the vaccine contains no pathogen that can multiply, defines every term it uses, and leaves side topics to a short 'not covered' list, so the reader gets every quiz point without detours." +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/rating/.key.json b/evals/artifact/results/2026-10-04-dev/rating/.key.json new file mode 100644 index 0000000..8f77b4a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/.key.json @@ -0,0 +1,17 @@ +{ + "key-exchange": { + "A": "published", + "B": "principles", + "C": "quick" + }, + "greenhouse": { + "A": "published", + "B": "principles", + "C": "quick" + }, + "gps-relativity": { + "A": "quick", + "B": "published", + "C": "principles" + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/A.md b/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/A.md new file mode 100644 index 0000000..6705031 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/A.md @@ -0,0 +1,51 @@ +# How does GPS find your position, and why does it need relativity? + + + +**GPS finds your position by timing radio signals. Each satellite sends the time it transmitted. Your receiver measures how long each signal took to arrive, and multiplies by the speed of light to get a distance. Four distances fix your position. Relativity matters because the satellite clocks tick faster than ground clocks by 38 microseconds per day. At the speed of light, that error is about 11 km of distance per day.** + +## Step 1: time becomes distance + +About 30 GPS satellites orbit at 20,200 km altitude. Each satellite carries an atomic clock. Each satellite continuously broadcasts a **signal**: a radio message that says "I sent this at time t, and I was at position (x, y, z)". + +Your receiver notes when the signal arrives. The **travel time** is receive time minus send time. For a satellite straight overhead, the travel time is about 67 ms (20,200 km ÷ 299,792,458 m/s). + +Distance = travel time × c. Light travels about 30 cm per nanosecond, so the clocks must agree to within a few nanoseconds for metre-level accuracy. + +## Step 2: why four satellites, not three + +One distance puts you somewhere on a sphere around one satellite. Three spheres meet at a point. So three satellites should be enough. + +But your phone has a cheap quartz clock, not an atomic clock. Suppose the phone clock is 1 ms wrong. Then every distance is wrong by 1 ms × c = 300 km. The three spheres no longer meet at your true position. + +GPS solves this by treating the clock error as a fourth unknown. The **receiver clock bias** b is the offset of your clock from GPS time. Each measured distance is a **pseudorange** ρ: the true distance plus the same error c·b. For satellite i at (xᵢ, yᵢ, zᵢ): + +```text +ρᵢ = √((x − xᵢ)² + (y − yᵢ)² + (z − zᵢ)²) + c·b +``` + +There are four unknowns (x, y, z, b), so the receiver needs four equations: four satellites. It solves them by iteration: guess a position, compute the predicted pseudoranges, and correct the guess until the predicted values match the measured ones. A side effect: your phone gets the time accurate to nanoseconds. + +This method works only if the satellite clocks all keep GPS time. Relativity says that they do not, if nobody corrects them. + +## Step 3: two effects change the satellite clock rate + +**Predict first:** a satellite moves fast and sits high above Earth. Does its clock run fast or slow compared with a clock on the ground? + +Two effects pull in opposite directions. + +1. **Speed slows the clock (special relativity).** A moving clock ticks slower by a fraction of about v²/(2c²). The orbit speed is 3.87 km/s, which gives 8.3 × 10⁻¹¹. Over one day (86,400 s), the satellite clock loses about 7 µs. +2. **Weaker gravity speeds the clock up (general relativity).** A clock deeper in a gravity well ticks slower. A ground clock sits at 6,371 km from Earth's center. A satellite sits at 26,560 km, where gravity is weaker. The fraction is about (GM/c²)(1/R_Earth − 1/r) = 5.3 × 10⁻¹⁰. Over one day, the satellite clock gains about 45 µs. + +The gravity effect wins. Net: the satellite clock gains about 38 µs per day. + +## Step 4: why 38 µs matters + +38 µs looks small. Multiply by c: 38 µs × 300 m per µs ≈ 11 km. The system needs agreement to a few nanoseconds. The drift is 0.44 ns per second, so after one minute it is 26 ns, or 8 m. After one day, the satellite time stamps would be off by the light-travel time of 11 km. + +The fix is simple. Engineers set each satellite clock to tick slow before launch. The nominal frequency is 10.23 MHz; the clock is set to 10.22999999543 MHz. In orbit, relativity speeds the clock up to exactly 10.23 MHz as seen from the ground. + +## Limits + +- **Estimate:** "11 km per day" is the light-travel distance of 38 µs. The actual position error would grow in a more complex way, because the bias b absorbs part of an error that all satellites share. The drift would still break the system within hours. +- **Out of scope:** Earth's rotation during signal travel (the Sagnac effect), a small extra correction for slightly elliptical orbits (tens of ns, computed by the receiver), and signal delay in the ionosphere. diff --git a/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/B.md b/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/B.md new file mode 100644 index 0000000..ff7aaeb --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/B.md @@ -0,0 +1,75 @@ +# How does GPS find your position, and why does it need relativity? + +**A GPS receiver times radio signals from four satellites and solves for its position and its own clock error. This works only if the satellite clocks agree to a few billionths of a second. Relativity makes those clocks gain 38 µs a day on ground clocks. Light covers 11.4 km in 38 µs, so engineers build the clocks to tick slow.** + +## 1. Time gives distance + +About 30 GPS satellites orbit 20,200 km above Earth. Each one carries atomic clocks and broadcasts a signal that says: "I am satellite k, I am at position P, and I sent this at time t." Your receiver (the GPS chip in a phone) only listens. + +The receiver notes when each signal arrives. Then: + +- travel time = receive time − send time +- range = travel time × c, where c is light speed and the range is the true distance to the satellite. + +From a satellite directly overhead, the travel time is about 67 ms. Light covers 300 m in one microsecond (1 µs = 10⁻⁶ s) and 0.3 m in one nanosecond (1 ns = 10⁻⁹ s). So every timing error becomes a range error at that rate. + +One range puts you somewhere on a sphere around the satellite: every point at that distance. Three spheres meet in two points. The receiver rejects the one far out in space. Three satellites would be enough, if every clock were perfect. + +## 2. The receiver's clock is wrong, so it uses a fourth satellite + + +Your phone's clock is quartz, not atomic. It can easily be off by 1 ms, and 1 ms × c = 300 km. + + +So the receiver does not compute a range. It computes a **pseudorange**: light speed × (receive time on the receiver clock − send time on the satellite clock). A pseudorange equals the range plus **b**, the receiver clock error expressed in metres. One clock makes every measurement, so b is the same in every pseudorange. That makes b one more unknown to solve for, next to the position. + + +**Worked example, on a line.** Put beacon A at 0 km and beacon B at 30 km. The receiver is at 12 km, so the true travel times are 40 µs and 60 µs. The receiver clock is 5 µs fast, so it measures 45 µs and 65 µs. Those give pseudoranges of 13.5 km and 19.5 km. + +The receiver lies between the beacons, so the two ranges must add up to 30 km. The pseudoranges add up to 33 km. The excess, 3 km, is b counted twice, so b = 1.5 km (5 µs × c). Subtract b from each: 12 km and 18 km. The receiver is at 12 km. With only beacon A, the receiver could not detect the 1.5 km error. + +In three dimensions the unknowns are x, y, z, and b. Four unknowns need four equations, so the receiver needs four satellites. +## 3. The satellite clocks must be almost perfect + + +The receiver can solve for its own clock error because every pseudorange shares it. A satellite clock error sits in only one pseudorange, so no unknown absorbs it. Each satellite clock must therefore match **GPS time**, the time kept by atomic clocks at ground stations. + +How close? Each nanosecond of satellite clock error is 0.3 m of range error. For a fix within about 5 m, the satellite clocks must agree with GPS time to about 17 ns. + +## 4. Relativity moves the satellite clocks by microseconds a day + + +**Time dilation** means that clocks in different motion, or at different heights in gravity, tick at different rates. It is measured, not only predicted. Two effects act on a satellite clock, compared with a clock on the ground: + +| Cause | Theory | Effect on the satellite clock | +|---|---|---| +| Speed: 3.9 km/s in orbit | special relativity: a moving clock ticks slower | loses 7 µs per day | +| Height: gravity is weaker 20,200 km up | general relativity: a clock higher above Earth ticks faster | gains 45 µs per day | +| **Net** | | **gains 38 µs per day** | + +The gravity effect is larger, so the clock runs fast. 38 µs is only 4.5 × 10⁻¹⁰ of a day, but light covers 11.4 km in 38 µs. Each uncorrected satellite clock would move 11.4 km of range further from GPS time every day. + +## 5. Why the receiver's b does not cancel the drift + + +Every satellite clock drifts at the same rate. Why does b not absorb the drift? + +Only if every clock had the same error at the same moment. If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right. + +But each satellite clock is set to GPS time at a different moment. Suppose satellite A is set on day 0 and satellite B on day 3. On day 5, A is 190 µs fast (57 km) and B is 76 µs fast (22.8 km). The 34 km difference is not shared, so no single b can absorb it, and the position fix fails. Every clock set on a different day carries a different error. + +## 6. The remedy: make the clocks tick slow before launch + + +Each satellite clock counts the ticks of a 10.23 MHz signal. Before launch, engineers set it to 10.22999999543 MHz: slower by 4.5 × 10⁻¹⁰, the fraction the clock gains in orbit. On the ground the clock runs slow. In orbit, relativity speeds it up to the rate of GPS time. Ground stations upload small corrections for what remains; the built-in offset keeps those corrections small instead of growing by 38 µs a day. + +## Summary + +Four pseudoranges give x, y, z, and the receiver's clock error, but only if satellite clocks stay within about 17 ns of GPS time. Relativity makes them gain 38 µs a day, so each clock is built to tick slow. + +## Not covered here + +- **Atmosphere:** charged and humid layers of the atmosphere slow the signal. These delays are the largest remaining errors. +- **Earth's rotation during the 67 ms flight (Sagnac effect)** and the **orbit-shape correction** (orbits are slightly elliptical, so each satellite clock error swings by up to about 23 ns): receivers correct both. See N. Ashby, "Relativity in the Global Positioning System", *Living Reviews in Relativity* (2003). +- **Why a moving clock ticks slow:** see the "light clock" argument in special relativity. +- **How the receiver solves the equations:** iteration from a guess, with a best fit when it hears more than four satellites. diff --git a/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/C.md b/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/C.md new file mode 100644 index 0000000..a95d00a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/gps-relativity/C.md @@ -0,0 +1,104 @@ +# How GPS finds your position, and why it needs relativity + +**The short answer.** A GPS receiver does not send anything. It only listens. Each satellite broadcasts its position and the exact time it sent its signal. The receiver measures how long each signal took to arrive. Then it multiplies that time by the speed of light to get a distance. With distances to four satellites, it can solve for where it is. + +This works only if the satellite clocks are correct to a few billionths of a second (nanoseconds). Relativity says a clock in orbit ticks at a different rate from a clock on the ground: about 38 microseconds per day faster. GPS builds that correction into every satellite and every receiver. + +Below, each step follows from the one before it. + +--- + +## 1. Time becomes distance + +Radio signals travel at the speed of light, *c* ≈ 300,000 km/s. The distance to a satellite is: + +``` +distance = c × (time received − time sent) +``` + +**Example.** GPS satellites orbit about 20,200 km above the ground. A signal from a satellite directly overhead takes about 0.067 s (67 ms) to arrive. A satellite near the horizon is farther away, so its signal takes up to about 86 ms. + +This conversion is unforgiving. Light travels **30 cm in one nanosecond**. A timing error of 10 ns gives a distance error of 3 m. A timing error of 1 microsecond (1,000 ns) gives 300 m. So GPS is really a system for distributing very precise time. + +## 2. Distances become a position + +One distance tells you that you are somewhere on a sphere around that satellite. + +- Two spheres intersect in a circle. +- A third sphere cuts that circle at two points. One point is near the Earth. The other is far out in space, so the receiver discards it. + +So three exact distances give a position. A flat version of the same idea: if you are 3 km from tower A, 4 km from tower B, and 5 km from tower C, only one spot on the map fits all three. + +## 3. The receiver's clock is the weak link, so use a fourth satellite + +Each satellite carries atomic clocks. Your phone has a cheap quartz clock. Suppose the quartz clock is wrong by 1 millisecond. Then every distance is wrong by 300 km, and the three spheres do not meet at a sensible point. + +The simplest fix is an atomic clock in every receiver. That fails on cost, size, and power. GPS uses a better trick: treat the receiver clock error *b* as a fourth unknown. The receiver's error adds the same amount, *c·b*, to every measured distance. For each satellite *i* at known position (*xᵢ, yᵢ, zᵢ*): + +``` +measured_distanceᵢ = √((x − xᵢ)² + (y − yᵢ)² + (z − zᵢ)²) + c·b +``` + +There are four unknowns (*x, y, z, b*), so the receiver needs four satellites. It solves the equations numerically, usually with more than four satellites to average out noise. A bonus: once it knows *b*, the receiver knows the true time to within tens of nanoseconds. + +**What this trick cannot fix.** The term *c·b* removes only an error that is **the same for every signal**. If one satellite's clock is off by 30 ns and the others are not, that satellite's distance is off by 9 m, and the solve cannot detect it. So every satellite clock must agree with one shared time scale, called **GPS time**, to within a few nanoseconds. Relativity enters here. + +## 4. Clocks in orbit do not tick at ground rate + +*Before you read on, predict: does a satellite clock run fast or slow compared with a clock on the ground?* + +Two established effects act on the satellite clock. Both are measured in experiments, not only predicted. They act in opposite directions. + +**Effect 1: speed (special relativity).** A moving clock ticks slower than a clock at rest, as seen by the clock at rest. For speeds much less than *c*, the clock loses a fraction *v²/2c²* of its time. A GPS satellite moves at about 3.87 km/s: + +``` +v² / 2c² = (3,870 m/s)² / (2 × (3.0×10⁸ m/s)²) ≈ 8.3 × 10⁻¹¹ +× 86,400 s per day ≈ 7.2 µs per day slower +``` + +**Effect 2: gravity (general relativity).** A clock deeper in a gravitational field ticks slower. The ground clock sits closer to the Earth's centre (6,371 km) than the satellite (26,560 km). So the satellite clock ticks *faster*. The fraction is the difference in gravitational potential divided by *c²*: + +``` +(GM/c²) × (1/R_earth − 1/r_orbit) += 4.43 mm × (1/6,371 km − 1/26,560 km) ≈ 5.29 × 10⁻¹⁰ +× 86,400 s per day ≈ 45.7 µs per day faster +``` + +(*GM/c²* for the Earth is a length, about 4.43 mm. It sets how strong the effect is.) + +**Net result.** Gravity wins: + +``` ++45.7 µs/day − 7.2 µs/day ≈ +38.5 µs/day +``` + +**What that number means.** 38.5 µs × 30 cm/ns ≈ **11.5 km**. That is the distance error that one day of uncorrected drift represents. Compare it with the few-nanosecond budget from section 3. The drift is about 0.45 ns per second, so it uses up that budget in about 10 seconds. + +## 5. How GPS corrects it + +**Before launch.** Engineers set each satellite clock slightly slow on the ground. The nominal clock frequency is 10.23 MHz. The factory setting is **10.22999999543 MHz**, which is lower by the net fraction 4.46 × 10⁻¹⁰. In orbit, the two relativistic effects speed the clock up by exactly that fraction. Seen from the ground, it then ticks at 10.23 MHz. + +**In the receiver.** The fixed offset assumes a perfectly circular orbit. Real orbits are slightly elliptical. As a satellite moves closer to the Earth and then farther away, its speed and gravity change, so its clock rate changes too. For an orbit with eccentricity 0.02, this swings the clock by up to about ±46 ns (±14 m). The amount differs for each satellite, so the fourth-satellite trick cannot absorb it. The receiver calculates this term from the broadcast orbit data and subtracts it. + +**From the ground.** Control stations track every satellite clock and upload small corrections for the remaining drift. Relativity is built into the model that these corrections assume. + +## 6. What if relativity were ignored? + +You often read that GPS would "drift 10 km per day" without relativity. That figure assumes the receiver trusts the satellite clocks. It is a fair measure of the size of the effect. It is not an exact forecast: + +- **The part common to all satellites** would partly cancel. It shifts every distance equally, so the receiver would absorb it into *b*, like its own clock error. The *time* the receiver reports would still drift by 38.5 µs per day. Power grids, phone networks, and financial exchanges use GPS time and would fail. +- **The parts that differ between satellites** would not cancel. Each satellite's eccentricity term alone produces errors of up to about 14 m. Satellite clocks would also drift away from GPS time, and from the ground stations that track them. + +Either way, GPS cannot meet its metre-level accuracy without relativity. + +--- + +## Summary + +1. The receiver measures signal travel time; time × *c* = distance. 1 ns = 30 cm. +2. Three distances fix a point. A fourth satellite removes the receiver's own clock error. +3. That trick removes only a shared error, so satellite clocks must agree to nanoseconds. +4. In orbit, speed slows the clock by 7.2 µs/day and weaker gravity speeds it up by 45.7 µs/day: net +38.5 µs/day, or 11.5 km of light travel. +5. GPS sets the clocks slow before launch and corrects the per-satellite orbit effect in the receiver. + +**Not covered here:** signal delays in the atmosphere, the Sagnac effect (the Earth rotates while the signal travels), how orbits are predicted, and centimetre-level methods such as RTK. diff --git a/evals/artifact/results/2026-10-04-dev/rating/greenhouse/A.md b/evals/artifact/results/2026-10-04-dev/rating/greenhouse/A.md new file mode 100644 index 0000000..81b640b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/greenhouse/A.md @@ -0,0 +1,91 @@ +# How do greenhouse gases warm the Earth? + + + +**Greenhouse gases make the Earth radiate its energy to space from high, cold air instead of from the warm surface. Cold air emits less, so the whole lower atmosphere, and the surface with it, must be warmer before enough energy leaves.** + +The rest of this page puts numbers on each step. + +## 1. Without greenhouse gases: 255 K + + +Start from conservation of energy. Over years, the Earth must emit as much power to space as it absorbs from the Sun. This is the **energy balance**. + +- Sunlight delivers 1361 W/m² to a panel that faces the Sun. +- The Earth catches sunlight on a disc of area πR², but it emits from its whole sphere, of area 4πR². Per square metre of surface, the average input is 1361 / 4 = 340 W/m². +- The **albedo** is the fraction of sunlight that clouds, ice, and ground reflect straight back to space: 0.30. The Earth absorbs 340 × 0.70 ≈ 240 W/m². + +A body at temperature T emits σT⁴ watts per square metre (the Stefan–Boltzmann law, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴). Emission grows fast with temperature: 255 K emits 240 W/m², and 288 K emits 390 W/m². + +Set σT⁴ = 240 W/m² and solve: T = 255 K (−18 °C). This is the **emission temperature**: whatever emits the Earth's energy to space must be at 255 K. If the surface emitted directly to space, the surface would be at 255 K. + + +The division by 4 matters: with 1361 × 0.70 instead, T = 360 K (87 °C), as if every point on Earth faced the noon Sun all day. + +The measured average surface temperature is 288 K (15 °C). The surface is 33 K warmer than the emission temperature. The greenhouse gases cause that 33 K. + +## 2. What greenhouse gases do + + +Sunlight peaks near 0.5 µm, in the visible. The Earth is far cooler than the Sun, so it emits at longer wavelengths, near 10 µm. This is **infrared**: invisible radiation from the ground and the air. + +A **greenhouse gas** lets most sunlight pass but absorbs and emits infrared. Water vapour, CO₂, and methane are greenhouse gases. Nitrogen and oxygen, 99% of dry air, are not: they barely absorb infrared at all. + +So sunlight reaches the ground. But infrared from the ground is absorbed by the air above it and emitted again, in all directions, at the temperature of that air. + +## 3. Where the infrared escapes: 5 km up + + +Infrared reaches space only from air that has little greenhouse gas above it. Lower down, the air above absorbs it again. Infrared really escapes from a spread of heights, which differ by wavelength. Call their average the **emission height**. + +The air at the emission height must emit 240 W/m², so it must be at the emission temperature, 255 K. + +The air gets colder with height. Rising air expands and cools, and convection keeps mixing the lower atmosphere. The measured fall is about 6.5 K per km, the **lapse rate**. At this rate, the air reaches 255 K about 5 km up. That is the emission height. + +| Height | Temperature | +|---|---| +| 0 km (surface) | 288 K | +| 5 km (emission height) | 255 K | + +Now the 33 K has a cause: 255 K + 6.5 K/km × 5 km ≈ 288 K. The surface is warmer than the emitting air because it sits 5 km below it. + + +The surface at 288 K emits 390 W/m², more than the 240 W/m² that leaves the top. The surface does not lose that much, because the warm air just above it emits about 330 W/m² of infrared back down (a measured value). + +## 4. Adding CO₂ + +**Predict first:** CO₂ doubles, from its pre-industrial 280 ppm to 560 ppm. What happens to the emission height, and then to the surface? + + +1. More gas means the infrared must start higher before little gas remains above it. The emission height rises, on average by about 150 m (a result of radiation calculations). +2. The air 150 m higher is about 1 K colder (6.5 K/km × 0.15 km). +3. Colder air emits less: σ·254⁴ ≈ 236 W/m², against 240 W/m² at 255 K. The Earth now emits about 3.7 W/m² less than it absorbs. This imbalance, just after the change and before any warming, is the **radiative forcing**. It is small next to 240 W/m², but it persists every second until something changes. +4. The Earth gains energy and warms. Convection still sets the lapse rate, so the temperature at every height rises by the same amount. (This assumes the lapse rate stays at 6.5 K/km; in reality it changes a little, one of the feedbacks below.) +5. Warming stops when the air at the new emission height is back at 255 K. The surface is then about 1 K warmer, because it now sits 150 m further below the emitting air. + +This 1 K counts only this mechanism. Warmer air holds more water vapour, which is itself a greenhouse gas; with this and other feedbacks, the estimated warming per doubling is about 3 K, likely 2.5–4 K (assessment of the Intergovernmental Panel on Climate Change, IPCC, from models and observations). + +## 5. The condition: colder air above + + +The usual picture, "greenhouse gases trap heat like a blanket", says more gas always warms. A test case shows that it does not. + +Suppose the atmosphere had the same temperature at every height (a lapse rate of 0). Add CO₂. The emission height still rises, but the air there is still 255 K. It emits the same 240 W/m². There is no imbalance and no warming. + +So the warming depends on two things together: the gas lifts the emission height, and the air is colder higher up. + +## 6. "But CO₂ is already saturated" + + +Near the ground, CO₂ already absorbs almost all infrared at its main wavelength (15 µm). People conclude that more CO₂ can do nothing. But what reaches space is decided high up, not near the ground. High up, the air is thin, and infrared emitted there still escapes. Add CO₂, and that thin air absorbs more, so the escape moves up again, to colder air. Each doubling adds about the same forcing: 280 → 560 ppm gives 3.7 W/m², and 560 → 1120 ppm gives 3.7 W/m² again. Today's 420 ppm gives about 2.2 W/m². + +## Summary + +Energy balance requires the Earth to emit 240 W/m² to space, which holds the emitting air at 255 K. Greenhouse gases move that emitting air up to about 5 km. The lapse rate of 6.5 K/km makes the surface 33 K warmer, at 288 K. More greenhouse gas lifts the emission height and warms the surface. + +## Not covered here + +- Feedbacks in detail (water vapour, clouds, ice): IPCC AR6 Working Group I, Chapter 7. +- Why only some molecules absorb infrared: a molecular spectroscopy text. +- Why each doubling gives the same forcing: Pierrehumbert, *Principles of Planetary Climate*. +- Where added CO₂ goes (the carbon cycle), and how fast the ocean lets the surface warm. diff --git a/evals/artifact/results/2026-10-04-dev/rating/greenhouse/B.md b/evals/artifact/results/2026-10-04-dev/rating/greenhouse/B.md new file mode 100644 index 0000000..0bb9f11 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/greenhouse/B.md @@ -0,0 +1,127 @@ +# How greenhouse gases warm the Earth + +**Short answer.** Earth loses heat to space only as infrared light. Greenhouse gases absorb infrared light, so most of the infrared that reaches space comes from high in the atmosphere, not from the surface. High air is cold, and cold air emits less infrared. To send out as much energy as it gets from the Sun, the whole lower atmosphere and the surface must become warmer. More CO2 lifts the emitting altitude higher, into colder air, so the surface must warm more. + +The sections below build this answer step by step, with numbers. + +## 1. The puzzle: Earth is 33 K too warm + +Energy that enters a planet must leave it, or the planet keeps warming. Start with the energy that enters. + +- Sunlight at Earth's distance delivers **1361 W/m²** to a surface that faces the Sun. +- Earth intercepts sunlight with a disc of area πR², but it has a surface of area 4πR². Spread over the whole sphere, the mean input is 1361 / 4 ≈ **340 W/m²**. +- Clouds, ice, and other bright surfaces reflect about 30% of it. Earth absorbs 340 × 0.70 ≈ **238 W/m²**. + +Now the energy that leaves. Every object emits thermal radiation. The **Stefan–Boltzmann law** gives the power per square metre from an object at absolute temperature T: + + P = σT⁴, with σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴. + +The T⁴ makes emission very sensitive to temperature: 10% warmer means about 46% more emission. + +Set emission equal to absorption and solve for T: + + T = (238 / 5.67 × 10⁻⁸)^(1/4) ≈ **255 K** (−18 °C). + +The measured mean surface temperature is **288 K** (15 °C). The surface is 33 K warmer than the simple balance predicts. A surface at 288 K emits σ × 288⁴ ≈ **390 W/m²**, but only about 238 W/m² leaves the top of the atmosphere. The atmosphere stops about **150 W/m²** of the surface emission from reaching space. The rest of this explanation shows how. + +## 2. Sunlight and Earth-light are different colours + +The wavelength at which an object emits most strongly falls as the object gets hotter (**Wien's law**: peak wavelength ≈ 2898 µm·K / T). + +| Emitter | Temperature | Peak wavelength | Type | +|---|---|---|---| +| Sun | 5800 K | 2898 / 5800 ≈ 0.5 µm | visible light | +| Earth | 288 K | 2898 / 288 ≈ 10 µm | infrared | + +The two bands hardly overlap. So the atmosphere can be nearly transparent to the incoming energy and nearly opaque to the outgoing energy. The gases that do this are the greenhouse gases. + +## 3. Why CO2 and water absorb infrared, but N2 and O2 do not + +A molecule absorbs light when the light's electric field can push its charges into a new motion. For infrared, that motion is a vibration of the atoms. The push works only if the vibration changes the molecule's **dipole**, the separation of its positive and negative charge. + +- **N2 and O2** (99% of dry air) have two identical atoms. When they stretch, the charge stays symmetric, so the dipole stays zero. They are almost transparent to infrared. +- **CO2** is O=C=O. When it bends, the carbon moves off the line of the oxygens and a dipole appears. This bend absorbs strongly near **15 µm**, close to the peak of Earth's emission. +- **H2O** is already bent and has many vibration and rotation modes. It absorbs across much of the infrared. CH4 and N2O act in the same way. + +A gas that absorbs at a wavelength also emits at that wavelength, at a rate set by its own temperature. Greenhouse gases do not reflect infrared. They absorb it, share the energy with the surrounding air through collisions, and emit new infrared in all directions. + +## 4. A first model: one absorbing layer + +Take the simplest version. Put one layer of air above the surface. It is transparent to sunlight and absorbs all infrared. + +*Predict before you read on: will the surface be warmer than 255 K, colder, or the same?* + +- Space sees only the layer. The layer must emit 238 W/m² upward, so the layer is at 255 K. +- The layer emits equally up and down. It sends 238 W/m² down to the surface as well. +- The surface receives 238 (Sun) + 238 (layer) = 476 W/m². It must emit that much: T = (476 / σ)^(1/4) ≈ **303 K**. + +So the absorbing layer warms the surface. But 303 K is 15 K too warm. The model fails for two reasons: + +1. Real air does not absorb all infrared. Between about 8 and 13 µm, few gases absorb, and some surface emission escapes directly through this **window**. +2. Warm surface air rises, so **convection** carries heat upward. Radiation is not the only path. + +The one-layer model shows *that* absorption warms the surface. It cannot show why *more* CO2 warms it further. If the layer already absorbs everything, extra CO2 would change nothing. The next model fixes this. + +## 5. The real mechanism: the emission level and the lapse rate + +Two facts replace the single layer. + +**Fact 1: infrared escapes from a height, not from the surface.** An infrared photon emitted low in the air is soon absorbed again. A photon reaches space only if little absorbing gas lies above it. So, at each wavelength, the infrared that reaches space comes mostly from one altitude, the **emission level**. Averaged over all wavelengths, Earth's emission level is about **5 km** up. + +**Fact 2: air gets colder with height.** Rising air expands as pressure falls, and expansion cools it. Convection keeps mixing the lower atmosphere (the **troposphere**), so its temperature falls at a nearly fixed rate, the **lapse rate**. The mean lapse rate is about **6.5 K per km**. + +Combine them: + +```text + altitude temperature + ~5 km ─── emission level ─── 255 K → emits 238 W/m² to space + │ air cools 6.5 K/km + │ as you go up + 0 km ─── surface ────────── 255 + 6.5 × 5 ≈ 288 K +``` + +The emission level must be at 255 K, because it must emit 238 W/m². The surface sits 5 km lower, on the same temperature slope, so it is about 33 K warmer. This matches the measured 288 K. + +## 6. Why adding CO2 warms the surface + +*Predict: the centre of the CO2 band already absorbs almost all 15 µm light within the lowest few hundred metres. Does adding more CO2 still matter?* + +It does. More CO2 means more absorbing gas above each altitude. So a photon must start higher before it can escape, and the emission level rises. The chain is: + +1. More CO2 → emission level rises. +2. Higher air is colder (lapse rate) → the emission level emits less (σT⁴). +3. Less energy leaves than arrives → the planet gains energy. +4. The troposphere and surface warm until the new, higher emission level reaches 255 K again. +5. The surface is now further below the emission level on the same slope → it is warmer. + +The imbalance in step 3 is the **radiative forcing**. A fit to detailed radiation calculations gives: + + ΔF ≈ 5.35 × ln(C / C₀) W/m², where C is the CO2 concentration and C₀ the starting value. + +The logarithm appears because the band centre is already opaque. Extra CO2 acts mostly at the edges of the band, where absorption is weaker, and each doubling widens the band by a similar amount. + +**Worked example: double CO2.** + +- Forcing: 5.35 × ln 2 ≈ **3.7 W/m²** less energy leaving. +- How much warming removes this? Differentiate P = σT⁴: dP/dT = 4σT³ = 4 × 5.67 × 10⁻⁸ × 255³ ≈ **3.8 W/m² per K**. +- Warming needed: 3.7 / 3.8 ≈ **1 K**, with nothing else changed. (Detailed calculations give about 1.2 K.) +- Equivalent rise of the emission level: 1 K / 6.5 K per km ≈ **150 m**. + +**Feedbacks amplify this.** Warmer air holds about 7% more water vapour per K, and water vapour is a greenhouse gas, so warming adds more absorption. Ice loss and clouds also respond. Water vapour cannot start a warming by itself, because extra water condenses and rains out within days; its amount follows temperature. CO2 stays in the air for centuries, so CO2 sets the level and water vapour amplifies it. Including feedbacks, the warming for doubled CO2 is *estimated* at **2.5 to 4 K** (IPCC AR6 likely range). + +## 7. When the mechanism does not warm: the stratosphere + +The argument in section 6 needs one assumption: the emission level must be colder than the air below it. Remove that assumption and the result breaks. + +- **Holds (troposphere):** temperature falls 6.5 K per km. Raising the emission level by 150 m makes it about 1 K colder, so it emits less and the surface must warm. +- **Breaks (isothermal air):** if temperature were the same at all heights, raising the emission level would not change its temperature. Emission would not change, and there would be no warming. +- **Reverses (stratosphere):** above about 12 km, ozone absorbs sunlight and temperature *rises* with height. At the very centre of the CO2 band, the emission level is already in the stratosphere. Here, more CO2 raises the emission level into warmer air, and the stratosphere cools. Satellites observe this stratospheric cooling at the same time as surface warming. A brighter Sun would warm both, so this pattern points to greenhouse gases as the cause. + +## 8. What the numbers mean + +- **238 W/m²** — the energy Earth must shed. It is fixed by sunlight and reflectivity, not by greenhouse gases. +- **255 K** — the temperature of the layer that sheds it. Greenhouse gases decide *where* that layer is, not its temperature. +- **5 km** — the emission height. Near 0 km, there would be almost no greenhouse effect (surface near 255 K). At 10 km, with the same lapse rate, the surface would be near 320 K. +- **3.7 W/m²** — about 1.5% of 238 W/m². A small change in the energy balance, held for decades, is enough to warm the surface by several kelvin. + +**Not covered here:** cloud effects in detail, how fast the oceans take up heat (the delay between forcing and warming), and the carbon cycle that sets CO2 levels. A glass greenhouse also works differently: it warms mostly because the glass stops convection, not because of infrared absorption. diff --git a/evals/artifact/results/2026-10-04-dev/rating/greenhouse/C.md b/evals/artifact/results/2026-10-04-dev/rating/greenhouse/C.md new file mode 100644 index 0000000..a16d23c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/greenhouse/C.md @@ -0,0 +1,52 @@ +# How do greenhouse gases warm the Earth? + + + +**Answer.** Sunlight passes through the air and warms the surface. The surface sends the energy back out as infrared. Greenhouse gases absorb infrared, so the infrared escapes to space only from high, cold air, about 5 km up. Cold air emits less. To send out enough energy, the planet must be warmer than it would be without the gases. Adding gas lifts the escape point into colder air, so the surface warms more. + +## Level 1 — Where the energy goes + +```mermaid +flowchart TB + SUN["Sun"] -->|"sunlight, peak 0.5 µm
240 W/m² absorbed"| SURF["Surface
288 K, emits 390 W/m²"] + SURF -->|"infrared, peak 10 µm"| GHG["Greenhouse gas
absorbs and re-emits infrared"] + GHG -->|"re-emitted downward"| SURF + GHG -->|"re-emitted upward, absorbed again"| TOP["Air at emission height
about 5 km, 255 K"] + TOP -->|"infrared to space
240 W/m²"| SPACE["Space"] +``` + +The two kinds of radiation have different wavelengths. Sunlight peaks near 0.5 µm, where the air is mostly transparent. The Earth is much colder than the Sun, so its radiation, **infrared**, peaks near 10 µm. A **greenhouse gas** absorbs and emits infrared. N2 and O2 make up 99% of dry air but do not absorb infrared. A molecule absorbs infrared only if its vibration moves electric charge unevenly; a two-atom molecule of one element cannot do that. CO2, only 0.04% of air, absorbs strongly near 15 µm. Water vapour absorbs across many bands. + +**Energy balance** means that the planet emits to space as much as it absorbs. Sunlight brings 1361 W/m²; 30% reflects. The Earth catches sunlight on a disc (area πr²) but emits from a whole sphere (4πr²), so the average absorbed power is 1361 × 0.7 / 4 = 238 W/m², about 240. If you forget the 4, you get 360 K, which is wrong. + +A body emits power σT⁴ per m² (Stefan–Boltzmann law, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴). To emit 238 W/m², a body must be at 255 K (−18 °C). The real mean surface is 288 K (15 °C) and emits 390 W/m². Only 240 W/m² reaches space, so the atmosphere holds back 150 W/m². That 33 K difference is the greenhouse effect. + +## Level 2 — Why more gas means a warmer surface + +**Predict first:** a greenhouse gas absorbs infrared and emits it again. If it emits as much as it absorbs, how can more gas change anything? + +```mermaid +flowchart TB + A["More greenhouse gas"] --> B["Emission height rises
about 150 m for doubled CO2"] + B --> C["Emitting air is colder
6.5 K/km × 0.15 km = 1 K"] + C --> D["Less infrared to space
3.8 W/m² less"] + D --> E["Energy balance broken:
planet gains energy"] + E --> F["Whole column warms,
surface included"] + F -->|"until outgoing infrared
again equals 240 W/m²"| G["Energy balance restored,
warmer surface"] +``` + + Infrared that leaves near the ground gets absorbed again higher up. Infrared escapes to space only from air with little gas above it. The average altitude of that air is the **emission height**, about 5 km today. The **lapse rate** is how fast air cools with height: about 6.5 K per km. So the emitting air sits at 288 − 6.5 × 5 ≈ 255 K. That is exactly the temperature energy balance requires. + +Add gas, and the emission height rises. Doubled CO2 lifts it about 150 m (estimate). The new emitting air is 1 K colder. Because emission goes as T⁴, a 1 K drop at 255 K cuts the output by 4σT³ × 1 K = 3.8 W/m². The planet now gains energy. The whole column warms until the air at the new emission height is back at 255 K. The surface, 5 km of lapse rate below, warms by the same amount: about 1.2 K before feedbacks. + + This works only because air gets colder with height. In air of one temperature at every height, a higher emitter would be just as warm. Output would not fall, and more gas would not warm the surface. + +## Level 3 — Common mistakes + +| Mistake | Correction | +|---|---| +| "Greenhouse gases trap heat like the glass of a greenhouse." | A glass greenhouse works mainly by stopping warm air from rising. Greenhouse gases move the place where the planet emits to space. | +| "CO2 already absorbs all 15 µm infrared, so more CO2 does nothing." | Near the ground, absorption is saturated. The escape point still moves up into colder air, so the effect grows with the logarithm of CO2: each doubling adds about 3.7 W/m². | +| "The gas makes energy." | The source is sunlight. The gas slows energy loss to space. | + +Feedbacks multiply the 1.2 K. Warmer air holds more water vapour, a greenhouse gas; melting ice reflects less sunlight. The estimated total for doubled CO2 is likely 2.5–4 K (IPCC AR6). Clouds and ocean heat uptake are outside this explainer. diff --git a/evals/artifact/results/2026-10-04-dev/rating/key-exchange/A.md b/evals/artifact/results/2026-10-04-dev/rating/key-exchange/A.md new file mode 100644 index 0000000..2de73d1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/key-exchange/A.md @@ -0,0 +1,83 @@ +# How can two strangers agree on a secret key when anyone can read their messages? + +**They never send the key. Each person keeps a private number and sends a public number computed from it. Each then combines the other person's public number with their own private number, and both get the same result. An eavesdropper sees only the public numbers, and no known fast method gets the private numbers back from them.** + +This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections. + +## The problem + +Alice and Bob want to encrypt their messages. Encryption needs a key that both of them know. They have never met, so every message goes over a public channel. Eve, an eavesdropper, reads every message but does not change any. So the key itself can never be sent. + +The calculations below use one piece of notation. "mod 23" means "the remainder after division by 23". For example, 100 mod 23 = 8, because 100 = 4 × 23 + 8. + + +A first idea fails. Alice chooses a key, the number 6, and sends 5^6 mod 23 = 8. This operation is hard to reverse (later sections show why). Bob receives 8, and so does Eve. If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number. Each side then does one half of a calculation, and the other side finishes it without undoing it. + +## The exchange, with small numbers + + +1. Alice and Bob agree in public on two numbers: a prime **modulus** p = 23 (the number after "mod") and a **base** g = 5. +2. Each picks a **private number** at random and never sends it. Alice picks a = 6. Bob picks b = 15. +3. Each sends a **public number**: g raised to their private number, mod p. Alice sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. +4. Each raises the other's public number to their own private number, mod p. Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2. + +The result, s = 2, is the **shared secret**. Alice and Bob both feed s into the same hash function, a fixed scrambling recipe, and the output is the key for ordinary encryption. + +**Why both get 2.** Alice computed (5^15)^6, and Bob computed (5^6)^15. By the exponent rule, both equal 5^(6×15). Both also took remainders along the way, and that does not change the final remainder. Replacing a factor with its remainder removes a multiple of 23 from that factor, so it removes a multiple of 23 from the product: (x + 23k) × y = x × y + 23ky, for any whole number k. A power is a chain of products, so the rule holds at every step. Example: 5^3 = 125, and 125 mod 23 = 10. Reducing first gives (25 mod 23) × 5 = 2 × 5 = 10. + +## What Eve sees + + +| Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2 | +|---|---|---|---|---|---| +| Alice | yes | yes | no | yes | yes | +| Bob | yes | no | yes | yes | yes | +| Eve | yes | no | no | yes | no | + +*Predict before you read on:* Eve has 8 and 19. Can Eve combine them into 2? + +Multiplying fails: 8 × 19 mod 23 = 152 mod 23 = 14. Multiplying powers adds the exponents, so 14 = 5^(6+15) mod 23. The shared secret needs the exponents multiplied: 5^(6×15). No one knows a way from A and B to s other than, in effect, finding a or b first. That problem, getting 6 back from 8, is the discrete logarithm, and the next section shows why no fast method for it is known. + +## Why powers, and why mod + +Each choice blocks a simpler way for Eve to find a private number. + + +- **Without mod**, A = 5^6 = 15625. Eve takes the logarithm base 5 of 15625 and gets 6. Plain powers grow in order, so a logarithm reverses them. + + +- **With multiplication instead of a power**, A = 5 × 6 mod 23 = 7. Multiplying by 14 undoes multiplying by 5, because 5 × 14 = 70 = 3 × 23 + 1. A standard fast method (Euclid's algorithm) finds this undoing number even for a huge p. So Eve computes 7 × 14 mod 23 = 6 = a, then computes the shared secret as Alice does. +- **With powers mod p**, the results jump around. For exponents 1 to 22, 5^x mod 23 gives: 5, 2, 10, 4, 20, 8, 17, 16, 11, 9, 22, 18, 21, 13, 19, 3, 15, 6, 7, 12, 14, 1. A result near 8 says nothing about where 6 is. + +So g^x mod p is a **one-way operation**: fast to compute, with no known fast reverse. The reverse problem (given g, p, and g^x mod p, find x) is the **discrete logarithm** named above. + +## Why real numbers are huge + +With p = 23, Eve can try exponents in order: 5, 2, 10, 4, 20, 8. The 6th try gives 8, so a = 6. Real exchanges use a p of 2048 bits, about 617 decimal digits. They pick a and b at random from that whole range. Then trying exponents one by one is hopeless. The best known methods are faster, but still need about 2^112 operations (an estimate). At a trillion operations per second, that is about 10^14 years. + +Alice's own work stays small. Alice squares or multiplies by 5, and takes the remainder at each step (→ marks the remainder): 5^2 = 25 → 2, 5^3 = 2 × 5 = 10, 5^6 = 10^2 = 100 → 8. Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications: about 4,000 steps. + + +**Assumption, not proof.** No one has proven that the discrete logarithm is hard, or that Eve cannot get s from A and B some other way. The security rests on decades of failed attempts. + +## The limit: an attacker who changes messages + + +The exchange stops a reader. It does not stop Mallory, an attacker who can replace messages. Mallory picks 3, and sends 5^3 mod 23 = 10 to Alice and to Bob in place of their public numbers. + +- Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6 from Alice's public number. +- Bob computes 10^15 mod 23 = 5. Mallory computes 19^3 mod 23 = 5 from Bob's public number. + +Alice shares 6 with Mallory, and Bob shares 5 with Mallory. Mallory decrypts each message, reads it, and encrypts it again for the other side. Nothing in the exchange tells Alice who sent a public number. Real systems add **authentication**: Bob attaches to B a value that only Bob can produce and anyone can check (a digital signature). + +## Answer + +The two strangers never send the key. Each sends g raised to a private number, mod p. Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p. + +## Scope + +- **Authentication**: digital signatures and certificates, which is how HTTPS stops Mallory. +- **Elliptic-curve Diffie–Hellman**: the same idea with points on a curve in place of powers mod p. +- **Using the key**: a cipher such as AES encrypts the messages with it. +- **Choosing p and g safely**: a prime p lets a good g reach every non-zero remainder, as 5 does for 23. A bad g fails: for example, with g = 22 the powers mod 23 are only 1 and 22, so s has only two possible values. +- **Quantum computers**: a large quantum computer could solve the discrete logarithm fast. Post-quantum key exchange methods are designed to resist that. diff --git a/evals/artifact/results/2026-10-04-dev/rating/key-exchange/B.md b/evals/artifact/results/2026-10-04-dev/rating/key-exchange/B.md new file mode 100644 index 0000000..6905756 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/key-exchange/B.md @@ -0,0 +1,75 @@ +# Agreeing on a secret key over a public channel + +**The question.** Alice and Bob have never met. They can talk only over a channel that an eavesdropper, Eve, reads in full. How can they end up with the same secret number that Eve does not know? + +**The answer in one paragraph.** Each person picks a private number and never sends it. Each person sends a *public value* made from their private number by an operation that is fast to do and very hard to undo. Each person then combines their own private number with the other person's public value. The rules of arithmetic make both results equal. Eve sees only the two public values. To get the secret, she must undo the operation, and nobody knows a fast way to do that. This method is the **Diffie–Hellman key exchange** (1976). + +**Why it matters.** Your browser does this each time it opens an `https://` connection. The shared number becomes the key for a fast cipher that encrypts the rest of the conversation. + +## The tool: arithmetic "mod p" + +"x mod p" means the remainder when you divide x by p. For example, 30 mod 23 = 7. Arithmetic mod p keeps every number between 0 and p − 1, however large the calculation becomes. A clock does the same with 12. + +## First attempt: send the key + +Alice picks a key and sends it to Bob. Eve reads it. This fails at once, so neither person can ever send the secret itself or anything that reveals it. + +## Second attempt: hide the private number with multiplication + +Alice and Bob agree in public on a prime **p = 23** and a base **g = 5**. Alice picks a private number **a = 6** and sends **g × a mod p** = 30 mod 23 = **7**. + +*Before you read on: can Eve get 6 back from 7?* + +Yes. Eve knows g = 5 and p = 23. She finds the number that undoes "× 5" mod 23. That number is 14, because 5 × 14 = 70 = 3 × 23 + 1. Then 7 × 14 = 98, and 98 mod 23 = **6**. Eve has Alice's private number. A method called the extended Euclidean algorithm finds this "undo" number quickly, even when p has hundreds of digits. Multiplication is easy to undo, so it fails. + +## The working method: hide the private number with exponentiation + +Use powers instead of products. Everything else stays the same. + +| Step | Alice (private: a = 6) | Public channel (Eve sees this) | Bob (private: b = 15) | +|---|---|---|---| +| 1. Agree on public settings | | p = 23, g = 5 | | +| 2. Make a public value | A = 5⁶ mod 23 = **8** | A = 8, B = 19 | B = 5¹⁵ mod 23 = **19** | +| 3. Combine with the other's value | 19⁶ mod 23 = **2** | | 8¹⁵ mod 23 = **2** | + +Alice and Bob both get **2**. Eve saw 23, 5, 8 and 19. She never saw 6, 15 or 2. + +**Why the two results are equal.** Alice computes (g^b)^a. Bob computes (g^a)^b. Both equal g^(a×b), because the order of the exponents does not matter. Taking "mod p" at each step does not change this. Here, both are 5⁹⁰ mod 23 = 2. + +## Why Eve cannot do the same + +To find the secret, Eve needs a or b. To get a, she must solve "5 to what power, mod 23, gives 8?" This is the **discrete logarithm problem**. Mod p, the powers of g jump around with no pattern that points back to the exponent: + +5¹ → 5, 5² → 2, 5³ → 10, 5⁴ → 4, 5⁵ → 20, 5⁶ → **8**, 5⁷ → 17, … + +With p = 23, Eve simply tries all 22 exponents and finds a = 6 in under a second. The defence is the size of p: + +- **Forward direction (Alice and Bob).** Repeated squaring computes g^a with about 2 multiplications per binary digit of a. For a 2048-bit a, that is about 4,000 multiplications. A laptop does this in milliseconds. +- **Backward direction (Eve).** Trying every exponent takes about 2²⁰⁴⁸ steps. The best known algorithms do much better than that, but for a 2048-bit p they still need roughly 2¹¹² operations *(published estimate)*. At a billion billion (10¹⁸) operations per second, that is more than 100 million years. + +So the size of p sets the gap. A small p (like 23) gives Eve a few dozen tries. A 2048-bit p gives her more work than any computer can do. + +*Epistemic status:* nobody has proved that the discrete logarithm is hard. It is an **assumption** that has survived 50 years of attack. A large quantum computer running Shor's algorithm would break it; that is why newer "post-quantum" key exchanges exist. + +## Where the guarantee breaks: an attacker who can change messages + +The method defeats Eve, who only reads. It does not defeat Mallory, who can intercept and replace messages. + +Mallory picks her own private number m = 3 and computes M = 5³ mod 23 = **10**. She blocks A = 8 and B = 19, and sends 10 to both Alice and Bob instead. + +- Alice computes 10⁶ mod 23 = **6**. Mallory computes 8³ mod 23 = **6**. They share a key. +- Bob computes 10¹⁵ mod 23. Mallory computes 19³ mod 23. They share a different key. + +Alice and Bob each think they talk to the other. Mallory decrypts each message, reads it, re-encrypts it, and passes it on. This is a **man-in-the-middle attack**. + +The fix is **authentication**: Bob signs his public value with a digital signature, and Alice checks the signature against a key she already trusts. On the web, a certificate authority vouches for that key. So "never met" is not quite free: key exchange removes the need to share a *secret* in advance, but each side still needs a trusted way to check *who* the other side is. + +## Summary + +1. Agree in public on p and g. +2. Each person keeps a private exponent and sends g^(private) mod p. +3. Each person raises the other's public value to their own private exponent. Both get g^(a×b) mod p. +4. Eve must solve a discrete logarithm to copy this, which is infeasible for large p. +5. Authenticate the public values, or an active attacker can sit in the middle. + +**Not covered here:** elliptic-curve Diffie–Hellman (the same idea on a different group, with shorter keys and the same structure), how to choose safe p and g, and how digital signatures work. diff --git a/evals/artifact/results/2026-10-04-dev/rating/key-exchange/C.md b/evals/artifact/results/2026-10-04-dev/rating/key-exchange/C.md new file mode 100644 index 0000000..9e8bd97 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/rating/key-exchange/C.md @@ -0,0 +1,50 @@ +# How can two strangers agree on a secret key in public? + +**Each person keeps one number private and sends a value computed from it. Each person then combines the other person's value with their own private number. Both get the same shared key, but the shared key is never sent. An eavesdropper sees every message and still cannot compute it, because one step is easy to do and, as far as anyone knows, very hard to undo.** + +The method is the Diffie–Hellman key exchange. Below, Alice and Bob want a shared key. Eve reads every message on the channel. + +"x mod p" means the remainder after you divide x by p. For example, 25 mod 23 = 2. + +## How it works + + + +1. Agree in public on a prime p and a base g. Here, p = 23 and g = 5. Eve knows them too. +2. Each person picks a **private number** at random and never sends it. Alice picks a = 6. Bob picks b = 15. +3. Each person computes a **public value**, g to the power of their private number, mod p, and sends it: + - Alice: A = 5^6 mod 23 = 8. + - Bob: B = 5^15 mod 23 = 19. +4. Each person raises the *other* public value to their *own* private number: + - Alice: 19^6 mod 23 = **2**. + - Bob: 8^15 mod 23 = **2**. + +The two results are equal because 19 = 5^15 and 8 = 5^6, so both people compute 5^(6·15) mod 23. Order does not matter: (g^b)^a = (g^a)^b = g^(ab). The **shared key** is 2. + +## Why Eve fails + + + +Eve sees 23, 5, 8 and 19. To get the key she needs a or b. That means solving 5^a mod 23 = 8 for a. This problem is the **discrete logarithm**. + +At p = 23 Eve can try every exponent: 5^1, 5^2, … mod 23 gives 5, 2, 10, 4, 20, 8. She finds a = 6 on try 6. So p must be large. Real systems use a p of 2048 bits (about 617 digits). The best known methods then need about 2^112 operations, far beyond any computer (NIST estimate). Alice and Bob still compute their values quickly, because repeated squaring needs only about 2 × 2048 multiplications. + +Eve cannot combine 8 and 19 directly either. 8 × 19 gives 5^(6+15), not 5^(6·15). + +## Why this form + + + +**Why powers, not multiplication?** Suppose Alice sent A = g·a mod p = 5·6 mod 23 = 7. Eve multiplies 7 by 14, because 5·14 mod 23 = 1. She gets 7·14 mod 23 = 6, which is a. Division undoes multiplication fast. No similar shortcut is known for powers mod p. + + + +**Why mod p?** Without it, Alice sends 5^6 = 15625. Eve computes log base 5 of 15625 = 6. Plain powers grow smoothly, so an ordinary logarithm undoes them. Mod p scrambles the order: the list 5, 2, 10, 4, 20, 8 above has no pattern to follow. + +## Limits + + + +- **Eve must only read (assumption).** Suppose Mallory can also replace messages. Mallory picks 3 and sends 5^3 mod 23 = 10 to Alice in place of 19. Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6. Alice now shares key 6 with Mallory and believes it is Bob. Mallory does the same with Bob. The fix is to authenticate the public values, for example with signatures and certificates, as HTTPS does. +- **Hardness is assumed, not proven.** No fast method for the discrete logarithm is known on ordinary computers. A large quantum computer would break it (Shor's algorithm), so newer systems add post-quantum key exchange. +- Most systems today use the same idea on elliptic curves (X25519), which gives equal security with much smaller numbers. diff --git a/evals/artifact/results/2026-10-04-dev/report.md b/evals/artifact/results/2026-10-04-dev/report.md new file mode 100644 index 0000000..1dbcd01 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/report.md @@ -0,0 +1,44 @@ +# Artifact eval, 2026-10-04 (dev, 10 subjects) + +Builder, reader, and grader: fresh `claude -p` agents. The reader saw only the renderings; an answer counts only when its quote is found in them (grounded). The grader saw each subject's explanations under shuffled labels. + +Conditions: **none** = `none`; **principles** = `principles`; **quick** = `quick`; **published** = `published` + +| condition | mean rank | ranked best | quiz (grounded) | predict | quiz (raw) | ungrounded | misconception taught | corrected | errors | stops / expl. | excess / expl. | words | forms | cost / expl. | min / expl. | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| none | 3.1 | 0/10 | 100% | 100% | 100% | 0 | 0 | 7 | 4 | 8.7 | 6.0 | 1689 | prose 5, prose+html 4, html 1 | $0.28 | 1.5 | +| principles | 1.8 | 3/10 | 99% | 95% | 99% | 0 | 0 | 7 | 1 | 7.7 | 4.3 | 1006 | prose 10 | $0.21 | 1.0 | +| quick | 2.9 | 2/10 | 95% | 95% | 95% | 0 | 0 | 5 | 6 | 7.1 | 3.8 | 606 | prose 8, diagram 2 | $0.46 | 1.6 | +| published | 2.2 | 5/10 | 95% | 100% | 95% | 0 | 0 | 7 | 1 | 7.9 | 5.0 | 1016 | prose 9, diagram 1 | $2.81 | 11.8 | + +Mean rank: the grader's ranking of each subject's 4 explanations by the reader's effort (1 is best). Quiz: fixed questions per subject, 0–2 each; predict: the last question of each, a case the explanation need not mention. Stops and excess: the reader's lists (at most 10 each). Words: visible words in all renderings. Cost: the builder and the reviews it ran; minutes: the builder. + +## Per subject (grounded quiz score, of 8) + +| subject | none | principles | quick | published | +|---|---|---|---|---| +| gps-relativity | 8 (1500 w) | 8 (1158 w) | 8 (676 w) | 8 (999 w) | +| vaccine-memory | 8 (1953 w) | 8 (1067 w) | 7 (600 w) | 8 (992 w) | +| key-exchange | 8 (1822 w) | 8 (931 w) | 8 (506 w) | 8 (1086 w) | +| two-tides | 8 (1929 w) | 8 (884 w) | 6 (599 w) | 6 (948 w) | +| bloom-filter | 8 (1439 w) | 8 (1083 w) | 8 (599 w) | 8 (1059 w) | +| heat-pump | 8 (1339 w) | 7 (784 w) | 7 (814 w) | 8 (1437 w) | +| quorum | 8 (1072 w) | 8 (777 w) | 8 (502 w) | 8 (896 w) | +| antibiotic-resistance | 8 (2511 w) | 8 (1194 w) | 8 (600 w) | 8 (949 w) | +| float-sum | 8 (1056 w) | 8 (661 w) | 8 (412 w) | 8 (721 w) | +| greenhouse | 8 (2271 w) | 8 (1522 w) | 8 (749 w) | 6 (1072 w) | + +✗: the explanation teaches the misconception. + +## Why the best is best + +- **gps-relativity**: published > principles > quick > none. D builds each idea in order with no detours: it says the receiver only listens, makes the clock bias b concrete with a small 1-D worked example, sets a nanosecond budget, and explains why a shared b cannot absorb satellite drift that differs between satellites. +- **vaccine-memory**: published > principles > none > quick. C carries one consistent worked model (rare precursors, division count, memory pool size, the race against a pathogen growing 10× per day) from start to finish, says explicitly that the vaccine contains no pathogen that can multiply, defines every term it uses, and leaves side topics to a short 'not covered' list, so the reader gets every quiz point without detours. +- **key-exchange**: quick > principles > published > none. B covers the operation, why it is hard, why both sides get the same key, the worked p=23 example and the man-in-the-middle fix in the fewest words, with no detours beyond what this reader needs. +- **two-tides**: principles > none > published > quick. A covers every quiz point and names and refutes the spin-centrifugal misconception, using compact worked numbers, without B's long detours (code, a broken companion-page dump) and without the spring-tide gap in C and D. +- **bloom-filter**: published > principles > quick > none. C states up front that the filter keeps only bits, and it runs one small example through add, a correct 'no', a false positive and a failed delete. It then derives the fill fraction f and the rate p ≈ f^k and shows overfilling numerically, so the reader can answer even the all-bits-set case straight from the text, without detours. +- **heat-pump**: published > none > principles > quick. A builds the model in clear steps: the energy balance, then the four-part loop with consistent numbers at every stage, then the kelvin formula with a worked entropy check of an allowed and a forbidden COP. That made all four answers exact, without D's detours (analogy, code, air-conditioning) or the refrigerant-temperature tables in B and C that led their readers to the wrong limit in Q4. +- **quorum**: quick > principles > published > none. A gives the counting rule up front, then derives the majority from a single split-brain example and the 2q > n overlap condition, and shows that an even-sized cluster adds no tolerance, all in the fewest words and with no detours. +- **antibiotic-resistance**: published > principles > quick > none. A covers variation, selection, and cross-species plasmid transfer in the fewest words, and its worked numbers and Luria–Delbrück variance argument show directly why the drug selects resistant cells rather than creating them. +- **float-sum**: principles > none > quick > published. A gives the full causal chain (the prime-factor rule, stored values as integer × power of 2, the tie rounding and the printing behaviour) in a short, exact form with no detours, so the reader gets a correct model with little effort. +- **greenhouse**: principles > none > quick > published. C builds every link the reader needs (energy balance, the wavelength split, the dipole reason, a one-layer model and why it fails, then emission height with the lapse rate, its limits, and worked numbers) in a tight order with prediction prompts, without A's extra detours and length or the gaps in D and B. diff --git a/evals/artifact/results/2026-10-04-dev/scores.json b/evals/artifact/results/2026-10-04-dev/scores.json new file mode 100644 index 0000000..c0f5863 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/scores.json @@ -0,0 +1,1483 @@ +{ + "none": { + "subjects": [ + { + "id": "gps-relativity", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 9, + "excess": 5, + "words": 1500, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.17838859999999995, + "seconds": 57, + "failed": false + }, + { + "id": "vaccine-memory", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "Inactivated (e.g. many flu shots, polio IPV) | Whole pathogen, killed | Can't cause infection; usually needs adjuvant and boosters: most inactivated flu shots and IPV contain no adjuvant." + ], + "stops": 8, + "excess": 6, + "words": 1953, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.16533019999999998, + "seconds": 62, + "failed": false + }, + { + "id": "key-exchange", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 8, + "words": 1822, + "forms": [ + "prose", + "html" + ], + "rank": 4, + "cost": 0.3186072, + "seconds": 96, + "failed": false + }, + { + "id": "two-tides", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 10, + "excess": 5, + "words": 1929, + "forms": [ + "prose", + "html" + ], + "rank": 2, + "cost": 0.4116578, + "seconds": 131, + "failed": false + }, + { + "id": "bloom-filter", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 6, + "words": 1439, + "forms": [ + "html" + ], + "rank": 4, + "cost": 0.40253520000000004, + "seconds": 118, + "failed": false + }, + { + "id": "heat-pump", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [ + "absolute zero, where molecular motion stops", + "the loop diagram routes the compressor's output down to the expansion valve and the condenser's output to the compressor, which contradicts the text order evaporator → compressor → condenser → valve" + ], + "stops": 9, + "excess": 7, + "words": 1339, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.1563, + "seconds": 49, + "failed": false + }, + { + "id": "quorum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 7, + "excess": 4, + "words": 1072, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.0993212, + "seconds": 32, + "failed": false + }, + { + "id": "antibiotic-resistance", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 10, + "excess": 6, + "words": 2511, + "forms": [ + "prose", + "html" + ], + "rank": 4, + "cost": 0.502619, + "seconds": 162, + "failed": false + }, + { + "id": "float-sum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [ + "The digits after the point (the *significand*) get a fixed budget: **53 significant binary digits**." + ], + "stops": 7, + "excess": 6, + "words": 1056, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.1607036, + "seconds": 49, + "failed": false + }, + { + "id": "greenhouse", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 9, + "excess": 7, + "words": 2271, + "forms": [ + "prose", + "html" + ], + "rank": 2, + "cost": 0.3816954, + "seconds": 124, + "failed": false + } + ] + }, + "principles": { + "subjects": [ + { + "id": "gps-relativity", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 5, + "words": 1158, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.28801780000000005, + "seconds": 89, + "failed": false + }, + { + "id": "vaccine-memory", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "Across the body's hundreds of billions of lymphocytes, the library contains millions of different shapes: this understates the repertoire, which is on the order of 10^8 or more distinct receptors." + ], + "stops": 9, + "excess": 5, + "words": 1067, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.2302082, + "seconds": 68, + "failed": false + }, + { + "id": "key-exchange", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 8, + "excess": 4, + "words": 931, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.16838920000000002, + "seconds": 43, + "failed": false + }, + { + "id": "two-tides", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 9, + "excess": 3, + "words": 884, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.1563002, + "seconds": 42, + "failed": false + }, + { + "id": "bloom-filter", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 6, + "excess": 3, + "words": 1083, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.1830298, + "seconds": 48, + "failed": false + }, + { + "id": "heat-pump", + "raw": [ + 2, + 2, + 2, + 1 + ], + "score": [ + 2, + 2, + 2, + 1 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 5, + "words": 784, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.2534982, + "seconds": 67, + "failed": false + }, + { + "id": "quorum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 6, + "excess": 4, + "words": 777, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.18852040000000003, + "seconds": 52, + "failed": false + }, + { + "id": "antibiotic-resistance", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 4, + "words": 1194, + "forms": [ + "prose" + ], + "rank": 2, + "cost": 0.1721862, + "seconds": 49, + "failed": false + }, + { + "id": "float-sum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 7, + "excess": 4, + "words": 661, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.2104636, + "seconds": 55, + "failed": false + }, + { + "id": "greenhouse", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 8, + "excess": 6, + "words": 1522, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.29882359999999997, + "seconds": 89, + "failed": false + } + ] + }, + "quick": { + "subjects": [ + { + "id": "gps-relativity", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "Three spheres meet at a point. (Three spheres generally meet at two points; the receiver discards one of them.)" + ], + "stops": 6, + "excess": 5, + "words": 676, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.44009420000000005, + "seconds": 91, + "failed": false + }, + { + "id": "vaccine-memory", + "raw": [ + 1, + 2, + 2, + 2 + ], + "score": [ + 1, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 9, + "excess": 5, + "words": 600, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.4694588000000001, + "seconds": 94, + "failed": false + }, + { + "id": "key-exchange", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "\"19 = 5^15 and 8 = 5^6\" (these hold only mod 23, not as plain equalities)" + ], + "stops": 6, + "excess": 3, + "words": 506, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.36011959999999993, + "seconds": 80, + "failed": false + }, + { + "id": "two-tides", + "raw": [ + 2, + 2, + 0, + 2 + ], + "score": [ + 2, + 2, + 0, + 2 + ], + "grounded": [ + true, + true, + false, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 7, + "excess": 4, + "words": 599, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 0.5251108, + "seconds": 114, + "failed": false + }, + { + "id": "bloom-filter", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 5, + "excess": 1, + "words": 599, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.5246917999999999, + "seconds": 99, + "failed": false + }, + { + "id": "heat-pump", + "raw": [ + 2, + 2, + 2, + 1 + ], + "score": [ + 2, + 2, + 2, + 1 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 5, + "words": 814, + "forms": [ + "diagram" + ], + "rank": 4, + "cost": 0.47279319999999997, + "seconds": 98, + "failed": false + }, + { + "id": "quorum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 7, + "excess": 3, + "words": 502, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 0.41601699999999997, + "seconds": 71, + "failed": false + }, + { + "id": "antibiotic-resistance", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [ + "Then every cell would have the same chance to survive. Instead, a rare fraction survives, and those cells were resistant before they met the drug.", + "Change the target | The target protein changes shape, so the drug cannot bind | MRSA makes a cell-wall enzyme that penicillin-type drugs do not bind" + ], + "stops": 7, + "excess": 3, + "words": 600, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.430636, + "seconds": 104, + "failed": false + }, + { + "id": "float-sum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 7, + "excess": 4, + "words": 412, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 0.4691284, + "seconds": 83, + "failed": false + }, + { + "id": "greenhouse", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [ + "\"the infrared escapes to space only from high, cold air, about 5 km up\" (overstated: through the 8–13 µm window much surface infrared escapes directly, and the text never mentions this)", + "\"The new emitting air is 1 K colder... The surface... warms by the same amount: about 1.2 K before feedbacks\" (inconsistent: the 1 K drop it just derived implies about 1 K of warming, and the 1.2 K figure is given without its source)" + ], + "stops": 9, + "excess": 5, + "words": 749, + "forms": [ + "diagram" + ], + "rank": 3, + "cost": 0.507285, + "seconds": 120, + "failed": false + } + ] + }, + "published": { + "subjects": [ + { + "id": "gps-relativity", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 7, + "words": 999, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 2.1912534, + "seconds": 567, + "failed": false + }, + { + "id": "vaccine-memory", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 9, + "excess": 5, + "words": 992, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 2.1222304000000003, + "seconds": 543, + "failed": false + }, + { + "id": "key-exchange", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 7, + "excess": 5, + "words": 1086, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 4.4839193999999996, + "seconds": 1162, + "failed": false + }, + { + "id": "two-tides", + "raw": [ + 2, + 2, + 0, + 2 + ], + "score": [ + 2, + 2, + 0, + 2 + ], + "grounded": [ + true, + true, + false, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 9, + "excess": 6, + "words": 948, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 2.838019, + "seconds": 705, + "failed": false + }, + { + "id": "bloom-filter", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 5, + "words": 1059, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 2.3029966, + "seconds": 553, + "failed": false + }, + { + "id": "heat-pump", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 5, + "words": 1437, + "forms": [ + "diagram" + ], + "rank": 1, + "cost": 3.399423, + "seconds": 845, + "failed": false + }, + { + "id": "quorum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 3, + "words": 896, + "forms": [ + "prose" + ], + "rank": 3, + "cost": 2.3876656, + "seconds": 589, + "failed": false + }, + { + "id": "antibiotic-resistance", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [], + "stops": 8, + "excess": 6, + "words": 949, + "forms": [ + "prose" + ], + "rank": 1, + "cost": 2.7889748, + "seconds": 686, + "failed": false + }, + { + "id": "float-sum", + "raw": [ + 2, + 2, + 2, + 2 + ], + "score": [ + 2, + 2, + 2, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "corrected", + "errors": [ + "The other 11 bits hold the sign and the power." + ], + "stops": 6, + "excess": 4, + "words": 721, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 2.377155, + "seconds": 607, + "failed": false + }, + { + "id": "greenhouse", + "raw": [ + 2, + 2, + 0, + 2 + ], + "score": [ + 2, + 2, + 0, + 2 + ], + "grounded": [ + true, + true, + true, + true + ], + "predict": [ + 3 + ], + "misconception": "absent", + "errors": [], + "stops": 8, + "excess": 4, + "words": 1072, + "forms": [ + "prose" + ], + "rank": 4, + "cost": 3.234709, + "seconds": 804, + "failed": false + } + ] + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/explainers/antibiotic-resistance/explanation.md b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/explainers/antibiotic-resistance/explanation.md new file mode 100644 index 0000000..3b651b3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/explainers/antibiotic-resistance/explanation.md @@ -0,0 +1,139 @@ +# How bacteria become resistant to an antibiotic + +## The short answer + +Individual bacteria don't learn to resist a drug, and they don't adapt because they've been exposed to it. What happens is this: + +1. **Variation already exists.** A large bacterial population almost always contains a few cells that differ genetically from the rest. They differ because of random mutations, or because they picked up genes from other bacteria. A few of those differences happen to make a cell harder to kill with a particular drug. +2. **The antibiotic does the selecting.** The drug kills or stops the susceptible cells. The resistant few survive, and they now have food and space to themselves. +3. **The survivors multiply.** Bacteria divide fast, so within hours or days the population is mostly made of the resistant type. + +This is natural selection, and it runs fast because bacteria have huge populations and short generation times. The rest of this document explains each step, with rough numbers so you can see why it is so reliable. + +--- + +## 1. How antibiotics work, briefly + +An antibiotic is a molecule that interferes with something a bacterium needs and that our own cells either don't have or have in a different form. Common targets: + +| Target | Example drugs | What goes wrong for the bacterium | +|---|---|---| +| Cell wall construction | penicillins, cephalosporins (β-lactams); vancomycin | The wall is built badly, and the cell bursts under its own internal pressure | +| Ribosome (protein synthesis) | tetracyclines, macrolides, aminoglycosides | The cell can't make proteins, or makes faulty ones | +| DNA copying (DNA gyrase/topoisomerase) | fluoroquinolones such as ciprofloxacin | DNA gets tangled and broken | +| RNA polymerase | rifampicin | Genes can't be read | +| Folate synthesis (a metabolic pathway) | sulfonamides, trimethoprim | The cell can't make the building blocks for DNA | + +Every one of these works the same basic way: the drug molecule has to **get into or onto the cell** and then **bind its target tightly enough** to jam it. That gives the bacterium several possible ways to resist. + +## 2. What resistance looks like at the molecular level + +A resistant bacterium has some trait that breaks that chain. There are five main strategies: + +- **Destroy or disable the drug.** Some bacteria make enzymes that chemically inactivate the antibiotic. The best-known are the **β-lactamases**, which cut open the ring structure that penicillin-type drugs need in order to work. Other enzymes stick chemical groups onto aminoglycosides so they no longer bind the ribosome. +- **Change the target.** If the target's shape changes slightly, the drug no longer fits, but the target can still do its normal job. Examples: a single amino-acid change in RNA polymerase gives rifampicin resistance. A few changes in DNA gyrase give fluoroquinolone resistance. MRSA (methicillin-resistant *Staphylococcus aureus*) carries an extra cell-wall-building enzyme, PBP2a, that β-lactams barely bind. +- **Pump the drug out.** **Efflux pumps** are membrane proteins that push molecules out of the cell. If a cell makes more of them, or makes a version that recognises the drug, the drug's concentration inside the cell stays too low to work. +- **Keep the drug out.** Some bacteria (the "Gram-negative" group, such as *E. coli*) have an outer membrane with channels called porins. Losing or narrowing those channels slows the drug's entry. +- **Bypass the blocked step.** A cell can gain an alternative enzyme that does the same job but ignores the drug. Some trimethoprim-resistance genes work this way: they encode a different version of the enzyme the drug inhibits. + +Resistance is usually a **matter of degree**. It is measured as the **MIC (minimum inhibitory concentration)**: the lowest drug concentration that stops the bacteria from growing in a lab test. A resistant strain has a higher MIC, sometimes slightly higher and sometimes thousands of times higher. In a clinic, a strain is labelled "resistant" when its MIC is above what can be safely reached in a patient's body. + +## 3. Where the variation comes from + +### 3a. Mutation (passed from parent to offspring) + +Every time a bacterium copies its DNA, errors creep in. A typical rate is about 1 error per 10⁹–10¹⁰ base pairs copied. That is very accurate, but consider the numbers: + +- A particular resistance-causing change, such as one specific letter in one specific gene, might arise roughly once per **10⁸–10⁹ cell divisions**. The exact figure varies a lot by gene and drug. +- A serious infection can contain **10⁹–10¹¹** bacteria. One millilitre of a dense lab culture has around 10⁹. + +So a large population **very likely already contains** a few resistant mutants before any drug is given. The drug doesn't create them. It reveals them, by removing everyone else. + +> Quick estimate: if 10¹⁰ cells exist and 1 in 10⁸ carries a resistance mutation, about 10¹⁰ / 10⁸ = **100 resistant cells** are present at the start of treatment. + +This was shown directly in classic experiments in the 1940s and 50s (Luria–Delbrück, and the Lederbergs' replica plating). Resistant mutants turned up in cultures that had **never touched** the selecting agent. Their numbers also fluctuated from culture to culture in the way you'd expect if the mutations happened randomly at earlier times, rather than as a response to exposure. + +Some antibiotics, and stress in general, can raise mutation rates somewhat. That speeds things up, but the basic logic of random variation followed by selection is unchanged. + +### 3b. Horizontal gene transfer (getting genes from other bacteria) + +Bacteria can also acquire **ready-made resistance genes** from other bacteria, even from different species. This is a major reason resistance spreads so fast in hospitals. There are three main routes: + +- **Conjugation.** One cell builds a bridge to another and passes over a **plasmid**: a small, circular piece of DNA, separate from the main chromosome, that can copy itself. Plasmids often carry several resistance genes together. +- **Transformation.** Some bacteria can take up loose DNA from their surroundings, for example DNA released by dead cells, and stitch it into their own genome. +- **Transduction.** Viruses that infect bacteria (**bacteriophages**) sometimes accidentally package a piece of bacterial DNA and deliver it to the next cell they infect. + +Within a genome, **transposons** ("jumping genes") and **integrons** (gene-capture systems) let resistance genes move between chromosome and plasmid and gather into clusters. As a result, one transfer event can make a cell resistant to several drug classes at once. + +If you program, here is a loose analogy. Mutation is like random bit-flips in your own source code. Horizontal transfer is like importing a tested library someone else already wrote. + +## 4. Selection: why the resistant cells take over + +With no drug present, resistant and susceptible cells grow at similar rates. Resistant ones are often slightly *slower* (see section 6). Add the drug and the picture flips. + +### A toy calculation + +Suppose: + +- Start of treatment: 10¹⁰ susceptible cells and 100 resistant cells. +- The drug kills susceptible cells quickly, so their numbers fall about 10-fold every hour or two. +- Resistant cells keep dividing every ~30 minutes, which means doubling twice per hour. + +After 10 hours: + +- Susceptible: 10¹⁰ shrinks to roughly 10⁵ or fewer. +- Resistant: 100 × 2²⁰ ≈ 100 × 10⁶ = **10⁸**. + +The population has gone from 1 resistant cell in 10⁸ to almost entirely resistant in less than half a day. It doesn't matter much how rare the resistant cells were to start with. Exponential growth makes up the difference quickly. + +In a real infection, the immune system also kills bacteria and nutrients run out, so growth is slower than this. Often the immune system mops up the small number of resistant survivors. Resistance becomes a clinical problem when that doesn't happen: in severe infections, in people with weakened immune systems, or when resistant bacteria settle in and spread to other people. + +### The "selection window" + +Drug concentration matters, and in a way that isn't obvious. Let MIC_S be the susceptible strain's MIC and MIC_R the resistant strain's MIC. + +- **Below MIC_S:** both types grow. The drug barely affects them, though even low concentrations can give resistant cells a slight edge. +- **Between MIC_S and MIC_R:** susceptible cells die and resistant cells grow. This range is the **mutant selection window**, and it is where resistance is selected most strongly. +- **Above MIC_R:** both types are suppressed. Resistance gains a foothold only if a cell gets a *second*, bigger resistance step. + +Inside a patient, drug levels rise and fall between doses and differ from one tissue to another. So bacteria somewhere often spend time inside the window. A low dose that nevertheless kills the susceptible cells is the textbook recipe for selecting resistance. + +The interactive page (`index.html`) lets you move the drug concentration through these zones and watch the effect. + +## 5. Stepwise resistance + +Often a single mutation gives only a small rise in MIC. Each extra mutation pushes the MIC higher: for example, one change in DNA gyrase, then a second, then a change that switches on an efflux pump. Each step lets the bacteria survive in a slightly higher concentration, where the next mutation can then be selected. Repeated or prolonged treatment can therefore climb a ladder from slightly resistant to highly resistant. + +## 6. Why resistance doesn't simply disappear + +Resistance often has a **fitness cost**. An altered target may work a bit less well, and running pumps or making extra enzymes uses energy. With no drug around, resistant cells may then be slowly outcompeted by susceptible ones. + +In practice, though, resistance tends to stick around: + +- **Compensatory mutations.** Later mutations can restore the lost growth rate *without* removing the resistance, so the cost shrinks to nearly zero. +- **Co-selection.** If a plasmid carries genes for resistance to drug A, drug B and a heavy metal, then using *any* of those keeps the whole plasmid around. +- **Low-cost mechanisms.** Some resistance genes cost very little from the start. +- **Ongoing exposure.** Antibiotics are used widely in medicine and in farming, and they turn up at low levels in the environment, so selection pressure rarely stops completely. + +## 7. Related ideas that are *not* resistance + +- **Tolerance.** The bacteria die more *slowly* under the drug, but the MIC doesn't change. Slow-growing or dormant cells are often tolerant, because many antibiotics attack processes that are only busy in actively growing cells. +- **Persistence.** A small fraction of genetically identical cells happen to be dormant when the drug arrives. They survive, then regrow into a population that is just as susceptible as before. Persisters can lead to infections that keep coming back, and they give resistance mutations more time and more chances to appear. +- **Biofilms.** Communities of bacteria stuck to a surface inside a slimy matrix. Drugs penetrate them poorly and many of the cells grow slowly, so biofilms survive treatment even without resistance genes. + +## 8. What this implies + +The mechanism suggests these strategies: + +- **Use antibiotics only when they're needed.** Every use selects for resistance somewhere, including among the harmless bacteria in your gut, which can later pass genes to harmful ones. +- **Use enough drug, for a suitable time.** The goal is to get above the selection window and clear the infection. The best course length is still being researched. For many infections, shorter courses turn out to work as well as longer ones, and the old blanket rule "always finish the course" is now debated. Follow the prescriber's advice for your situation. +- **Combine drugs, sometimes.** If resistance to drug A arises in roughly 1 in 10⁸ cells, and resistance to drug B in roughly 1 in 10⁸, then a cell resistant to both by chance is about 1 in 10¹⁶, more cells than are in the infection. This is why tuberculosis and HIV are treated with combinations. +- **Stop transmission.** Hand hygiene, infection control and sanitation stop resistant strains spreading from person to person, which matters as much as how the drugs are used. + +## Summary + +- Bacterial populations are huge and varied. Random mutation and gene swapping mean a few cells are already resistant before any drug is given. +- Resistance mechanisms work in one of five ways: break the drug, change the target, pump the drug out, keep the drug out, or bypass the blocked step. +- The antibiotic doesn't cause resistance. It **selects** for it, by removing the competition. Exponential growth then turns a handful of survivors into the whole population. +- Drug concentrations between the susceptible and resistant MICs select resistance most strongly. +- Fitness costs, compensatory mutations and co-selection decide whether resistance fades or persists once the drug is gone. diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/explainers/antibiotic-resistance/index.html b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/explainers/antibiotic-resistance/index.html new file mode 100644 index 0000000..7dbab9e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/explainers/antibiotic-resistance/index.html @@ -0,0 +1,375 @@ + + + + + +Antibiotic resistance: a selection simulator + + + + +

Watching resistance get selected

+

A small simulation of one bacterial population with two kinds of cells: susceptible (S) and resistant (R). +At the start there are no resistant cells at all. They appear only through random mutation as susceptible cells divide. Then the antibiotic decides who survives.

+ +
+
+ + +
Shown as a multiple of the susceptible strain's MIC. The resistant strain's MIC is × higher.
+
+
+ + +
The drug starts at hour 8 and is held at a constant level.
+
+
+ + +
Chance that a single cell division produces a resistant cell.
+
+
+ + +
How much slower resistant cells grow when no drug is present.
+
+
+ +
+ + + + + +
+ + +
+ Susceptible cells + Resistant cells + Drug present + Vertical axis is logarithmic: each gridline is 10× more cells. +
+ +

+ +

Why concentration matters: net growth rate vs. drug level

+

This chart shows how fast each type grows (above zero) or dies (below zero) at each drug concentration, ignoring crowding. +The shaded band is the mutant selection window: concentrations high enough to kill susceptible cells but too low to stop resistant ones. +The dashed line marks the concentration you've chosen above.

+ + +

What to try

+
    +
  • No drug: resistant mutants appear but stay a tiny minority. With a fitness cost, they're outcompeted.
    • +
  • In the window: the susceptible population collapses and the resistant line, which started from a handful of cells, climbs to take over. Press "Run again" a few times. When the first mutants appear, and whether they survive, changes from run to run.
    • +
  • Above both MICs: both types die. (Real drugs can't always reach this level safely in every part of the body. Bacteria can also climb out of this regime one mutation step at a time.)
    • +
  • Short treatment in the window: the drug stops before the susceptible cells are gone, and both types regrow. The resistant fraction is now far higher than before treatment.
    • +
  • Lower the mutation rate to 10⁻¹⁰: in many runs, no resistant cell exists when the drug arrives, and treatment succeeds. Resistance is a numbers game: (population size) × (mutation rate).
    • +
+ +
+How the model works +

Time advances in steps of 0.02 hours. Each type i has a birth rate g_i × (1 − N/K), where N is the total population and K = 10¹⁰ is the carrying capacity. It also has a drug-caused death rate

+

kill_i(c) = (g_max − g_min) · x^κ / (x^κ − g_min/g_max),   with x = c / MIC_i

+

This is a standard pharmacodynamic curve, chosen so that net growth is exactly zero at the MIC. The parameters are g_max = 1/hour for S (about one doubling every 42 minutes), g_min = −3/hour, κ = 2. The resistant type's g is reduced by the fitness cost, and its MIC is 64× higher.

+

Births and deaths are drawn at random (Poisson) while a type has fewer than 10,000 cells, so small resistant lineages can die out by chance. Above that size the model switches to the deterministic average. In each step, every susceptible birth has probability μ of producing a resistant cell instead. There is no immune system and no horizontal gene transfer, so this is deliberately the simplest case.

+
+ + + + diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/read.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/read.json new file mode 100644 index 0000000..483bf99 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/read.json @@ -0,0 +1,46 @@ +{ + "answers": { + "1": { + "answer": "It comes from random mutations made when DNA is copied, or from genes picked up from other bacteria. In a large population a few resistant cells are already there before any drug is given. The drug doesn't create them.", + "quote": "They differ because of random mutations, or because they picked up genes from other bacteria. A few of those differences happen to make a cell harder to kill with a particular drug." + }, + "2": { + "answer": "It selects rather than causes. It kills or stops the susceptible cells, which leaves the resistant few with food and space to themselves, so they multiply and take over.", + "quote": "The drug kills or stops the susceptible cells. The resistant few survive, and they now have food and space to themselves." + }, + "3": { + "answer": "By horizontal gene transfer, which can happen even between species. There are three routes: conjugation (one cell passes a plasmid to another through a bridge), transformation (a cell takes up loose DNA from its surroundings) and transduction (a bacteriophage delivers bacterial DNA to the next cell it infects).", + "quote": "Bacteria can also acquire **ready-made resistance genes** from other bacteria, even from different species. This is a major reason resistance spreads so fast in hospitals. There are three main routes:" + }, + "4": { + "answer": "An infection holds 10^9–10^11 cells, so 1 in a billion still means several resistant cells at the start. The full course kills the susceptible cells, and the resistant ones keep dividing about every 30 minutes, so they grow exponentially into most of the population. The full course doesn't help, because the drug level may be inside the selection window (between the two MICs), where the resistant cells grow anyway. If the immune system doesn't clear the few survivors, the infection ends up resistant.", + "quote": "It doesn't matter much how rare the resistant cells were to start with. Exponential growth makes up the difference quickly." + } + }, + "stops": [ + "\"bind its target tightly enough\": the reader may not know what binding means at the molecular level.", + "\"cut open the ring structure that penicillin-type drugs need\": 'ring structure' is never explained.", + "\"PBP2a, that β-lactams barely bind\": an unexplained acronym.", + "\"the \"Gram-negative\" group\": the term is used without saying what it means.", + "\"1 error per 10⁹–10¹⁰ base pairs copied\": 'base pairs' is never defined.", + "\"Luria–Delbrück, and the Lederbergs' replica plating\": the experiment names are dropped in without describing how they worked.", + "\"**integrons** (gene-capture systems)\": the gloss is too short to say what they do.", + "\"Resistance gains a foothold only if a cell gets a *second*, bigger resistance step.\": it's unclear why one step can't reach above MIC_R.", + "\"kill_i(c) = (g_max − g_min) · x^κ / (x^κ − g_min/g_max)\": the formula is given with no intuition behind it.", + "\"Shown as a multiple of the susceptible strain's MIC\": the slider unit takes a moment to work out." + ], + "excess": [ + "\"If you program, here is a loose analogy. Mutation is like random bit-flips in your own source code.\"", + "\"Biofilms. Communities of bacteria stuck to a surface inside a slimy matrix.\"", + "\"Tolerance. The bacteria die more *slowly* under the drug, but the MIC doesn't change.\"", + "\"Stop transmission. Hand hygiene, infection control and sanitation stop resistant strains spreading\"", + "\"Time advances in steps of 0.02 hours.\"", + "\"The antibiotic doesn't cause resistance. It **selects** for it, by removing the competition.\": this repeats the short answer." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/run.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/run.json new file mode 100644 index 0000000..c8219e2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/none/run.json @@ -0,0 +1,13 @@ +{ + "result": "I made `explainers/antibiotic-resistance/explanation.md` and `explainers/antibiotic-resistance/index.html`. The prose covers how resistance works and where it comes from, with rough numbers. The page is a small simulation where you change the drug dose and watch resistant cells take over from almost nothing, which this reader can explore and check against the stated model better than text alone. I haven't opened the page in a browser.", + "error": null, + "cost": 0.502619, + "seconds": 162, + "turns": 3, + "files": [ + "explainers/antibiotic-resistance/explanation.md", + "explainers/antibiotic-resistance/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/explainers/antibiotic-resistance/explanation.md b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/explainers/antibiotic-resistance/explanation.md new file mode 100644 index 0000000..4f26c3f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/explainers/antibiotic-resistance/explanation.md @@ -0,0 +1,90 @@ +# How bacteria become resistant to an antibiotic + +**Short answer.** A single bacterium does not learn to resist a drug. A population becomes resistant through selection. A few cells already carry a gene that protects them. The antibiotic kills the other cells, and the protected cells multiply to replace them. The protective genes come from two sources: random copying errors (mutation) and genes passed from other bacteria (gene transfer). + +This matters in practice. An infection can improve on treatment and then return, and the drug no longer works on the bacteria that come back. + +The rest of this file builds that answer in four steps. It shows how a drug kills a cell, how a cell can block the drug, where the blocking genes come from, and how the drug itself makes those genes common. + +## 1. How an antibiotic kills a cell + +An antibiotic is a molecule that binds to a **target**: a part of the bacterial cell that the cell needs to live or divide. Two examples: + +- Penicillin binds the enzymes that build the cell wall. The wall weakens, and the cell bursts. +- Streptomycin binds the ribosome, the machine that makes proteins. The cell makes faulty proteins and dies. + +Human cells do not have these exact targets, so the drug harms bacteria much more than it harms the patient. + +To work, the drug must do three things: get into the cell, stay intact, and bind its target. + +## 2. How a cell blocks the drug + +Each of those three requirements can fail. Each failure is a resistance **mechanism**, and a gene controls each one. + +| Requirement | How the cell blocks it | Example | +|---|---|---| +| Get in | Make fewer entry channels in the outer membrane | Fewer porin channels in *E. coli* | +| Get in (and stay in) | Pump the drug out faster than it enters (an **efflux pump**) | Tetracycline pumps | +| Stay intact | Make an enzyme that destroys the drug | Beta-lactamase cuts penicillin | +| Bind the target | Change the target shape so the drug no longer fits | A changed ribosome protein blocks streptomycin | + +Resistance is usually a matter of degree, not all or nothing. Microbiologists measure it as the **MIC** (minimum inhibitory concentration): the lowest drug concentration that stops a strain from growing. A low MIC, such as 1 mg/L, means a little drug stops the strain. A high MIC, such as 64 mg/L, means the strain grows at drug levels that a patient's blood may never safely reach. "Resistant" means the MIC is above the level that treatment can achieve. + +## 3. Where the resistance genes come from + +### Mutation + +Each time a bacterium copies its DNA to divide, it makes rare errors. Most errors do nothing or cause harm. A few change a target or switch on a pump, and these give resistance. + +Worked example (illustrative numbers; real rates vary by drug and species from about 1 in a million to 1 in 10 billion per division): + +- A serious infection contains about 10¹⁰ bacteria (estimate). +- To grow from one cell to 10¹⁰ cells takes about 10¹⁰ cell divisions. +- Suppose a specific resistance mutation occurs once per 10⁸ divisions. +- Expected resistant cells: 10¹⁰ × 1/10⁸ = **about 100**, present *before* the first dose. + +So in a large population, resistant cells usually already exist. The drug does not create them. + +The obvious alternative idea is that the drug makes bacteria change to survive. Luria and Delbrück tested this in 1943. They grew many separate cultures and then exposed them to a killer (a virus, in their case). If the exposure caused resistance, every culture would have a similar, small number of survivors. Instead, most cultures had none and a few had hundreds. That pattern only fits mutations that happened at random times *before* exposure: an early mutation has many generations to multiply, a late one has few. + +### Gene transfer + +Bacteria can also receive resistance genes that evolved in other bacteria. This is **horizontal gene transfer**: genes move between living cells, not only from parent to offspring. There are three routes: + +- **Conjugation:** one cell passes a **plasmid** (a small, separate DNA circle) to another through direct contact. +- **Transformation:** a cell takes up free DNA from a dead cell nearby. +- **Transduction:** a virus that infects bacteria carries a piece of DNA from one cell to the next. + +Gene transfer is the main reason resistance spreads fast in hospitals. One plasmid can carry genes for several mechanisms, so one transfer event can make a cell resistant to several drugs at once. Transfer also crosses species: a harmless gut bacterium can give a gene to a dangerous one. + +## 4. How the drug makes resistance common + +Return to the infection with 10¹⁰ bacteria, of which about 100 are resistant. That is 1 resistant cell in 100 million. + +Treatment starts. The drug kills the susceptible cells. Suppose the immune system and the drug together reduce them to about 10⁴ within two days. The 100 resistant cells are not affected by the drug. Suppose they double once per hour in the body (estimate). After one day, which is 24 doublings: + +100 × 2²⁴ ≈ 100 × 1.7 × 10⁷ ≈ **1.7 × 10⁹ resistant cells** + +The resistant fraction has gone from 1 in 100 million to nearly 100%. No cell changed. The drug removed the competition. This is natural selection, and it runs fast because bacteria divide in hours, not years. + +### Dose controls which cells survive + +Suppose the susceptible strain has an MIC of 1 mg/L and a partly resistant mutant has an MIC of 8 mg/L (illustrative). + +- At **16 mg/L**, the drug stops both strains. Nothing is selected. +- At **4 mg/L**, the drug stops the susceptible strain but not the mutant. This concentration selects for resistance most strongly. +- At **0.5 mg/L**, the drug stops neither. It selects only weakly. + +The range between the two MICs is the **mutant selection window**. Low doses, missed doses, and drug levels that fall slowly after a course all keep bacteria in this window. This is one reason a prescribed course has a set dose and length. It is also why some treatments use two drugs at once: a cell would need two independent mutations, about 1 in 10⁸ × 1 in 10⁸ = 1 in 10¹⁶ divisions, which is larger than the whole infection. + +### What happens when the drug is removed + +Resistance often has a cost. A pump uses energy, and a changed target can work less well. Suppose a resistant strain grows 5% slower per generation than the susceptible strain. Without the drug, after 100 generations its share falls by a factor of 0.95¹⁰⁰ ≈ 0.006, about 170 times. In this case, stopping use of the drug makes resistance rare again. + +That reversal often fails in practice. Resistant strains acquire further **compensatory mutations** that remove most of the cost but keep the resistance. Some resistance genes cost almost nothing from the start. Then resistance stays in the population for years after the drug is no longer used. + +## Summary + +Bacteria become resistant to an antibiotic in two steps. First, variation: a few cells carry a gene that keeps the drug out, pumps it out, destroys it, or changes its target. These genes arise from random mutation or arrive by gene transfer from other bacteria. Second, selection: the drug kills the cells without the gene, and the cells with it multiply until they make up most of the population. + +**Not covered here:** *tolerance* and *persistence* (cells that survive a drug by growing slowly, without a resistance gene), biofilms, and the spread of resistant bacteria between people and animals. diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/read.json new file mode 100644 index 0000000..3dbe9ea --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/read.json @@ -0,0 +1,42 @@ +{ + "answers": { + "1": { + "answer": "Resistance comes from two sources. The main one before treatment is random copying errors (mutations) made when bacteria copy their DNA to divide. In a large population, a few resistant cells already exist before the first dose (about 100 in 10¹⁰ cells in the worked example). The drug does not create them. Resistance genes can also arrive by gene transfer from other bacteria.", + "quote": "So in a large population, resistant cells usually already exist. The drug does not create them." + }, + "2": { + "answer": "It does not change any cell. It kills the susceptible cells and leaves the resistant ones alone. With their competition gone, the resistant cells multiply until they make up nearly the whole population. This is selection.", + "quote": "The resistant fraction has gone from 1 in 100 million to nearly 100%. No cell changed. The drug removed the competition." + }, + "3": { + "answer": "Through horizontal gene transfer, which can cross species. There are three routes: conjugation (one cell passes a plasmid to another by direct contact), transformation (a cell takes up free DNA from a dead cell) and transduction (a virus carries DNA from one cell to the next). For example, a harmless gut bacterium can give a resistance gene to a dangerous one.", + "quote": "Transfer also crosses species: a harmless gut bacterium can give a gene to a dangerous one." + }, + "4": { + "answer": "Applying the explanation's reasoning: in an infection of about 10¹⁰ cells, 1 in a billion is still about 10 resistant cells, and they are there before treatment starts. The drug kills the susceptible cells but does not affect the resistant ones. Those multiply quickly (24 doublings a day in the example) and replace the cells that died. So the infection can seem to improve and then come back, made of resistant bacteria the drug no longer works on.", + "quote": "The drug kills the susceptible cells. Suppose the immune system and the drug together reduce them to about 10⁴ within two days. The 100 resistant cells are not affected by the drug." + } + }, + "stops": [ + "'susceptible' is used many times but never defined: \"The drug kills the susceptible cells.\"", + "The cell-division count is stated without a reason: \"To grow from one cell to 10¹⁰ cells takes about 10¹⁰ cell divisions.\"", + "A probability is compared to a population size in a confusing way: \"1 in 10¹⁶ divisions, which is larger than the whole infection.\"", + "It is not explained why a dose that stops neither strain still selects at all: \"At **0.5 mg/L**, the drug stops neither. It selects only weakly.\"", + "The resistant cells are said to grow for a full day while the drug works for two days. How these timelines fit together is unclear: \"After one day, which is 24 doublings\"", + "The Luria–Delbrück reasoning goes by fast, and it is unclear how a virus experiment applies to antibiotics: \"exposed them to a killer (a virus, in their case)\"", + "The arithmetic step is not explained: \"its share falls by a factor of 0.95¹⁰⁰ ≈ 0.006, about 170 times\"", + "'porin' appears with no explanation: \"Fewer porin channels in *E. coli*\"" + ], + "excess": [ + "The intro restates the plan of the file: \"The rest of this file builds that answer in four steps.\"", + "The summary repeats the short answer: \"Bacteria become resistant to an antibiotic in two steps. First, variation\"", + "The list of excluded topics is off the question: \"**Not covered here:** *tolerance* and *persistence*\"", + "The point about human cells is not needed for the question about resistance: \"Human cells do not have these exact targets, so the drug harms bacteria much more than it harms the patient.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/run.json new file mode 100644 index 0000000..7cf1bbe --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/antibiotic-resistance/explanation.md` as prose only, because the answer is a short chain of variation then selection that worked numbers show well, and a diagram or interactive page would not make it easier to follow.", + "error": null, + "cost": 0.1721862, + "seconds": 49, + "turns": 2, + "files": [ + "explainers/antibiotic-resistance/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/explanation.md b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/explanation.md new file mode 100644 index 0000000..0daf3f3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/explanation.md @@ -0,0 +1,73 @@ +# How do bacteria become resistant to an antibiotic? + + + +**The drug does not change the bacteria. In a large infection, a few bacteria already carry a resistance gene, made by a random copying error or received from another bacterium. The drug removes the susceptible bacteria around them, and the resistant bacteria multiply.** The bacteria become resistant, not the patient's body. + +## What a cell must block + +An antibiotic is a molecule that kills bacteria or stops their growth. It does this by binding a **target**: a molecule the cell needs. For example, penicillin binds the enzymes that build the cell wall. + +A **resistance gene** is a gene, changed by mutation or received from another bacterium, that blocks this. It works in one of four ways: + +| Way | Example | +|---|---| +| Destroy the drug | A beta-lactamase enzyme cuts penicillin before penicillin reaches its target. | +| Change the target | One DNA letter change alters the target protein, so rifampicin no longer binds. | +| Pump the drug out | An efflux pump, a membrane protein, exports tetracycline faster than it enters. | +| Keep the drug out | The cell loses a porin, an entry channel in its surface, so less drug gets in. | + +Each way means the cell needs more drug to stop it. The lowest concentration that stops a strain's growth is its **MIC** (minimum inhibitory concentration). Suppose a drug reaches 8 mg/L in the blood. A strain with an MIC of 0.5 mg/L is stopped: it is **susceptible**. A strain with an MIC of 32 mg/L grows: it is **resistant**. (Illustrative values.) + +## Resistant cells are there before the drug + +A given resistance mutation (a mutation that turns a normal gene into a resistance gene) happens about once per 10⁸ cell divisions. A serious infection can hold 10¹⁰ cells. Each division adds one cell, so growing to that size took about 10¹⁰ divisions. Expect + +10¹⁰ × 10⁻⁸ = **100** resistance mutations, before anyone gives a drug. + +Early mutants have divided since, so there are at least 100 resistant cells. Compare a small infection of 10⁶ cells: it expects 0.01 mutations, so there is only about a 1% chance that even one resistant cell exists. + +## The drug selects them + + +1. The drug stops the susceptible cells: it kills them or halts their growth, and the immune system clears the halted cells. +2. The resistant cells keep dividing. The immune system removes some, but alone it cannot keep up with a population that keeps doubling. The drug did most of the work. +3. 100 cells need **27** doublings to reach 10¹⁰ again (100 × 2²⁷ ≈ 100 × 1.3 × 10⁸ ≈ 10¹⁰). At a lab doubling time of 30 minutes, that is 13 hours. +4. The infection returns, and almost every cell is resistant. + +This is **selection**: the drug did not make resistant cells, it removed the susceptible cells. + +**How do we know the drug did not cause the change?** If exposure caused resistance, each cell would have the same small chance when it met the drug. Equal cultures would then give similar counts. Luria and Delbrück tested this in 1943. They used a virus that kills bacteria; the test asks only whether resistance arose before contact, so it applies to drugs too. They grew 20 equal cultures of bacteria, then exposed each one and counted the resistant colonies. The mean was about 11. For independent rare events with a mean of 11, the random scatter is about √11 ≈ 3.3, so the "drug causes it" idea predicts most cultures between 5 and 18. They found 11 cultures with 0 and one with **107**. Random mutation before exposure explains this spread: a mutant that appears early leaves many resistant descendants, and a late one leaves few. + +## Genes from other bacteria + + +A cell does not have to wait for its own rare mutation. A **plasmid** is a small circle of DNA, separate from the chromosome. In **conjugation**, two bacteria touch and one copies a plasmid into the other. This is **horizontal gene transfer**: a gene from another bacterium, not from the parent cell. + +Plasmids carry complete working genes, such as the beta-lactamase gene, which one letter change could never build. The same plasmid gene turns up in different species, which shows that the gene moved between them. One plasmid often carries several resistance genes at once. + +## Without the drug, resistance usually costs + + +Resistant cells take over only while the drug is present. Without the drug, resistance often has a **fitness cost**: the extra protein, or the changed target, makes the cell grow more slowly. + +Suppose resistant cells grow 5% slower: each time the susceptible cells double, the resistant cells grow only 0.95 times as much. So each doubling multiplies the ratio of resistant to susceptible cells by 0.95. The effect compounds: after 100 doublings the ratio is 0.95¹⁰⁰ ≈ **0.006** of its start value. + +But resistance rarely vanishes, for two reasons: + +- A **compensatory mutation** is a second change, in another gene, that restores growth; the first change and the resistance stay. If it cuts the cost to 1%, then 0.99¹⁰⁰ ≈ 0.37 of the ratio remains after 100 doublings. +- Plasmids persist. One plasmid often carries resistance genes for several drugs, so any one of those drugs still selects the whole plasmid. + +## What this predicts + +- Large infections are the risk, because they hold resistant cells before treatment. +- Two drugs with independent targets need two mutations in one cell. The two are independent events, so their chances multiply: 10⁻⁸ × 10⁻⁸ = 10⁻¹⁶. In 10¹⁰ cells, expect 10⁻⁶ such cells: a one-in-a-million chance. The treatment clears the infection. This fails if one event gives resistance to both drugs, for example one plasmid with both genes, or one pump that exports both molecules. Then about 100 cells survive again. + +**In short:** resistance genes come first, by chance or from other bacteria. The drug removes the susceptible cells, and the resistant cells multiply. + +## Not covered + +- Persister cells, which survive a drug without a resistance gene ("antibiotic persistence"). +- Other routes of gene transfer: transformation and transduction. +- Drugs that raise the mutation rate ("SOS response"). +- Course length, prescribing policy, farm use, biofilms. diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.md b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.md new file mode 100644 index 0000000..328cfa5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.md @@ -0,0 +1,158 @@ +# Semantic model: how bacteria become resistant to an antibiotic + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How does a population of bacteria that an antibiotic used to kill become a population that the antibiotic no longer kills? + +## Audience and prior knowledge + +A technical reader: school mathematics and science, basic programming, new to microbiology. Knows: cells, DNA and genes (a gene codes for a protein), enzymes, that a mutation is a change in DNA, exponential growth and doubling, powers of ten, expected value, the idea of natural selection in outline. Does not know: how antibiotics act on a cell, what a plasmid is, horizontal gene transfer, minimum inhibitory concentration (MIC), fitness cost, the Luria–Delbrück result. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Bacterium | A single-cell organism that copies its DNA and divides in two. | fact | +| Antibiotic | A molecule that kills bacteria or stops their growth by binding to a target in the cell. | fact | +| Target | The molecule in the bacterium that the antibiotic binds to, usually a protein the cell needs (for example, the enzymes that build the cell wall). | fact | +| Resistance gene | A gene, changed by mutation or gained from another bacterium, that lets the cell grow at a drug concentration that stops normal cells. | fact | +| Susceptible cell | A cell that the drug stops at the concentration reached in the patient. | definition | +| Resistant cell | A cell that grows at the concentration reached in the patient. | definition | +| Mutation | A random copying error in DNA, made at a division, independently of the drug. | fact | +| Plasmid | A small circle of DNA, separate from the chromosome, that a bacterium can copy and pass to another bacterium. | fact | +| Horizontal gene transfer | A bacterium gets a gene from another bacterium, not from its parent cell. Main route: conjugation (two cells touch and one copies a plasmid into the other). | fact | +| Selection | The drug removes susceptible cells; resistant cells survive and divide, so their fraction in the population rises. | fact | +| MIC (minimum inhibitory concentration) | The lowest drug concentration that stops a strain's growth, measured in mg/L. | definition | +| Fitness cost | A resistant cell often grows more slowly than a susceptible cell when there is no drug, because it makes an extra protein or its changed target works less well. | fact (size varies) | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Mutation | makes (at random, before any drug) | resistance gene | about 1 in 10^8 divisions for one given resistance mutation | +| Horizontal gene transfer | gives | resistance gene | in one step, can cross species | +| Plasmid | carries | resistance genes | often several at once | +| Antibiotic | stops | susceptible cell | | +| Resistance gene | blocks | antibiotic action | four ways: destroy, change target, pump out, keep out | +| Antibiotic | selects | resistant cell | by removing its competitors | +| Fitness cost | slows | resistant cell, when there is no drug | | + +## Resistance mechanisms (what the resistance gene does) + +| Way | Example | +|---|---| +| Destroy the drug | A beta-lactamase enzyme cuts penicillin before it reaches the cell-wall enzymes. | +| Change the target | One DNA letter change in the rpoB gene changes the target protein, so rifampicin no longer binds. | +| Pump the drug out | An efflux pump (a membrane protein) exports tetracycline faster than it enters. | +| Keep the drug out | Loss of a porin (an entry channel in the outer membrane) lets less drug in. | + +Each way raises the MIC. Illustrative: beta-lactamase raises a strain's MIC from 0.5 to 32 mg/L. + +## Dependencies (learning order) + +1. An antibiotic works by binding a target in the cell. +2. A resistance gene stops that binding or its effect (four ways, with examples). +3. Resistance genes appear at random (mutation) or arrive from other bacteria (horizontal gene transfer), before the drug. +4. The drug removes the susceptible cells; the resistant ones remain and multiply (selection). +5. Without the drug, the fitness cost works against resistant cells. + +## Causal chain + +```text +random variation (mutation or gene transfer) → a few resistant cells among many susceptible ones +→ antibiotic stops the susceptible cells → resistant cells grow into the space and nutrients left +→ the population is now resistant → resistant cells spread to other hosts, and plasmids spread to other species +``` + +## Temporal sequence + +1. Before treatment: the infection grows to 10^10 cells. About 10^10 divisions made it, so about 100 resistance mutations happened. Early mutants have divided since, so resistant cells number at least 100, typically several hundred. +2. Treatment: the drug stops the susceptible cells, and the immune system clears them; the resistant cells keep dividing. +3. Regrowth: 100 cells double about 27 times to reach 10^10 again (2^27 ≈ 1.3 × 10^8 ≈ 10^10 / 100). Assumption: the immune system removes some cells but does not keep up with a population that doubles. Weakened patients clear fewer. +4. The infection returns, and the same drug no longer works on it. + +## Quantities + +- Rate of one given resistance mutation: about 10^-8 per division (order of magnitude; real values span about 10^-6 to 10^-10). +- Doubling: every cell in the population divides once. Growing from 1 cell to N cells takes about N divisions. +- Cells in a serious infection: 10^9 to 10^10 or more (estimate). Example uses 10^10. +- Expected resistance mutations before treatment: 10^10 × 10^-8 = 100. Resistant cells ≥ 100, because early mutants divide (mean about μN ln N ≈ 2,300; typical several hundred; heavy spread between infections). +- Small infection, 10^6 cells: 10^6 × 10^-8 = 0.01 expected mutations. Probability of at least one resistant cell = 1 − e^-0.01 ≈ 1%. For 10^10 cells, probability ≈ 1. +- Regrowth: 100 → 10^10 needs log2(10^8) ≈ 26.6, so about 27 doublings. At 30 min per doubling (laboratory, ideal) that is about 13 hours; in a patient, doublings are slower. +- MIC example (illustrative): susceptible strain 0.5 mg/L, resistant strain 32 mg/L, concentration reached in blood 8 mg/L. 8 > 0.5: the drug stops the first strain. 8 < 32: the second strain grows. +- Fitness cost (illustrative): resistant cells grow 5% slower, no drug: each time the susceptible cells double, the resistant cells grow 0.95 times as much. The resistant-to-susceptible ratio is multiplied by 0.95 each doubling, because both groups compound: after 10 doublings 0.95^10 ≈ 0.60, after 100 doublings 0.95^100 ≈ 0.006. (Not 1 − 100 × 0.05, which goes negative.) +- Compensatory mutation: a second mutation, in another gene, that cuts the cost from 5% to 1% without losing resistance: 0.99^100 ≈ 0.37 after 100 doublings. +- Plasmid persistence: one plasmid often carries resistance genes for several drugs, so any one of those drugs still selects the whole plasmid. +- Luria–Delbrück scatter: independent rare events with mean 11 have a spread of about √11 ≈ 3.3; mean ± 2 spreads ≈ 5 to 18. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Drug present | resistant fraction rises from about 10^-8 to near 1 | susceptible cells are removed | +| Drug absent | resistant fraction falls slowly, or stays | fitness cost; compensatory mutations reduce it | +| Small population (10^6) | about 1% chance of any resistant cell; drug clears it | 10^6 × 10^-8 = 0.01 | +| Two drugs with independent targets | needs two independent mutations: 10^-16 | 10^10 × 10^-16 = 10^-6 expected double-resistant cells | +| Two drugs, one plasmid or pump resists both | one event, about 10^-8 | 10^10 × 10^-8 = 100: the combination fails | +| Plasmid gene | one transfer gives a whole resistance gene, even to another species | not a rare copying error | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Explain resistance as selection of variants that existed before the drug | "The drug makes the bacteria change" | Luria and Delbrück (1943) grew 20 parallel cultures and plated each on a killing agent (a virus, not an antibiotic; same logic). Mean about 11 resistant colonies. If the agent caused resistance, every culture would show about 11 (most between 5 and 18). Observed: 11 cultures had 0, one had 107. That spread fits mutations at random times before exposure: an early mutant leaves many resistant descendants, a late one few. | +| Count horizontal gene transfer as a second source | "Every resistance must be a new mutation in that lineage" | A beta-lactamase is a complete enzyme; one letter change does not build a new enzyme. Bacteria get such genes on plasmids, and the same plasmid gene turns up in different species. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| Resistant cells exist before treatment (mechanism) | 10^10 cells × 10^-8 = 100 mutations: at least 100 resistant cells | 10^6 cells × 10^-8 = 0.01: about a 1% chance of any; the drug clears the infection | +| Resistance takes over only when the drug is present (guarantee) | With drug: fraction about 10^-8 → near 1 after regrowth (100 cells → 10^10 in 27 doublings) | Without drug, 5% cost: ratio × 0.95^100 ≈ 0.006 after 100 generations | +| Two drugs together prevent selection (guarantee, assumes independent resistance) | Independent: 10^-8 × 10^-8 = 10^-16; 10^10 cells give 10^-6 expected double-resistant cells | One plasmid or pump resists both: 10^10 × 10^-8 = 100 survivors | + +## Terms + +- **Antibiotic:** a molecule that kills bacteria or stops their growth by binding a target. +- **Target:** the molecule the antibiotic binds to. +- **Resistance gene:** a gene, changed by mutation or gained from another bacterium, that lets the cell grow in the drug. +- **Resistant / susceptible:** grows / is stopped at the concentration the drug reaches in the patient. Resistance is a property of the bacteria, not of the patient. +- **Mutation:** a random copying error in DNA at a division. +- **Plasmid:** a small circle of DNA, separate from the chromosome, that can pass between bacteria. +- **Horizontal gene transfer:** getting a gene from another bacterium, not from the parent. +- **Conjugation:** two bacteria touch, and one copies a plasmid into the other. +- **Selection:** the drug removes the susceptible cells, so the resistant fraction rises. +- **MIC:** the lowest concentration that stops a strain's growth. Low MIC (0.5 mg/L, below the 8 mg/L reached): the drug works. High MIC (32 mg/L, above it): the drug fails. +- **Fitness cost:** slower growth of a resistant cell when no drug is present. +- **Doubling:** every cell divides once. "Division" is one cell splitting. +- Mutation rate 10^-8 per division: one in a hundred million divisions makes this mutation. At 10^-6, resistance appears even in small populations; at 10^-10, only in very large ones. + +## Epistemic status + +- **Observed / established:** the four resistance mechanisms; mutations arise before exposure (Luria–Delbrück 1943); plasmids carry and transfer resistance genes; selection by antibiotic use. +- **Mathematical consequence:** expected counts (N × rate); doublings needed; product of independent rates. +- **Assumed:** the two resistances in combination therapy are independent; constant doubling time; the immune system does not keep up with growth. +- **Estimated / illustrative:** the 10^-8 rate, 10^10 cell count, 30 min doubling, 5% cost and MIC values are orders of magnitude. + +## Confusion points + +- "The patient becomes resistant" → the bacteria are resistant; the patient's body does not change. +- "The drug makes bacteria mutate / bacteria learn" → the variants exist before the drug; the drug removes their competitors. +- "Each cell slowly gets used to the drug" → a cell's resistance comes from its genes. The population changes because the fraction of resistant cells changes. Resistance can rise in steps (a first mutation raises MIC from 0.5 to 2 mg/L, a second to 16 mg/L), but each step is a new gene change, selected in turn. +- "Resistance disappears when we stop the drug" → it falls slowly at best; compensatory mutations reduce the cost, and plasmids persist. + +## Scope + +- **Out of scope:** tolerance and persister cells (survive without being resistant) — search "antibiotic persistence". +- **Out of scope:** transformation and transduction (other routes of gene transfer) — "horizontal gene transfer" in any microbiology text. +- **Out of scope:** stress-induced mutagenesis (some drugs raise the mutation rate via the SOS response) — search "SOS response antibiotic". +- **Out of scope:** course length, stewardship policy, agricultural use, biofilms. + +## Representation decision + +- **Stage:** 1 +- **Reason:** the subject is a causal sequence (variation → selection → regrowth → spread) with one table of mechanisms and short arithmetic. Nothing needs a parameter explored or motion seen. +- **Levels:** L1 resistance is selection of variants that existed before the drug / L2 the four mechanisms and the two sources / L3 numbers: N × rate, doublings, cost. diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.yaml b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.yaml new file mode 100644 index 0000000..230d7fe --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/model.yaml @@ -0,0 +1,99 @@ +# Machine-readable half of model.md. `explainer check antibiotic-resistance` holds every rendering to it. + +values: + expected_mutations_large: 100 # 10^10 cells × 10^-8 per division + small_infection_chance_pct: 1 # 1 − e^-0.01 ≈ 1% + regrowth_doublings: 27 # log2(10^8) ≈ 26.6 + two_pow_27: 130000000 # 2^27 ≈ 1.34 × 10^8 + lab_regrowth_hours: 13 # 27 × 30 min + doubling_minutes: 30 + mic_susceptible: 0.5 # mg/L + mic_resistant: 32 # mg/L + blood_concentration: 8 # mg/L + cost_pct: 5 + cost_generations: 100 + ratio_after_100_generations: 0.006 # 0.95^100 + compensated_cost_pct: 1 + compensated_ratio: 0.37 # 0.99^100 + ld_cultures: 20 + ld_mean: 11 + ld_zero_cultures: 11 + ld_max: 107 + ld_poisson_low: 5 + ld_poisson_high: 18 + ld_year: 1943 + ld_scatter: 3.3 # √11 + +allow: [1, 2, 4, 6, 8, 10, 16, -6, -8, -16, 0.95, 0.99, 2.0] + +require: + - path: expected_mutations_large + in: [prose] + - path: regrowth_doublings + in: [prose] + - path: ratio_after_100_generations + in: [prose] + - path: ld_max + in: [prose] + +claims: + - id: preexisting + kind: mechanism + statement: "Resistant cells exist before the drug arrives, made by random mutation at about 10^-8 per division; a large infection already holds some." + example: "10^10 cells × 10^-8 = 100 resistance mutations before treatment; a 10^6-cell infection has about a 1% chance of any." + ask: "An infection of 10^10 bacteria has never met drug X. A given resistance mutation happens once per 10^8 divisions. Does the infection contain resistant cells, and roughly how many?" + - id: selection-not-induction + kind: why + statement: "The drug does not cause resistance; it removes the susceptible cells, so resistant cells that were already there take over." + alternative: "The drug makes the bacteria change, as an adaptation." + counterexample: "Luria and Delbrück 1943: 20 parallel cultures, mean about 11 resistant colonies. Induction predicts about 11 in each (most 5 to 18); observed 11 cultures with 0 and one with 107." + ask: "Why do scientists say the antibiotic selects resistance rather than causes it? What evidence separates the two ideas?" + - id: mechanisms + kind: definition + statement: "A resistance gene blocks the drug in one of four ways: destroy it, change its target, pump it out, or keep it out; each raises the MIC." + - id: hgt + kind: why + statement: "Bacteria also get complete resistance genes from other bacteria, often on plasmids, even across species." + alternative: "Every resistance is a new mutation in that lineage." + counterexample: "A beta-lactamase is a complete enzyme that one letter change cannot build; the same plasmid gene appears in different species." + ask: "How can a species become resistant to a drug in one step, without any mutation of its own?" + - id: takeover-needs-drug + kind: guarantee + statement: "Resistant cells take over only while the drug is present; without the drug a fitness cost makes their fraction fall." + example: "With drug: 100 survivors regrow to 10^10 in about 27 doublings (about 13 h at 30 min per doubling)." + counterexample: "Without drug, 5% slower growth: resistant-to-susceptible ratio × 0.95^100 ≈ 0.006 after 100 doublings; with a compensatory mutation (1% cost), 0.99^100 ≈ 0.37; plasmids with genes for several drugs stay selected by any of them." + ask: "If doctors stop using an antibiotic, what happens to the resistant fraction, and why might it not disappear?" + - id: mic + kind: definition + statement: "MIC is the lowest concentration that stops a strain's growth; a strain is resistant when its MIC is above the concentration the drug reaches in the patient." + example: "Blood concentration 8 mg/L: MIC 0.5 mg/L strain stopped, MIC 32 mg/L strain grows." + - id: combination + kind: guarantee + statement: "Two drugs given together stop selection when resistance to each needs its own independent change." + example: "10^-8 × 10^-8 = 10^-16; 10^10 cells give 10^-6 expected double-resistant cells." + counterexample: "If one plasmid or pump resists both drugs, one event at about 10^-8 suffices: 10^10 × 10^-8 = 100 survivors." + +accept_unjustified: + "√": "the scatter of independent rare-event counts is √mean (Poisson); stated as a fact the reader can use, deriving it is off the main line" + ln: "model.md only: the Luria–Delbrück mean μN ln N, a background check that resistant cells exceed 100; no rendering shows it" + +terms: + - term: resistant + means: "a property of the bacteria, not of the patient or the body" + avoid: [patient becomes resistant, body becomes resistant, become immune to] + - term: selection + means: "the drug removes susceptible cells; it does not cause the change" + avoid: [bacteria learn, drug causes mutations, adapt to the drug] + +budget: {prose: 950, reason: "two sources of resistance genes, four mechanisms in a table, and the selection guarantee with its no-drug case and the Luria–Delbrück evidence"} + +quiz: + - q: "A patient's infection has 10^10 bacteria. A resistance mutation to drug X happens once in 10^8 divisions. The patient takes drug X; it clears symptoms for a day, then the infection returns and drug X no longer works. Explain what happened." + expect: "About 100 resistant cells existed before treatment (10^10 × 10^-8). The drug removed the susceptible cells; the resistant cells survived and regrew (about 27 doublings to 10^10). The bacteria population is now resistant." + misconception: "The drug caused the bacteria to mutate or adapt, or the patient's body became resistant." + - q: "Would you expect the same outcome if the infection had only 10^6 bacteria? Why?" + expect: "Probably not: expected resistant mutations 10^6 × 10^-8 = 0.01, about a 1% chance any resistant cell exists, so the drug most likely clears it." + misconception: "Same outcome, because resistance develops during treatment regardless of population size." + - q: "A hospital stops using drug X for years. Does resistance to X vanish?" + expect: "It usually falls, because resistant cells often grow slower without the drug (fitness cost), but slowly and not always to zero: compensatory mutations reduce the cost and plasmids persist." + misconception: "Yes, it disappears quickly, or no, nothing changes because resistance is permanent." diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/narrative.md b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/narrative.md new file mode 100644 index 0000000..4354736 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/narrative.md @@ -0,0 +1,59 @@ +# Narrative: antibiotic-resistance + +> Pass 3. `model.md` says what is true. This file says the path a first-time reader takes to it. + +## 1. Reader before and after + +- **Before:** a technical reader (school science and mathematics, basic programming) who knows cells, genes, mutation, exponential growth, and natural selection in outline, and nothing of how antibiotics act or how bacteria exchange genes. +- **After:** "Resistance genes appear at random or arrive from other bacteria before the drug; the drug removes the susceptible cells, so the resistant ones are left to multiply." + +## 2. Question and motive + +- **Question:** How does an infection that a drug used to clear become one it cannot clear? Result in words first: the drug does not change the bacteria. A few bacteria already carry a resistance gene; the drug removes their competitors. +- **Why care:** the common picture ("bacteria get used to the drug") predicts the wrong things: that resistance appears in every infection, that it needs the drug to arise, and that it is in the patient. +- **Why this approach:** start from what the drug does to one cell, because resistance is a way to block that. Then count: how many cells could already have a resistance gene? The count is the surprise. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| target | the molecule the antibiotic binds | penicillin binds the enzymes that build the cell wall | 2 | +| resistance gene | gene, mutated or gained, that lets the cell grow in the drug | beta-lactamase gene | 3 | +| beta-lactamase, efflux pump, porin, rpoB | the four mechanism examples | one row each in the table | 3 | +| MIC | lowest concentration that stops a strain's growth | 0.5 vs 32 mg/L against 8 mg/L in blood | 3 | +| resistant / susceptible | grows / stopped at the concentration reached | the two MIC strains | 3 | +| mutation rate per division | chance that one division makes this mutation | 10^-8 | 4 | +| selection | drug removes susceptible cells, resistant fraction rises | 100 survivors regrow | 5 | +| doubling | every cell divides once | 100 → 10^10 in 27 doublings | 5 | +| Luria–Delbrück | the experiment that separates selection from induction | 20 cultures, 0 to 107 | 6 | +| plasmid, horizontal gene transfer, conjugation | gene from another bacterium | beta-lactamase gene on a plasmid in different species | 7 | +| fitness cost, compensatory mutation | slower growth without drug; a second mutation that reduces it | 0.95^100 ≈ 0.006; 0.99^100 ≈ 0.37 | 8 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How does an infection that a drug used to clear become one it cannot clear? | — | the question as title, bold answer | The drug does not change the bacteria. A few bacteria already carry a resistance gene, by chance or from other bacteria; the drug removes their competitors, and they multiply. The bacteria are resistant, not the patient's body. | the answer | +| 2 | mechanism | What does a drug do to a cell? | so, to resist, a cell must block this | — | An antibiotic binds a target, a molecule the cell needs: penicillin binds the enzymes that build the cell wall. | target | +| 3 | mechanism | How can a cell stop that? | therefore | table: four ways, each with one example in words | Destroy the drug (beta-lactamase enzyme cuts penicillin); change the target so the drug no longer fits (one letter in rpoB, rifampicin); pump it out (efflux pump); keep it out (lose a porin, an entry channel). Each way means a higher concentration is needed to stop growth. That concentration is the MIC, minimum inhibitory concentration. Blood reaches about 8 mg/L: a strain with MIC 0.5 is stopped (susceptible), a strain with MIC 32 grows (resistant). | mechanisms, MIC, resistant | +| 4 | instance | Where does a resistance gene come from in an infection that never met the drug? | but the drug was never there… | arithmetic | A copying error at one division makes a given resistance mutation about once per 10^8 divisions. An infection of 10^10 cells took about 10^10 divisions to grow, so expect 10^10 × 10^-8 = 100 mutations. Early mutants have divided since, so resistant cells number at least 100. An infection of 10^6 cells expects 0.01 mutations: about a 1% chance that even one resistant cell exists. | variants exist before the drug | +| 5 | mechanism | What does the drug do to that mix? | so | numbered steps | The drug stops the susceptible cells; the immune system clears those, but it does not keep up with a population that doubles. The 100 resistant cells divide: 27 doublings (every cell divides once) take them back to 10^10, about 13 hours at a lab doubling time of 30 min. This is selection. | selection | +| 6 | objection | How do we know the drug did not cause the change? | but | the numbers of one experiment | If the drug caused the change (induction), each cell would have the same small chance when exposed, so equal cultures would show similar counts. Luria and Delbrück (1943) grew 20 equal cultures and then exposed them; mean 11 resistant colonies, so induction predicts most cultures between 5 and 18. They found 11 cultures with 0 and one with 107. An early mutant leaves many descendants, a late one few: that spread means the mutations happened before exposure. | evidence | +| 7 | generalize | Must every lineage wait for its own rare mutation? | but | — | A plasmid is a small circle of DNA, separate from the chromosome. In conjugation, two bacteria touch and one copies a plasmid into the other. So a cell gets a complete working gene, such as the beta-lactamase gene, in one step, even from another species. One letter change cannot build a whole new enzyme; this is how such genes spread. | second source | +| 8 | test | Does resistance stay after the drug stops? | therefore: takeover needs the drug | numbers | Resistance often has a fitness cost: the extra protein or the changed target makes the cell grow slower when there is no drug. With 5% slower growth, the ratio of resistant to susceptible cells is multiplied by 0.95 each generation: after 100 generations, 0.006. But a compensatory mutation (a second change that cuts the cost to 1%) gives 0.99^100 ≈ 0.37, so resistance can persist. | takeover needs the drug, and its limit | +| 9 | payoff | What does this predict? | so | two lines | Large infections are the risk, by the count in beat 4. Two drugs with independent targets need two mutations in one cell: 10^-8 × 10^-8 = 10^-16, so 10^10 cells expect 10^-6. This fails if one plasmid or pump resists both: 100 survivors again. | predictions | +| 10 | close | — | — | — | Resistance genes come first, by chance or from other bacteria; the drug removes the competitors; the resistant cells multiply. A short "not covered" list follows, outside the argument. | — | + +## 5. Concrete to symbol + +Expected count = cells × rate: "here 10^10 cells and 10^-8 per division give 100". Checked at 10^6 → 0.01. + +## 6. Links between representations + +The MIC numbers (0.5, 8, 32) appear in the table caption and in the definition of resistant. + +## 7. Close the loop + +- **Answer:** resistance genes come first, by chance or from other bacteria; the drug removes the competitors; the resistant cells multiply. +- **General argument:** any heritable variant that grows under the drug while others are stopped takes over while the drug is present. +- **Payoff:** predicts when resistance appears (large populations, single drugs, long use) and why it persists. Scope: persisters, other transfer routes, SOS mutagenesis, policy. diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/decisions.yaml new file mode 100644 index 0000000..4cfc2fc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/decisions.yaml @@ -0,0 +1,221 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T151610-probe#1: + decision: adopted +2026-10-04T151610-probe#10: + decision: adopted +2026-10-04T151610-probe#11: + decision: adopted +2026-10-04T151610-probe#12: + decision: declined +2026-10-04T151610-probe#13: + decision: adopted +2026-10-04T151610-probe#14: + decision: declined +2026-10-04T151610-probe#15: + decision: adopted +2026-10-04T151610-probe#2: + decision: adopted +2026-10-04T151610-probe#3: + decision: adopted +2026-10-04T151610-probe#4: + decision: adopted +2026-10-04T151610-probe#5: + decision: adopted +2026-10-04T151610-probe#6: + decision: adopted +2026-10-04T151610-probe#7: + decision: adopted +2026-10-04T151610-probe#8: + decision: adopted +2026-10-04T151610-probe#9: + decision: adopted +2026-10-04T151845-coldread-narrative#1: + decision: adopted +2026-10-04T151845-coldread-narrative#10: + decision: adopted +2026-10-04T151845-coldread-narrative#11: + decision: adopted +2026-10-04T151845-coldread-narrative#12: + decision: adopted +2026-10-04T151845-coldread-narrative#13: + decision: adopted +2026-10-04T151845-coldread-narrative#14: + decision: adopted +2026-10-04T151845-coldread-narrative#15: + decision: adopted +2026-10-04T151845-coldread-narrative#16: + decision: adopted +2026-10-04T151845-coldread-narrative#17: + decision: adopted +2026-10-04T151845-coldread-narrative#18: + decision: adopted +2026-10-04T151845-coldread-narrative#19: + decision: adopted +2026-10-04T151845-coldread-narrative#2: + decision: adopted +2026-10-04T151845-coldread-narrative#20: + decision: adopted +2026-10-04T151845-coldread-narrative#21: + decision: adopted +2026-10-04T151845-coldread-narrative#22: + decision: adopted +2026-10-04T151845-coldread-narrative#23: + decision: adopted +2026-10-04T151845-coldread-narrative#24: + decision: adopted +2026-10-04T151845-coldread-narrative#25: + decision: adopted +2026-10-04T151845-coldread-narrative#26: + decision: adopted +2026-10-04T151845-coldread-narrative#27: + decision: adopted +2026-10-04T151845-coldread-narrative#28: + decision: adopted +2026-10-04T151845-coldread-narrative#29: + decision: adopted +2026-10-04T151845-coldread-narrative#3: + decision: adopted +2026-10-04T151845-coldread-narrative#30: + decision: adopted +2026-10-04T151845-coldread-narrative#31: + decision: adopted +2026-10-04T151845-coldread-narrative#32: + decision: adopted +2026-10-04T151845-coldread-narrative#33: + decision: adopted +2026-10-04T151845-coldread-narrative#34: + decision: adopted +2026-10-04T151845-coldread-narrative#35: + decision: adopted +2026-10-04T151845-coldread-narrative#36: + decision: adopted +2026-10-04T151845-coldread-narrative#37: + decision: adopted +2026-10-04T151845-coldread-narrative#38: + decision: adopted +2026-10-04T151845-coldread-narrative#4: + decision: adopted +2026-10-04T151845-coldread-narrative#5: + decision: declined +2026-10-04T151845-coldread-narrative#6: + decision: declined +2026-10-04T151845-coldread-narrative#7: + decision: adopted +2026-10-04T151845-coldread-narrative#8: + decision: adopted +2026-10-04T151845-coldread-narrative#9: + decision: adopted +2026-10-04T152038-coldread-prose#1: + decision: adopted +2026-10-04T152038-coldread-prose#10: + decision: adopted +2026-10-04T152038-coldread-prose#11: + decision: adopted +2026-10-04T152038-coldread-prose#12: + decision: adopted +2026-10-04T152038-coldread-prose#13: + decision: declined +2026-10-04T152038-coldread-prose#14: + decision: declined +2026-10-04T152038-coldread-prose#2: + decision: adopted +2026-10-04T152038-coldread-prose#3: + decision: adopted +2026-10-04T152038-coldread-prose#4: + decision: adopted +2026-10-04T152038-coldread-prose#5: + decision: adopted +2026-10-04T152038-coldread-prose#6: + decision: adopted +2026-10-04T152038-coldread-prose#7: + decision: adopted +2026-10-04T152038-coldread-prose#8: + decision: adopted +2026-10-04T152038-coldread-prose#9: + decision: declined +2026-10-04T152206-quiz-prose#1: + decision: adopted +2026-10-04T152206-quiz-prose#10: + decision: declined +2026-10-04T152206-quiz-prose#11: + decision: adopted +2026-10-04T152206-quiz-prose#12: + decision: adopted +2026-10-04T152206-quiz-prose#13: + decision: adopted +2026-10-04T152206-quiz-prose#14: + decision: adopted +2026-10-04T152206-quiz-prose#15: + decision: adopted +2026-10-04T152206-quiz-prose#16: + decision: declined +2026-10-04T152206-quiz-prose#17: + decision: declined +2026-10-04T152206-quiz-prose#18: + decision: declined +2026-10-04T152206-quiz-prose#19: + decision: declined +2026-10-04T152206-quiz-prose#2: + decision: adopted +2026-10-04T152206-quiz-prose#20: + decision: adopted +2026-10-04T152206-quiz-prose#21: + decision: adopted +2026-10-04T152206-quiz-prose#22: + decision: declined +2026-10-04T152206-quiz-prose#3: + decision: adopted +2026-10-04T152206-quiz-prose#4: + decision: declined +2026-10-04T152206-quiz-prose#5: + decision: adopted +2026-10-04T152206-quiz-prose#6: + decision: adopted +2026-10-04T152206-quiz-prose#7: + decision: adopted +2026-10-04T152206-quiz-prose#8: + decision: declined +2026-10-04T152206-quiz-prose#9: + decision: declined +2026-10-04T152433-quiz-prose#1: + decision: declined +2026-10-04T152433-quiz-prose#10: + decision: declined +2026-10-04T152433-quiz-prose#11: + decision: declined +2026-10-04T152433-quiz-prose#12: + decision: declined +2026-10-04T152433-quiz-prose#13: + decision: declined +2026-10-04T152433-quiz-prose#14: + decision: declined +2026-10-04T152433-quiz-prose#15: + decision: declined +2026-10-04T152433-quiz-prose#16: + decision: declined +2026-10-04T152433-quiz-prose#17: + decision: declined +2026-10-04T152433-quiz-prose#18: + decision: declined +2026-10-04T152433-quiz-prose#19: + decision: declined +2026-10-04T152433-quiz-prose#2: + decision: declined +2026-10-04T152433-quiz-prose#20: + decision: declined +2026-10-04T152433-quiz-prose#21: + decision: declined +2026-10-04T152433-quiz-prose#3: + decision: declined +2026-10-04T152433-quiz-prose#4: + decision: declined +2026-10-04T152433-quiz-prose#5: + decision: declined +2026-10-04T152433-quiz-prose#6: + decision: declined +2026-10-04T152433-quiz-prose#7: + decision: adopted +2026-10-04T152433-quiz-prose#8: + decision: declined +2026-10-04T152433-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151610-probe.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151610-probe.json new file mode 100644 index 0000000..e2de577 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151610-probe.json @@ -0,0 +1,230 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:16:10", + "model": "default", + "prompt_sha": "e5129749a45f", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "'About 100 carry a resistance mutation' (N × rate) counts mutation events but calls them resistant cells. Descendants are left out.", + "why_it_matters": "The file's own Luria–Delbrück row says an early mutation leaves many resistant descendants. So the number of resistant cells is larger than N × rate, and it varies a lot from culture to culture. The causal chain and the temporal sequence both rely on this number, and the step from mutation events to resistant cells is left implicit.", + "suggested_addition": "Keep two separate names: 'mutation events ≈ N × μ = 10^10 × 10^-8 = 100' and 'resistant cells'. Then add: mutants that arose in earlier generations have divided since, so the expected number of resistant cells is about μN·ln(N) ≈ 10^-8 × 10^10 × 23 ≈ 2,300, and the median is about μN·ln(μN) ≈ 100 × 4.6 ≈ 460. State that 100 is a lower bound." + }, + { + "rule": 2, + "severity": "main", + "about": "The model never says why the roughly 100 surviving cells are not cleared by the host. It assumes nothing kills resistant cells after treatment.", + "why_it_matters": "A learner will expect the immune system to mop up 100 cells. Without this, step 3 (regrowth to 10^10) looks unjustified, and the 'small population is cleared' case seems to rest only on the drug.", + "suggested_addition": "State the assumption: host defences remove some cells but cannot keep up with a population that doubles. Give an example: if immune killing removes 50% of cells per doubling time, net growth is zero; if it removes 20%, 100 cells still reach 10^10 in about 27/log2(1.6) ≈ 40 doublings. Say that weakened patients (ICU, chemotherapy) clear fewer cells." + }, + { + "rule": 4, + "severity": "main", + "about": "The four resistance mechanisms (destroy, change target, pump out, keep out) are only listed. None has a concrete example, and the learning order puts this as step 2.", + "why_it_matters": "The audience explicitly does not know how antibiotics act on a cell. 'A change in DNA can stop that binding' stays abstract with no instance.", + "suggested_addition": "Give one line each. Destroy: a beta-lactamase enzyme cuts penicillin's beta-lactam ring before it reaches the wall-building enzymes. Change target: one base change in the rpoB gene stops rifampicin binding. Pump out: an efflux pump exports tetracycline. Keep out: loss of a porin channel cuts drug entry. Link one example to MIC numbers, e.g. beta-lactamase raises MIC from 0.5 to 32 mg/L (64-fold)." + }, + { + "rule": 2, + "severity": "main", + "about": "The 'breaks' column for combination therapy compares with one drug instead of removing the stated assumption, which is that the two resistances are independent.", + "why_it_matters": "The file itself says plasmids carry several resistance genes at once, and a single efflux pump can export both drugs. Either way the 10^-16 product fails. The claim also silently assumes both drugs are given together and that each drug alone kills cells resistant to the other.", + "suggested_addition": "Breaks case: if one plasmid or pump gives resistance to both drugs at about 10^-8, then expected double-resistant cells = 10^10 × 10^-8 = 100. Second breaks case: drug A alone leaves about 100 A-resistant cells. They regrow to 10^10 and then drug B is added: 10^10 × 10^-8 = 100 cells resistant to both." + }, + { + "rule": 3, + "severity": "main", + "about": "Several concepts have two names or two definitions. 'Resistance gene / resistance mutation' is one entity. 'Resistant' is defined both as 'survives the dose used' and as 'higher MIC'. Antibiotic 'kills or stops growth' (Entities) vs 'kills' (Terms). MIC 'stops growth' but the example says 8 mg/L 'kills'. Division / doubling / generation are used for the same thing.", + "why_it_matters": "A reader cannot tell whether resistance is relative to a dose or to a strain. They also cannot tell whether exceeding the MIC kills cells or only stops their growth.", + "suggested_addition": "Use one term, 'resistance determinant (mutation or acquired gene)'. Define resistant as 'MIC above the concentration reachable in the patient (e.g. MIC 32 > 8 mg/L)'. Say MIC measures growth inhibition, not killing. Pick one word, 'doubling', for one division of every cell." + }, + { + "rule": 5, + "severity": "edge", + "about": "'Low MIC ratio means the drug still works' uses an undefined ratio and gives no threshold value.", + "why_it_matters": "The reader cannot judge what number counts as resistant.", + "suggested_addition": "Define the ratio as reachable concentration ÷ MIC. 8/0.5 = 16 means the drug works; 8/32 = 0.25 means it fails. Give the cutoff (a clinical breakpoint), e.g. ratio ≥ about 1 to 4 counts as susceptible (illustrative)." + }, + { + "rule": 1, + "severity": "edge", + "about": "The fitness cost compounds multiplicatively (0.95^100). There is no justification against the linear alternative.", + "why_it_matters": "A learner may compute 1 − 0.05 × 100 = −4 (or 0) and get confused.", + "suggested_addition": "Each generation the resistant/susceptible ratio is multiplied by 0.95, because both lineages compound. So after 100 generations the ratio is 0.95^100 ≈ 0.006, not 1 − 5 = negative. After 10 generations it is ≈ 0.60." + }, + { + "rule": 1, + "severity": "edge", + "about": "The 'many 0, a few hundreds' result of Luria–Delbrück has no concrete numbers or comparison statistic.", + "why_it_matters": "The argument against 'the drug induces resistance' depends on variance versus mean, and that is never shown.", + "suggested_addition": "Example: 20 parallel cultures, mean about 10 resistant colonies. Induction (Poisson) predicts most counts between about 4 and 16, with variance ≈ mean ≈ 10. Observed counts were 11 zeros, several 1–5, and one of 107, so variance ≫ mean (illustrative; recompute)." + }, + { + "rule": 4, + "severity": "edge", + "about": "Plasmid transfer and spread to other hosts and species have no mechanism (conjugation) and no numbers.", + "why_it_matters": "'No waiting for a rare error' implies transfer is fast, but no rate is given to compare with 10^-8.", + "suggested_addition": "Name conjugation: cell-to-cell contact through a pilus copies the plasmid. Give an illustrative rate of 10^-4 to 10^-2 transfers per donor per hour in dense gut populations, versus 10^-8 per division for mutation." + }, + { + "rule": 3, + "severity": "edge", + "about": "'Compensatory mutations' appears in the confusion points without definition or numbers.", + "why_it_matters": "It explains why resistance persists after the drug is stopped, which is a likely follow-up question.", + "suggested_addition": "Definition: a second mutation that restores growth rate without losing resistance. Example: cost goes from 5% to 1%, so after 100 generations the fraction factor is 0.99^100 ≈ 0.37 instead of 0.006." + }, + { + "rule": 2, + "severity": "edge", + "about": "'A cell has the resistance gene or not' conflicts with stepwise resistance, where MIC can rise in steps.", + "why_it_matters": "The binary claim breaks for low-level mutations that accumulate.", + "suggested_addition": "Add the case: a first mutation raises MIC from 0.5 to 2 mg/L (still killed at 8), and a second raises it to 16 mg/L (survives). Each step is selected by sub-MIC or low exposure." + }, + { + "rule": 5, + "severity": "edge", + "about": "In-patient regrowth time is described only as 'slower' with no value.", + "why_it_matters": "The reader cannot tell when the relapse occurs.", + "suggested_addition": "Illustrative: a 2–4 h doubling in tissue gives 27 doublings in about 2–4.5 days, versus 13 h in the lab." + }, + { + "rule": 5, + "severity": "edge", + "about": "The 0.01 expected count is not turned into a probability.", + "why_it_matters": "'Usually none' is vague.", + "suggested_addition": "P(at least one resistant cell) = 1 − e^-0.01 ≈ 1% for 10^6 cells. For 10^10 cells (expected 100), P ≈ 1." + } + ], + "excess": [ + "Representation decision section (Stage, Levels): this is production metadata, not content the reader needs", + "Epistemic 'Disputed: shortened course' line: it is out of scope and not needed for the central question" + ], + "next_questions": [ + { + "q": "Why doesn't my immune system kill the 100 resistant cells that survive?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "What physically happens inside the cell to block the drug, e.g. what does a beta-lactamase do?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Does stopping the antibiotic course early cause resistance?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "How does a plasmid get from one bacterium to another, and how fast?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If resistance has a fitness cost, why doesn't it disappear when we stop using the drug?", + "answered_by_model": true, + "declared_out_of_scope": false + } + ] + }, + "usage": { + "cost": 0.1673884, + "seconds": 62 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "'About 100 carry a resistance mutation' (N × rate) counts mutation events but calls them resistant cells. Descendants are left out. → Keep two separate names: 'mutation events ≈ N × μ = 10^10 × 10^-8 = 100' and 'resistant cells'. Then add: mutants that arose in earlier generations have divided since, so the expected number of resistant cells is about μN·ln(N) ≈ 10^-8 × 10^10 × 23 ≈ 2,300, and the median is about μN·ln(μN) ≈ 100 × 4.6 ≈ 460. State that 100 is a lower bound." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "The model never says why the roughly 100 surviving cells are not cleared by the host. It assumes nothing kills resistant cells after treatment. → State the assumption: host defences remove some cells but cannot keep up with a population that doubles. Give an example: if immune killing removes 50% of cells per doubling time, net growth is zero; if it removes 20%, 100 cells still reach 10^10 in about 27/log2(1.6) ≈ 40 doublings. Say that weakened patients (ICU, chemotherapy) clear fewer cells." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 4", + "text": "The four resistance mechanisms (destroy, change target, pump out, keep out) are only listed. None has a concrete example, and the learning order puts this as step 2. → Give one line each. Destroy: a beta-lactamase enzyme cuts penicillin's beta-lactam ring before it reaches the wall-building enzymes. Change target: one base change in the rpoB gene stops rifampicin binding. Pump out: an efflux pump exports tetracycline. Keep out: loss of a porin channel cuts drug entry. Link one example to MIC numbers, e.g. beta-lactamase raises MIC from 0.5 to 32 mg/L (64-fold)." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 2", + "text": "The 'breaks' column for combination therapy compares with one drug instead of removing the stated assumption, which is that the two resistances are independent. → Breaks case: if one plasmid or pump gives resistance to both drugs at about 10^-8, then expected double-resistant cells = 10^10 × 10^-8 = 100. Second breaks case: drug A alone leaves about 100 A-resistant cells. They regrow to 10^10 and then drug B is added: 10^10 × 10^-8 = 100 cells resistant to both." + }, + { + "n": 5, + "severity": "main", + "kind": "rule 3", + "text": "Several concepts have two names or two definitions. 'Resistance gene / resistance mutation' is one entity. 'Resistant' is defined both as 'survives the dose used' and as 'higher MIC'. Antibiotic 'kills or stops growth' (Entities) vs 'kills' (Terms). MIC 'stops growth' but the example says 8 mg/L 'kills'. Division / doubling / generation are used for the same thing. → Use one term, 'resistance determinant (mutation or acquired gene)'. Define resistant as 'MIC above the concentration reachable in the patient (e.g. MIC 32 > 8 mg/L)'. Say MIC measures growth inhibition, not killing. Pick one word, 'doubling', for one division of every cell." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 5", + "text": "'Low MIC ratio means the drug still works' uses an undefined ratio and gives no threshold value. → Define the ratio as reachable concentration ÷ MIC. 8/0.5 = 16 means the drug works; 8/32 = 0.25 means it fails. Give the cutoff (a clinical breakpoint), e.g. ratio ≥ about 1 to 4 counts as susceptible (illustrative)." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 1", + "text": "The fitness cost compounds multiplicatively (0.95^100). There is no justification against the linear alternative. → Each generation the resistant/susceptible ratio is multiplied by 0.95, because both lineages compound. So after 100 generations the ratio is 0.95^100 ≈ 0.006, not 1 − 5 = negative. After 10 generations it is ≈ 0.60." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 1", + "text": "The 'many 0, a few hundreds' result of Luria–Delbrück has no concrete numbers or comparison statistic. → Example: 20 parallel cultures, mean about 10 resistant colonies. Induction (Poisson) predicts most counts between about 4 and 16, with variance ≈ mean ≈ 10. Observed counts were 11 zeros, several 1–5, and one of 107, so variance ≫ mean (illustrative; recompute)." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 4", + "text": "Plasmid transfer and spread to other hosts and species have no mechanism (conjugation) and no numbers. → Name conjugation: cell-to-cell contact through a pilus copies the plasmid. Give an illustrative rate of 10^-4 to 10^-2 transfers per donor per hour in dense gut populations, versus 10^-8 per division for mutation." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 3", + "text": "'Compensatory mutations' appears in the confusion points without definition or numbers. → Definition: a second mutation that restores growth rate without losing resistance. Example: cost goes from 5% to 1%, so after 100 generations the fraction factor is 0.99^100 ≈ 0.37 instead of 0.006." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 2", + "text": "'A cell has the resistance gene or not' conflicts with stepwise resistance, where MIC can rise in steps. → Add the case: a first mutation raises MIC from 0.5 to 2 mg/L (still killed at 8), and a second raises it to 16 mg/L (survives). Each step is selected by sub-MIC or low exposure." + }, + { + "n": 12, + "severity": "edge", + "kind": "rule 5", + "text": "In-patient regrowth time is described only as 'slower' with no value. → Illustrative: a 2–4 h doubling in tissue gives 27 doublings in about 2–4.5 days, versus 13 h in the lab." + }, + { + "n": 13, + "severity": "edge", + "kind": "rule 5", + "text": "The 0.01 expected count is not turned into a probability. → P(at least one resistant cell) = 1 − e^-0.01 ≈ 1% for 10^6 cells. For 10^10 cells (expected 100), P ≈ 1." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Representation decision section (Stage, Levels): this is production metadata, not content the reader needs" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Epistemic 'Disputed: shortened course' line: it is out of scope and not needed for the central question" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151845-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151845-coldread-narrative.json new file mode 100644 index 0000000..a85dd69 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T151845-coldread-narrative.json @@ -0,0 +1,417 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:18:45", + "model": "default", + "prompt_sha": "3c4725ae6cae", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Said: 'Selection of variants that existed before the drug; not the drug changing them, not the patient.' Shown: bold answer.", + "unresolved": [ + "'not the patient': the reader hasn't been told the common belief this rejects (that the patient's body becomes resistant), so the denial means little" + ], + "unsaid": [], + "leap": [ + "The answer comes before any question is said. The reader can only guess that the topic is how resistance happens." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (mechanism)", + "text": "Said: 'binds a target the cell needs'. Shown: nothing.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Said: 'four ways; each raises MIC; resistant = MIC above concentration reached'. Shown: table of four ways, MIC example.", + "unresolved": [ + "'MIC': the abbreviation is never expanded or defined in the Said column", + "'concentration reached': it doesn't say where (in the blood? at the infection site?)" + ], + "unsaid": [ + "The names of the four ways appear only in the table, and none is explained in words", + "The numbers in the MIC example (two strains against the concentration reached) are shown but not walked through" + ], + "leap": [ + "It doesn't say how each of the four ways blocks the 'binds a target' step from beat 2, or why that 'raises MIC'" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (instance)", + "text": "Said: '10^10 × 10^-8 = 100; at least 100 resistant cells; 10^6 gives about 1%'. Shown: arithmetic.", + "unresolved": [ + "'10^10': not said to be the number of cells in an infection", + "'10^-8': not said to be a mutation rate per cell division" + ], + "unsaid": [], + "leap": [ + "'at least 100': 10^10 × 10^-8 is an expected count of new mutations. 'At least' needs a reason, such as earlier mutants having divided, and none is given", + "'10^6 gives about 1%': 10^6 × 10^-8 = 0.01 is an expected count. Calling it '1%' leaves unclear whether this is a 1% chance of one resistant cell or 1% of the cells" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Said: 'removes susceptible; 100 regrow in 27 doublings, about 13 h in the lab; immune system does not keep up'. Shown: steps.", + "unresolved": [], + "unsaid": [], + "leap": [ + "'immune system does not keep up' is asserted with no reason", + "'about 13 h' assumes a doubling time (about 29 min) that is never stated" + ], + "excess": [ + "'about 13 h in the lab': a lab figure that the argument doesn't need" + ], + "purpose_unclear": false + }, + { + "at": "Beat 6 (objection)", + "text": "Said: '20 cultures, mean 11, induction predicts 5–18 each; observed 11 zeros and a 107'. Shown: LD numbers.", + "unresolved": [ + "'induction': never defined as the drug causing the change", + "'LD': a label that is never expanded (Luria–Delbrück)", + "'cultures': it doesn't say what was grown, what was counted, or that it was counted after exposure to the drug" + ], + "unsaid": [ + "The 'LD numbers' display isn't tied to a named experiment in the Said column" + ], + "leap": [ + "It doesn't say why induction predicts 5–18 for each culture (similar counts everywhere)", + "It doesn't say why 11 zeros and a 107 point to mutations that existed before the drug. Mutants arising early and multiplying into jackpots is never given as the reasoning" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (generalize)", + "text": "Said: 'plasmids, conjugation, across species; whole enzymes'. Shown: nothing.", + "unresolved": [ + "'plasmids'", + "'conjugation'", + "'whole enzymes': unclear what this refers to" + ], + "unsaid": [], + "leap": [ + "It doesn't say how a gene moves from one bacterium to another, or why that removes the wait for each lineage's own mutation" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (test)", + "text": "Bridge: 'the guarantee needs the drug'. Said: 'cost 5%: 0.006; compensatory 1%: 0.37'. Shown: numbers.", + "unresolved": [ + "'cost': fitness cost is never defined", + "'compensatory': never defined", + "'the guarantee': no earlier beat named a guarantee" + ], + "unsaid": [ + "The exponent 100 (generations) that produces 0.006 and 0.37 is never said", + "It doesn't say what 0.006 and 0.37 are fractions of" + ], + "leap": [ + "It doesn't say why resistance costs growth when the drug is absent" + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 9 (payoff)", + "text": "Said: 'small infections rarely; two independent drugs 10^-16; breaks if one plasmid covers both'. Shown: two lines.", + "unresolved": [], + "unsaid": [], + "leap": [ + "10^-16 is presumably 10^-8 squared, but the multiplication isn't said" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 10 (close)", + "text": "Said: 'answer in the opening words; scope list'.", + "unresolved": [ + "'scope list': its contents aren't in the script" + ], + "unsaid": [], + "leap": [], + "excess": [ + "Restating the opening answer word for word adds nothing new unless the general argument is added" + ], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 4: '10^10 × 10^-8 = 100'. The two numbers aren't identified as cells and mutation rate, so the count that the whole argument rests on can't be followed", + "Beat 6: 'induction predicts 5–18 each; observed 11 zeros and a 107'. 'Induction' is undefined and the reasoning from the spread of counts to 'existed before the drug' is missing, so the key evidence doesn't land", + "Beat 7: 'plasmids, conjugation'. Both are undefined for a reader who knows nothing of gene exchange", + "Beat 8: 'cost 5%: 0.006; compensatory 1%: 0.37'. Cost, compensatory, the 100 generations and 'the guarantee' are all undefined" + ], + "cuts": [ + "Beat 5: 'about 13 h in the lab'. Cutting it removes an unexplained conversion", + "Beat 3: name and explain one mechanism (for example, an enzyme that destroys the drug) instead of a table of four unexplained names", + "Beat 10: replace the scope list with the one-sentence general argument" + ], + "question": { + "text": "How does an infection that a drug once cleared become one it can't clear? The question is never said in the Said column; the reader infers it from the hook's answer in Beat 1.", + "known_at": "Beat 1, only by inference", + "result_stated": "Yes, Beat 1: 'Selection of variants that existed before the drug; not the drug changing them, not the patient.'" + }, + "motive": "Missing from the script. The plan's reasons to care (the 'bacteria get used to the drug' picture predicts the wrong things) and the reason for the approach (start from what the drug does, then count) never appear in the Said column.", + "close": "Partly. Beat 10 repeats 'the answer in the opening words', but the general argument ('any heritable variant that grows under the drug while others are stopped takes over') is never said in the beats.", + "recap": "Resistant bacteria are already there by chance, or they get the gene from other bacteria; the drug just kills their competitors so they take over.", + "strongest": "Beat 4: one multiplication turns 'resistance must be caused by the drug' into 'about 100 resistant cells are already present', which is the surprise the explanation is built around.", + "weakest": "Beat 6: the Luria–Delbrück evidence is given as bare numbers, with 'induction' undefined and no reasoning from the jackpot pattern to the conclusion.", + "suggestions": [ + "Say in Beat 4: 'an infection holds about 10^10 bacteria; each division makes this mutation about once in 10^8 times'", + "Define induction in Beat 6, then say: if the drug caused it, every culture would have similar counts; if mutations came first, a mutant that arose early multiplies into a jackpot, so most cultures have none and a few have many", + "Define a plasmid as a small, separate ring of DNA that bacteria copy and pass to each other through contact (conjugation)", + "In Beat 8, say that the resistant cells grow 5% slower per generation, so after 100 generations they are 0.95^100 ≈ 0.6% of what they would otherwise be", + "State the motive in Beat 1: the 'bacteria get used to it' picture predicts the wrong things" + ] + }, + "usage": { + "cost": 0.12325699999999998, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: '10^10 × 10^-8 = 100'. The two numbers aren't identified as cells and mutation rate, so the count that the whole argument rests on can't be followed" + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 6: 'induction predicts 5–18 each; observed 11 zeros and a 107'. 'Induction' is undefined and the reasoning from the spread of counts to 'existed before the drug' is missing, so the key evidence doesn't land" + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 7: 'plasmids, conjugation'. Both are undefined for a reader who knows nothing of gene exchange" + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 8: 'cost 5%: 0.006; compensatory 1%: 0.37'. Cost, compensatory, the 100 generations and 'the guarantee' are all undefined" + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 5: 'about 13 h in the lab'. Cutting it removes an unexplained conversion" + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 3: name and explain one mechanism (for example, an enzyme that destroys the drug) instead of a table of four unexplained names" + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 10: replace the scope list with the one-sentence general argument" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): 'not the patient': the reader hasn't been told the common belief this rejects (that the patient's body becomes resistant), so the denial means little" + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): The answer comes before any question is said. The reader can only guess that the topic is how resistance happens." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): 'MIC': the abbreviation is never expanded or defined in the Said column" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): 'concentration reached': it doesn't say where (in the blood? at the infection site?)" + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): The names of the four ways appear only in the table, and none is explained in words" + }, + { + "n": 13, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): The numbers in the MIC example (two strains against the concentration reached) are shown but not walked through" + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): It doesn't say how each of the four ways blocks the 'binds a target' step from beat 2, or why that 'raises MIC'" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (instance): '10^10': not said to be the number of cells in an infection" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (instance): '10^-8': not said to be a mutation rate per cell division" + }, + { + "n": 17, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (instance): 'at least 100': 10^10 × 10^-8 is an expected count of new mutations. 'At least' needs a reason, such as earlier mutants having divided, and none is given" + }, + { + "n": 18, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (instance): '10^6 gives about 1%': 10^6 × 10^-8 = 0.01 is an expected count. Calling it '1%' leaves unclear whether this is a 1% chance of one resistant cell or 1% of the cells" + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 'immune system does not keep up' is asserted with no reason" + }, + { + "n": 20, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 'about 13 h' assumes a doubling time (about 29 min) that is never stated" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (objection): 'induction': never defined as the drug causing the change" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (objection): 'LD': a label that is never expanded (Luria–Delbrück)" + }, + { + "n": 23, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (objection): 'cultures': it doesn't say what was grown, what was counted, or that it was counted after exposure to the drug" + }, + { + "n": 24, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (objection): The 'LD numbers' display isn't tied to a named experiment in the Said column" + }, + { + "n": 25, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (objection): It doesn't say why induction predicts 5–18 for each culture (similar counts everywhere)" + }, + { + "n": 26, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (objection): It doesn't say why 11 zeros and a 107 point to mutations that existed before the drug. Mutants arising early and multiplying into jackpots is never given as the reasoning" + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (generalize): 'plasmids'" + }, + { + "n": 28, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (generalize): 'conjugation'" + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (generalize): 'whole enzymes': unclear what this refers to" + }, + { + "n": 30, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (generalize): It doesn't say how a gene moves from one bacterium to another, or why that removes the wait for each lineage's own mutation" + }, + { + "n": 31, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (test): 'cost': fitness cost is never defined" + }, + { + "n": 32, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (test): 'compensatory': never defined" + }, + { + "n": 33, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (test): 'the guarantee': no earlier beat named a guarantee" + }, + { + "n": 34, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (test): The exponent 100 (generations) that produces 0.006 and 0.37 is never said" + }, + { + "n": 35, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (test): It doesn't say what 0.006 and 0.37 are fractions of" + }, + { + "n": 36, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (test): It doesn't say why resistance costs growth when the drug is absent" + }, + { + "n": 37, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (payoff): 10^-16 is presumably 10^-8 squared, but the multiplication isn't said" + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 10 (close): 'scope list': its contents aren't in the script" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152038-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152038-coldread-prose.json new file mode 100644 index 0000000..e3a83bf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152038-coldread-prose.json @@ -0,0 +1,328 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:20:38", + "model": "default", + "prompt_sha": "fbc694b7c598", + "reply": { + "paragraphs": [ + { + "at": "Lead (bold paragraph under the title)", + "text": "The drug does not change the bacteria. In a large infection, a few bacteria already carry a resistance gene... The bacteria become resistant, not the patient's body.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What a cell must block, para 1", + "text": "An antibiotic is a molecule that kills bacteria or stops their growth... penicillin binds the enzymes that build the cell wall.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What a cell must block, para 2", + "text": "A resistance gene is a gene, changed by mutation or received from another bacterium, that blocks this. It works in one of four ways:", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Table of four ways", + "text": "Destroy the drug / Change the target / Pump the drug out / Keep the drug out", + "unresolved": [ + "rpoB gene", + "outer membrane" + ], + "leap": [], + "excess": [ + "One DNA letter change in the rpoB gene" + ], + "purpose_unclear": false + }, + { + "at": "What a cell must block, MIC paragraph", + "text": "Each way means the cell needs more drug to stop it. The lowest concentration ... is its MIC ... (Illustrative values.)", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Resistant cells are there before the drug, para 1", + "text": "A given resistance mutation happens about once per 10⁸ cell divisions ... Growing to that size took about 10¹⁰ divisions, so expect", + "unresolved": [], + "leap": [ + "Growing to that size took about 10¹⁰ divisions: the reason (going from 1 cell to N cells takes about N−1 divisions) is not given" + ], + "excess": [ + "(order of magnitude; real values range from about 10⁻⁶ to 10⁻¹⁰)" + ], + "purpose_unclear": false + }, + { + "at": "Resistant cells..., equation line", + "text": "10¹⁰ × 10⁻⁸ = 100 resistance mutations, before anyone gives a drug.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Resistant cells..., para 3", + "text": "Early mutants have divided since, so there are at least 100 resistant cells. Compare a small infection of 10⁶ cells...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The drug selects them, item 1", + "text": "The drug stops the susceptible cells, and the immune system clears them.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The drug selects them, item 2", + "text": "The resistant cells keep dividing... (A weakened patient clears fewer.)", + "unresolved": [], + "leap": [], + "excess": [ + "(A weakened patient clears fewer.)" + ], + "purpose_unclear": false + }, + { + "at": "The drug selects them, item 3", + "text": "A doubling is one round in which every cell divides once. 100 cells need 27 doublings to reach 10¹⁰ again (2²⁷ ≈ 1.3 × 10⁸)...", + "unresolved": [], + "leap": [ + "(2²⁷ ≈ 1.3 × 10⁸): the reader has to multiply by the starting 100 cells to get 10¹⁰, and the text never says so" + ], + "excess": [ + "A doubling is one round in which every cell divides once." + ], + "purpose_unclear": false + }, + { + "at": "The drug selects them, item 4", + "text": "The infection returns, and now almost every cell is resistant.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The drug selects them, para after list", + "text": "This is selection: the drug did not make resistant cells, it removed everything else.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The drug selects them, Luria–Delbrück paragraph", + "text": "How do we know the drug did not cause the change? ... They found 11 cultures with 0 and one with 107...", + "unresolved": [ + "with a virus as the killing agent" + ], + "leap": [ + "so the \"drug causes it\" idea predicts most cultures between 5 and 18: the text never says where the 5–18 range comes from" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Genes from other bacteria, para 1", + "text": "A cell does not have to wait for its own rare mutation. A plasmid is a small circle of DNA... horizontal gene transfer...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Genes from other bacteria, para 2", + "text": "Plasmids carry complete working genes... The same plasmid gene turns up in different species, and one plasmid often carries several resistance genes at once.", + "unresolved": [], + "leap": [ + "The same plasmid gene turns up in different species: the text doesn't say this is evidence of transfer, or what it is meant to show" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Without the drug, para 1", + "text": "Resistant cells take over only while the drug is present. Without the drug, resistance often has a fitness cost...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Without the drug, para 2", + "text": "Suppose resistant cells grow 5% slower... Plasmids also persist.", + "unresolved": [], + "leap": [ + "Plasmids also persist.: no reason is given" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What this predicts, bullet 1", + "text": "Large infections are the risk, because they hold resistant cells before treatment.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What this predicts, bullet 2", + "text": "Two drugs with independent targets need two mutations in one cell... This fails if one plasmid or one pump resists both drugs: then about 100 cells survive again.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "In short", + "text": "resistance genes come first, by chance or from other bacteria. The drug removes the competitors, and the resistant cells multiply.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered", + "text": "Persister cells... transformation and transduction... SOS response... biofilms.", + "unresolved": [ + "transformation and transduction", + "biofilms" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Resistant cells..., para 1: drop the parenthetical \"(order of magnitude; real values range from about 10⁻⁶ to 10⁻¹⁰)\" or shorten it", + "Selects them, item 3: drop the definition \"A doubling is one round in which every cell divides once.\" because this reader knows exponential growth", + "Selects them, item 2: drop \"(A weakened patient clears fewer.)\"", + "Table: shorten \"One DNA letter change in the rpoB gene\" to \"One DNA letter change\"" + ], + "question": { + "text": "How do bacteria come to resist an antibiotic: does the drug cause the change, or does it select bacteria that were already resistant?", + "known_at": "Title and the bold lead paragraph", + "result_stated": "Yes, in the lead: \"The drug does not change the bacteria... The drug removes their competitors, and the resistant bacteria multiply.\"" + }, + "motive": "Neither motive is stated outright. A reason to care is only implied, by \"The infection returns, and now almost every cell is resistant.\" There is no stated reason for the counting approach, though the arithmetic in \"10¹⁰ × 10⁻⁸ = 100 resistance mutations, before anyone gives a drug\" makes the case by example.", + "close": "Yes: \"In short: resistance genes come first, by chance or from other bacteria. The drug removes the competitors, and the resistant cells multiply.\"", + "recap": "A big infection already holds about a hundred resistant bacteria before treatment, from random mutation or genes passed between bacteria, and the drug just clears away everything else so they take over.", + "strongest": "'Resistant cells are there before the drug': the 10¹⁰ × 10⁻⁸ = 100 calculation, together with the Luria–Delbrück paragraph that tests the idea.", + "weakest": "'Genes from other bacteria', para 2: the facts in it (the same gene in different species, several genes on one plasmid) are stated without saying what they show.", + "suggestions": [ + "Say where 5–18 comes from (Poisson spread: about mean ± 2√mean).", + "Explain that a virus can infect and kill bacteria (a bacteriophage).", + "Add one sentence on why resistance matters: treatment fails and the patient relapses.", + "Say why plasmids persist without the drug, or cut that sentence.", + "Write the doubling arithmetic as 100 × 2²⁷ ≈ 1.3 × 10¹⁰." + ] + }, + "usage": { + "cost": 0.13425979999999998, + "seconds": 44 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Resistant cells..., para 1: drop the parenthetical \"(order of magnitude; real values range from about 10⁻⁶ to 10⁻¹⁰)\" or shorten it" + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Selects them, item 3: drop the definition \"A doubling is one round in which every cell divides once.\" because this reader knows exponential growth" + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Selects them, item 2: drop \"(A weakened patient clears fewer.)\"" + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "Table: shorten \"One DNA letter change in the rpoB gene\" to \"One DNA letter change\"" + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Table of four ways: rpoB gene" + }, + { + "n": 6, + "severity": "edge", + "kind": "unresolved", + "text": "Table of four ways: outer membrane" + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Resistant cells are there before the drug, para 1: Growing to that size took about 10¹⁰ divisions: the reason (going from 1 cell to N cells takes about N−1 divisions) is not given" + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "The drug selects them, item 3: (2²⁷ ≈ 1.3 × 10⁸): the reader has to multiply by the starting 100 cells to get 10¹⁰, and the text never says so" + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "The drug selects them, Luria–Delbrück paragraph: with a virus as the killing agent" + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "The drug selects them, Luria–Delbrück paragraph: so the \"drug causes it\" idea predicts most cultures between 5 and 18: the text never says where the 5–18 range comes from" + }, + { + "n": 11, + "severity": "edge", + "kind": "leap", + "text": "Genes from other bacteria, para 2: The same plasmid gene turns up in different species: the text doesn't say this is evidence of transfer, or what it is meant to show" + }, + { + "n": 12, + "severity": "edge", + "kind": "leap", + "text": "Without the drug, para 2: Plasmids also persist.: no reason is given" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: transformation and transduction" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: biofilms" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152206-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152206-quiz-prose.json new file mode 100644 index 0000000..1fab995 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152206-quiz-prose.json @@ -0,0 +1,220 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:21:51", + "model": "default", + "prompt_sha": "57d2fb9e88c1", + "reply": { + "answers": { + "1": "The main objects are: antibiotics (molecules that kill bacteria or stop their growth); targets (molecules the cell needs, such as the enzymes that build the cell wall); resistance genes and resistance mutations; and bacterial cells, both susceptible and resistant. Also involved are the MIC (minimum inhibitory concentration) and the drug concentration in blood, plasmids, conjugation and horizontal gene transfer, the immune system, fitness cost, compensatory mutations, and combination therapy.", + "2": "An antibiotic works by binding a target. A resistance gene blocks this in one of four ways: it destroys the drug (beta-lactamase), changes the target (rifampicin), pumps the drug out (efflux pump), or keeps the drug out (loss of a porin). Each of these raises the strain's MIC. If the MIC is above the drug level reached in blood, the strain is resistant; if below, it is susceptible. Resistance genes come from random copying errors during division or from other bacteria via plasmids passed by conjugation. The drug, with help from the immune system, removes susceptible cells, and the resistant ones multiply. When the drug is absent, a fitness cost works against resistant cells, but compensatory mutations can reduce that cost.", + "3": "In a large population, random mutation or gene transfer has already produced a few resistant cells before treatment. The drug removes their susceptible competitors, so the resistant cells grow back exponentially and the infection returns, now resistant.", + "4": "The key variable is population size. Expected resistant mutants are roughly N × 10⁻⁸. At 10¹⁰ cells about 100 mutants already exist, so treatment fails. At 10⁶ cells there is only about a 1% chance that even one exists, so treatment usually works. Using two drugs with independent targets multiplies the rates to 10⁻¹⁶, giving an expected 10⁻⁶ double mutants in 10¹⁰ cells, so the infection clears. That protection is lost if one plasmid or pump resists both drugs.", + "5": "Assumption or simplification: each division adds one cell, so reaching 10¹⁰ cells took about 10¹⁰ divisions. The mutation rate of 10⁻⁸ is an order of magnitude, and the MIC values are labelled illustrative. Observation: in Luria and Delbrück's 1943 experiment, 11 of 20 cultures had 0 resistant colonies and one had 107, far more spread than random scatter around a mean of 11 would give.", + "6": "Antibiotics kill bacteria by binding a target the cell needs. Resistance genes block this by destroying the drug, changing the target, pumping the drug out, or keeping it out, which raises the MIC above the drug level the body reaches. In a large infection, random mutation (about 1 per 10⁸ divisions) or plasmids received from other bacteria have already produced some resistant cells before any drug is given. The drug removes the susceptible cells, and the resistant ones regrow within hours to days. Without the drug, resistance often slows growth, but compensatory mutations let it persist.", + "7": "The Luria–Delbrück paragraph was hardest. The text does not show where the range '5 to 18' comes from, and it does not explain why a virus counts as a 'killing agent' or how it relates to a drug. A short note that ordinary random scatter is about ±2×√11, plus a picture of early versus late mutant lineages, would make it easier to follow.", + "8": "Growing to 10¹⁰ cells took about 10¹⁰ divisions, so about 100 resistant mutants were already present before treatment. Drug X and the immune system cleared the susceptible majority, so symptoms improved. The roughly 100 resistant cells kept dividing; about 27 doublings bring them back to 10¹⁰, which takes about 13 hours at a lab doubling time and longer in a patient. The returning infection is almost entirely resistant. The drug selected these cells; it did not create them.", + "9": "No, not usually. 10⁶ × 10⁻⁸ = 0.01 expected mutants, so there is only about a 1% chance that a resistant cell exists, and the drug would usually clear the infection. The exception is if a resistance gene arrives on a plasmid, which does not depend on the population's own mutations.", + "10": "Not necessarily. If resistance costs 5% growth, the resistant-to-susceptible ratio falls to about 0.006 of its starting value after 100 generations, so it shrinks a lot but does not reach zero. If a compensatory mutation cuts the cost to 1%, the ratio is still about 0.37 after 100 generations. The explanation gives no timescale linking 'years' to generations, and it does not say whether plasmid-borne resistance carries a cost.", + "11": "Yes. About 100 resistance mutations are expected (10¹⁰ × 10⁻⁸), and because early mutants have divided since, there are at least 100 resistant cells, possibly more.", + "12": "If the drug caused resistance, every cell would have the same small chance when it met the drug, so equal cultures would give similar counts with only ordinary scatter (here, mostly 5 to 18 around a mean of 11). Luria and Delbrück instead found 11 cultures with 0 and one with 107. This fits random mutation before exposure: a mutant that arose early leaves many resistant descendants, and a late one leaves few. Separately, the regrowth story shows the drug only removes the competitors.", + "13": "Through horizontal gene transfer. In conjugation, one bacterium copies a plasmid carrying a complete working resistance gene (such as beta-lactamase) into another bacterium, which can be of a different species. One plasmid often carries several resistance genes at once.", + "14": "Without the drug, the fitness cost makes the resistant fraction shrink: with a 5% cost the ratio is multiplied by 0.95 each generation, giving about 0.006 after 100 generations. It may not disappear because a compensatory mutation can cut the cost to about 1% (0.37 remains after 100 generations). Also, the decline is gradual rather than to zero." + }, + "audit": { + "terms": [ + "The same thing is counted in three units: 'divisions' (lines 24, 26), 'doublings' (line 35) and 'generations' (line 54). The reader must work out that they relate.", + "'resistance gene' (line 11), 'resistance mutation' (lines 24, 26) and 'One DNA letter change' / 'One letter change' (lines 15, 47) are used for overlapping ideas. A mutation counts as a gene in one place and is contrasted with whole plasmid genes in another.", + "'kills bacteria or stops their growth' (line 9) vs 'stops' (lines 20, 33) vs 'removes' (line 38) vs 'killing agent' (line 40): it is unclear whether stopping growth and killing are treated as the same thing.", + "'Genes from other bacteria' (heading, line 42), 'received from another bacterium' (lines 5, 11), 'conjugation' and 'horizontal gene transfer' (line 45): the relationship between conjugation and HGT is given only once.", + "'competitors' (lines 5, 61) vs 'susceptible cells' (line 33) vs 'everything else' (line 38) all name the same group." + ], + "unexplained": [ + "'most cultures between 5 and 18' (line 40): the text does not say how this range follows from a mean of 11.", + "'5% slower' becomes 'multiplies the ratio … by 0.95' each generation (line 54): it does not say whose generation, or what a 5% growth difference means per unit of time.", + "'in a patient it takes longer' (line 35): no figure is given, so the reader cannot judge the timescale of relapse.", + "'at least 100 resistant cells' (line 28): it does not say how many more could exist when early mutants have divided.", + "'cut the cost to 1%' (line 54): no indication of how typical this is or how fast a compensatory mutation appears." + ], + "missing_why": [ + "Line 54: why a growth-rate penalty multiplies the ratio by (1 − cost) each generation.", + "Line 59: why independent targets let the probabilities multiply, and why a single pump can defeat two different drugs.", + "Line 34: why the immune system 'does not keep up' with the resistant cells when it did clear the susceptible ones.", + "Line 40: why a virus is equivalent to a drug for the logic of the experiment.", + "Line 54: why a compensatory mutation reduces the cost without losing the resistance." + ], + "excess": [ + "Line 3: an HTML comment with process metadata ('Stage 1, rendered from model.md in STE-80').", + "Lines 63–68: the 'Not covered' list names topics the reader cannot use." + ] + }, + "gaps": [ + "It does not say how fast a plasmid spreads through a population once it arrives, or whether plasmid-borne resistance also carries a fitness cost.", + "There is no timescale linking 'generations' to real time, so the reader cannot judge what 'years without the drug' means (Q10).", + "It does not say what keeps the immune system from clearing about 100 resistant cells.", + "The source of the Luria–Delbrück 'expected scatter' range is not given.", + "It does not explain how resistance spreads between patients or within a hospital, though Q10 implies it." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 1, + "11": 2, + "12": 2, + "13": 2, + "14": 2 + }, + "reasons": { + "10": "Correctly says resistance falls through fitness cost (0.95^100 ≈ 0.006), slowly and not to zero, and that compensatory mutations reduce the cost (0.99^100 ≈ 0.37). It omits the expected point that plasmids persist, and instead treats plasmid behaviour as unexplained. That is one of the two listed reasons resistance need not vanish, so the answer is partial." + }, + "pass": false + }, + "usage": { + "cost": 0.20742880000000002, + "seconds": 53 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same thing is counted in three units: 'divisions' (lines 24, 26), 'doublings' (line 35) and 'generations' (line 54). The reader must work out that they relate." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'resistance gene' (line 11), 'resistance mutation' (lines 24, 26) and 'One DNA letter change' / 'One letter change' (lines 15, 47) are used for overlapping ideas. A mutation counts as a gene in one place and is contrasted with whole plasmid genes in another." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'kills bacteria or stops their growth' (line 9) vs 'stops' (lines 20, 33) vs 'removes' (line 38) vs 'killing agent' (line 40): it is unclear whether stopping growth and killing are treated as the same thing." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'Genes from other bacteria' (heading, line 42), 'received from another bacterium' (lines 5, 11), 'conjugation' and 'horizontal gene transfer' (line 45): the relationship between conjugation and HGT is given only once." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "'competitors' (lines 5, 61) vs 'susceptible cells' (line 33) vs 'everything else' (line 38) all name the same group." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'most cultures between 5 and 18' (line 40): the text does not say how this range follows from a mean of 11." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'5% slower' becomes 'multiplies the ratio … by 0.95' each generation (line 54): it does not say whose generation, or what a 5% growth difference means per unit of time." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'in a patient it takes longer' (line 35): no figure is given, so the reader cannot judge the timescale of relapse." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'at least 100 resistant cells' (line 28): it does not say how many more could exist when early mutants have divided." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "'cut the cost to 1%' (line 54): no indication of how typical this is or how fast a compensatory mutation appears." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Line 54: why a growth-rate penalty multiplies the ratio by (1 − cost) each generation." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "Line 59: why independent targets let the probabilities multiply, and why a single pump can defeat two different drugs." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "Line 34: why the immune system 'does not keep up' with the resistant cells when it did clear the susceptible ones." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "Line 40: why a virus is equivalent to a drug for the logic of the experiment." + }, + { + "n": 15, + "severity": "gap", + "kind": "missing_why", + "text": "Line 54: why a compensatory mutation reduces the cost without losing the resistance." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "Line 3: an HTML comment with process metadata ('Stage 1, rendered from model.md in STE-80')." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "Lines 63–68: the 'Not covered' list names topics the reader cannot use." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "It does not say how fast a plasmid spreads through a population once it arrives, or whether plasmid-borne resistance also carries a fitness cost." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "There is no timescale linking 'generations' to real time, so the reader cannot judge what 'years without the drug' means (Q10)." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "It does not say what keeps the immune system from clearing about 100 resistant cells." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "The source of the Luria–Delbrück 'expected scatter' range is not given." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "It does not explain how resistance spreads between patients or within a hospital, though Q10 implies it." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152433-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152433-quiz-prose.json new file mode 100644 index 0000000..bca5bce --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/runs/2026-10-04T152433-quiz-prose.json @@ -0,0 +1,226 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "antibiotic-resistance", + "at": "2026-10-04T15:24:20", + "model": "default", + "prompt_sha": "57d2fb9e88c1", + "reply": { + "answers": { + "1": "The main objects are: bacteria (susceptible and resistant cells and strains); the antibiotic, a molecule that kills bacteria or stops their growth; its target, a molecule the cell needs, such as the cell-wall enzymes penicillin binds; resistance genes; mutations (random copying errors); plasmids, which are small circles of DNA separate from the chromosome; conjugation and horizontal gene transfer; MIC (minimum inhibitory concentration); fitness cost; compensatory mutations; the immune system; and drug concentration in the blood.", + "2": "An antibiotic binds a target the cell needs. A resistance gene blocks this in one of four ways: it destroys the drug, changes the target, pumps the drug out, or keeps it out. Any of these raises the strain's MIC. If the MIC is above the drug level reached in the blood, the strain is resistant; if below, it is susceptible. A cell gets a resistance gene by mutation, which is rare (about 1 per 10^8 divisions), or by receiving a plasmid from another bacterium through conjugation. The drug removes susceptible cells, so resistant cells multiply. Without the drug, resistance often carries a fitness cost, so resistant cells lose ground. A compensatory mutation can cancel that cost, and a plasmid carrying genes for several drugs can be kept by any one of those drugs.", + "3": "Resistant cells already exist in a large population before treatment, from random mutation or gene transfer. The drug removes the susceptible cells, and the surviving resistant cells regrow until they make up nearly the whole infection.", + "4": "Population size is the key variable. Expected resistant mutants = number of divisions × mutation rate. At 10^10 cells about 100 mutants are expected, so treatment fails. At 10^6 cells only 0.01 are expected, a 1% chance that even one exists, so treatment usually succeeds. Combining two independent drugs multiplies the two chances (10^-16 per cell), so even 10^10 cells expect about 10^-6 doubly resistant cells. That fails if a single event, such as one plasmid or one pump, gives resistance to both drugs.", + "5": "Assumptions and simplifications: the MIC values are marked illustrative; the 30-minute lab doubling time is applied to an infection; the 5% fitness cost is a supposed value; and 10^10 cells is taken to mean about 10^10 divisions. Established observation: Luria and Delbrück (1943) grew 20 equal cultures. Eleven had 0 resistant colonies and one had 107, against a mean of about 11. That spread fits mutation before exposure, not resistance caused by the drug.", + "6": "Antibiotics act on a target molecule, and bacteria resist by destroying the drug, changing the target, pumping the drug out, or keeping it out, which raises the MIC above the drug level in the blood. In a large infection, rare random mutations (about 1 per 10^8 divisions) mean about 100 resistant cells exist before any drug is given, and plasmids passed between bacteria can deliver whole resistance genes in one step. The drug kills or halts the susceptible cells, and the resistant ones regrow to the full infection size in about 27 doublings. This is selection, not induction, as Luria and Delbrück showed: resistant counts across equal cultures varied far too much for resistance to be caused by exposure. Without the drug, resistance usually costs growth and fades, but compensatory mutations and multi-drug plasmids keep it around; combining two independent drugs makes double resistance extremely unlikely.", + "7": "The Luria–Delbrück paragraph was hardest. It uses √11 as 'random scatter' and a 5–18 range without saying why scatter equals the square root of the mean, or why the range is about two scatters wide. It also does not show the expected distribution under the mutation-first idea. A short note on the Poisson rule, or a small simulated comparison of the two ideas, would make it easier. The 0.95-per-doubling ratio step is also a little loose.", + "8": "About 10^10 divisions × 10^-8 means roughly 100 or more cells were already resistant to X before treatment. Drug X killed or halted the susceptible majority, so symptoms cleared. The resistant cells kept dividing, and 100 cells reach 10^10 in about 27 doublings, about 13 hours at a 30-minute doubling time. The returning infection is almost entirely resistant, so X no longer works. The drug selected these cells; it did not create them.", + "9": "Probably not. With 10^6 cells the expected number of resistance mutations is 0.01, about a 1% chance that even one resistant cell exists, so drug X would usually clear the infection. The exception is if a resistance plasmid arrives from another bacterium, which the size argument does not cover.", + "10": "Not necessarily. If resistance has a fitness cost (for example 5%), the ratio of resistant to susceptible cells shrinks by 0.95 each doubling, to about 0.006 of its starting value after 100 doublings. But a compensatory mutation can cut the cost (at 1%, about 0.37 remains after 100 doublings). Also, if the gene sits on a plasmid with resistance genes for other drugs, using any of those drugs keeps selecting it. So resistance usually declines but rarely vanishes. The explanation does not give real time scales in years.", + "11": "Yes. About 10^10 × 10^-8 = 100 resistance mutations are expected, and because early mutants have divided since, there are at least about 100 resistant cells, possibly many more.", + "12": "Because the drug removes susceptible cells; it does not create resistant ones. If exposure caused resistance, each cell would have the same small chance on contact, so equal cultures would give similar counts scattered by about √mean (about 5–18 around a mean of 11). Luria and Delbrück found 11 cultures with 0 resistant colonies and one with 107. That huge spread fits random mutations before exposure: an early mutant leaves many descendants, a late one few.", + "13": "By horizontal gene transfer. Through conjugation, another bacterium, even one of a different species, copies a plasmid into the cell. The plasmid carries a complete working resistance gene, such as a beta-lactamase, often along with several other resistance genes.", + "14": "The resistant fraction usually declines because of the fitness cost, compounding each doubling (0.95^100 ≈ 0.006 at a 5% cost). It may not disappear: compensatory mutations restore growth while keeping the resistance (0.99^100 ≈ 0.37 at a 1% cost), and multi-drug plasmids are still selected whenever any one of their drugs is used." + }, + "audit": { + "terms": [ + "Mutation has several names: 'random copying error' (line 5), 'mutation' (lines 11, 24), 'One DNA letter change' (table, line 16), 'one letter change' (line 47). The reader can probably map these, but the link is never stated.", + "Gene transfer has several names: 'received from another bacterium' (lines 5, 11), 'Genes from other bacteria' (heading, line 42), 'conjugation' and 'horizontal gene transfer' (line 45). Conjugation is one route of horizontal transfer, and that is only hinted at in 'Not covered' (line 71).", + "'resistance gene' (line 11) and 'resistance mutation' (line 24) are used for close but different things: a gene versus the event that makes it.", + "Growth being stopped has three names: 'stops their growth' (line 9), 'halts their growth' (line 33), 'inhibitory' in MIC (line 20)." + ], + "unexplained": [ + "'At a lab doubling time of 30 minutes, that is 13 hours' (line 35): it does not say whether bacteria in a patient double this fast, or how a slower rate would change the regrowth time.", + "'after 100 doublings' (lines 54, 58): it does not say how long 100 doublings takes in real time, which is needed for the 'hospital stops using drug X for years' case.", + "'In 10¹⁰ cells, expect 10⁻⁶ such cells' (line 64): it is not translated into about a one-in-a-million chance that even one doubly resistant cell exists.", + "'about once per 10⁸ cell divisions' (line 24): it does not say whether this rate is typical or varies across genes and drugs, or how a rate of 10⁻⁶ or 10⁻¹⁰ would change the conclusion.", + "'0.95¹⁰⁰ ≈ 0.006' (line 54): this is a ratio relative to the start value, but it does not say what resistant fraction that leaves if resistance started common or rare." + ], + "missing_why": [ + "'the random scatter is about √11 ≈ 3.3, so ... predicts most cultures between 5 and 18' (line 40): no reason is given for why scatter is the square root of the mean, or why the range is about ±2 scatters.", + "'each time the susceptible cells double, the resistant cells grow only 0.95 times as much' (line 54): it does not say why a 5% slower growth translates to a 0.95 factor per doubling.", + "'A compensatory mutation is a second change that restores growth ... so the first change, and the resistance, stay' (line 58): it does not say why the cell gains a second mutation rather than losing the first one.", + "'Early mutants have divided since, so there are at least 100 resistant cells' (line 28): this mixes an expected count of mutation events with a minimum number of cells, and does not say how many more there may be.", + "'the test asks only whether resistance arose before contact, so it applies to drugs too' (line 40): the transfer from virus to antibiotic is asserted rather than argued." + ], + "excess": [ + "'The bacteria become resistant, not the patient's body.' (line 5): this is a common misconception, but the questions and the later argument never use it.", + "The 'Not covered' list (lines 68–73) names topics (persisters, SOS response, biofilms) the reader cannot use. It is harmless but adds no understanding." + ] + }, + "gaps": [ + "The explanation does not give real time scales for decline in a hospital after a drug is withdrawn, so 'for years' cannot be answered quantitatively.", + "It does not say whether plasmids themselves carry a fitness cost, or how fast they spread between cells or species.", + "It does not say whether bacteria divide at the lab rate inside a patient; the 13-hour regrowth depends on this.", + "It does not describe the expected distribution under the mutation-before-exposure idea, only that it 'explains this spread'.", + "It does not explain how mutation and transfer compare in importance in practice." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2, + "14": 2 + }, + "reasons": { + "1": "No points lost: lists the model's entities with correct definitions (antibiotic, target, plasmid, MIC, fitness cost, and others).", + "2": "No points lost: covers target binding, the four mechanisms, MIC versus blood level, both sources, selection, fitness cost, compensatory mutations and multi-drug plasmids.", + "3": "No points lost: states that resistant variants exist before treatment and the drug selects them.", + "4": "No points lost: gives N × rate with 10^10 versus 10^6, and the two-drug 10^-16 case with its plasmid/pump exception.", + "5": "No points lost: separates the illustrative assumptions from the established Luria–Delbrück observation with correct numbers.", + "6": "No points lost: a correct, specific summary of the full causal chain, selection versus induction, cost, compensation and combination therapy.", + "7": "No points lost: names a specific difficulty (the unexplained √mean scatter and the 5–18 range) and proposes a concrete fix.", + "8": "No points lost: about 100 pre-existing resistant cells, susceptible cells removed, regrowth in about 27 doublings, and explicitly selected, not created.", + "9": "No points lost: 0.01 expected mutations, about a 1% chance, drug likely clears it; the plasmid caveat is consistent with the model.", + "10": "No points lost: declines through fitness cost (0.95^100 ≈ 0.006) but persists through compensatory mutation (0.37) and multi-drug plasmids.", + "11": "No points lost: 10^10 × 10^-8 = 100, with at least about 100 resistant cells because early mutants divide.", + "12": "No points lost: the drug removes susceptible cells, and the Luria–Delbrück evidence is given with the induction prediction (5–18) against the observation (11 zeros, one 107).", + "13": "No points lost: horizontal gene transfer by conjugation, a plasmid carrying a complete gene such as beta-lactamase, across species.", + "14": "No points lost: the fraction falls through compounding cost, and compensatory mutations and multi-drug plasmids keep it from disappearing, with correct numbers." + }, + "pass": true + }, + "usage": { + "cost": 0.2225246, + "seconds": 57 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "Mutation has several names: 'random copying error' (line 5), 'mutation' (lines 11, 24), 'One DNA letter change' (table, line 16), 'one letter change' (line 47). The reader can probably map these, but the link is never stated." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Gene transfer has several names: 'received from another bacterium' (lines 5, 11), 'Genes from other bacteria' (heading, line 42), 'conjugation' and 'horizontal gene transfer' (line 45). Conjugation is one route of horizontal transfer, and that is only hinted at in 'Not covered' (line 71)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'resistance gene' (line 11) and 'resistance mutation' (line 24) are used for close but different things: a gene versus the event that makes it." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "Growth being stopped has three names: 'stops their growth' (line 9), 'halts their growth' (line 33), 'inhibitory' in MIC (line 20)." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'At a lab doubling time of 30 minutes, that is 13 hours' (line 35): it does not say whether bacteria in a patient double this fast, or how a slower rate would change the regrowth time." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'after 100 doublings' (lines 54, 58): it does not say how long 100 doublings takes in real time, which is needed for the 'hospital stops using drug X for years' case." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'In 10¹⁰ cells, expect 10⁻⁶ such cells' (line 64): it is not translated into about a one-in-a-million chance that even one doubly resistant cell exists." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'about once per 10⁸ cell divisions' (line 24): it does not say whether this rate is typical or varies across genes and drugs, or how a rate of 10⁻⁶ or 10⁻¹⁰ would change the conclusion." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'0.95¹⁰⁰ ≈ 0.006' (line 54): this is a ratio relative to the start value, but it does not say what resistant fraction that leaves if resistance started common or rare." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'the random scatter is about √11 ≈ 3.3, so ... predicts most cultures between 5 and 18' (line 40): no reason is given for why scatter is the square root of the mean, or why the range is about ±2 scatters." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'each time the susceptible cells double, the resistant cells grow only 0.95 times as much' (line 54): it does not say why a 5% slower growth translates to a 0.95 factor per doubling." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'A compensatory mutation is a second change that restores growth ... so the first change, and the resistance, stay' (line 58): it does not say why the cell gains a second mutation rather than losing the first one." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "'Early mutants have divided since, so there are at least 100 resistant cells' (line 28): this mixes an expected count of mutation events with a minimum number of cells, and does not say how many more there may be." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "'the test asks only whether resistance arose before contact, so it applies to drugs too' (line 40): the transfer from virus to antibiotic is asserted rather than argued." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "'The bacteria become resistant, not the patient's body.' (line 5): this is a common misconception, but the questions and the later argument never use it." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "The 'Not covered' list (lines 68–73) names topics (persisters, SOS response, biofilms) the reader cannot use. It is harmless but adds no understanding." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The explanation does not give real time scales for decline in a hospital after a drug is withdrawn, so 'for years' cannot be answered quantitatively." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "It does not say whether plasmids themselves carry a fitness cost, or how fast they spread between cells or species." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "It does not say whether bacteria divide at the lab rate inside a patient; the 13-hour regrowth depends on this." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "It does not describe the expected distribution under the mutation-before-exposure idea, only that it 'explains this spread'." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "It does not explain how mutation and transfer compare in importance in practice." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/understanding.md b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/understanding.md new file mode 100644 index 0000000..e695efc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/explainers/antibiotic-resistance/review/understanding.md @@ -0,0 +1,15 @@ +# Review: antibiotic-resistance (published tier, Stage 1 prose) + +## Runs + +| Run | Tool | Result | Action | +|---|---|---|---| +| 2026-10-04T151610-probe | probe | 5 main, 8 edge, 2 excess | Adopted 1–11, 13, 15: mutation events vs resistant cells, immune assumption, one example per mechanism, combination break case, one term per concept, MIC threshold, compounding cost, Luria–Delbrück numbers, conjugation, compensatory mutation, stepwise MIC, 1% probability. Declined 12 (in-patient doubling time: no reliable value) and 14 (representation section is template-required). | +| 2026-10-04T151845-coldread-narrative | cold read | 4 blocking | Rewrote every beat's Said column in full. Declined cut of the 13 h figure (now tied to a stated 30 min doubling), cut of the mechanisms table, removal of scope. | +| 2026-10-04T152038-coldread-prose | cold read | 0 blocking | Applied the short edge and excess fixes. Declined: virus agent (later explained), transformation/transduction and biofilms in Not covered (pointers only). | +| 2026-10-04T152206-quiz-prose | blind test | FAIL (Q10 scored 1) | Cause: the cold-read cut of "plasmids also persist" removed a rubric point. Restored with its reason (a plasmid with genes for several drugs stays selected by any of them). Also: one word "doubling" for the cost arithmetic, why the immune system cannot clear the survivors, √11 scatter for 5–18, why probabilities multiply, how one pump resists two drugs. | +| 2026-10-04T152433-quiz-prose | blind test | PASS, all 14 items 2 | Adopted the one-in-a-million wording. Declined the rest: real-time scales for hospital decline and in-patient growth, plasmid cost and spread rate, mutation vs transfer importance — each adds length past the 950-word budget for questions this reader need not answer to grasp the mechanism. | + +## Rule learned + +A cold-read "leap: no reason given" on a sentence that carries a quiz answer should be fixed by adding the reason, not by cutting the sentence. Check the rubric before you cut a claim the cold read questions. diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/read.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/read.json new file mode 100644 index 0000000..d889b8d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "Resistance genes exist before any drug is given. They come from random copying errors (mutations) during cell division, at about 1 per 10⁸ divisions, or from genes received from another bacterium. A 10¹⁰-cell infection therefore expects about 100 resistance mutations.", + "quote": "a few bacteria already carry a resistance gene, made by a random copying error or received from another bacterium." + }, + "2": { + "answer": "The drug does not create resistance or change the bacteria. It kills or halts the susceptible cells, which removes them, and the resistant cells that were already there keep dividing and take over. This is selection.", + "quote": "This is **selection**: the drug did not make resistant cells, it removed the susceptible cells." + }, + "3": { + "answer": "By horizontal gene transfer, mainly through conjugation. Two bacteria touch and one copies a plasmid (a small circle of DNA that carries complete resistance genes) into the other. The same plasmid gene is found in different species, which shows it moved between them.", + "quote": "In **conjugation**, two bacteria touch and one copies a plasmid into the other. This is **horizontal gene transfer**: a gene from another bacterium, not from the parent cell." + }, + "4": { + "answer": "Reasoning from the text: a serious infection can hold 10¹⁰ cells, so a one-in-a-billion rate means about 10 resistant cells are already present before treatment. The full course kills or halts the susceptible cells but cannot affect the resistant ones. These keep doubling, the immune system alone cannot keep up, and they regrow to a large infection made almost entirely of resistant cells. Finishing the course does not help, because the drug was never acting on them.", + "quote": "The resistant cells keep dividing. The immune system removes some, but alone it cannot keep up with a population that keeps doubling." + } + }, + "stops": [ + "\"Each division adds one cell, so growing to that size took about 10¹⁰ divisions.\" The reason why the number of divisions equals the number of cells is packed into a few words.", + "\"Early mutants have divided since, so there are at least 100 resistant cells.\" The jump from 100 mutations to 'at least 100' cells needs a moment's thought.", + "\"At a lab doubling time of 30 minutes, that is 13 hours.\" It is unclear whether bacteria in the body double this fast.", + "\"They used a virus that kills bacteria; the test asks only whether resistance arose before contact, so it applies to drugs too.\" The switch from a virus to drugs needs a reread.", + "\"For independent rare events with a mean of 11, the random scatter is about √11 ≈ 3.3\" The square-root rule for scatter is stated without explanation.", + "\"each time the susceptible cells double, the resistant cells grow only 0.95 times as much\" It is unclear what '0.95 times as much' growth means.", + "\"Then about 100 cells survive again.\" Why it goes back to 100 is left for the reader to work out (one event at 10⁻⁸ in 10¹⁰ cells).", + "\"the immune system clears the halted cells\" The immune system's role is introduced without saying how big it is." + ], + "excess": [ + "\"\"", + "\"The bacteria become resistant, not the patient's body.\"", + "\"| Keep the drug out | The cell loses a porin, an entry channel in its surface, so less drug gets in. |\"", + "\"Not covered\"", + "\"Course length, prescribing policy, farm use, biofilms.\"", + "\"**In short:** resistance genes come first, by chance or from other bacteria.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/run.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/run.json new file mode 100644 index 0000000..94063c2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/antibiotic-resistance/explanation.md` as prose only, because the subject is a chain of causes (random variation, then selection by the drug, regrowth and spread) carried by a few worked numbers, so a diagram or interactive page wouldn't add anything. It passed the probe, the cold reads (nothing blocking) and the final blind test (all 14 items at full marks; the first run failed one item, and I fixed that); details are in `review/understanding.md`.", + "error": null, + "cost": 1.9340748, + "seconds": 686, + "turns": 81, + "files": [ + "explainers/antibiotic-resistance/explanation.md" + ], + "review_cost": 0.8549, + "reviews": 5 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/explanation.md b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/explanation.md new file mode 100644 index 0000000..8435c4e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/explanation.md @@ -0,0 +1,51 @@ +# How do bacteria become resistant to an antibiotic? + +**A few bacteria in a large population already carry a gene that protects them, by random chance. The antibiotic kills the other bacteria, and the protected ones divide to replace them. The bacteria do not learn; the drug selects.** + +## What an antibiotic does + +An antibiotic is a molecule that binds a **target**: a protein or structure that the bacterium needs. Penicillin, for example, blocks the enzymes that build the cell wall, so the cell bursts. A cell is **susceptible** if the drug kills it or stops its growth. A cell is **resistant** if it survives a dose that kills susceptible cells. The bacteria become resistant, not the patient. + +## Step 1: random variation makes a few resistant cells + +Resistance needs a **resistance gene**: a version of a gene whose protein stops the drug from working on that cell. A cell gets one in two ways. + +1. **Mutation.** Each time a cell copies its DNA, it makes rare random errors. A specific error that changes the target has a chance of about 10^-9 per division: once per billion divisions. +2. **Gene transfer.** Bacteria pass DNA to each other, often on a **plasmid**: a small circle of DNA, separate from the chromosome, that cells copy and hand over. One plasmid can carry several resistance genes and move between species, so one transfer can give resistance to several drugs. + +## Step 2: the antibiotic selects the resistant cells + +Worked numbers (illustrative): + +- A serious infection holds about 10^10 bacteria. +- Expected resistant cells before any drug: 10^10 × 10^-9 = 10. +- The antibiotic kills the susceptible cells. The 10 resistant cells survive. +- Each survivor divides. 10 × 2^30 ≈ 10^10, so 30 doublings rebuild the population. At 30 minutes per doubling (a lab rate; slower in a body), that takes about 15 hours. + +The new population is almost all resistant. This process is **selection**: the drug removes one kind of cell, and the remaining kind fills the space. + +Predict before you read on: what happens with a small population of 10^7 cells? Expected resistant cells = 10^7 × 10^-9 = 0.01. Usually no resistant cell exists, so the drug clears the infection. + +## Why selection, and not adaptation + + +The simplest alternative idea is that each cell meets the drug and adapts. Then every cell would have the same chance to survive. Instead, a rare fraction survives, and those cells were resistant before they met the drug. Luria and Delbrück showed this in 1943: the number of resistant bacteria varied between cultures in a way that only fits mutations that occurred before exposure. So the antibiotic does not cause resistance. It reveals and enriches resistance that random variation already made. + +## How a resistance gene protects the cell + + +Each resistance gene works in one of four ways: + +| Mechanism | What the cell does | Example | +|---|---|---| +| Destroy the drug | An enzyme cuts or changes the drug | Beta-lactamase cuts penicillin | +| Change the target | The target protein changes shape, so the drug cannot bind | MRSA makes a cell-wall enzyme that penicillin-type drugs do not bind | +| Pump the drug out | A membrane pump removes the drug faster than it enters | Efflux pumps against tetracycline | +| Keep the drug out | The cell makes fewer entry channels in its outer membrane | Fewer porins against several drugs | + +## What follows from this + +- **Two drugs at once help.** A cell needs two independent rare events: 10^-9 × 10^-9 = 10^-18 per cell. In 10^10 cells, that is 10^-8 expected cells, so effectively none. This fails if one plasmid carries both resistance genes. +- **Resistance is often a cost.** Without the drug, resistant cells often grow slower, so they stay rare. Every use of the drug gives them the advantage. + +**Not covered here:** persister cells, which survive a drug without a resistance gene; and stress responses that raise mutation rates. diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.md b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.md new file mode 100644 index 0000000..a3ff463 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.md @@ -0,0 +1,100 @@ +# Semantic model: antibiotic resistance + +> Source of truth. Every rendering derives from this file. + +## Central question + +How do bacteria become resistant to an antibiotic? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (cells, DNA, genes code for proteins, exponential growth), basic programming. New to microbiology. Does not know: what an antibiotic targets, mutation rates, plasmids, horizontal gene transfer, selection as a mechanism. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Antibiotic | A molecule that kills bacteria or stops their growth by binding a target that bacteria need. | fact | +| Target | The bacterial protein or structure that the antibiotic binds (for penicillin: the enzymes that build the cell wall). | fact | +| Population | The bacteria in one infection: about 10^10 cells, all descended from a few cells. | estimate | +| Resistance gene | A version of a gene whose protein stops the antibiotic from working on that cell. | fact | +| Mutation | A random copying error in DNA when a cell divides. | fact | +| Gene transfer | One bacterium passes DNA to another, usually on a plasmid (a small circle of DNA). | fact | +| Selection | The antibiotic kills the susceptible cells, so the resistant cells make up the next population. | fact | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| Mutation, gene transfer | create | resistance gene in a few cells | before or during treatment, at random | +| Resistance gene | blocks | antibiotic action | four mechanisms | +| Antibiotic | kills | susceptible cells | not resistant cells | +| Selection | enriches | resistant cells | they divide into the space left | + +## Causal chain + +```text +random variation (mutation or gene transfer) → a few resistant cells +→ antibiotic kills susceptible cells → resistant cells survive and divide +→ resistant population +``` + +## Quantities + +- Mutation rate for one specific resistance mutation: about 10^-9 per cell division (illustrative; real values range about 10^-6 to 10^-10). +- Population in a serious infection: about 10^10 cells. +- Expected resistant cells before any drug: 10^10 × 10^-9 = 10 cells. +- Regrowth: 10 cells × 2^30 ≈ 10^10 cells. 30 doublings at 30 min each ≈ 15 hours (lab rate; slower in a body). +- Two drugs with independent targets: chance per cell 10^-9 × 10^-9 = 10^-18; expected cells in 10^10: 10^-8, so effectively none. + +## Alternative states + +| State | What changes | Why | +|---|---|---| +| No antibiotic | resistant cells stay rare | resistance often costs growth rate; no selection | +| Antibiotic present | resistant cells take over | selection | +| Two drugs together | resistance very unlikely | a cell needs two independent rare events | +| Resistance gene on a plasmid | spreads between cells and species | gene transfer, not only division | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Resistance comes from pre-existing random variation, then selection | Each cell "learns" or adapts when it meets the drug | If cells adapted on contact, all 10^10 would survive or none would; observed: a fixed rare fraction (about 10 in 10^10) survives, and those cells existed before exposure (Luria–Delbrück, 1943) | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| Selection enriches resistance | 10^10 cells with drug: 10 resistant survive, regrow to 10^10 in about 30 doublings | Small population of 10^7 cells: expected resistant cells = 0.01, so the drug usually clears it | +| Combination therapy prevents resistance | Two independent drugs: 10^-18 per cell, 10^-8 expected | A plasmid carrying both resistance genes gives both at once, so the multiplication no longer applies | + +## Terms + +- **Susceptible:** the antibiotic kills or stops this cell. +- **Resistant:** the cell survives a dose that kills susceptible cells. +- **Plasmid:** a small DNA circle, separate from the chromosome, that cells copy and pass to each other. +- **Selection:** see Entities. + +## Epistemic status + +- **Established:** four mechanisms; mutation and gene transfer as sources; resistance arises before exposure (Luria–Delbrück). +- **Illustrative:** the specific numbers 10^-9, 10^10, 30 min. + +## Confusion points + +- "The antibiotic causes the mutation" → the mutation is random and usually exists already; the antibiotic only selects. (Some antibiotics raise mutation rates a little; out of scope.) +- "The patient becomes resistant" → the bacteria do, not the person. +- "Resistant cells are stronger in general" → they often grow slower without the drug. + +## Scope + +- **Out of scope:** tolerance and persister cells (survive without a resistance gene) — see "antibiotic persistence". +- **Out of scope:** stress-induced mutagenesis (SOS response). +- **Out of scope:** clinical policy (stewardship, dosing) beyond one line. + +## Representation decision + +- **Stage:** 1 +- **Reason:** a short causal sequence plus a four-row list of mechanisms; prose with one table and worked numbers carries it. +- **Levels:** L1 answer / L2 the two steps / L3 why it is selection, not adaptation / L4 mechanisms. diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.yaml b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.yaml new file mode 100644 index 0000000..f33a8ba --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/explainers/antibiotic-resistance/model.yaml @@ -0,0 +1,57 @@ +# Machine-readable half of model.md. `explainer check antibiotic-resistance` holds every rendering to it. + +values: + mutation_rate: "10^-9" + population: "10^10" + resistant_before: 10 + doublings: 30 + doubling_minutes: 30 + regrowth_hours: 15 + small_population: "10^7" + small_expected: 0.01 + combo_rate: "10^-18" + combo_expected: "10^-8" + luria_year: 1943 + +# exponents of the power-of-ten values above (the checker reads 10^-9 as 10 and -9) +allow: [2, 4, -9, 7, -18, -8] + +require: + - path: resistant_before + in: [prose] + - path: combo_rate + in: [prose] + +claims: + - id: selection-not-adaptation + kind: why + about: [selection] + statement: "Resistance comes from rare random variants that exist before the drug; the drug selects them, it does not create them." + alternative: "Each cell adapts when it meets the drug." + counterexample: "If cells adapted on contact, every cell would have the same chance; observed: about 10 of 10^10 survive, and they were resistant before exposure (Luria–Delbrück, 1943)." + ask: "Does the antibiotic cause the resistance mutation?" + - id: selection-worked + kind: mechanism + statement: "The drug kills susceptible cells; the resistant cells regrow the population." + example: "10^10 cells × 10^-9 = 10 resistant cells; 10 × 2^30 ≈ 10^10 after 30 doublings, about 15 hours." + - id: sources + kind: mechanism + statement: "Resistance genes arrive by mutation or by gene transfer on plasmids." + example: "A plasmid can carry several resistance genes and move between species." + - id: mechanisms + kind: mechanism + statement: "A resistance gene works in one of four ways: destroy the drug, change the target, pump the drug out, or keep it out." + example: "Beta-lactamase cuts penicillin before it reaches its target." + - id: combination + kind: guarantee + statement: "Two independent drugs make resistance far less likely." + example: "10^-9 × 10^-9 = 10^-18 per cell; 10^-8 expected cells in 10^10." + counterexample: "A plasmid with both resistance genes gives both at once." + ask: "Why does giving two drugs at once help?" + +terms: + - term: resistant + means: "the bacterial cell survives the drug; the patient is not resistant" + avoid: [patient becomes resistant, immune to the antibiotic] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/read.json new file mode 100644 index 0000000..2810587 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/read.json @@ -0,0 +1,40 @@ +{ + "answers": { + "1": { + "answer": "Random variation that happens before any drug is present. It comes from rare random mutations made when a cell copies its DNA (about once per billion divisions), or from genes passed between bacteria on plasmids.", + "quote": "A few bacteria in a large population already carry a gene that protects them, by random chance." + }, + "2": { + "answer": "The drug does not cause resistance. It selects for it: it kills the susceptible cells, and the resistant survivors divide to fill the space, so the new population is almost all resistant.", + "quote": "So the antibiotic does not cause resistance. It reveals and enriches resistance that random variation already made." + }, + "3": { + "answer": "Through gene transfer. Bacteria hand DNA to each other, often on plasmids (small circles of DNA separate from the chromosome). A plasmid can carry several resistance genes and move between species.", + "quote": "One plasmid can carry several resistance genes and move between species, so one transfer can give resistance to several drugs." + }, + "4": { + "answer": "In a large infection (about 10^10 cells), one in a billion means about 10 resistant cells exist before treatment. However long the course runs, the drug kills only the susceptible cells. The resistant ones survive and divide, and about 30 doublings rebuild a population that is almost all resistant. The explanation does not say how the length of the course matters.", + "quote": "The antibiotic kills the susceptible cells. The 10 resistant cells survive." + } + }, + "stops": [ + "'A full course' and how long treatment lasts are never discussed, so it is unclear how duration interacts with selection.", + "\"the number of resistant bacteria varied between cultures in a way that only fits mutations that occurred before exposure\": it does not say why this variation rules out adaptation.", + "\"A specific error that changes the target has a chance of about 10^-9 per division\": it is unclear how a per-division rate becomes the per-cell count of 10^10 × 10^-9.", + "\"MRSA makes a cell-wall enzyme\": MRSA is never defined.", + "\"Fewer porins against several drugs\": porins are only implied by 'entry channels'.", + "\"Efflux pumps against tetracycline\": the term efflux is not explained.", + "\"The bacteria become resistant, not the patient.\": this is stated without context, and it is unclear what misconception it corrects." + ], + "excess": [ + "\"Penicillin, for example, blocks the enzymes that build the cell wall, so the cell bursts.\"", + "\"Not covered here: persister cells, which survive a drug without a resistance gene; and stress responses that raise mutation rates.\"", + "\"At 30 minutes per doubling (a lab rate; slower in a body), that takes about 15 hours.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/run.json new file mode 100644 index 0000000..f9b0e31 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/antibiotic-resistance/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/antibiotic-resistance/explanation.md`, which is prose with worked numbers and one table, plus its `model.md` and `model.yaml`; I chose prose because the answer is a short two-step story (random variation, then the drug kills the rest), so a diagram or interactive page wouldn't add anything, and `explainer check` passes.", + "error": null, + "cost": 0.430636, + "seconds": 104, + "turns": 21, + "files": [ + "explainers/antibiotic-resistance/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/explainers/bloom-filter/index.html b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/explainers/bloom-filter/index.html new file mode 100644 index 0000000..a473b2e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/explainers/bloom-filter/index.html @@ -0,0 +1,456 @@ + + + + + +Bloom filters: "have I seen this before?" in a few bits + + + +
+ +

Bloom filters: answering "have I seen this before?" with very little memory

+

A Bloom filter is a row of bits plus a few hash functions. It can tell you that something is definitely new or probably seen before. It uses a small, fixed amount of memory no matter how long the items are. The cost is that the "probably" answer is sometimes wrong. This page shows how it works, lets you try one, and explains how to work out how often it will be wrong.

+ +

1. The problem

+

Lots of programs need to ask the same question over and over: is this item in my set? Has this user name been taken? Has the web crawler already visited this URL? Is the row I'm looking for in this file on disk, or should I skip the file?

+

The obvious answer is to store every item, say in a hash set, and look it up. That gives exact answers, but you pay for the items themselves. A billion URLs of about 60 bytes each comes to roughly 60 GB before any overhead. Often you can't afford that much fast memory, or the full set lives somewhere slow (on disk, across the network), and you just want a cheap check before you go there.

+

A Bloom filter gives up a little accuracy in return for a large saving in memory. At a 1% error rate it needs about 10 bits per item, whether the item is a 6-letter word or a 2 KB document. For those billion URLs that's about 1.2 GB.

+ +

2. The machinery: bits and hash functions

+

You need two ingredients:

+
    +
  • An array of m bits, all starting at 0.
    • +
  • k different hash functions. A hash function turns any input (such as a string) into a number that looks random but is always the same for the same input. We take each number modulo m, so it picks a position in the array.
    • +
+

So each item is linked to k positions in the array. You can think of them as the item's "fingerprint" of bit positions.

+
+

Add an item: compute its k positions and set all of those bits to 1. (If a bit is already 1, it stays 1.)

+

Check an item: compute its k positions and look at the bits.

+
    +
  • If any of them is 0, the item was definitely never added. Adding it would have set that bit to 1, and bits never go back to 0.
    • +
  • If all of them are 1, the item was probably added. Those bits might have been set by other items.
    • +
+
+

The filter never stores the item itself. It only stores the bits that items have switched on. That's why it's small, and also why it can be fooled.

+ +

3. Try it

+

This filter has m = 32 bits. Add a few words, then check some words you added and some you didn't. The outlined squares are the positions the last word hashed to: orange means the bit was 1, red means it was 0.

+ +
+
+ + + + +
+
+ + + +
+
+
bit is 0   bit is 1   probed, was 1   probed, was 0
+
Type a word and press Add or Check.
+
+
Words actually added (the filter itself does not keep this list; it's shown here only so you can see which answers are wrong):
+
+
+ +

Things to notice:

+
    +
  • A word you added always comes back "probably yes". There are no false negatives.
    • +
  • When the filter is nearly empty, most new words land on at least one 0 and are rejected straight away.
    • +
  • As you add more words, the array fills up with 1s. Eventually a word you never added lands only on 1s that other words set, and the filter says "probably yes". That's a false positive. The Hunt button tries made-up words until it finds one.
    • +
  • Different words sometimes share positions. That sharing is what lets the filter use so few bits, and it's also the cause of false positives.
    • +
+ +

4. How it can be wrong

+

There are four possible outcomes:

+ + + + +
Filter says "definitely not"Filter says "probably yes"
Item was addedImpossible: its bits were set to 1 and never clearedCorrect
Item was not addedCorrectFalse positive: other items happened to cover all its bits
+

So the error only goes one way. "No" is always trustworthy, and "yes" means "check properly if it matters". That fits the usual way Bloom filters are used, as a cheap guard in front of an expensive exact lookup:

+
    +
  • A database keeps a Bloom filter for each file on disk. Before reading a file to look for a key, it asks the filter. "Definitely not" saves a disk read. "Probably yes" means it reads the file, and now and then finds nothing there, which costs a wasted read but never gives a wrong answer.
    • +
  • A web cache only stores a page the second time it's requested. A Bloom filter of "URLs seen once" answers "have I seen this before?" cheaply, and the occasional false positive just means a page gets cached a little early.
    • +
+

Two more limits come from the same design:

+
    +
  • You can't delete. Setting an item's bits back to 0 might erase bits that other items also depend on, and then those items would get false negatives. (A variant called a counting Bloom filter stores a small counter in each slot instead of a single bit. Adding increments the counters and deleting decrements them, at several times the memory cost.)
    • +
  • You can't list the contents. The items themselves were never stored, only bits.
    • +
+ +

5. How often is it wrong? The arithmetic

+

Here's how to estimate the false positive rate. Assume the hash functions behave like independent, fair random choices of position. Real hash functions are close enough to this.

+ +

Step 1: chance that a particular bit is still 0. A single hash picks a particular bit with probability 1/m, so it misses that bit with probability 1 − 1/m. Adding n items performs kn hashes in total, and the bit stays 0 only if every one of them misses it:

+P(bit still 0) = (1 − 1/m)kn ≈ e−kn/m +

(The approximation uses the standard fact that (1 − 1/m)m ≈ 1/e when m is large.)

+ +

Step 2: chance of a false positive. Now check an item that was never added. Its k positions are effectively random, and each one is 1 with probability 1 − e−kn/m. The filter wrongly says "yes" only if all k are 1:

+p ≈ (1 − e−kn/m)k +

Put another way, if a fraction f of the bits are 1, the false positive rate is about fk. The demo above shows this estimate under the bit array.

+ +

Step 3: choosing k. The number of hash functions pulls in two directions:

+
    +
  • More hashes per check means a stranger has to get lucky more times, which lowers the error.
    • +
  • More hashes per insertion fills the array with 1s faster, which raises the error.
    • +
+

The best balance turns out to be when about half the bits are 1. That happens when

+k = (m/n) · ln 2 ≈ 0.69 · (m/n) +

With that choice, each extra hash halves the error, so p ≈ (1/2)k. Rearranging gives the number of bits needed per item for a target error rate p:

+m/n = −ln p / (ln 2)2 ≈ 1.44 · log2(1/p) + + + + + + + +
Target false positive rateBits per itemBest k
10%4.83
1%9.67
0.1%14.410
0.01%19.213
+

Two things in this table are worth noticing. The item size doesn't appear anywhere. And each tenfold improvement in accuracy costs only about 4.8 more bits per item.

+ +

6. Explore the trade-off

+

Move the sliders to see how memory per item (m/n) and the number of hashes (k) affect the false positive rate. The curve plots the rate against k for the chosen memory budget. Its lowest point is the best k.

+
+ + +
+ +
+

Try overfilling the filter: keep k fixed and slide the bits per item down. This is what happens when you build a filter for a million items and then put ten million in it. The error rate doesn't creep up slowly; it rises steeply toward 100% as the array fills with 1s. Bloom filters need to be sized for the largest number of items you expect.

+ +

7. Summary

+
    +
  • A Bloom filter is m bits plus k hash functions. Adding an item sets its k bits. Checking an item tests them.
    • +
  • Any 0 means definitely not seen. All 1s means probably seen.
    • +
  • It can be wrong in only one direction: false positives, which happen when other items have already set all of a new item's bits. False negatives can't happen.
    • +
  • The false positive rate is about (1 − e−kn/m)k. With the best k, about 10 bits per item gives 1%, and every extra ~4.8 bits per item cuts the rate tenfold.
    • +
  • It saves memory because it stores no items, which is also why it can't delete items or list them.
    • +
+ +
+ + + + diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/read.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/read.json new file mode 100644 index 0000000..055f622 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "Adding an item: work out its k positions using the k hash functions (each taken modulo m) and set all of those bits to 1. A bit that is already 1 stays 1. Looking an item up: work out the same k positions and look at those bits. If any of them is 0, the item was definitely never added. If all of them are 1, the item was probably added.", + "quote": "Add an item: compute its k positions and set all of those bits to 1. (If a bit is already 1, it stays 1.)" + }, + "2": { + "answer": "It can give false positives: it says \"probably yes\" for an item that was never added, because other items happened to set all of that item's bits. It can never give a false negative. Adding an item sets its bits to 1, and bits never go back to 0, so an item that was added always finds all its bits set.", + "quote": "If any of them is 0, the item was definitely never added. Adding it would have set that bit to 1, and bits never go back to 0." + }, + "3": { + "answer": "Other items may share those bits. Setting them back to 0 could erase bits that other items depend on, and those items would then get false negatives. A counting Bloom filter, which keeps a counter in each slot, can delete.", + "quote": "Setting an item's bits back to 0 might erase bits that other items also depend on, and then those items would get false negatives." + }, + "4": { + "answer": "It says \"probably yes\", which is a false positive. Whatever positions the item hashes to, all of them are 1, so the check finds no 0. The page's own estimate agrees: with f = 1, the false positive rate f^k comes to 100%. The page never mentions this case directly; this is reasoned from the check rule.", + "quote": "If all of them are 1, the item was probably added. Those bits might have been set by other items." + } + }, + "stops": [ + "\"We take each number modulo m, so it picks a position in the array.\" The word 'modulo' is not explained for a reader new to the subject.", + "\"store every item, say in a hash set\". 'Hash set' is used before hashing has been explained.", + "\"The best balance turns out to be when about half the bits are 1.\" This is stated without any reason.", + "\"With that choice, each extra hash halves the error\". It is not shown why this follows.", + "\"m/n = −ln p / (ln 2)^2 ≈ 1.44 · log2(1/p)\". The 'Rearranging' step is skipped.", + "\"(1 − 1/m)^m ≈ 1/e when m is large\". This 'standard fact' may not be familiar to the reader.", + "\"Assume the hash functions behave like independent, fair random choices of position.\" It is unclear what 'independent' means here and why it matters.", + "\"For those billion URLs that's about 1.2 GB.\" The reader has to work out that 10 bits per item gives this figure." + ], + "excess": [ + "\"A web cache only stores a page the second time it's requested.\" This second use case is not needed to understand how the filter works.", + "\"A variant called a counting Bloom filter stores a small counter in each slot instead of a single bit.\" This is off the main question.", + "\"Words actually added (the filter itself does not keep this list; it's shown here only so you can see which answers are wrong)\"", + "\"Any 0 means definitely not seen. All 1s means probably seen.\" The summary repeats the earlier rules.", + "\"It saves memory because it stores no items, which is also why it can't delete items or list them.\" This also repeats earlier text.", + "\"Put another way, if a fraction f of the bits are 1, the false positive rate is about f^k.\" This restates the formula just given." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/run.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/run.json new file mode 100644 index 0000000..dd13086 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/bloom-filter/index.html`, a single interactive page. You learn a Bloom filter fastest by adding words, watching bits turn on and finding a false positive yourself, and the page puts the prose and the false-positive arithmetic right next to the hands-on filter and a trade-off chart (I haven't opened it in a browser yet).", + "error": null, + "cost": 0.40253520000000004, + "seconds": 118, + "turns": 3, + "files": [ + "explainers/bloom-filter/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/explainers/bloom-filter/explanation.md b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/explainers/bloom-filter/explanation.md new file mode 100644 index 0000000..dff85e6 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/explainers/bloom-filter/explanation.md @@ -0,0 +1,111 @@ +# Bloom filters: "have I seen this before?" in about 10 bits per item + +**The short answer.** A Bloom filter does not store the items. It stores a row of bits. To add an item, it hashes the item to a few bit positions and sets those bits to 1. To check an item, it hashes the item the same way and looks at the same positions. If any of those bits is 0, the item was never added. If all of them are 1, the item was *probably* added. The bits for that item could also have been set by other items. That last case is the only way the filter is wrong. It can say "yes" to an item that it never saw. It never says "no" to an item that it did see. + +**Why you would want this.** Suppose a database keeps keys on disk. A disk read is slow, and many lookups ask for keys that do not exist. A small Bloom filter in memory answers "definitely not here" for most of those lookups, so the database skips the disk read. A "probably here" answer costs one real disk read, and that read gives the true answer. So a rare wrong "yes" costs a little time. It never gives an incorrect result. Databases such as LevelDB, RocksDB, and Cassandra use Bloom filters in this way. + +## The parts + +- **Bit array**: `m` bits, all 0 at the start. +- **Hash function**: a function that turns an item into a number that looks random. The same item always gives the same number. Here, each hash gives a position from 0 to `m − 1`. +- **`k`**: the number of hash functions. Each item maps to `k` positions. +- **`n`**: the number of items added so far. + +There are two operations: + +1. **Add(x)**: compute the `k` positions for `x`. Set each bit to 1. +2. **Query(x)**: compute the `k` positions for `x`. If any bit is 0, answer "definitely not seen". If all bits are 1, answer "probably seen". + +## A worked example + +Use `m = 16` bits and `k = 3`. The hash outputs below are invented for the example. A real filter gets them from real hash functions. + +| Step | Positions | Bits that are 1 after the step | +|---|---|---| +| Add "cat" | 1, 6, 11 | 1, 6, 11 | +| Add "dog" | 3, 6, 14 | 1, 3, 6, 11, 14 | + +"cat" and "dog" share bit 6. That sharing is normal, and it is the source of every error. + +Now query three items: + +| Query | Positions | Bits at those positions | Answer | Correct? | +|---|---|---|---|---| +| "cat" | 1, 6, 11 | 1, 1, 1 | probably seen | yes, "cat" was added | +| "fox" | 2, 6, 11 | **0**, 1, 1 | definitely not seen | yes | +| "owl" | 1, 3, 14 | 1, 1, 1 | probably seen | **no: false positive** | + +"owl" was never added. Its three bits were set by "cat" (bit 1) and "dog" (bits 3 and 14). The filter has no record of *which* item set a bit, so it cannot tell the difference. + +## Why it never misses an item it saw + +Add only changes bits from 0 to 1. Nothing changes a bit back to 0. When you add `x`, its `k` bits become 1, and they stay 1. A later query for `x` computes the same `k` positions, finds all 1s, and answers "probably seen". So a **false negative** (a "no" for an item that was added) cannot happen. + +**Where this guarantee breaks.** The guarantee depends on "nothing clears a bit". Suppose you try to delete "cat" by clearing bits 1, 6, and 11. Bit 6 also belongs to "dog". A query for "dog" now finds bit 6 = 0 and answers "definitely not seen", which is wrong. For this reason a basic Bloom filter does not support delete. (A *counting Bloom filter* stores a small counter in place of each bit, so it can delete. It uses about 4 times the memory.) + +## Why so little memory + +A normal set stores each item. One million URLs of about 50 bytes each need about 50 MB, plus the overhead of the set structure. A Bloom filter stores only bits, and the number of bits does not depend on the size of an item. With 1 million items and a 1% false-positive target, the filter needs about 9.6 bits per item: about 1.2 MB in total. A long URL and a short URL cost the same. + +The price is that the filter can only answer the membership question. It cannot list its items, count them, or give back an item. + +## How often it is wrong + +Each false positive happens when all `k` bits of a new item are already 1. So the error rate depends on how full the array is. + +**Assumption:** each hash position is uniform and independent of the others. Good hash functions come close to this. + +One hash sets one specific bit with probability `1/m`. After `n` adds, there were `k·n` bit-sets. The chance that a given bit is still 0 is + +``` +(1 − 1/m)^(k·n) ≈ e^(−k·n/m) +``` + +The query for an item that was never added checks `k` bits. It is a false positive if all `k` of them are 1: + +``` +false-positive rate p ≈ (1 − e^(−k·n/m))^k +``` + +(This is a mathematical consequence of the assumption above, and it is a very close approximation for large `m`.) + +**Example.** `m/n = 10` bits per item, `k = 7`. Then `k·n/m = 0.7`, and `e^(−0.7) = 0.497`. So about half the bits are 1. Then `p ≈ 0.503^7 ≈ 0.0082`, which is 0.82%: about 1 wrong "yes" in 120 queries for absent items. + +### Why use several hash functions, not one + +The simplest design uses one hash (`k = 1`). With 10 bits per item, that gives `p = 1 − e^(−0.1) ≈ 9.5%`. More hashes make each item's "fingerprint" harder to fake by accident. But each hash also sets more bits, so the array fills faster. These two effects pull in opposite directions: + +| `k` (10 bits per item) | 1 | 2 | 4 | 7 | 10 | 20 | +|---|---|---|---|---|---|---| +| false-positive rate | 9.5% | 3.3% | 1.2% | **0.82%** | 1.0% | 5.5% | + +The best value is `k = (m/n)·ln 2`, which is about 7 here. At that value, about half of the bits are 1. The array then carries as much information as possible: a bit is equally likely to be 0 or 1. + +### What happens when the filter fills up + +The size `m` is fixed when you create the filter. If you add more items than planned, the error rate climbs fast. Take `m` = 10 million bits (1.25 MB) and `k = 7`: + +| items added `n` | 0.5 million | 1 million | 2 million | 5 million | +|---|---|---|---|---| +| false-positive rate | 0.02% | 0.82% | 14% | 81% | + +At 5 million items, almost every bit is 1, so almost every query answers "probably seen". The filter then gives almost no information. The bits cannot be resized, because the filter does not keep the items it would need to hash again. So choose `m` from the largest `n` you expect. + +### Sizing rule + +For a target rate `p`, you need + +``` +bits per item m/n = −ln(p) / (ln 2)² ≈ 1.44 · log2(1/p) +``` + +| target `p` | 10% | 1% | 0.1% | +|---|---|---|---| +| bits per item | 4.8 | 9.6 | 14.4 | +| best `k` | 3 | 7 | 10 | + +Each 10× drop in the error rate costs about 4.8 more bits per item. + +## Summary + +A Bloom filter maps each item to `k` positions in a bit array and sets those bits. A query checks the same positions. A 0 proves the item is absent. All 1s mean "probably present", because other items can set the same bits. That shared use of bits is what makes the filter small, and it is also what makes it wrong sometimes. The error is one-sided (false "yes", never false "no"). Its rate depends only on bits per item and `k`, and about 10 bits per item gives about 1%. diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/read.json new file mode 100644 index 0000000..76817aa --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/read.json @@ -0,0 +1,39 @@ +{ + "answers": { + "1": { + "answer": "Add: the item is hashed to k bit positions, and each of those bits is set to 1. Lookup: the item is hashed to the same k positions. If any of those bits is 0, the answer is \"definitely not seen\". If all of them are 1, the answer is \"probably seen\".", + "quote": "To add an item, it hashes the item to a few bit positions and sets those bits to 1. To check an item, it hashes the item the same way and looks at the same positions." + }, + "2": { + "answer": "It can give a false positive: a \"yes\" for an item it never saw, because other items may have set all of that item's bits. It can never give a false negative. Adding only turns bits from 0 to 1 and nothing turns them back, so an item that was added always finds all of its bits still set to 1.", + "quote": "Add only changes bits from 0 to 1. Nothing changes a bit back to 0. When you add `x`, its `k` bits become 1, and they stay 1." + }, + "3": { + "answer": "Other items may share those bits. Clearing them can set to 0 a bit that another item needs, and a later lookup for that other item would then wrongly answer \"definitely not seen\". That is a false negative, which breaks the filter's guarantee.", + "quote": "Suppose you try to delete \"cat\" by clearing bits 1, 6, and 11. Bit 6 also belongs to \"dog\". A query for \"dog\" now finds bit 6 = 0 and answers \"definitely not seen\", which is wrong." + }, + "4": { + "answer": "\"Probably seen\". Every bit it checks is 1, so the answer is a false positive. A full filter answers yes to every query and so tells you nothing.", + "quote": "At 5 million items, almost every bit is 1, so almost every query answers \"probably seen\". The filter then gives almost no information." + } + }, + "stops": [ + "\"in about 10 bits per item\" appears in the title before the reader knows why 10 bits is the figure.", + "\"(1 − 1/m)^(k·n) ≈ e^(−k·n/m)\" gives the approximation step without saying why it holds.", + "\"The array then carries as much information as possible: a bit is equally likely to be 0 or 1.\" uses an idea from information theory without explaining it.", + "\"−ln(p) / (ln 2)² ≈ 1.44 · log2(1/p)\" gives the sizing formula without deriving it.", + "\"The best value is `k = (m/n)·ln 2`\" is stated with no reason given.", + "\"It uses about 4 times the memory.\" does not explain why counters cost 4 times as much." + ], + "excess": [ + "\"Databases such as LevelDB, RocksDB, and Cassandra use Bloom filters in this way.\" A list of example systems that the reader does not need.", + "\"Each 10× drop in the error rate costs about 4.8 more bits per item.\" This repeats what the sizing table already shows.", + "\"Its rate depends only on bits per item and `k`, and about 10 bits per item gives about 1%.\" The summary repeats points made earlier." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/run.json new file mode 100644 index 0000000..daf0d60 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/bloom-filter/explanation.md`, a prose explanation, because the mechanism is a short sequence of steps; worked tables (a 16-bit example, error rates by `k` and by fill level) show the trade-offs without an interactive page; `explainer check` was not run and I made no `model.md`/`model.yaml`, but I calculated all numbers by hand.", + "error": null, + "cost": 0.1830298, + "seconds": 48, + "turns": 2, + "files": [ + "explainers/bloom-filter/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/explanation.md b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/explanation.md new file mode 100644 index 0000000..3403260 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/explanation.md @@ -0,0 +1,88 @@ +# Bloom filters: "have I seen this before?" in a few bits per item + + + +**A Bloom filter does not keep the items. It keeps a row of bits, and each item turns on a few of them. If any of an item's bits is still off, the answer is "definitely not seen". If all of them are on, the answer is "probably seen", because other items can turn on the same bits. So the filter can be wrong in one direction only: it can say "probably seen" for an item it never saw.** + + +The motive is memory. A hash set of 1 million URLs, at about 50 bytes per URL, takes about 50 MB. A Bloom filter for the same URLs takes 1.25 MB (10 bits per URL). The price for the smaller memory is error: the filter gives a wrong "probably seen" for 0.82% of URLs it never saw. The memory does not depend on how long the items are, because the filter never keeps them. + +## How it works: add and query + + +Start with a **bit array** of m bits, all 0. In this small example, m = 16, with positions 0 to 15. + +Choose k different hash functions; here k = 3. Take each hash output mod m to get a **position**. For example, a hash output of 3,141,593 gives position 3,141,593 mod 16 = 9. Each item thus gets k positions, one from each hash function. + +To **add** an item, set its k positions to 1. Nothing else is kept. Add two items: + +- cat → positions 1, 6, 11 +- dog → positions 3, 9, 14 + +| Position | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| Bit | 0 | **1** | 0 | **1** | 0 | 0 | **1** | 0 | 0 | **1** | 0 | **1** | 0 | 0 | **1** | 0 | +| Set by | | cat | | dog | | | cat | | | dog | | cat | | | dog | | + +To **query** an item, compute the same k positions and read those bits: + +- If any bit is 0, the answer is **definitely not seen**. +- If all bits are 1, the answer is **probably seen**. + +Query fox → positions 2, 6, 14. Bit 2 is 0, so fox is definitely not seen. That answer is right. + +## How it can be wrong: false positives + + +Query owl → positions 3, 6, 11. Owl was never added. But bit 3 came from dog, and bits 6 and 11 came from cat. All three bits are 1, so the filter says "probably seen". A **false positive** is this error: "probably seen" for an item that was never added. The filter cannot tell owl from an added item, because it kept only bits, and bits do not record which item set them. + +## Why it is never wrong the other way + + +The other error, a **false negative**, is "definitely not seen" for an item that was added. A false negative cannot happen. Adding cat set bits 1, 6 and 11, and nothing ever sets a bit back to 0. So any later query for cat finds all three bits at 1. + +This guarantee depends on bits never going back to 0. Suppose you add owl for real, and then try to delete cat by clearing bits 1, 6 and 11. Owl also uses bits 6 and 11, so they are now 0, and the filter says owl is definitely not seen. That is a false negative. Bits are shared between items, so a Bloom filter has no delete operation. + +## How often it is wrong + + +Let n be the number of items added. Assume that each hash position is a uniform random choice, independent of the others. Then: + +1. Each time an add sets a bit, it misses a given bit with probability 1 − 1/m. +2. n adds set kn bits in total. So a given bit is still 0 with probability (1 − 1/m)^(kn). Because (1 − 1/m)^m ≈ 1/e for large m, this is about e^(−kn/m). +3. So the **fill** f, the fraction of bits that are 1, is about 1 − e^(−kn/m). +4. A never-added item gets k random positions. Each one is 1 with probability f. The **false-positive rate** p, the chance that a never-added item reads "probably seen", is the chance that all k are 1: **p ≈ f^k**. + +Take n = 1 million items and m = 10 million bits: 10 bits per item (m/n = 10). With k = 7: kn/m = 0.7, e^(−0.7) = 0.497, so f = 0.503 and p = 0.503^7 = 0.0082, or 0.82%. That is about 1 wrong answer in 120 queries for never-added items. + + +Why 7 hash functions? The number of hash functions k pulls in two directions, at 10 bits per item: + +| k | Fill f | False-positive rate p | Why | +|---|---|---|---| +| 1 | 0.095 | 9.5% | Most bits are 0, but a query checks only one bit, so p = f. | +| 7 | 0.503 | 0.82% | About half the bits are 1; a query checks 7 of them. | +| 20 | 0.865 | 5.5% | Each add sets 20 bits, so almost every bit is 1. | + +More hash functions make each query check more bits, but they also fill the array faster. Below half full, a query checks too few bits; above half full, almost every bit it checks is 1. The best balance is near f = 0.5, which the formula gives at k = (m/n)·ln 2, about 0.69 times the bits per item. Here 10 × 0.69 = 6.9, so k = 7. At the best k, p ≈ 0.5^k. Each extra bit per item adds 0.69 to k, so it multiplies p by 0.5^0.69 ≈ 0.62. Thus each extra 4.8 bits per item cuts p by about 10 times. + +## Overfilling + + +The rate p stays at 0.82% only while n stays near the **capacity**: the number of items the filter was sized for. Keep the same filter (10 million bits, k = 7) and add more items. At 2 million items, p is 13.8%. At 3 million items, kn/m = 2.1, f = 1 − e^(−2.1) = 0.88, and p = 0.88^7 = 40%. The filter never forgets an added item, but its "probably seen" answer stops telling you much. + +## What it is good for + + +Put the filter in front of a slow exact lookup, such as a database on disk. If the filter says "definitely not seen", skip the lookup; that is safe, because there are no false negatives. If the filter says "probably seen", do the lookup, which gives the true answer. A false positive thus costs one wasted lookup, never a wrong final answer. Of 1,000,000 queries for never-added items, about 8,200 do a wasted lookup at p = 0.82%; at p = 40%, about 400,000 do. + +## Summary + +A Bloom filter answers "have I seen this before?" by remembering only the bits each item turned on. A 0 bit proves an item is new. All 1s mean "probably seen", which is wrong at a rate p ≈ f^k that you choose with bits per item, and that rises when you add more items than the capacity. + +## Not covered here + +- **Deletion:** counting Bloom filters keep a small counter per position instead of a bit. Cuckoo filters are another option. +- **Growing:** changing m changes every position (hash mod m), so you cannot resize a filter in place. See "scalable Bloom filter". +- **Sizing:** for n items and a target p, m = −n·ln p / (ln 2)². +- **Cheap hash functions:** real code makes k positions from two hash values ("double hashing"). diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/model.md b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/model.md new file mode 100644 index 0000000..7d83da9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/model.md @@ -0,0 +1,162 @@ +# Semantic model: Bloom filter + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How does a Bloom filter answer "have I seen this before?" with little memory, and how can it be wrong? + +## Audience and prior knowledge + +A technical reader: school mathematics (powers, probability of independent events, e ≈ 2.718), basic programming (arrays, bits, functions, what a hash set is). New to Bloom filters. Knows that a hash function turns any input into a number that looks random, and that the same input always gives the same number; does not know how a Bloom filter uses several of them. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| bit array | A row of m bits, all 0 at the start; the only storage the filter has. | fact | +| hash function | A function that maps an item to one position in the bit array; the same item always gets the same position. | fact | +| k | The number of hash functions; each item gets k positions. | fact | +| m | The number of bits in the bit array. | fact | +| n | The number of items added so far. | fact | +| add | Set the item's k bits to 1. | fact | +| query | Look at the item's k bits; any 0 → "definitely not seen"; all 1 → "probably seen". | fact | +| false positive | The filter says "probably seen" for an item that was never added. | fact | +| false negative | The filter says "definitely not seen" for an item that was added. Impossible while bits only go 0 → 1. | mathematical consequence | +| false-positive rate p | The chance that a query for an item never added gives "probably seen". | mathematical consequence (under the random-hash assumption) | +| bits per item m/n | Memory spent per added item. | fact | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| add | sets to 1 | k bits | never sets a bit to 0 | +| query | reads | the same k bits | same hash functions as add | +| other items' bits | can cover | all k bits of a new item | cause of false positives | +| n / m | raises | fraction of bits that are 1 | fuller array → more false positives | +| k | raises the bits each query checks, and raises the fill | p | two opposite effects → a best k | + +## Dependencies (learning order) + +1. Hash function gives a fixed position per item. +2. Bit array, add (k bits set to 1). +3. Query; the 0 → "definitely not" rule. +4. Why a 0 proves absence (bits never go back to 0). +5. False positive: all k bits set by other items. +6. Rate p, its dependence on m/n and k; best k. +7. What breaks: overfilling; deleting. + +## Causal chain + +```text +add item → its k bits become 1 → bits never become 0 + → a later query on the item finds all 1 → "probably seen" is always right for added items (no false negatives) +other items fill bits → a never-added item's k bits can all be 1 already → false positive +more items per bit → more 1s → higher p +``` + +## Temporal sequence + +1. Start: m bits, all 0. +2. Add each item: compute k positions, set them to 1. +3. Query: compute k positions, read them. + +## Quantities + +- Toy: m = 16, k = 3. Positions are stipulated (stand-ins for hash outputs), numbered 0–15. + - "cat" → 1, 6, 11. "dog" → 3, 9, 14. After both: bits 1, 3, 6, 9, 11, 14 are 1 (6 of 16). + - Query "fox" → 2, 6, 14: bit 2 is 0 → definitely not seen. Correct. + - Query "owl" → 3, 6, 11: all 1 (3 from dog; 6, 11 from cat) → probably seen. Wrong: false positive. + - Estimate from the observed fill: (6/16)³ ≈ 0.053, about 5%. (The formula's average fill for m = 16, n = 2, k = 3 is 0.31, giving 3.1%; the toy positions were picked with no collisions, so the toy is fuller than average.) +- Hash output → position: a hash function gives a large number; take it mod m. Illustrative: h1("cat") = 3,141,593; 3,141,593 mod 16 = 9. +- Fill f: the fraction of bits that are 1. +- Formula (assumption: hash positions behave like independent uniform random choices): + 1. One bit-set misses a given bit with probability 1 − 1/m. + 2. n adds make kn bit-sets, so the bit is still 0 with probability (1 − 1/m)^(kn) ≈ e^(−kn/m). (Toy check: (15/16)^6 = 0.679 vs e^(−0.375) = 0.687.) + 3. So f ≈ 1 − e^(−kn/m). + 4. A never-added item gives a false positive when all k of its positions land on 1 bits: p ≈ f^k. + - Main example m/n = 10, k = 7: kn/m = 0.7, e^(−0.7) = 0.497, f = 0.503, p = 0.503^7 = 0.0082. +- Use: put the filter in front of a slow exact lookup (disk, database, network). "Definitely not seen" skips the lookup; "probably seen" does the lookup, which settles the answer. A false positive costs one wasted lookup, never a wrong final answer. 1,000,000 queries for absent items at p = 0.82% → about 8,200 wasted lookups; at p = 40% → about 400,000. +- At m/n = 10 bits per item: + - k = 1: p ≈ 9.5% + - k = 7: p ≈ 0.82% (best; (m/n)·ln 2 = 6.9) + - k = 20: p ≈ 5.5% +- Best k = (m/n)·ln 2. At best k, f ≈ 0.5. Why half: at k = 1, f = 0.095, but the query checks only one bit; at k = 20, f = 0.865, so almost every bit is 1 and a query rarely finds a 0. Between, more checks and a fuller array balance near f = 0.5. At best k, p ≈ 0.5^k = 0.6185^(m/n); this gives "each ~4.8 bits per item cuts p by 10×" (ln 10 / (ln 2)² = 4.79). +- Sizing: m = −n·ln p / (ln 2)². n = 1 M, p = 0.82% → m ≈ 10 M bits. +- Bits per item meaning: 5 bits/item (k = 3) → p ≈ 9.2%; 10 → 0.82%; 20 (k = 14) → about 0.0067%. Each extra ~4.8 bits per item cuts p by about 10×. +- Memory, 1 million items: 10 bits each = 10 million bits = 1.25 MB. Storing 1 million URLs of about 50 bytes each = about 50 MB (estimate; the URL length is assumed), before hash-set overhead. +- Overfill: filter sized for 1 million items (m = 10 million, k = 7). With 2 million items p ≈ 13.8%; with 3 million, p ≈ 40%. +- The memory does not depend on item size: a 2 KB document and a 5-byte word each cost the same k bit-sets. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Overfilled (n above plan) | p rises from 0.82% to 40% at 3× | most bits become 1 | +| Delete by clearing bits | false negatives appear | a cleared bit can belong to another item | +| k too small | p high (k = 1: 9.5%) | each query checks only one bit | +| k too large | p high (k = 20: 5.5%) | each add sets many bits, the array fills | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| store bits, not items | a hash set of the items | 1 million URLs ≈ 50 MB vs 1.25 MB for the filter at 0.82% | +| k > 1 hash functions | one hash function (k = 1) | at 10 bits/item: 9.5% false positives vs 0.82% at k = 7 | +| not too many hash functions | k = 20 | 5.5%: each add sets 20 bits, so the array fills up | +| never clear bits (no delete) | clear the k bits to delete | add owl too, delete cat by clearing 1, 6, 11 → owl's bits 6, 11 are 0 → "definitely not seen" for owl, which is in the filter | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| "definitely not seen" is always correct (no false negatives) | cat → 1, 6, 11 set; query cat later finds all 1 | delete allowed: clearing cat's bits makes owl (added) read as absent | +| p ≈ (1 − e^(−kn/m))^k stays small | 1 M items, 10 M bits, k = 7: 0.82% | 3 M items in the same filter: 40% | +| mechanism: add/query | toy: cat, dog, fox, owl | — | + +## Terms + +- **bit array**: the m bits. +- **position**: an index into the bit array, 0 to m − 1, given by a hash function. +- **add**: set the k positions to 1. (Not "insert" or "store": the item itself is not kept.) +- **query**: read the k positions. +- **definitely not seen** / **probably seen**: the two answers. +- **false positive**: "probably seen" for an item never added. +- **false negative**: "definitely not seen" for an item that was added. +- **false-positive rate p**: low 0.0067% (20 bits/item) means almost never wrong; 0.82% (10 bits/item) about 1 in 120 never-added queries; 40% (overfilled) nearly useless. +- **bits per item m/n**: 10 in the main example. +- **fill f**: fraction of bits that are 1; 0.503 in the main example; near 0 the filter wastes memory, near 1 nearly every query says "probably seen". +- **capacity**: the n used to choose m and k (1 million in the main example). Overfilling = adding more than capacity. + +## Epistemic status + +- **Established:** the add/query rules; no false negatives without deletion. +- **Mathematical consequence (under an assumption):** p formula and best k, assuming hash positions act as independent uniform random choices. The formula is an approximation; the true rate is close for large m. +- **Estimated:** 50 bytes per URL. + +## Confusion points + +- "The filter stores the items" → it stores only bits; you cannot list or recover items. +- "It can say no wrongly" → no; a 0 bit proves the item was never added. +- "More hash functions is always better" → no; past (m/n)·ln 2 the array fills and p rises. +- "p is the chance an added item is reported wrongly" → p applies only to items never added. +- "Remove an item by clearing its bits" → causes false negatives. + +## Scope + +- **Out of scope:** counting Bloom filters (counters instead of bits, allow deletion) — search "counting Bloom filter". +- **Out of scope:** cuckoo filters, quotient filters — alternatives that support deletion. +- **Out of scope:** deriving k from two hashes (double hashing, Kirsch–Mitzenmacher) — how real code gets k positions cheaply. +- **Out of scope:** the exact (non-approximate) false-positive formula. +- **Out of scope:** growing a filter when n is unknown — changing m changes every position (hash mod m), so a resized filter gives false negatives; see "scalable Bloom filter". + +## Why this form (representation) + +Prose with one small bit-array table. The mechanism is a short sequence (add, query) and the failure is one case; a table of the toy array carries the picture. The parameter trade-off has three values, so a small table is enough; a slider page would add little. + +## Representation decision + +- **Stage:** 1 +- **Reason:** sequential mechanism + one counterexample + a 3-row trade-off; a table in prose shows all of it. +- **Levels:** L1 what it answers / L2 add and query / L3 why no false negatives, why false positives, the rate / L4 sizing and limits. diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/model.yaml b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/model.yaml new file mode 100644 index 0000000..c486ed8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/model.yaml @@ -0,0 +1,135 @@ +# Machine-readable half of model.md. `explainer check bloom-filter` holds every rendering to it. + +values: + toy_m: 16 + toy_k: 3 + toy_last_position: 15 + toy_cat: [1, 6, 11] + toy_dog: [3, 9, 14] + toy_fox: [2, 6, 14] + toy_owl: [3, 6, 11] + hash_example_output: 3141593 + hash_example_position: 9 + items_million: 1 + bits_per_item: 10 + m_bits_million: 10 + filter_mb: 1.25 + url_bytes: 50 + urls_mb: 50 + k_best: 7 + k_best_exact: 6.9 + kn_over_m: 0.7 + minus_kn_over_m: -0.7 + e_term: 0.497 + fill: 0.503 + p_percent: 0.82 + p_decimal: 0.0082 + one_in: 120 + queries: 1000000 + wasted_lookups: 8200 + k1_p_percent: 9.5 + k1_fill: 0.095 + k20: 20 + k20_p_percent: 5.5 + k20_fill: 0.865 + overfill_2x_million: 2 + overfill_2x_p_percent: 13.8 + overfill_3x_million: 3 + overfill_3x_p_percent: 40 + overfill_3x_wasted: 400000 + overfill_3x_kn_over_m: 2.1 + overfill_3x_minus_kn_over_m: -2.1 + overfill_3x_fill: 0.88 + best_k_fill: 0.5 + per_bit_factor: 0.62 + ten_x_bits: 4.8 + ln2: 0.69 + +allow: [0, 1, 2, 3, 4, 5, 8, 10, 12, 13, 100] + +require: + - path: p_percent + in: [prose] + - path: filter_mb + in: [prose] + - path: overfill_3x_p_percent + in: [prose] + - path: k1_p_percent + in: [prose] + +claims: + - id: mechanism-add-query + kind: mechanism + statement: "Add sets the item's k positions to 1. Query reads them: any 0 means definitely not seen; all 1 means probably seen." + example: "m = 16, k = 3. cat → 1, 6, 11; dog → 3, 9, 14. fox → 2, 6, 14: bit 2 is 0, definitely not seen." + ask: "A filter has bits 1, 3, 6, 9, 11, 14 set. A query's positions are 2, 6, 14. What does the filter answer, and can that answer be wrong?" + + - id: guarantee-no-false-negative + kind: guarantee + statement: "A 'definitely not seen' answer is always right, because add only sets bits to 1 and nothing sets them back to 0." + example: "cat sets 1, 6, 11; any later query for cat finds all three at 1." + counterexample: "Allow delete by clearing bits: add owl (3, 6, 11), then delete cat by clearing 1, 6, 11. Owl's bits 6 and 11 are now 0, so the filter says owl is definitely not seen, though owl was added." + ask: "Why can't you remove an item from a Bloom filter by setting its bits back to 0?" + + - id: mechanism-false-positive + kind: mechanism + statement: "A false positive happens when other items have already set all k positions of an item that was never added." + example: "owl → 3, 6, 11. Bit 3 came from dog, bits 6 and 11 from cat. The filter says probably seen; owl was never added." + + - id: why-bits-not-items + kind: why + statement: "The filter stores bits, not items, so memory depends on the number of items and the target error, not on item size." + alternative: "Store the items in a hash set." + counterexample: "1 million URLs of about 50 bytes = about 50 MB; a Bloom filter at 10 bits per item = 1.25 MB with 0.82% false positives." + + - id: why-several-hashes + kind: why + statement: "Several hash functions make a false positive need several 1 bits at once; too many fill the array." + alternative: "One hash function (k = 1), or many (k = 20)." + counterexample: "At 10 bits per item: k = 1 gives 9.5%; k = 7 gives 0.82%; k = 20 gives 5.5%." + ask: "At 10 bits per item, what happens to the false-positive rate if you use 20 hash functions instead of 7, and why?" + + - id: mechanism-rate + kind: mechanism + statement: "Each bit is still 0 with probability about e^(-kn/m), so the fill is f ≈ 1 − e^(-kn/m) and p ≈ f^k." + example: "m/n = 10, k = 7: e^(-0.7) = 0.497, f = 0.503, p = 0.503^7 = 0.0082." + + - id: limit-overfill + kind: guarantee + statement: "The false-positive rate stays small only while n stays near the capacity the filter was sized for." + example: "10 million bits, k = 7, 1 million items: 0.82%." + counterexample: "Same filter, 3 million items: 40%." + ask: "A filter sized for 1 million items gives 0.82% false positives. What happens if 3 million items are added?" + + - id: why-useful + kind: why + statement: "One-sided error is useful: put the filter before a slow exact lookup; 'definitely not seen' skips it, a false positive costs one wasted lookup." + alternative: "Do the slow lookup for every query." + counterexample: "1,000,000 queries for absent items: at 0.82%, about 8,200 lookups instead of 1,000,000." + +accept_unjustified: + ln: "ln 2 appears only in the best-k result (m/n)·ln 2, given as a result of the formula with its meaning (about half the bits are 1); deriving it needs calculus, out of scope for this reader." + +terms: + - term: add + means: "set the item's k positions to 1; the item itself is not kept" + avoid: [stores the item, stores the items, store the item] + - term: false positive + means: "'probably seen' for an item never added" + avoid: [false alarm, false hit] + - term: definitely not seen + means: "the answer when any position is 0" + avoid: [maybe not seen, probably not seen] + +budget: {prose: 1100, reason: "mechanism with a toy, two guarantees each with a breaking case, a derived rate with a worked number, and a use case"} + +quiz: + - q: "A Bloom filter says 'probably seen' for an item. Was the item added?" + expect: "Not necessarily: other items may have set all its bits (a false positive). Only 'definitely not seen' is certain." + misconception: "Yes, the item was certainly added." + - q: "You add more and more items to a fixed-size Bloom filter. What happens to its answers?" + expect: "More bits become 1, so false positives rise (0.82% at capacity to 40% at 3×). Items that were added still always read 'probably seen'." + misconception: "It starts forgetting items or gives false negatives." + - q: "Why does a Bloom filter use less memory than a hash set of the items?" + expect: "It keeps only about 10 bits per item, not the item; the price is a small chance of false positives." + misconception: "It compresses the items and can list them back." diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/narrative.md b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/narrative.md new file mode 100644 index 0000000..4bbb524 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/narrative.md @@ -0,0 +1,60 @@ +# Narrative: bloom-filter + +## 1. Reader before and after + +- **Before:** a technical reader (school maths incl. probability of independent events and e; arrays, bits, hash functions, hash sets). Has never seen a Bloom filter. +- **After:** "A Bloom filter sets k bits per item and checks those k bits later; a 0 proves the item is new, but all 1s can come from other items, so 'yes' is only probable, and it gets less reliable as the array fills." + +## 2. Question and motive + +- **Question:** How can a program answer "have I seen this before?" for millions of items without storing them, and when is the answer wrong? +- **Result in words:** keep a row of bits; each item turns on a few of them. If any of an item's bits is off, it is definitely new. If all are on, it is probably not new, and the filter is wrong only in that direction, with a rate you choose by memory per item. +- **Why care:** 1 million URLs take about 50 MB; the filter takes 1.25 MB and is wrong 0.82% of the time, only toward "probably seen". +- **Why this approach:** a hash function already turns an item into a fixed number. Turn the number into a bit position and remember only that the bit was hit. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| bit array, m | row of m bits, all 0 at the start | m = 16, positions 0–15 | 2 | +| position | index in the bit array, from hash mod m | 3,141,593 mod 16 = 9 | 2 | +| k | number of hash functions per item | k = 3 | 2 | +| add | set the item's k positions to 1 | cat → 1, 6, 11 | 2 | +| query | read the item's k positions | fox → 2, 6, 14 | 3 | +| definitely not seen / probably seen | the two answers | fox / owl | 3, 4 | +| false positive | "probably seen" for an item never added | owl | 4 | +| false negative | "definitely not seen" for an added item | owl after clearing cat | 5 | +| n | number of items added | 1 million | 6 | +| fill f | fraction of bits that are 1 | 0.503 | 6 | +| p | false-positive rate | 0.82% | 6 | +| m/n, bits per item | memory per item | 10 | 6 | +| capacity | the n the filter was sized for | 1 million | 8 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | Why not just a hash set? | — | 50 MB vs 1.25 MB | "A hash set of 1 million URLs (about 50 bytes each) takes about 50 MB. A Bloom filter keeps no URLs: it keeps a row of bits that each URL turned on, 10 bits per URL, 1.25 MB. It answers 'definitely not seen' or 'probably seen', and only the second answer can be wrong." | what the filter promises | +| 2 | mechanism | What does it store? | So | 16-bit table after cat, dog | "Start with m = 16 bits, all 0, numbered 0–15. Use k = 3 different hash functions. Each gives a large number; take it mod 16 to get a position (3,141,593 mod 16 = 9). To add an item, set its 3 positions to 1. cat → 1, 6, 11. dog → 3, 9, 14." | add, m, k, position | +| 3 | mechanism | How does it answer? | Then | fox query | "To query an item, compute the same 3 positions and read them. If any bit is 0, the answer is 'definitely not seen'. If all are 1, the answer is 'probably seen'. fox → 2, 6, 14; bit 2 is 0, so fox is definitely not seen." | query, both answers | +| 4 | notice | Can it be wrong? | But | owl query | "owl → 3, 6, 11, never added. Bit 3 came from dog; bits 6 and 11 from cat. All are 1, so the filter says 'probably seen'. This wrong 'probably seen' is a false positive." | false positive | +| 5 | guarantee | Can it be wrong the other way? | Therefore | cat query; delete case | "The other error, a false negative, is 'definitely not seen' for an item that was added. It cannot happen, because add only sets bits to 1 and nothing sets them back. But suppose you try to delete cat by clearing 1, 6, 11, after adding owl for real. Owl's bits 6 and 11 are now 0, so owl reads 'definitely not seen': a false negative. Bits are shared, so a Bloom filter has no delete." | one-sided error and why | +| 6 | mechanism | How often is it wrong? | So | 4-step derivation, 10 bits/item | "Let n be the number of items added, and assume each hash position is a uniform random choice, independent of the others. One bit-set misses a given bit with probability 1 − 1/m. n adds make kn bit-sets, so the bit stays 0 with probability (1 − 1/m)^(kn) ≈ e^(−kn/m). So the fill f, the fraction of bits that are 1, is about 1 − e^(−kn/m). A never-added item is a false positive when all k of its positions are 1: p ≈ f^k. With n = 1 million, m = 10 million (10 bits per item, m/n = 10) and k = 7: kn/m = 0.7, f = 0.503, p = 0.503^7 = 0.82%." | n, f, p, m/n | +| 7 | approach | Why k = 7, not 1 or 20? | But | 3-row table | "At 10 bits per item: k = 1 gives f = 0.095 but checks only one bit, p = 9.5%. k = 20 makes f = 0.865, so nearly every bit is 1, p = 5.5%. k = 7 gives f ≈ 0.5, the balance, p = 0.82%. The best k is (m/n)·ln 2." | k trade-off | +| 8 | limit | What if I add too many? | And | 2× → 13.8%, 3× → 40% | "The filter was sized for a capacity of 1 million items. Add 2 million and p is 13.8%; 3 million, 40%. Added items still always read 'probably seen'." | capacity | +| 9 | payoff | What is it good for? | So | filter before disk | "Check the filter before a slow lookup on disk. 'Definitely not seen' skips the lookup, safely, because there are no false negatives. 'Probably seen' does the lookup, which gives the true answer. Of 1,000,000 queries for absent items, about 8,200 do a wasted lookup." | use | +| 10 | close | Answered? | — | — | "A Bloom filter remembers only the bits each item turned on. A 0 bit proves an item is new. All 1s mean 'probably seen', wrong at rate p, which rises as the array fills." | after-sentence | + +## 5. Concrete to symbol + +Toy first (m = 16, k = 3, two items), then the symbols m, k, n bound aloud, then the formula with m/n = 10, k = 7. + +## 6. Links between representations + +The toy table shows bit numbers above, 0/1 below, and which item set each bit. + +## 7. Close the loop + +- **Answer:** It keeps only bits each item turned on. A 0 proves "new"; all 1s mean "probably seen", wrong at rate p. +- **General argument:** bits only go from 0 to 1, so added items always pass; other items' bits can cover a new item, with probability f^k. +- **Payoff:** a cheap pre-check before slow lookups. Scope: deletion (counting Bloom filters), growing, k from two hashes. diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/decisions.yaml new file mode 100644 index 0000000..b87b0fd --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/decisions.yaml @@ -0,0 +1,211 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T150533-probe#1: + decision: adopted +2026-10-04T150533-probe#10: + decision: declined +2026-10-04T150533-probe#11: + decision: declined +2026-10-04T150533-probe#12: + decision: declined +2026-10-04T150533-probe#2: + decision: adopted +2026-10-04T150533-probe#3: + decision: adopted +2026-10-04T150533-probe#4: + decision: declined +2026-10-04T150533-probe#5: + decision: declined +2026-10-04T150533-probe#6: + decision: adopted +2026-10-04T150533-probe#7: + decision: adopted +2026-10-04T150533-probe#8: + decision: adopted +2026-10-04T150533-probe#9: + decision: declined +2026-10-04T150725-coldread-narrative#1: + decision: adopted +2026-10-04T150725-coldread-narrative#10: + decision: adopted +2026-10-04T150725-coldread-narrative#11: + decision: adopted +2026-10-04T150725-coldread-narrative#12: + decision: adopted +2026-10-04T150725-coldread-narrative#13: + decision: adopted +2026-10-04T150725-coldread-narrative#14: + decision: adopted +2026-10-04T150725-coldread-narrative#15: + decision: adopted +2026-10-04T150725-coldread-narrative#16: + decision: adopted +2026-10-04T150725-coldread-narrative#17: + decision: adopted +2026-10-04T150725-coldread-narrative#18: + decision: adopted +2026-10-04T150725-coldread-narrative#19: + decision: adopted +2026-10-04T150725-coldread-narrative#2: + decision: adopted +2026-10-04T150725-coldread-narrative#20: + decision: adopted +2026-10-04T150725-coldread-narrative#21: + decision: adopted +2026-10-04T150725-coldread-narrative#22: + decision: adopted +2026-10-04T150725-coldread-narrative#23: + decision: adopted +2026-10-04T150725-coldread-narrative#24: + decision: adopted +2026-10-04T150725-coldread-narrative#25: + decision: adopted +2026-10-04T150725-coldread-narrative#26: + decision: adopted +2026-10-04T150725-coldread-narrative#27: + decision: adopted +2026-10-04T150725-coldread-narrative#28: + decision: adopted +2026-10-04T150725-coldread-narrative#29: + decision: adopted +2026-10-04T150725-coldread-narrative#3: + decision: adopted +2026-10-04T150725-coldread-narrative#30: + decision: adopted +2026-10-04T150725-coldread-narrative#31: + decision: adopted +2026-10-04T150725-coldread-narrative#32: + decision: adopted +2026-10-04T150725-coldread-narrative#33: + decision: adopted +2026-10-04T150725-coldread-narrative#34: + decision: adopted +2026-10-04T150725-coldread-narrative#35: + decision: adopted +2026-10-04T150725-coldread-narrative#36: + decision: declined +2026-10-04T150725-coldread-narrative#37: + decision: adopted +2026-10-04T150725-coldread-narrative#38: + decision: adopted +2026-10-04T150725-coldread-narrative#39: + decision: adopted +2026-10-04T150725-coldread-narrative#4: + decision: adopted +2026-10-04T150725-coldread-narrative#40: + decision: adopted +2026-10-04T150725-coldread-narrative#41: + decision: adopted +2026-10-04T150725-coldread-narrative#5: + decision: adopted +2026-10-04T150725-coldread-narrative#6: + decision: adopted +2026-10-04T150725-coldread-narrative#7: + decision: adopted +2026-10-04T150725-coldread-narrative#8: + decision: adopted +2026-10-04T150725-coldread-narrative#9: + decision: adopted +2026-10-04T150831-coldread-narrative#1: + decision: adopted +2026-10-04T150831-coldread-narrative#10: + decision: adopted +2026-10-04T150831-coldread-narrative#2: + decision: adopted +2026-10-04T150831-coldread-narrative#3: + decision: declined +2026-10-04T150831-coldread-narrative#4: + decision: adopted +2026-10-04T150831-coldread-narrative#5: + decision: adopted +2026-10-04T150831-coldread-narrative#6: + decision: adopted +2026-10-04T150831-coldread-narrative#7: + decision: adopted +2026-10-04T150831-coldread-narrative#8: + decision: adopted +2026-10-04T150831-coldread-narrative#9: + decision: declined +2026-10-04T151003-coldread-prose#1: + decision: adopted +2026-10-04T151003-coldread-prose#10: + decision: declined +2026-10-04T151003-coldread-prose#11: + decision: declined +2026-10-04T151003-coldread-prose#12: + decision: declined +2026-10-04T151003-coldread-prose#13: + decision: declined +2026-10-04T151003-coldread-prose#14: + decision: declined +2026-10-04T151003-coldread-prose#2: + decision: adopted +2026-10-04T151003-coldread-prose#3: + decision: adopted +2026-10-04T151003-coldread-prose#4: + decision: declined +2026-10-04T151003-coldread-prose#5: + decision: declined +2026-10-04T151003-coldread-prose#6: + decision: adopted +2026-10-04T151003-coldread-prose#7: + decision: adopted +2026-10-04T151003-coldread-prose#8: + decision: adopted +2026-10-04T151003-coldread-prose#9: + decision: adopted +2026-10-04T151043-quiz-prose#1: + decision: adopted +2026-10-04T151043-quiz-prose#10: + decision: adopted +2026-10-04T151043-quiz-prose#11: + decision: declined +2026-10-04T151043-quiz-prose#12: + decision: adopted +2026-10-04T151043-quiz-prose#13: + decision: adopted +2026-10-04T151043-quiz-prose#14: + decision: declined +2026-10-04T151043-quiz-prose#2: + decision: declined +2026-10-04T151043-quiz-prose#3: + decision: declined +2026-10-04T151043-quiz-prose#4: + decision: adopted +2026-10-04T151043-quiz-prose#5: + decision: adopted +2026-10-04T151043-quiz-prose#6: + decision: adopted +2026-10-04T151043-quiz-prose#7: + decision: adopted +2026-10-04T151043-quiz-prose#8: + decision: adopted +2026-10-04T151043-quiz-prose#9: + decision: declined +2026-10-04T151208-quiz-prose#1: + decision: adopted +2026-10-04T151208-quiz-prose#10: + decision: declined +2026-10-04T151208-quiz-prose#11: + decision: declined +2026-10-04T151208-quiz-prose#12: + decision: declined +2026-10-04T151208-quiz-prose#13: + decision: declined +2026-10-04T151208-quiz-prose#14: + decision: declined +2026-10-04T151208-quiz-prose#2: + decision: declined +2026-10-04T151208-quiz-prose#3: + decision: declined +2026-10-04T151208-quiz-prose#4: + decision: declined +2026-10-04T151208-quiz-prose#5: + decision: declined +2026-10-04T151208-quiz-prose#6: + decision: declined +2026-10-04T151208-quiz-prose#7: + decision: declined +2026-10-04T151208-quiz-prose#8: + decision: declined +2026-10-04T151208-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150533-probe.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150533-probe.json new file mode 100644 index 0000000..e3725c0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150533-probe.json @@ -0,0 +1,185 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "bloom-filter", + "at": "2026-10-04T15:05:33", + "model": "default", + "prompt_sha": "7ab239541a1d", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "The derivation of p ≈ (1 − e^(−kn/m))^k skips steps and never connects the fill fraction to p", + "why_it_matters": "The reader is shown the final formula but not why. Three steps are missing. First, one add misses a given bit with probability 1 − 1/m. Second, there are kn independent tries, so the bit is still 0 with probability (1 − 1/m)^(kn). Third, (1 − 1/m)^(kn) ≈ e^(−kn/m), and the audience knows e but not this limit. The final step, p = (fraction of 1 bits)^k, also assumes a never-added item's k positions are independent of each other and may repeat. None of this is stated, so the formula appears without support.", + "suggested_addition": "Work through the main example. With m/n = 10 and k = 7: kn/m = 0.7 and e^(−0.7) ≈ 0.497, so the fraction of 1 bits is ≈ 0.503 and p ≈ 0.503^7 ≈ 0.0082. Check the approximation on the toy: (15/16)^6 ≈ 0.679 versus e^(−0.375) ≈ 0.687. Say that p is the chance that k independent uniform draws all land on 1 bits." + }, + { + "rule": 4, + "severity": "edge", + "about": "The toy 'rough rate' and the formula disagree, and the file doesn't say why", + "why_it_matters": "The toy uses the observed fill, 6/16 = 0.375, so p ≈ 5.3%. The formula with m = 16, n = 2, k = 3 predicts a fill of 1 − e^(−0.375) ≈ 0.31, so p ≈ 3.1%. The gap comes from the toy positions being chosen with no collisions. A careful reader will compute both numbers and get confused.", + "suggested_addition": "Say that (6/16)^3 ≈ 5.3% uses the bits actually set. The formula predicts the average fill, ≈ 0.31, which gives p ≈ 3.1%. Random hashes would often collide and set fewer than 6 bits." + }, + { + "rule": 1, + "severity": "edge", + "about": "Best k = (m/n)·ln 2 and 'about half the bits are 1' are stated without a reason", + "why_it_matters": "Without a reason, ln 2 looks arbitrary, and 'half full' isn't linked to it.", + "suggested_addition": "Intuition: when the array is half full, each bit tells you the most. At k = 7, m/n = 10, the fill is 0.503. At k = 1 the fill is 0.095: most bits are 0, so each check carries little information. At k = 20 the fill is 0.865: almost every bit is 1, so checks rarely find a 0. Optionally add that at best k, p ≈ 0.5^k = 0.6185^(m/n). That is where 'each ~4.8 bits per item cuts p by 10×' comes from, since ln 10 / ln(1/0.6185) ≈ 4.79." + }, + { + "rule": 2, + "severity": "edge", + "about": "The no-false-negatives guarantee also assumes the query uses the same m and the same hash functions as the add. Only deletion is shown as breaking it.", + "why_it_matters": "A reader may try to 'grow' a full filter by enlarging the array. That silently breaks the guarantee.", + "suggested_addition": "Breaking case: resize the toy array from m = 16 to m = 32 without re-adding. If the positions are hash mod m, cat's positions change (for example 1 → 17), the new bit is 0, and cat reads as 'definitely not seen'." + }, + { + "rule": 2, + "severity": "edge", + "about": "The formula's random-hash assumption has no breaking instance", + "why_it_matters": "Epistemic status names the assumption, but Concrete cases breaks the formula only by overfilling, never by violating the assumption itself.", + "suggested_addition": "Breaking case: all 7 hash functions are identical. Each item then sets 1 bit, the fill is ≈ 9.5% at m/n = 10, and p ≈ 9.5% instead of 0.82%. The formula's prediction is off by 11×." + }, + { + "rule": 5, + "severity": "main", + "about": "The file never says what a 'probably seen' answer costs, or why an error that runs in only one direction is useful", + "why_it_matters": "The central question is 'have I seen this before?' The reader needs to know that a false positive usually costs one wasted exact check, never a wrong final answer. Without that, 0.82% means nothing in practice.", + "suggested_addition": "Use case: check the filter before a disk or database lookup. With 1,000,000 queries for absent URLs at p = 0.82%, about 8,200 queries hit the disk unnecessarily, and about 991,800 are answered from 1.25 MB of RAM. At p = 40%, about 400,000 hit the disk." + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts have more than one name, and some are used before they are defined", + "why_it_matters": "The fill fraction is called 'fraction of bits that are 1', 'more 1s', 'fill' and 'fills' (in Relationships, Alternative states and Confusion points), but it is not in Terms. Relationships uses n/m while the rest of the file uses bits per item m/n. Planned capacity is called 'n above plan' and 'sized for 1 million items' but is never defined. 'Rate' means both the toy's observed-fill estimate and p. 'Position' and 'bit' are used interchangeably.", + "suggested_addition": "Define fill f = fraction of bits that are 1 (for example 0.503 at m/n = 10, k = 7). Use m/n everywhere. Define capacity n₀ = the n used to choose m and k (1,000,000 in the example). Call the toy figure 'estimate from the observed fill'." + }, + { + "rule": 4, + "severity": "edge", + "about": "The step from hash output to position has no worked example", + "why_it_matters": "The audience knows a hash gives a large number but not how it becomes an index from 0 to m − 1. The toy positions are only stipulated.", + "suggested_addition": "Example: h1(\"cat\") = 3,141,593 and 3,141,593 mod 16 = 9 (an illustrative value). Repeat with h2 and h3 to get k positions." + }, + { + "rule": 1, + "severity": "edge", + "about": "The file doesn't say why each slot is 1 bit rather than a byte or a boolean", + "why_it_matters": "A programmer's natural alternative is a bool array, which typically uses 1 byte per slot.", + "suggested_addition": "With 1 byte per slot, 10 M slots cost 10 MB instead of 1.25 MB: 8× the memory for no gain in accuracy, since each slot only ever holds 0 or 1." + } + ], + "excess": [ + "The 'Why this form (representation)' and 'Representation decision' sections (stage, levels) are rendering metadata, not content for the reader", + "The k = 20 example appears in three places (Quantities, Alternative states, Why this form). One is enough.", + "The 20 bits/item, p ≈ 0.0067% data point is not needed once the rule '~4.8 bits per item per 10×' is given" + ], + "next_questions": [ + { + "q": "What do I actually do with 'probably seen'? Where is a Bloom filter used, and what does a false positive cost?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Given my n and a target p, how do I choose m and k (for example m = −n·ln p / (ln 2)^2)?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How do real implementations get k independent hash functions cheaply?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "If I can't delete by clearing bits, how do I support deletion?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "What if I don't know n in advance? Can a filter grow without breaking the no-false-negatives guarantee?", + "answered_by_model": false, + "declared_out_of_scope": false + } + ] + }, + "usage": { + "cost": 0.1518542, + "seconds": 55 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "The derivation of p ≈ (1 − e^(−kn/m))^k skips steps and never connects the fill fraction to p → Work through the main example. With m/n = 10 and k = 7: kn/m = 0.7 and e^(−0.7) ≈ 0.497, so the fraction of 1 bits is ≈ 0.503 and p ≈ 0.503^7 ≈ 0.0082. Check the approximation on the toy: (15/16)^6 ≈ 0.679 versus e^(−0.375) ≈ 0.687. Say that p is the chance that k independent uniform draws all land on 1 bits." + }, + { + "n": 2, + "severity": "edge", + "kind": "rule 4", + "text": "The toy 'rough rate' and the formula disagree, and the file doesn't say why → Say that (6/16)^3 ≈ 5.3% uses the bits actually set. The formula predicts the average fill, ≈ 0.31, which gives p ≈ 3.1%. Random hashes would often collide and set fewer than 6 bits." + }, + { + "n": 3, + "severity": "edge", + "kind": "rule 1", + "text": "Best k = (m/n)·ln 2 and 'about half the bits are 1' are stated without a reason → Intuition: when the array is half full, each bit tells you the most. At k = 7, m/n = 10, the fill is 0.503. At k = 1 the fill is 0.095: most bits are 0, so each check carries little information. At k = 20 the fill is 0.865: almost every bit is 1, so checks rarely find a 0. Optionally add that at best k, p ≈ 0.5^k = 0.6185^(m/n). That is where 'each ~4.8 bits per item cuts p by 10×' comes from, since ln 10 / ln(1/0.6185) ≈ 4.79." + }, + { + "n": 4, + "severity": "edge", + "kind": "rule 2", + "text": "The no-false-negatives guarantee also assumes the query uses the same m and the same hash functions as the add. Only deletion is shown as breaking it. → Breaking case: resize the toy array from m = 16 to m = 32 without re-adding. If the positions are hash mod m, cat's positions change (for example 1 → 17), the new bit is 0, and cat reads as 'definitely not seen'." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 2", + "text": "The formula's random-hash assumption has no breaking instance → Breaking case: all 7 hash functions are identical. Each item then sets 1 bit, the fill is ≈ 9.5% at m/n = 10, and p ≈ 9.5% instead of 0.82%. The formula's prediction is off by 11×." + }, + { + "n": 6, + "severity": "main", + "kind": "rule 5", + "text": "The file never says what a 'probably seen' answer costs, or why an error that runs in only one direction is useful → Use case: check the filter before a disk or database lookup. With 1,000,000 queries for absent URLs at p = 0.82%, about 8,200 queries hit the disk unnecessarily, and about 991,800 are answered from 1.25 MB of RAM. At p = 40%, about 400,000 hit the disk." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts have more than one name, and some are used before they are defined → Define fill f = fraction of bits that are 1 (for example 0.503 at m/n = 10, k = 7). Use m/n everywhere. Define capacity n₀ = the n used to choose m and k (1,000,000 in the example). Call the toy figure 'estimate from the observed fill'." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 4", + "text": "The step from hash output to position has no worked example → Example: h1(\"cat\") = 3,141,593 and 3,141,593 mod 16 = 9 (an illustrative value). Repeat with h2 and h3 to get k positions." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 1", + "text": "The file doesn't say why each slot is 1 bit rather than a byte or a boolean → With 1 byte per slot, 10 M slots cost 10 MB instead of 1.25 MB: 8× the memory for no gain in accuracy, since each slot only ever holds 0 or 1." + }, + { + "n": 10, + "severity": "excess", + "kind": "model entry", + "text": "The 'Why this form (representation)' and 'Representation decision' sections (stage, levels) are rendering metadata, not content for the reader" + }, + { + "n": 11, + "severity": "excess", + "kind": "model entry", + "text": "The k = 20 example appears in three places (Quantities, Alternative states, Why this form). One is enough." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "The 20 bits/item, p ≈ 0.0067% data point is not needed once the rule '~4.8 bits per item per 10×' is given" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150725-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150725-coldread-narrative.json new file mode 100644 index 0000000..d11a5a4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150725-coldread-narrative.json @@ -0,0 +1,443 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "bloom-filter", + "at": "2026-10-04T15:07:25", + "model": "default", + "prompt_sha": "84de7cead935", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: 50 MB vs 1.25 MB. Said: answer first: bits, one-sided error", + "unresolved": [ + "one-sided error", + "bits (bits of what, set by what?)" + ], + "unsaid": [ + "50 MB vs 1.25 MB: the words never say what is being stored (how many items, which kinds) or which number belongs to the hash set and which to the filter" + ], + "leap": [ + "How can storing 'bits' replace storing the items? Nothing yet links an item to a bit." + ], + "excess": [ + "answer first: this is a stage direction, not something the reader can use" + ], + "purpose_unclear": true + }, + { + "at": "Beat 2 (mechanism)", + "text": "Shown: 16-bit table after cat, dog. Said: hash mod m → position; add sets k bits", + "unresolved": [ + "m", + "k", + "add" + ], + "unsaid": [ + "16-bit table: the words never say that m = 16 here, or which positions cat and dog set", + "how one item gets k different positions (k hash functions?) is never said" + ], + "leap": [ + "'hash mod m → position' gives one position, but 'add sets k bits' needs k of them. Where do the other k−1 come from?" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Shown: fox query. Said: any 0 → definitely not seen", + "unresolved": [], + "unsaid": [ + "fox's positions, and which of them is 0, are not said" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (notice)", + "text": "Shown: owl query. Said: bits from cat and dog cover owl → false positive", + "unresolved": [ + "false positive (used as a label, never defined as 'probably seen' for an item never added)" + ], + "unsaid": [ + "the answer 'probably seen' when all bits are 1 is never said as a rule. Only the 0 case was stated in beat 3", + "owl's positions and which bits came from cat and which from dog" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (guarantee)", + "text": "Shown: cat query; delete case. Said: bits never go to 0 → no false negatives; clearing breaks it", + "unresolved": [ + "false negatives", + "clearing" + ], + "unsaid": [ + "delete case: nothing in the words introduces deletion or says why it comes up here" + ], + "leap": [ + "'clearing breaks it': the reason (a cleared bit may be shared with another added item, such as owl's bit from cat) is not given" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Shown: 4-step derivation, 10 bits/item. Said: f ≈ 1 − e^(−kn/m), p ≈ f^k = 0.82%", + "unresolved": [ + "f", + "n", + "p", + "bits/item (as m/n)" + ], + "unsaid": [ + "the 4-step derivation is shown but none of its steps are spoken", + "the value of k behind 0.82% (k = 7) is not said", + "10 bits/item is not tied to m/n in the words" + ], + "leap": [ + "why f ≈ 1 − e^(−kn/m): the chance a given bit is missed by one hash, raised to the power kn, is not given", + "why p ≈ f^k: the independence assumption is not stated" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (approach)", + "text": "Shown: 3-row table. Said: too few checks vs too full; best k at f ≈ 0.5", + "unresolved": [ + "k = 7 (in the reader's question, but no beat has said k = 7)" + ], + "unsaid": [ + "the 3-row table's contents: which k values and which p values" + ], + "leap": [ + "why the best k lands at f ≈ 0.5 is asserted, not shown" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (limit)", + "text": "Shown: 2× → 13.8%, 3× → 40%. Said: p depends on n", + "unresolved": [ + "2×, 3× (twice and three times what? The intended n is never said aloud)", + "capacity (in 'Reader now knows' but never spoken)" + ], + "unsaid": [ + "the shown percentages are not mentioned in the words" + ], + "leap": [], + "excess": [ + "p depends on n: already plain from the beat 6 formula, so nothing new is said aloud" + ], + "purpose_unclear": false + }, + { + "at": "Beat 9 (payoff)", + "text": "Shown: filter before disk. Said: a false positive costs one wasted lookup", + "unresolved": [], + "unsaid": [ + "filter before disk: the setup (check the filter, then go to disk only on 'probably seen') is not said" + ], + "leap": [ + "why it is safe: no false negatives means a real item is never skipped. This is not linked back to beat 5" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 10 (close)", + "text": "Shown: —. Said: answer the question in its words", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": true + } + ], + "blocking": [ + "Beat 2: 'add sets k bits'. k is undefined, and one 'hash mod m' gives one position, not k, so the core mechanism can't be rebuilt", + "Beat 4: 'false positive'. The rule 'all 1s → probably seen' is never said, so the reader cannot tell why owl gets the wrong answer", + "Beat 6: 'f ≈ 1 − e^(−kn/m), p ≈ f^k = 0.82%'. f, n and p are undefined, the derivation is shown but not spoken, and k = 7 is missing", + "Beat 10: 'answer the question in its words'. A placeholder: the close says nothing" + ], + "cuts": [ + "Beat 1 'answer first:' and beat 10's placeholder: replace with the real sentences, or drop them", + "Beat 8 'p depends on n': already follows from beat 6. Say the 2×/3× numbers instead" + ], + "question": { + "text": "How can a program answer 'have I seen this before?' for many items without storing them, and when is the answer wrong?", + "known_at": "Only implied at beat 1 (reader's question 'Why not just a hash set?'). The Said column never asks it.", + "result_stated": "No, not in full words. Beat 1 gives only 'bits, one-sided error', and beat 10 is a placeholder." + }, + "motive": "Reason to care: only hinted by beat 1 Shown '50 MB vs 1.25 MB', with nothing said about what is stored. Reason for the approach (a hash already maps an item to a number, so keep one bit for it): missing from the beats.", + "close": "Not answered. Beat 10 Said is 'answer the question in its words' with nothing shown. The general argument (bits only go 0→1, so added items always pass; others' bits cover a new item with probability f^k) is never said at the end.", + "recap": "A Bloom filter turns on a few bits per item; any off bit means definitely new, all on means probably seen, and it gets less reliable as it fills.", + "strongest": "Beat 5: 'bits never go to 0 → no false negatives'. A clear one-line reason for the one-sided guarantee.", + "weakest": "Beat 6: the formula arrives with f, n and p undefined and the derivation unspoken. Beat 10 is a close close behind, as an empty placeholder.", + "suggestions": [ + "In beat 2, say 'k different hash functions, each mod m, give k positions; here m = 16, k = 3, and cat sets 1, 6, 11'.", + "In beat 3 or 4, state the rule 'all k bits 1 → probably seen' before showing owl.", + "In beat 6, define n, f and p aloud, speak the steps (chance one bit is missed = (1 − 1/m)^(kn) ≈ e^(−kn/m)), and say that k = 7 gives 0.82%.", + "In beat 1, say '1 million URLs: a hash set takes about 50 MB, the filter 1.25 MB, wrong 0.82% of the time, and only toward \"probably seen\"'.", + "In beat 5, give the reason clearing fails: a bit may be shared with another added item.", + "Write beat 10 using the plan's own answer and general argument from section 7." + ] + }, + "usage": { + "cost": 0.10809079999999999, + "seconds": 36 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'add sets k bits'. k is undefined, and one 'hash mod m' gives one position, not k, so the core mechanism can't be rebuilt" + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'false positive'. The rule 'all 1s → probably seen' is never said, so the reader cannot tell why owl gets the wrong answer" + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 6: 'f ≈ 1 − e^(−kn/m), p ≈ f^k = 0.82%'. f, n and p are undefined, the derivation is shown but not spoken, and k = 7 is missing" + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 10: 'answer the question in its words'. A placeholder: the close says nothing" + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 1 'answer first:' and beat 10's placeholder: replace with the real sentences, or drop them" + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 8 'p depends on n': already follows from beat 6. Say the 2×/3× numbers instead" + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): one-sided error" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): bits (bits of what, set by what?)" + }, + { + "n": 9, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): 50 MB vs 1.25 MB: the words never say what is being stored (how many items, which kinds) or which number belongs to the hash set and which to the filter" + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): How can storing 'bits' replace storing the items? Nothing yet links an item to a bit." + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): m" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): k" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): add" + }, + { + "n": 14, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): 16-bit table: the words never say that m = 16 here, or which positions cat and dog set" + }, + { + "n": 15, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): how one item gets k different positions (k hash functions?) is never said" + }, + { + "n": 16, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (mechanism): 'hash mod m → position' gives one position, but 'add sets k bits' needs k of them. Where do the other k−1 come from?" + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): fox's positions, and which of them is 0, are not said" + }, + { + "n": 18, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (notice): false positive (used as a label, never defined as 'probably seen' for an item never added)" + }, + { + "n": 19, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (notice): the answer 'probably seen' when all bits are 1 is never said as a rule. Only the 0 case was stated in beat 3" + }, + { + "n": 20, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (notice): owl's positions and which bits came from cat and which from dog" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (guarantee): false negatives" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (guarantee): clearing" + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (guarantee): delete case: nothing in the words introduces deletion or says why it comes up here" + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (guarantee): 'clearing breaks it': the reason (a cleared bit may be shared with another added item, such as owl's bit from cat) is not given" + }, + { + "n": 25, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): f" + }, + { + "n": 26, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): n" + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): p" + }, + { + "n": 28, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): bits/item (as m/n)" + }, + { + "n": 29, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): the 4-step derivation is shown but none of its steps are spoken" + }, + { + "n": 30, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): the value of k behind 0.82% (k = 7) is not said" + }, + { + "n": 31, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): 10 bits/item is not tied to m/n in the words" + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): why f ≈ 1 − e^(−kn/m): the chance a given bit is missed by one hash, raised to the power kn, is not given" + }, + { + "n": 33, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): why p ≈ f^k: the independence assumption is not stated" + }, + { + "n": 34, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (approach): k = 7 (in the reader's question, but no beat has said k = 7)" + }, + { + "n": 35, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (approach): the 3-row table's contents: which k values and which p values" + }, + { + "n": 36, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (approach): why the best k lands at f ≈ 0.5 is asserted, not shown" + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (limit): 2×, 3× (twice and three times what? The intended n is never said aloud)" + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (limit): capacity (in 'Reader now knows' but never spoken)" + }, + { + "n": 39, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (limit): the shown percentages are not mentioned in the words" + }, + { + "n": 40, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 9 (payoff): filter before disk: the setup (check the filter, then go to disk only on 'probably seen') is not said" + }, + { + "n": 41, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (payoff): why it is safe: no false negatives means a real item is never skipped. This is not linked back to beat 5" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150831-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150831-coldread-narrative.json new file mode 100644 index 0000000..2bea3a1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T150831-coldread-narrative.json @@ -0,0 +1,209 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "bloom-filter", + "at": "2026-10-04T15:08:31", + "model": "default", + "prompt_sha": "84de7cead935", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Hash set of 1M URLs ≈ 50 MB; Bloom filter keeps 10 bits per URL, 1.25 MB; answers 'definitely not seen' / 'probably seen', only the second can be wrong.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (mechanism)", + "text": "m = 16 bits, k = 3 hash functions, hash mod 16 gives a position; add sets 3 positions; cat → 1, 6, 11; dog → 3, 9, 14.", + "unresolved": [], + "unsaid": [ + "The shown '16-bit table after cat, dog' may mark which item set each bit, but the words never say so." + ], + "leap": [ + "'(3,141,593 mod 16 = 9)' is not linked to any item. Position 9 later turns up under dog, so the reader may stop to check whether this was dog's hash." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Query reads the same 3 positions; any 0 → definitely not seen; all 1 → probably seen; fox → 2, 6, 14, bit 2 is 0.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (notice)", + "text": "owl → 3, 6, 11, never added, all bits set by dog and cat → false positive.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (guarantee)", + "text": "No false negatives because bits only go to 1; deleting cat after adding owl would cause one; so no delete.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Derivation: miss prob 1 − 1/m, (1 − 1/m)^(kn) ≈ e^(−kn/m), f ≈ 1 − e^(−kn/m), p ≈ f^k; with m/n = 10, k = 7 → p = 0.82%.", + "unresolved": [ + "'p' is never named as the false-positive rate. The beat says 'p ≈ f^k' straight after 'is a false positive when…', so the reader has to infer what p means." + ], + "unsaid": [], + "leap": [ + "'(1 − 1/m)^(kn) ≈ e^(−kn/m)': the step (1 − 1/m)^m ≈ 1/e is not stated.", + "'p ≈ f^k': it is not said that the fill fraction f is used as the probability that one random position is 1, or that the k positions are treated as independent for this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (approach)", + "text": "k = 1 → p = 9.5%, k = 20 → p = 5.5%, k = 7 → f ≈ 0.5, p = 0.82%; best k = (m/n)·ln 2.", + "unresolved": [], + "unsaid": [], + "leap": [ + "'The best k is (m/n)·ln 2' is stated with no reason, and the beat never says that 10·ln 2 ≈ 6.9 rounds to 7.", + "'f ≈ 0.5, the balance': it is not said why half-full is the optimum." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (limit)", + "text": "Sized for 1M; 2M → 13.8%, 3M → 40%; added items still read 'probably seen'.", + "unresolved": [], + "unsaid": [], + "leap": [ + "The beat does not say that m and k stay fixed (10M bits, k = 7) while n grows." + ], + "excess": [ + "'Added items still always read \"probably seen\"' repeats beat 5 with nothing new." + ], + "purpose_unclear": false + }, + { + "at": "Beat 9 (payoff)", + "text": "Filter before a disk lookup; 'definitely not seen' skips it; ≈ 8,200 wasted lookups per 1M absent queries.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [ + "'safely, because there are no false negatives' restates beat 5." + ], + "purpose_unclear": false + }, + { + "at": "Beat 10 (close)", + "text": "Remembers only bits; a 0 proves new; all 1s → probably seen, wrong at rate p, rising as the array fills.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Beat 8: drop 'Added items still always read \"probably seen\"' (repeats beat 5).", + "Beat 9: shorten 'safely, because there are no false negatives' to 'safely' (repeats beat 5)." + ], + "question": { + "text": "How can a program check 'have I seen this?' for millions of items without storing the items, and when is that answer wrong?", + "known_at": "Beat 1. It is implied by the 50 MB vs 1.25 MB contrast, but the words never ask it as a question.", + "result_stated": "Yes. Beat 1: 'It answers \"definitely not seen\" or \"probably seen\", and only the second answer can be wrong.' Beat 10: 'A 0 bit proves an item is new. All 1s mean \"probably seen\", wrong at rate p, which rises as the array fills.'" + }, + "motive": "Reason to care: beat 1, 'A hash set of 1 million URLs ... takes about 50 MB ... 10 bits per URL, 1.25 MB', and beat 9, 'Check the filter before a slow lookup on disk.' Reason for the approach: partly given in beat 1, 'keeps no URLs: it keeps a row of bits that each URL turned on'. The words never say why hashing to bit positions is the natural way to do this.", + "close": "Mostly. Beat 10: 'A Bloom filter remembers only the bits each item turned on. A 0 bit proves an item is new. All 1s mean \"probably seen\", wrong at rate p, which rises as the array fills.' It answers the question. It does not repeat the reason a 0 is proof (bits only ever go from 0 to 1), and it does not mention that the rate is set by memory per item.", + "recap": "A Bloom filter hashes each item to k bits and sets them; a 0 among an item's bits proves it is new, all 1s only means 'probably seen', and with about 10 bits per item it is wrong under 1% of the time, getting worse as it fills.", + "strongest": "Beats 4–5: the owl example produces a false positive from cat's and dog's bits, then reuses owl to show why deleting creates false negatives.", + "weakest": "Beat 7: 'The best k is (m/n)·ln 2' and 'f ≈ 0.5, the balance' are given with no reason. Beat 6's undefined 'p' is a close second.", + "suggestions": [ + "In beat 6, name p aloud: 'call the false-positive rate p'.", + "In beat 6, add one clause: (1 − 1/m)^m ≈ 1/e for large m.", + "In beat 7, say that 10·ln 2 ≈ 6.9, hence k = 7. Either give the one-line reason for the optimum or mark it as stated without proof.", + "In beat 2, tie the mod example to an item, e.g. 'dog's second hash is 3,141,593; mod 16 = 9'.", + "In beat 8, say that m and k stay fixed while n grows.", + "In beat 10, add 'because bits only ever go from 0 to 1' so the close carries the general argument." + ] + }, + "usage": { + "cost": 0.1236152, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Beat 8: drop 'Added items still always read \"probably seen\"' (repeats beat 5)." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Beat 9: shorten 'safely, because there are no false negatives' to 'safely' (repeats beat 5)." + }, + { + "n": 3, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): The shown '16-bit table after cat, dog' may mark which item set each bit, but the words never say so." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (mechanism): '(3,141,593 mod 16 = 9)' is not linked to any item. Position 9 later turns up under dog, so the reader may stop to check whether this was dog's hash." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): 'p' is never named as the false-positive rate. The beat says 'p ≈ f^k' straight after 'is a false positive when…', so the reader has to infer what p means." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): '(1 − 1/m)^(kn) ≈ e^(−kn/m)': the step (1 − 1/m)^m ≈ 1/e is not stated." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): 'p ≈ f^k': it is not said that the fill fraction f is used as the probability that one random position is 1, or that the k positions are treated as independent for this." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (approach): 'The best k is (m/n)·ln 2' is stated with no reason, and the beat never says that 10·ln 2 ≈ 6.9 rounds to 7." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (approach): 'f ≈ 0.5, the balance': it is not said why half-full is the optimum." + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (limit): The beat does not say that m and k stay fixed (10M bits, k = 7) while n grows." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151003-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151003-coldread-prose.json new file mode 100644 index 0000000..63bb141 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151003-coldread-prose.json @@ -0,0 +1,322 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "bloom-filter", + "at": "2026-10-04T15:10:03", + "model": "default", + "prompt_sha": "4717e3c3480f", + "reply": { + "paragraphs": [ + { + "at": "Title (line 1)", + "text": "Bloom filters: \"have I seen this before?\" in a few bits per item", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold lede (line 5)", + "text": "A Bloom filter does not keep the items. It keeps a row of bits...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Motive paragraph (line 8)", + "text": "The motive is memory. A hash set of 1 million URLs...", + "unresolved": [], + "leap": [ + "\"gives a wrong 'probably seen' for 0.82% of new URLs\": the 0.82% figure and the choice of 10 bits per URL are given without a source here. It is a forward claim that is only derived at line 56, and nothing says it will be derived later." + ], + "excess": [ + "\"(The 50 bytes per URL is an estimate.)\": this repeats the \"about\" already in the sentence." + ], + "purpose_unclear": false + }, + { + "at": "How it works, para 1 (line 13)", + "text": "Start with a bit array of m bits, all 0...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "How it works, para 2 (line 15)", + "text": "Choose k different hash functions; here k = 3...", + "unresolved": [], + "leap": [], + "excess": [ + "\"A hash function gives a large number, and the same item always gives the same number.\": this reader already knows hash functions." + ], + "purpose_unclear": false + }, + { + "at": "Add + list (lines 17-20)", + "text": "To add an item, set its k positions to 1... cat → 1, 6, 11; dog → 3, 9, 14", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Table (lines 22-25)", + "text": "Position / Bit / Set by", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Query rule (lines 27-30)", + "text": "To query an item... If any bit is 0... If all bits are 1...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Fox query (line 32)", + "text": "Query fox → positions 2, 6, 14...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "False positives (line 37)", + "text": "Query owl → positions 3, 6, 11...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "No false negatives, para 1 (line 42)", + "text": "The other error, a false negative...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "No false negatives, para 2 (line 44)", + "text": "This guarantee depends on bits never going back to 0...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "How often, intro (line 49)", + "text": "Let n be the number of items added. Assume...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Derivation list (lines 51-54)", + "text": "1. ...1 − 1/m. 2. ...e^(−kn/m). 3. fill f... 4. p ≈ f^k", + "unresolved": [], + "leap": [ + "Step 4, \"Each one is 1 with probability f\": the step from 'fraction of bits that are 1' to 'probability a random position is 1', and then multiplying k of these, uses the independence assumption only implicitly. The reader can manage this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Worked numbers (line 56)", + "text": "Take n = 1 million items and m = 10 million bits...", + "unresolved": [], + "leap": [], + "excess": [ + "\"That is 10 bits per item (m/n = 10).\": this says the same thing twice." + ], + "purpose_unclear": false + }, + { + "at": "Why 7, intro (line 59)", + "text": "Why 7 hash functions? The number of hash functions k pulls in two directions...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "k table (lines 61-65)", + "text": "k=1 / 7 / 20 with fill, p, why", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Best k (line 67)", + "text": "More hash functions make each query check more bits... best k is (m/n)·ln 2... Each extra 4.8 bits per item then cuts p by about 10 times.", + "unresolved": [], + "leap": [ + "\"The best k is (m/n)·ln 2\": stated with no reason.", + "\"At the best k, about half the bits are 1 (a mathematical result of the formula above)\": the reader is not shown why.", + "\"Each extra 4.8 bits per item then cuts p by about 10 times\": no reason given, and the reader cannot easily rebuild it." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Overfilling (line 72)", + "text": "The rate p stays at 0.82% only while n stays near the capacity...", + "unresolved": [], + "leap": [ + "\"the number of items used to choose m and k\": the text has not said how m is chosen from n. Only k = (m/n)·ln 2 was given, and the sizing formula appears later, in 'Not covered'." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What it is good for (line 77)", + "text": "Put the filter in front of a slow exact lookup...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Summary (line 81)", + "text": "A Bloom filter answers \"have I seen this before?\"...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered list (lines 85-88)", + "text": "Deletion / Growing / Sizing / Cheap hash functions", + "unresolved": [ + "\"counting Bloom filters\" (it gets a one-line gloss)", + "\"Cuckoo filters\"", + "\"scalable Bloom filter\"", + "\"double hashing\" (only glossed briefly)" + ], + "leap": [ + "\"m = −n·ln p / (ln 2)²\": given without derivation. That is acceptable for a pointer." + ], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Line 15: cut \"A hash function gives a large number, and the same item always gives the same number.\" because this reader already knows it.", + "Line 8: cut \"(The 50 bytes per URL is an estimate.)\"", + "Line 56: shorten \"That is 10 bits per item (m/n = 10).\" to a single statement." + ], + "question": { + "text": "How can you answer 'have I seen this item before?' using much less memory than a hash set, and what does it cost in accuracy?", + "known_at": "Title (line 1), made concrete in the lede (line 5) and the memory paragraph (line 8)", + "result_stated": "Yes. Line 5: \"So the filter can be wrong in one direction only: it can say 'probably seen' for an item it never saw.\" Line 8: \"takes 1.25 MB (10 bits per URL) and gives a wrong 'probably seen' for 0.82% of new URLs.\"" + }, + "motive": "Reason to care: \"The motive is memory. A hash set of 1 million URLs... takes about 50 MB. A Bloom filter... takes 1.25 MB\" (line 8), and the use case \"Put the filter in front of a slow exact lookup\" (line 77). Reason for the approach: \"The memory does not depend on how long the items are, because the filter never keeps them\" (line 8). The reason for using several hash functions is given at lines 59-67.", + "close": "Yes. Line 81: \"A Bloom filter answers 'have I seen this before?' by remembering only the bits each item turned on. A 0 bit proves an item is new. All 1s mean 'probably seen', which is wrong at a rate p ≈ f^k that you choose with bits per item, and that rises when you add more items than the capacity.\"", + "recap": "A Bloom filter stores only a few bits per item instead of the items themselves, so it can say 'definitely new' with certainty but only 'probably seen' otherwise; at about 10 bits per item that guess is wrong under 1% of the time.", + "strongest": "Lines 37-44: the owl example reuses cat's and dog's bits to produce a false positive. It then shows, with the same bits, why deleting cat would create a false negative. Both claims are made concrete with no new machinery.", + "weakest": "Line 67: three results arrive in a row with no reason given (best k = (m/n)·ln 2, half the bits are 1 at the best k, and 4.8 bits per item gives 10× lower p). This comes right after a step-by-step derivation that justified every step.", + "suggestions": [ + "At line 67, give a one-line reason: for fixed m/n, p = f^k is smallest when f = 1/2. Then k = (m/n)·ln 2, and p ≈ 0.5^k ≈ 0.6185^(m/n), which gives 4.8 bits per 10× reduction.", + "At line 8, add a short '(derived below)' after 0.82%.", + "In the Overfilling section, mention that m is chosen from the expected n and a target p, or move the sizing formula there." + ] + }, + "usage": { + "cost": 0.12415359999999999, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Line 15: cut \"A hash function gives a large number, and the same item always gives the same number.\" because this reader already knows it." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Line 8: cut \"(The 50 bytes per URL is an estimate.)\"" + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Line 56: shorten \"That is 10 bits per item (m/n = 10).\" to a single statement." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Motive paragraph (line 8): \"gives a wrong 'probably seen' for 0.82% of new URLs\": the 0.82% figure and the choice of 10 bits per URL are given without a source here. It is a forward claim that is only derived at line 56, and nothing says it will be derived later." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Derivation list (lines 51-54): Step 4, \"Each one is 1 with probability f\": the step from 'fraction of bits that are 1' to 'probability a random position is 1', and then multiplying k of these, uses the independence assumption only implicitly. The reader can manage this." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Best k (line 67): \"The best k is (m/n)·ln 2\": stated with no reason." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Best k (line 67): \"At the best k, about half the bits are 1 (a mathematical result of the formula above)\": the reader is not shown why." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Best k (line 67): \"Each extra 4.8 bits per item then cuts p by about 10 times\": no reason given, and the reader cannot easily rebuild it." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Overfilling (line 72): \"the number of items used to choose m and k\": the text has not said how m is chosen from n. Only k = (m/n)·ln 2 was given, and the sizing formula appears later, in 'Not covered'." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered list (lines 85-88): \"counting Bloom filters\" (it gets a one-line gloss)" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered list (lines 85-88): \"Cuckoo filters\"" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered list (lines 85-88): \"scalable Bloom filter\"" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered list (lines 85-88): \"double hashing\" (only glossed briefly)" + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "Not covered list (lines 85-88): \"m = −n·ln p / (ln 2)²\": given without derivation. That is acceptable for a pointer." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151043-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151043-quiz-prose.json new file mode 100644 index 0000000..2e3af51 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151043-quiz-prose.json @@ -0,0 +1,164 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "bloom-filter", + "at": "2026-10-04T15:10:34", + "model": "default", + "prompt_sha": "87e51d7b50fa", + "reply": { + "answers": { + "1": "The main objects are the bit array (m bits, all starting at 0), the k hash functions, the items, and each item's k positions (hash output mod m). The two operations are add and query. The answers are 'definitely not seen' and 'probably seen'. The errors are false positives and false negatives. The quantities are n (items added), the fill f, the false-positive rate p and the capacity.", + "2": "Each hash function maps an item to one position in the bit array, so each item gets k positions. Adding an item sets its k bits to 1. A query reads the same k bits. If any of them is 0 the answer is 'definitely not seen'; if all are 1 it is 'probably seen'. Different items can share bits, and a bit does not record which item set it. As n grows relative to m, the fill f rises. The false-positive rate p is about f^k.", + "3": "Adding items only ever turns bits on, so an added item always finds its bits at 1 and there are no false negatives. As more items are added, more bits are on by chance, so a never-added item can find all its k bits on. That is a false positive, with rate p ≈ (1 − e^(−kn/m))^k.", + "4": "The most important variable is the load: items per bit (n/m), or equivalently bits per item. If you add more items to a fixed m, the fill f = 1 − e^(−kn/m) rises and p = f^k rises sharply. With m = 10 million and k = 7, p is 0.82% at 1 million items, 13.8% at 2 million and 40% at 3 million. Giving each item more bits does the opposite: about 4.8 extra bits per item cuts p by 10 times (when k is chosen at its best value). k also matters: too few hashes means each query checks too few bits, and too many fills the array.", + "5": "Assumption: each hash position is a uniform random choice, independent of the others. The text also uses the approximation (1 − 1/m)^m ≈ 1/e. Established fact: false negatives cannot happen, because bits are never set back to 0. A worked fact: 10 bits per item with k = 7 gives p = 0.82%.", + "6": "A Bloom filter stores no items, only an m-bit array. Each item is hashed by k functions to k positions, and adding the item sets those bits to 1. A query checks the same positions: any 0 means definitely not seen, and all 1s means probably seen, because other items may have set those bits. With random hashing, the fill is f ≈ 1 − e^(−kn/m) and the false-positive rate is p ≈ f^k. The best k is (m/n)·ln 2, which gives 0.82% at 10 bits per item, so the filter is much smaller than a hash set and works well as a guard in front of a slow exact lookup. You cannot delete items, and adding many more items than the capacity makes p climb steeply.", + "7": "The hardest part was the optimum paragraph. The text says the best k is (m/n)·ln 2, that about half the bits are 1 at the best k, and that 4.8 extra bits per item cut p by 10 times. All three are stated as results with no derivation or intuition. A short line would have helped, for example: at f = 1/2, p = (1/2)^k and k = (m/n)·ln 2, so p = 2^(−(m/n)·ln 2); and ln 10 / (ln 2)² ≈ 4.8. The text also does not show how the 13.8% and 40% figures were computed.", + "8": "Not necessarily. The item was probably added, but it may be a false positive: all its bits could have been set by other items (like owl in the example).", + "9": "The fill rises and more queries for new items get 'probably seen'. With the example filter, the false-positive rate goes from 0.82% at 1 million items to 13.8% at 2 million and 40% at 3 million. Added items are still always reported as 'probably seen', but that answer stops telling you much.", + "10": "It keeps only a few bits per item (for example 10) and never stores the items themselves. Its memory therefore does not depend on how long the items are: 1.25 MB versus about 50 MB for 1 million URLs.", + "11": "Bit 2 is 0, so the filter answers 'definitely not seen'. That answer cannot be wrong: if the item had been added, bit 2 would have been set and never cleared.", + "12": "Bits are shared between items. Clearing an item's bits may clear bits that another added item needs, and that item would then get 'definitely not seen', which is a false negative. The example is deleting cat, which clears bits 6 and 11 that owl also uses.", + "13": "p rises from 0.82% to about 5.5%. Each add sets 20 bits, so the fill goes to about 0.865. Each query does check 20 bits, but nearly all of them are 1, so (0.865)^20 ≈ 5.5%.", + "14": "p rises to about 40%, which follows from f = 1 − e^(−2.1) ≈ 0.88 and p ≈ 0.88^7. About 400,000 of every 1,000,000 queries for new items would then do a wasted lookup. Added items are still never missed." + }, + "audit": { + "terms": [ + "The two answers are worded in two ways. Line 5 says the item is 'definitely new' / 'probably not new'; lines 29-30 and later say 'definitely not seen' / 'probably seen'.", + "'capacity' (line 72) and n, 'the number of items added' (line 49), are both used to describe the filter's design point. That the filter was 'sized for' n = 1 million is only implied.", + "The same error is called 'a wrong \"probably seen\"' (line 8), 'false positive' (line 37), 'false-positive rate p' (line 54) and 'wasted lookup' (line 77)." + ], + "unexplained": [ + "Line 67: 'Each extra 4.8 bits per item then cuts p by about 10 times.' The text does not say where 4.8 comes from.", + "Line 72: 'At 2 million items, p is 13.8%. At 3 million items, p is 40%.' No calculation is shown, unlike the worked 0.82% example.", + "Line 63, the k = 1 row: the text says p equals the fill (9.5%), but the reader has to work out themselves that with one hash p = f." + ], + "missing_why": [ + "Line 67: 'The best k is (m/n)·ln 2'. This is stated with no reason for why ln 2 appears.", + "Line 67: 'At the best k, about half the bits are 1 (a mathematical result of the formula above)'. The text asserts this without showing it.", + "Line 87: the sizing formula 'm = −n·ln p / (ln 2)²' is given with no derivation or link to p ≈ f^k." + ], + "excess": [ + "Line 15: 'A hash function gives a large number, and the same item always gives the same number.' This reader already knows hash functions.", + "Lines 85-88: the 'Not covered here' items (counting/cuckoo filters, scalable Bloom filters, double hashing) are not used by the argument or the questions." + ] + }, + "gaps": [ + "The text gives no derivation or intuition for the optimal k = (m/n)·ln 2, for the fill being one half at the optimum, or for the 4.8-bits-per-10× rule.", + "The overfill figures (13.8%, 40%) are given without the calculation.", + "The text does not explain why the formula is only approximate (repeated bits in step 2, independence of the hash positions) or how close it is for real hash functions." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 1, + "11": 2, + "12": 2, + "13": 2, + "14": 2 + }, + "reasons": { + "10": "Correctly says the filter keeps about 10 bits per item, not the items (1.25 MB vs about 50 MB). It omits the price the rubric expects: the memory saving comes with a small chance of false positives." + }, + "pass": false + }, + "usage": { + "cost": 0.172155, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The two answers are worded in two ways. Line 5 says the item is 'definitely new' / 'probably not new'; lines 29-30 and later say 'definitely not seen' / 'probably seen'." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'capacity' (line 72) and n, 'the number of items added' (line 49), are both used to describe the filter's design point. That the filter was 'sized for' n = 1 million is only implied." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "The same error is called 'a wrong \"probably seen\"' (line 8), 'false positive' (line 37), 'false-positive rate p' (line 54) and 'wasted lookup' (line 77)." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "Line 67: 'Each extra 4.8 bits per item then cuts p by about 10 times.' The text does not say where 4.8 comes from." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 72: 'At 2 million items, p is 13.8%. At 3 million items, p is 40%.' No calculation is shown, unlike the worked 0.82% example." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 63, the k = 1 row: the text says p equals the fill (9.5%), but the reader has to work out themselves that with one hash p = f." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 67: 'The best k is (m/n)·ln 2'. This is stated with no reason for why ln 2 appears." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 67: 'At the best k, about half the bits are 1 (a mathematical result of the formula above)'. The text asserts this without showing it." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 87: the sizing formula 'm = −n·ln p / (ln 2)²' is given with no derivation or link to p ≈ f^k." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "Line 15: 'A hash function gives a large number, and the same item always gives the same number.' This reader already knows hash functions." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "Lines 85-88: the 'Not covered here' items (counting/cuckoo filters, scalable Bloom filters, double hashing) are not used by the argument or the questions." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "The text gives no derivation or intuition for the optimal k = (m/n)·ln 2, for the fill being one half at the optimum, or for the 4.8-bits-per-10× rule." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "The overfill figures (13.8%, 40%) are given without the calculation." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The text does not explain why the formula is only approximate (repeated bits in step 2, independence of the hash positions) or how close it is for real hash functions." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151208-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151208-quiz-prose.json new file mode 100644 index 0000000..ab846b4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/runs/2026-10-04T151208-quiz-prose.json @@ -0,0 +1,162 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "bloom-filter", + "at": "2026-10-04T15:12:02", + "model": "default", + "prompt_sha": "87e51d7b50fa", + "reply": { + "answers": { + "1": "The main objects are: a bit array of m bits; k hash functions, whose outputs are taken mod m to give positions; the items being added and queried; the two operations, add and query; the two answers, \"definitely not seen\" and \"probably seen\"; false positives and false negatives; and the quantities n (items added), f (fill, the fraction of bits that are 1), p (false-positive rate) and capacity.", + "2": "Each item is hashed by the k hash functions into k positions in the bit array. Adding an item sets those bits to 1. Querying an item reads the same k bits: any 0 means \"definitely not seen\", all 1s mean \"probably seen\". Because different items can share bits, an item that was never added can find all its bits set, which is a false positive. Bits never go back to 0, so an added item always finds its bits set, which means no false negatives. Fill f grows with kn/m, and p ≈ f^k. The filter is sized for a capacity n, and adding more items raises f and so raises p.", + "3": "Each added item turns on k shared bits and nothing is ever turned off. So a query that finds a 0 proves the item is new, and a query that finds all 1s may only be matching bits set by other items. That chance, p ≈ (1 − e^(−kn/m))^k, grows as the array fills.", + "4": "The key variable is items per bit, n/m (or its inverse, bits per item). If more items go into the same m, the fill f = 1 − e^(−kn/m) rises, so a never-added item is more likely to find all k bits at 1, and p rises sharply: 0.82% at 1M, 13.8% at 2M, 40% at 3M. More bits per item lowers p: at the best k, each extra 4.8 bits per item cuts p about tenfold. Changing k pulls two ways. More hashes mean more bits checked per query, but also faster filling. The best is near f = 0.5, at k ≈ 0.69·m/n.", + "5": "Assumption: each hash position is a uniform random choice, independent of the others. There is also the approximation (1 − 1/m)^m ≈ 1/e. Fact: nothing ever sets a bit back to 0, so a false negative is impossible. A worked figure: 1M items in 10M bits with k = 7 gives p ≈ 0.82%.", + "6": "A Bloom filter replaces a set of items with an array of m bits. Adding an item sets the k bits picked by k hash functions. A query answers \"definitely not seen\" if any of its k bits is 0, and \"probably seen\" if all are 1. Since bits are shared and never cleared, it can give false positives but never false negatives, and for the same reason it can't delete. With independent uniform hashes, the fill is f ≈ 1 − e^(−kn/m) and the false-positive rate is p ≈ f^k: about 0.82% at 10 bits per item with k = 7. The best k makes the array about half full, and p rises sharply once you add more items than the filter was sized for. That makes it a cheap pre-check before a slow exact lookup.", + "7": "The hardest part was the paragraph on choosing k. It says the optimum k = (m/n)·ln 2 \"which the formula gives\" without showing why f = 0.5 is the best point. It also jumps through 0.5^0.69 ≈ 0.62 to \"each 4.8 bits cuts p by 10\". Two things would help: a line showing that f = 0.5 means e^(−kn/m) = 1/2, so k = (m/n)·ln 2, and a line showing 0.62^4.8 ≈ 0.1.", + "8": "Not necessarily. All of its k bits are 1, which happens if it was added, but other items may have set those bits (a false positive, as with owl). It was probably added, but the filter can't confirm it.", + "9": "The fill f rises towards 1, so more never-added items find all their bits set, and the false-positive rate climbs: 0.82% at 1M, 13.8% at 2M, 40% at 3M for the 10M-bit, k = 7 filter. Added items are still always reported as \"probably seen\" (no false negatives), but \"probably seen\" stops meaning much.", + "10": "It doesn't store the items. It stores only a fixed number of bits per item (for example 10), whatever the item's length. A hash set stores the items themselves (for example about 50 bytes per URL). For 1M URLs that is 1.25 MB against about 50 MB, paid for with a small false-positive rate.", + "11": "Bit 2 is not set, so the answer is \"definitely not seen\". That answer can't be wrong: an added item would have set bit 2, and bits never go back to 0. (This is the fox example.)", + "12": "Bits are shared between items. Clearing one item's bits can clear bits another added item depends on. For example, deleting cat clears 6 and 11, which owl also uses, so owl would then read \"definitely not seen\". That is a false negative, and it breaks the guarantee.", + "13": "It gets worse: about 5.5% instead of 0.82%. Each add sets 20 bits, so the fill rises to about 0.865, and checking 20 bits that are almost all 1 doesn't filter well (0.865^20 ≈ 5.5%).", + "14": "kn/m = 2.1, so f = 1 − e^(−2.1) ≈ 0.88 and p ≈ 0.88^7 ≈ 40%. Added items are still never missed, but about 40% of queries for new items wrongly say \"probably seen\". In front of a database, that means about 400,000 wasted lookups per million queries for new items, against 8,200." + }, + "audit": { + "terms": [ + "The same concept appears under several names: \"never-added item\" (step 4, 'How often it is wrong'), \"new items\" / \"new URLs\" (intro, line 8; 'How often', line 56; 'What it is good for'), and \"an item that was never added\" ('false positives'). These are all the same population that p is measured over.", + "Capacity has two names: \"capacity\" ('Overfilling') and \"sized for\" ('Overfilling'; Q14). The summary also says p is chosen \"with bits per item\" rather than with capacity." + ], + "unexplained": [ + "'Overfilling': \"At 2 million items, p is 13.8%\" is given without the working that the 3-million case gets (kn/m = 1.4, f ≈ 0.75).", + "'How often it is wrong', final paragraph: \"Thus each extra 4.8 bits per item cuts p by about 10 times\". It doesn't show where 4.8 comes from (0.62^4.8 ≈ 0.1).", + "'Not covered here': \"m = −n·ln p / (ln 2)²\" is given with no meaning or example. The reader can't tell what m it gives for, say, p = 1%." + ], + "missing_why": [ + "'How often it is wrong': \"The best balance is near f = 0.5, which the formula gives at k = (m/n)·ln 2\". The reader isn't shown why the optimum is f = 0.5, or how setting e^(−kn/m) = 1/2 yields ln 2.", + "Step 4: \"p ≈ f^k\" relies on the k checked bits being independent, each 1 with probability f. That is only implied by the earlier assumption and isn't tied to this step.", + "'Not covered here': the sizing formula \"m = −n·ln p / (ln 2)²\" is stated with no reason for its form. It follows from p ≈ 0.5^k with k = (m/n)·ln 2, but the text doesn't say so." + ], + "excess": [ + "'How it works': \"a hash output of 3,141,593 gives position 3,141,593 mod 16 = 9\". This reader already knows hash functions and mod; the worked mod example adds nothing for them.", + "'Summary' restates the bold intro paragraph and the rate formula with nothing new." + ] + }, + "gaps": [ + "The 13.8% figure at 2M items has no derivation.", + "There is no derivation of why the optimal k gives f = 0.5, or of k = (m/n)·ln 2.", + "The 4.8-bits-per-tenfold claim is not shown.", + "The sizing formula in 'Not covered' is unexplained, though it is explicitly marked out of scope." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2, + "14": 2 + }, + "reasons": {}, + "pass": true + }, + "usage": { + "cost": 0.172995, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same concept appears under several names: \"never-added item\" (step 4, 'How often it is wrong'), \"new items\" / \"new URLs\" (intro, line 8; 'How often', line 56; 'What it is good for'), and \"an item that was never added\" ('false positives'). These are all the same population that p is measured over." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Capacity has two names: \"capacity\" ('Overfilling') and \"sized for\" ('Overfilling'; Q14). The summary also says p is chosen \"with bits per item\" rather than with capacity." + }, + { + "n": 3, + "severity": "gap", + "kind": "unexplained", + "text": "'Overfilling': \"At 2 million items, p is 13.8%\" is given without the working that the 3-million case gets (kn/m = 1.4, f ≈ 0.75)." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'How often it is wrong', final paragraph: \"Thus each extra 4.8 bits per item cuts p by about 10 times\". It doesn't show where 4.8 comes from (0.62^4.8 ≈ 0.1)." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'Not covered here': \"m = −n·ln p / (ln 2)²\" is given with no meaning or example. The reader can't tell what m it gives for, say, p = 1%." + }, + { + "n": 6, + "severity": "gap", + "kind": "missing_why", + "text": "'How often it is wrong': \"The best balance is near f = 0.5, which the formula gives at k = (m/n)·ln 2\". The reader isn't shown why the optimum is f = 0.5, or how setting e^(−kn/m) = 1/2 yields ln 2." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Step 4: \"p ≈ f^k\" relies on the k checked bits being independent, each 1 with probability f. That is only implied by the earlier assumption and isn't tied to this step." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "'Not covered here': the sizing formula \"m = −n·ln p / (ln 2)²\" is stated with no reason for its form. It follows from p ≈ 0.5^k with k = (m/n)·ln 2, but the text doesn't say so." + }, + { + "n": 9, + "severity": "excess", + "kind": "excess", + "text": "'How it works': \"a hash output of 3,141,593 gives position 3,141,593 mod 16 = 9\". This reader already knows hash functions and mod; the worked mod example adds nothing for them." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "'Summary' restates the bold intro paragraph and the rate formula with nothing new." + }, + { + "n": 11, + "severity": "gap", + "kind": "gap", + "text": "The 13.8% figure at 2M items has no derivation." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "There is no derivation of why the optimal k gives f = 0.5, or of k = (m/n)·ln 2." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "The 4.8-bits-per-tenfold claim is not shown." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The sizing formula in 'Not covered' is unexplained, though it is explicitly marked out of scope." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/understanding.md b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/understanding.md new file mode 100644 index 0000000..c7c0e5b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/explainers/bloom-filter/review/understanding.md @@ -0,0 +1,16 @@ +# Review: bloom-filter (prose, published tier) + +## Probe (2026-10-04T150533) +Adopted 1, 2, 3, 6, 7, 8: worked derivation of p ≈ f^k, toy-vs-formula note, why best k is near half full, use case (filter before a slow lookup), fill and capacity terms, hash → position example. Declined: resize and identical-hash breaking cases (resizing went to Scope), byte-vs-bit slots, and the excess notes. + +## Cold read, narrative +- Run 150725: 4 blocking (beats too terse: k undefined, "all 1 → probably seen" not said, derivation not spoken, close a placeholder). Rewrote every beat's "Said" as full sentences. +- Run 150831: no blocking. Adopted the short edge fixes (name p, (1 − 1/m)^m ≈ 1/e, 6.9 → 7, m and k fixed when overfilling, cut repeats). + +## Cold read, prose (151003) +No blocking. Cut the "hash functions are deterministic" sentence and the estimate note; added reasons for best k ≈ half full and the 4.8-bit rule; reworded capacity. Declined: glosses for the Not-covered pointers (Scope only). + +## Blind test, prose +- Run 151043: FAIL, 13/14 at 2; item 10 (why less memory) scored 1: the answer omitted the price. Fixed: "The price for the smaller memory is error" next to the memory numbers. Also unified the two answers' wording, added p = f for k = 1, and worked the 3× overfill number. +- Run 151208: PASS, all 14 items at 2. +- Remaining audit notes not applied: no derivation of the 2× (13.8%) figure, of the optimum f = 0.5 (needs calculus), or of the sizing formula (Scope pointer). Unified "new items" → "never-added items" after this run. diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/read.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/read.json new file mode 100644 index 0000000..e45b75d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "To add an item, you compute its k positions (each hash output mod m) and set those bits to 1. Nothing else is stored. To look an item up, you compute the same k positions and read those bits. If any of them is 0, the answer is \"definitely not seen\". If all of them are 1, the answer is \"probably seen\".", + "quote": "To **add** an item, set its k positions to 1. Nothing else is kept." + }, + "2": { + "answer": "It can give a false positive: \"probably seen\" for an item that was never added. This happens because other items may have set all of that item's bits, and the bits don't record which item set them. It can never give a false negative (\"definitely not seen\" for an item that was added). Adding an item sets its bits to 1, and nothing ever sets a bit back to 0, so a later query always finds all of its bits at 1.", + "quote": "Adding cat set bits 1, 6 and 11, and nothing ever sets a bit back to 0. So any later query for cat finds all three bits at 1." + }, + "3": { + "answer": "Items share bits. Clearing one item's bits can set to 0 a bit that another added item also uses. That item would then read \"definitely not seen\", which is a false negative. So a Bloom filter has no delete operation.", + "quote": "Owl also uses bits 6 and 11, so they are now 0, and the filter says owl is definitely not seen. That is a false negative. Bits are shared between items, so a Bloom filter has no delete operation." + }, + "4": { + "answer": "It answers \"probably seen\", which is a false positive. If every bit is 1, all k of the item's positions are 1, so any item at all reads \"probably seen\". This matches the formula: when f = 1, p = f^k = 1, so the answer is wrong for every never-added item.", + "quote": "If all bits are 1, the answer is **probably seen**." + } + }, + "stops": [ + "Hash functions are never defined, and nothing says how to get k different ones: \"Choose k different hash functions; here k = 3.\"", + "\"mod\" is used without a definition: \"Take each hash output mod m to get a **position**.\"", + "The reason for this approximation isn't given: \"Because (1 − 1/m)^m ≈ 1/e for large m, this is about e^(−kn/m).\"", + "Step 2 isn't explained: why does each add count as setting a bit independently, even when a bit is already 1? \"n adds set kn bits in total.\"", + "The value for 2 million items is stated but not worked out: \"At 2 million items, p is 13.8%.\"", + "This number has no derivation: \"Thus each extra 4.8 bits per item cuts p by about 10 times.\"", + "\"The formula\" is ambiguous, and it isn't shown why f = 0.5 is the best point: \"The best balance is near f = 0.5, which the formula gives at k = (m/n)·ln 2\"", + "The reasoning in this step is compressed: \"Each extra bit per item adds 0.69 to k, so it multiplies p by 0.5^0.69 ≈ 0.62.\"" + ], + "excess": [ + "Leftover build comments: \"\"", + "The summary repeats the opening: \"A 0 bit proves an item is new. All 1s mean \"probably seen\"\"", + "The not-covered list wasn't needed: \"**Cheap hash functions:** real code makes k positions from two hash values (\"double hashing\").\"", + "Not needed for these questions: \"**Sizing:** for n items and a target p, m = −n·ln p / (ln 2)².\"", + "Not needed for these questions: \"**Growing:** changing m changes every position (hash mod m), so you cannot resize a filter in place.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/run.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/run.json new file mode 100644 index 0000000..408b753 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/bloom-filter/explanation.md`, prose only: the subject is a short add-then-query procedure and one failure case, and a 16-bit table plus a 3-row trade-off table show both without a diagram or interactive page; the blind test passed with all 14 items at 2 after one fix, and the cold read found nothing blocking.", + "error": null, + "cost": 1.4500966000000002, + "seconds": 553, + "turns": 54, + "files": [ + "explainers/bloom-filter/explanation.md" + ], + "review_cost": 0.8529, + "reviews": 6 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/explanation.md b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/explanation.md new file mode 100644 index 0000000..9d0e6a5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/explanation.md @@ -0,0 +1,67 @@ +# How does a Bloom filter remember what it has seen, and when does it get it wrong? + +**A Bloom filter does not store the items. It stores a row of bits, and each item switches on a few of them. To ask "have I seen this?", you check the item's bits. If any bit is off, the answer is a sure "no". If all are on, the answer is "probably yes", because other items can switch on the same bits.** + +## The parts + +- A **bit array**: m bits, all 0 at the start. +- k **hash functions**. A hash function turns an item into a position from 0 to m − 1. The same item always gives the same position; different items give positions that look random. + +## How it works + + +Take a toy filter with m = 10 bits and k = 2 hash functions. + +1. **Add** an item: compute its k positions and set those bits to 1. "cat" gives positions 1 and 6. "dog" gives positions 3 and 6. +2. **Query** an item: compute its k positions and look at those bits. If any bit is 0, answer "no". If all are 1, answer "probably yes". + +After adding cat and dog, the array is: + +| position | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | +|---|---|---|---|---|---|---|---|---|---|---| +| bit | 0 | **1** | 0 | **1** | 0 | 0 | **1** | 0 | 0 | 0 | + +Query "owl": its positions are 3 and 5. Bit 5 is 0, so owl was never added: a certain "no". + +## Why "no" is always right + + +Adding an item only changes bits from 0 to 1. Nothing changes a bit back to 0. So every item you added still has all its bits set, and it always gets "probably yes". + +This guarantee breaks if you try to delete. Suppose you remove cat by clearing bits 1 and 6. Dog also uses bit 6. Now a query for dog finds bit 6 at 0 and answers "no", which is wrong. For this reason, a plain Bloom filter does not support deletion. + +## How "yes" can be wrong + + +Query "fox": its positions are 1 and 3. Cat set bit 1 and dog set bit 3. Both bits are 1, so the filter answers "probably yes" for an item it never saw. This is a **false positive**. + +The filter cannot tell which item set a bit. As you add more items, more bits become 1, and an unseen item is more likely to find all its bits already on. + +## How much memory, and how often wrong + + +With n items in m bits and k hash functions, the **false-positive rate** p (the fraction of unseen items that get "probably yes") is about: + +p ≈ (1 − e^(−kn/m))^k + +This formula assumes the hash positions are independent and uniform. +Store 1,000,000 web addresses of about 50 bytes each: as a normal set, about 50 MB. A Bloom filter with 9,600,000 bits (1.2 MB, or 9.6 bits per item) and k = 7 gives p ≈ 1%. So 1 in 100 unseen addresses gets a wrong "probably yes". The filter size depends on n and the accepted rate, not on item length. + + +Why several hash functions? Each extra hash makes a false positive need one more lucky bit. But each extra hash also sets more bits to 1. At 9.6 bits per item: + +| k | false-positive rate | +|---|---| +| 1 | 9.9% | +| 7 | 1% | +| 20 | 7% | + +The best k is about (m/n) × ln 2, which is 9.6 × 0.69 ≈ 7 here. + +The rate also depends on how full the filter is. Put 3,000,000 items into the same 1.2 MB filter, and p rises to about 43%. Choose m from the largest n you expect. + +## When to use one + +Use a Bloom filter in front of a slow lookup, such as a disk or a network call. A "no" lets you skip the slow lookup. A "probably yes" sends you to the slow lookup, which gives the true answer. A false positive then costs one wasted lookup, not a wrong result. + +*Not covered: counting Bloom filters (they allow deletion by storing small counters instead of bits) and cuckoo filters.* diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/model.md b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/model.md new file mode 100644 index 0000000..a8f3b01 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/model.md @@ -0,0 +1,78 @@ +# Semantic model: Bloom filter + +## Central question + +How does a Bloom filter answer "have I seen this before?" with little memory, and how can it be wrong? + +## Audience and prior knowledge + +A technical reader: school mathematics (powers, e^x), basic programming (arrays, functions). Knows what a set is. Does not know what a hash function is in detail, or what a Bloom filter is. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| bit array | A row of m bits, all 0 at the start. | fact | +| hash function | A function that turns an item into a position 0..m−1; the same item always gives the same position, different items give positions that look random. | fact (assumed uniform) | +| add | Set the k bits at the item's k positions to 1. | fact | +| query | Check the item's k positions; "no" if any bit is 0, "probably yes" if all are 1. | fact | +| false positive | A "probably yes" for an item that was never added. | fact | + +## Causal chain + +```text +item → k hash functions → k positions → (add: set to 1) / (query: all 1? → probably yes; any 0 → no) +other items' bits overlap → all k positions 1 by chance → false positive +``` + +## Quantities + +- m = number of bits; n = number of items added; k = number of hash functions. +- False-positive rate p ≈ (1 − e^(−kn/m))^k (assumes independent uniform hashes). +- Best k ≈ (m/n)·ln 2. +- n = 1,000,000, m = 9,600,000 bits (1.2 MB), k = 7 → p ≈ 1%. About 9.6 bits per item. +- Same data as a set of 50-byte strings: about 50 MB. +- Same m and n, k = 1 → p ≈ 9.9%. k = 20 → p ≈ 7%. +- Same m, k = 7, n = 3,000,000 → p ≈ 43%. + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| k hashes, not one | k = 1 | at 9.6 bits per item, p ≈ 9.9% instead of 1% | +| not too many hashes | k = 20 | array fills with 1s; p ≈ 7% | +| store bits, not items | a set of the strings | 50 MB instead of 1.2 MB for 1,000,000 items | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| No false negatives | added "dog" → bits 3, 6 are 1 forever → query says probably yes | delete "cat" by clearing bits 1, 6 → bit 6 is 0 → "dog" gets "no" | +| False positives exist | "owl" → bits 3, 5; bit 5 is 0 → no | "fox" → bits 1, 3; both set by cat and dog → probably yes | + +Worked example: m = 10, k = 2. cat → 1, 6. dog → 3, 6. + +## Terms + +- bit array, hash function, position, add, query, false positive, false-positive rate (p: 1% means 1 in 100 unseen items gets "probably yes"; at 43% the filter is close to useless). + +## Epistemic status + +- **Mathematical consequence:** no false negatives (bits only go 0 → 1). +- **Estimate:** the p formula; it assumes independent uniform hashes and is a close approximation. + +## Confusion points + +- "Probably yes" means certain → no; it can be a false positive. +- You can remove items by clearing bits → no; that creates false negatives. +- More hashes are always better → no; there is a best k. + +## Scope + +- **Out of scope:** counting Bloom filters (allow deletion); cuckoo filters; how to build k hashes from two (double hashing). + +## Representation decision + +- **Stage:** 1 +- **Reason:** The mechanism is a short sequence of steps; one 10-bit worked example and one formula carry it. A static table shows the bit array. +- **Levels:** L1 what it answers / L2 add and query / L3 why errors are one-sided / L4 sizing. diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/model.yaml b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/model.yaml new file mode 100644 index 0000000..e164d8e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/explainers/bloom-filter/model.yaml @@ -0,0 +1,70 @@ +values: + p_percent: 1 + bits_per_item: 9.6 + toy: + m: 10 + k: 2 + cat: [1, 6] + dog: [3, 6] + fox: [1, 3] + owl: [3, 5] + real: + n: 1000000 + m_bits: 9600000 + megabytes: 1.2 + k: 7 + p_percent: 1 + bits_per_item: 9.6 + set_megabytes: 50 + string_bytes: 50 + k1_p_percent: 9.9 + k20_p_percent: 7 + k_many: 20 + overfull_n: 3000000 + overfull_p_percent: 43 + ln2: 0.69 + +allow: [0, 1, 2, 3, 4, 8, 9, 100] + +require: + - path: p_percent + in: [prose] + - path: bits_per_item + in: [prose] + - path: overfull_p_percent + in: [prose] + +claims: + - id: mech-add-query + kind: mechanism + statement: "Add sets the item's k bits to 1; query answers no if any of them is 0." + example: "m=10, k=2: cat sets 1 and 6, dog sets 3 and 6; owl checks 3 and 5, bit 5 is 0, so no." + - id: no-false-negatives + kind: guarantee + statement: "An added item always gets 'probably yes', because bits only change from 0 to 1." + example: "dog's bits 3 and 6 stay 1, so dog always gets probably yes." + counterexample: "Delete cat by clearing 1 and 6: bit 6 becomes 0 and dog now gets no." + ask: "Why can you not remove an item by setting its bits back to 0?" + - id: false-positive + kind: limit + statement: "An item never added can find all its bits already set by other items." + example: "fox checks 1 and 3, set by cat and dog, so fox gets probably yes." + ask: "How can a Bloom filter say 'probably yes' for an item it never saw?" + - id: why-k + kind: why + statement: "Several hash functions lower the false-positive rate, up to a best k of about (m/n) ln 2." + alternative: "one hash function, or very many" + counterexample: "At 9.6 bits per item: k=1 gives 9.9%, k=7 gives 1%, k=20 gives 7%." + - id: why-bits + kind: why + statement: "The filter stores bits, not items, so its size does not depend on item length." + alternative: "store the items in a set" + counterexample: "1,000,000 items of 50 bytes: about 50 MB as a set, 1.2 MB as a filter with 1% false positives." + +accept_unjustified: + ln: "ln 2 in the best-k rule comes from minimizing the formula; the k table shows the minimum with numbers." + +terms: + - term: probably yes + means: "the answer when all k bits are 1; it can be a false positive" + avoid: [definitely yes] diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/read.json new file mode 100644 index 0000000..fbf4df5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/read.json @@ -0,0 +1,36 @@ +{ + "answers": { + "1": { + "answer": "Adding an item: you compute its k positions with the hash functions and set those bits to 1. Looking an item up: you compute the same k positions and check those bits. If any of them is 0 the answer is \"no\"; if all are 1 the answer is \"probably yes\".", + "quote": "Query** an item: compute its k positions and look at those bits. If any bit is 0, answer \"no\". If all are 1, answer \"probably yes\"." + }, + "2": { + "answer": "It can give a false positive: a \"probably yes\" for an item that was never added, because other items may have already turned on all of that item's bits. It can never give a false negative: adding only ever changes bits from 0 to 1 and nothing resets them, so every item that was added still has all its bits set.", + "quote": "Adding an item only changes bits from 0 to 1. Nothing changes a bit back to 0. So every item you added still has all its bits set, and it always gets \"probably yes\"." + }, + "3": { + "answer": "Other items may share those bits. Clearing them can turn off a bit that another added item needs, so that item then gets a wrong \"no\". In the example, clearing cat's bit 6 breaks dog.", + "quote": "Suppose you remove cat by clearing bits 1 and 6. Dog also uses bit 6. Now a query for dog finds bit 6 at 0 and answers \"no\", which is wrong." + }, + "4": { + "answer": "\"Probably yes\". All of the item's bits are 1, so the filter gives a false positive for every unseen item. This is the extreme end of the over-full case the explanation describes. The explanation does not discuss this case directly; I reasoned it from the query rule.", + "quote": "If all are 1, answer \"probably yes\"." + } + }, + "stops": [ + "Hash function positions: the reader may wonder how k different hash functions are obtained in practice. Quote: \"k **hash functions**. A hash function turns an item into a position from 0 to m − 1.\"", + "The formula appears with no derivation or explanation of e and its terms. Quote: \"p ≈ (1 − e^(−kn/m))^k\"", + "\"one more lucky bit\" is vague wording. Quote: \"Each extra hash makes a false positive need one more lucky bit.\"", + "The optimal k formula is stated without a reason. Quote: \"The best k is about (m/n) × ln 2\"", + "\"independent and uniform\" is not explained. Quote: \"This formula assumes the hash positions are independent and uniform.\"" + ], + "excess": [ + "The closing note names topics it then leaves out. Quote: \"*Not covered: counting Bloom filters (they allow deletion by storing small counters instead of bits) and cuckoo filters.*\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/run.json new file mode 100644 index 0000000..a2067fa --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/bloom-filter/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/bloom-filter/explanation.md`, a prose explanation (plus its `model.md` and `model.yaml`; `explainer check` passes). I chose prose because the mechanism is a short series of steps: one 10-bit worked example, a small table of k values and one formula show the memory saving (1.2 MB instead of about 50 MB, at about 1% wrong answers) and both ways it can be wrong, so a diagram or interactive page would not add enough to justify it.", + "error": null, + "cost": 0.5246917999999999, + "seconds": 99, + "turns": 29, + "files": [ + "explainers/bloom-filter/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/none/explainers/float-sum/explanation.md b/evals/artifact/results/2026-10-04-dev/work/float-sum/none/explainers/float-sum/explanation.md new file mode 100644 index 0000000..183334e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/none/explainers/float-sum/explanation.md @@ -0,0 +1,176 @@ +# Why 0.1 + 0.2 ≠ 0.3 + +Type this into Python, JavaScript, Java, C, Ruby, Go, or most other languages: + +``` +0.1 + 0.2 == 0.3 → false +0.1 + 0.2 → 0.30000000000000004 +``` + +This isn't a bug in the language or in your CPU. The computer is doing exactly what it was designed to do. The surprise comes from two facts that work together: + +1. The computer stores numbers in **binary**, and **0.1, 0.2 and 0.3 can't be written exactly in binary**. Each one gets rounded to the nearest number that *can* be stored. +2. Those three small rounding errors don't cancel out. + +The rest of this document explains both facts and then what to do about them. + +--- + +## 1. A familiar version of the problem: 1/3 in decimal + +Before binary, look at something you already know. In decimal, 1/3 has no finite form: + +``` +1/3 = 0.3333333333... (the 3s never end) +``` + +Say you could only keep 4 decimal places. You'd store 1/3 as `0.3333`. Then: + +``` +0.3333 + 0.3333 + 0.3333 = 0.9999 ≠ 1 +``` + +Nothing went wrong with the addition. The error was already there once you rounded 1/3 to fit the space you had. + +Which fractions end and which repeat depends on the **base**. In base 10, a fraction has a finite form only if its denominator (after simplifying) has no prime factors other than 2 and 5, the prime factors of 10. So 1/2, 1/4, 1/5 and 1/8 end, while 1/3, 1/6 and 1/7 repeat forever. + +## 2. The same problem in binary + +Base 2 has only one prime factor: 2. So in binary, a fraction has a finite form **only if its denominator is a power of 2**: 1/2, 1/4, 3/8, 5/16, and so on. + +Now look at our numbers as fractions: + +| Decimal | Fraction | Denominator a power of 2? | +|---|---|---| +| 0.5 | 1/2 | yes, exact in binary | +| 0.25 | 1/4 | yes, exact | +| 0.1 | 1/10 | no (10 = 2 × **5**) | +| 0.2 | 1/5 | no | +| 0.3 | 3/10 | no | + +So 0.1 repeats forever in binary, the same way 1/3 does in decimal: + +``` +0.1 (decimal) = 0.0001100110011001100110011... (binary, "0011" repeats forever) +``` + +You can check this by repeated doubling, which is how you convert a fraction to binary. Double the number. If the result is 1 or more, write a 1 and subtract 1. Otherwise write a 0. + +``` +0.1 × 2 = 0.2 → 0 +0.2 × 2 = 0.4 → 0 +0.4 × 2 = 0.8 → 0 +0.8 × 2 = 1.6 → 1 (keep 0.6) +0.6 × 2 = 1.2 → 1 (keep 0.2) +0.2 × 2 = 0.4 → 0 ← we've been at 0.2 before, so the pattern repeats +... +``` + +The digits come out `0.0001 1001 1001 1001 ...`, and they never stop. + +## 3. How much room the computer has + +Most languages store non-integer numbers in the **IEEE 754 double-precision** format (a "double", or `float` in Python and `number` in JavaScript). It's scientific notation in base 2: + +``` +value = ± (1.xxxxxxxx...) × 2^exponent +``` + +The digits after the point (the *significand*) get a fixed budget: **53 significant binary digits**. That's roughly 15–17 significant decimal digits. Whatever doesn't fit is rounded off, to the *nearest* number that can be stored. + +So "0.1" in your source code is never stored as 0.1. It's stored as the closest double to 0.1. Here are the exact stored values, written out in full decimal. Every double has a finite decimal expansion, because its denominator is a power of 2. + +| You wrote | What is actually stored | Error | +|---|---|---| +| `0.1` | 0.1000000000000000055511151231257827… | a little **too big** | +| `0.2` | 0.2000000000000000111022302462515654… | a little **too big** | +| `0.3` | 0.2999999999999999888977697537484345… | a little **too small** | + +That already explains most of the surprise. The 0.1 and 0.2 you add are both slightly too big. The 0.3 you compare against is slightly too small. They're about to land on different sides of the true 0.3. + +## 4. The addition itself + +The computer adds the two stored values. The exact mathematical sum is: + +``` + 0.1000000000000000055511151231257827… ++ 0.2000000000000000111022302462515654… += 0.3000000000000000166533453693773481… +``` + +That result also has to be stored as a double, so it gets rounded again. Near 0.3, the doubles are spaced about 5.55 × 10⁻¹⁷ apart. The two candidates on either side of our sum are: + +``` +lower: 0.29999999999999998889776975… (this is the double we call "0.3") +sum: 0.30000000000000001665334536… +upper: 0.30000000000000004440892098… +``` + +The sum sits **exactly halfway** between them: 2.775 × 10⁻¹⁷ from each. IEEE 754's tie rule is "round half to even": pick the neighbour whose last binary digit is 0. That's the upper one. So the result is: + +``` +0.1 + 0.2 → 0.3000000000000000444089209850062616… +``` + +That's a *different double* from the one `0.3` gives you. They're neighbours, one step apart, but they aren't equal, so `==` returns `false`. + +> For readers who like exact arithmetic: the double for 0.1 is exactly +> 3602879701896397 × 2⁻⁵⁵, and the double for 0.2 is twice that. Their sum is +> 10808639105689191 × 2⁻⁵⁵. Near 0.3 the stored values are whole multiples of +> 2⁻⁵⁴, and 10808639105689191 / 2 = 5404319552844595.5, an exact tie. Rounding to +> even gives 5404319552844596 × 2⁻⁵⁴. But `0.3` is stored as 5404319552844595 × 2⁻⁵⁴. + +## 5. Why it *prints* as 0.30000000000000004, while 0.1 prints as 0.1 + +If 0.1 isn't stored exactly, why does `print(0.1)` show `0.1`? + +Modern languages print the **shortest decimal string that maps back to the same double**. "0.1" is shorter than "0.1000000000000000055511…", and reading "0.1" back gives exactly the same stored value. So the language shows you "0.1". It hides the error, but nothing is lost. + +For the sum, "0.3" would map back to the *other* double (the lower neighbour). The shortest string that identifies the upper one takes 17 significant digits: `0.30000000000000004`. That trailing `4` isn't noise. It's the shortest way to tell the two doubles apart. + +Seeing the error at all is the exception. Most of the time the rounding stays hidden below the printed digits. For example, `0.1 + 0.7` happens to print `0.7999999999999999`, but `0.1 + 0.4` prints `0.5`, because the rounding happened to land on the double nearest 0.5. + +## 6. Why computers use binary floating point anyway + +- **Speed and hardware:** binary arithmetic is what the circuits do natively. Doubles are added and multiplied in a few nanoseconds by dedicated hardware. +- **Range:** the exponent lets one 64-bit format hold numbers from about 10⁻³⁰⁸ to 10³⁰⁸. +- **Bounded, predictable error:** IEEE 754 guarantees that each basic operation (+, −, ×, ÷, √) gives the exact result *correctly rounded* to the nearest double. The relative error per operation is at most about 1.1 × 10⁻¹⁶. That's far more precision than almost any physical measurement has. + +The cost is the one you've seen: decimal fractions that look simple to us are not simple in base 2. + +## 7. What to do about it + +**Don't compare floats with `==`** when they come out of a calculation. Check whether they are *close enough*: + +```python +import math +math.isclose(0.1 + 0.2, 0.3) # True (default relative tolerance 1e-9) +abs((0.1 + 0.2) - 0.3) < 1e-9 # True, a hand-written version +``` + +What counts as "close enough" depends on your problem. A relative tolerance (a fraction of the numbers' size) usually works better than a fixed absolute one. + +**For money and other exact decimal quantities, don't use binary floats.** Two common options: + +- Store **integers in the smallest unit** (cents rather than dollars): `10 + 20 == 30` is exact. +- Use a **decimal type**, which stores digits in base 10: Python's `decimal.Decimal`, Java's `BigDecimal`, C#'s `decimal`, SQL's `NUMERIC`. + +```python +from decimal import Decimal +Decimal("0.1") + Decimal("0.2") == Decimal("0.3") # True +``` + +(Create the Decimal from the *string* `"0.1"`. `Decimal(0.1)` would faithfully copy the already-rounded binary value.) + +**For exact rational arithmetic**, some languages provide fractions, such as Python's `fractions.Fraction(1, 10)`. They're slower, but never round. + +**Whole numbers are safe.** Every integer up to 2⁵³ (about 9 × 10¹⁵) is stored exactly in a double, so integer arithmetic within that range has no rounding at all. + +--- + +## Summary + +- Binary can only write fractions with power-of-2 denominators exactly. 0.1, 0.2 and 0.3 are tenths and fifths, so they repeat forever in binary, like 1/3 does in decimal. +- A double keeps 53 binary digits, so each of those numbers is rounded to the nearest storable value. 0.1 and 0.2 round up slightly, and 0.3 rounds down slightly. +- Adding the rounded 0.1 and 0.2, then rounding the sum, lands one step above the stored 0.3. The two are different numbers, so `==` is false. +- That's correct IEEE 754 behaviour, not a bug. Compare with a tolerance, or use integers or decimal types when you need exact decimal results. diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/none/read.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/none/read.json new file mode 100644 index 0000000..a11d19f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/none/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "A computer stores numbers in binary. In binary, a fraction has a finite form only if its denominator is a power of 2. 0.1 is 1/10, and 10 = 2 × 5, so in binary it repeats forever, the way 1/3 does in decimal. A double keeps only 53 significant binary digits, so 0.1 is rounded to the nearest number it can store, which is slightly larger than 0.1.", + "quote": "So in binary, a fraction has a finite form **only if its denominator is a power of 2**: 1/2, 1/4, 3/8, 5/16, and so on." + }, + "2": { + "answer": "The stored 0.1 and 0.2 are both slightly too big, and the stored 0.3 is slightly too small. Their exact sum falls exactly halfway between two doubles. The tie rule, round half to even, picks the upper one. So the result is the double one step above the stored 0.3. They are different numbers, so == returns false.", + "quote": "Adding the rounded 0.1 and 0.2, then rounding the sum, lands one step above the stored 0.3. The two are different numbers, so `==` is false." + }, + "3": { + "answer": "It should check whether the two values are close enough, for example with math.isclose or abs(a - 0.3) < tolerance. A relative tolerance usually works better than a fixed one. For money or other exact decimal amounts, it should use integers in the smallest unit (such as cents) or a decimal type.", + "quote": "**Don't compare floats with `==`** when they come out of a calculation. Check whether they are *close enough*:" + }, + "4": { + "answer": "Yes, it should be true. The explanation doesn't test this sum, but its rule settles it. 0.5 = 1/2, 0.25 = 1/4 and 0.75 = 3/4 all have power-of-2 denominators, so all three are stored exactly. Addition gives the exact result correctly rounded. The exact sum 0.75 can be stored, so no rounding happens and the comparison is true.", + "quote": "IEEE 754 guarantees that each basic operation (+, −, ×, ÷, √) gives the exact result *correctly rounded* to the nearest double." + } + }, + "stops": [ + "The digits after the point (the *significand*) get a fixed budget: **53 significant binary digits**.", + "Near 0.3, the doubles are spaced about 5.55 × 10⁻¹⁷ apart.", + "pick the neighbour whose last binary digit is 0. That's the upper one.", + "Near 0.3 the stored values are whole multiples of\n> 2⁻⁵⁴", + "Every double has a finite decimal expansion, because its denominator is a power of 2.", + "Modern languages print the **shortest decimal string that maps back to the same double**.", + "A relative tolerance (a fraction of the numbers' size) usually works better than a fixed absolute one." + ], + "excess": [ + "For readers who like exact arithmetic: the double for 0.1 is exactly\n> 3602879701896397 × 2⁻⁵⁵", + "**Speed and hardware:** binary arithmetic is what the circuits do natively.", + "**Range:** the exponent lets one 64-bit format hold numbers from about 10⁻³⁰⁸ to 10³⁰⁸.", + "For example, `0.1 + 0.7` happens to print `0.7999999999999999`, but `0.1 + 0.4` prints `0.5`", + "Python's `decimal.Decimal`, Java's `BigDecimal`, C#'s `decimal`, SQL's `NUMERIC`.", + "**For exact rational arithmetic**, some languages provide fractions, such as Python's `fractions.Fraction(1, 10)`." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/none/run.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/none/run.json new file mode 100644 index 0000000..40a7d09 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I wrote `explainers/float-sum/explanation.md`. Prose works best because the explanation is a step-by-step argument (1/3 in decimal, then 0.1 in binary, then the exact stored values, then the tie-breaking rounding) with small worked tables, and neither a diagram nor an interactive page would add enough to justify a second file.", + "error": null, + "cost": 0.1607036, + "seconds": 49, + "turns": 2, + "files": [ + "explainers/float-sum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/explanation.md b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/explanation.md new file mode 100644 index 0000000..c56e195 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/explanation.md @@ -0,0 +1,109 @@ +# Why 0.1 + 0.2 is not 0.3 + +In Python, JavaScript, Java, C, and most other languages: + +```python +>>> 0.1 + 0.2 +0.30000000000000004 +>>> 0.1 + 0.2 == 0.3 +False +``` + +**Short answer:** the computer cannot store 0.1, 0.2, or 0.3 exactly. It stores the nearest value it can hold. The sum of the stored 0.1 and the stored 0.2 lands on a different stored value than the stored 0.3. This is not a bug. It is how the standard number format works, and the same thing happens with decimals on paper. + +This matters because equality tests on decimal fractions fail silently, and money calculations drift by fractions of a cent. + +## 1. The same problem in decimal + +Imagine a calculator that keeps only 4 decimal digits. + +- 1/3 is stored as 0.3333 (rounded down). +- 2/3 is stored as 0.6667 (rounded up). +- 1/3 + 1/3 = 0.3333 + 0.3333 = 0.6666. + +So on this calculator, 1/3 + 1/3 ≠ 2/3. Each number was rounded on its own, and the rounding errors do not cancel. Computers have exactly this problem, but in base 2, and 1/10 plays the role of 1/3. + +## 2. Why 0.1 has no exact binary form + +A fraction has a finite expansion in base 10 only if its denominator (in lowest terms) has no prime factors other than 2 and 5. That is why 1/4 = 0.25 ends, but 1/3 repeats. + +In base 2, the only allowed prime factor is 2. The denominator of 1/10 is 2 × 5. The factor 5 makes the binary expansion repeat forever: + +``` +0.1 (decimal) = 0.0001100110011001100110011... (binary, "0011" repeats) +``` + +The computer must cut this off somewhere, so it must round. + +## 3. What the computer actually stores + +Most languages use the **double** format (IEEE 754 binary64), which the processor supports in hardware. A double holds a number as + +``` +(an integer of at most 53 bits) × (a power of 2) +``` + +So the values a double can hold form a grid. Between 0.25 and 0.5, the grid step is 2⁻⁵⁴ ≈ 5.55 × 10⁻¹⁷. Every literal you type is rounded to the nearest grid point: + +| You type | Stored as integer × power of 2 | Exact stored value | Error | +|---|---|---|---| +| `0.1` | 7205759403792794 × 2⁻⁵⁶ | 0.1000000000000000055511151231… | slightly high | +| `0.2` | 7205759403792794 × 2⁻⁵⁵ | 0.2000000000000000111022302462… | slightly high | +| `0.3` | 5404319552844595 × 2⁻⁵⁴ | 0.2999999999999999888977697537… | slightly low | + +Note that the stored 0.2 is exactly twice the stored 0.1. Doubling is exact in binary: only the power of 2 changes. + +The errors already point in different directions. 0.1 and 0.2 are a little too high. 0.3 is a little too low. + +## 4. The addition, step by step + +Write everything in units of 2⁻⁵⁴, the grid step near 0.3: + +- stored 0.1 = 1801439850948198.5 units +- stored 0.2 = 3602879701896397 units +- exact sum = **5404319552844595.5** units + +The exact sum is not a grid point. The processor must round it again. It sits exactly halfway between two grid points: + +``` + stored 0.3 result + 5404319552844595 exact sum 5404319552844596 + ───────────●─────────────────○─────────────────●─────────── + 0.29999999999999998890 0.30000000000000001665 0.30000000000000004441 + ←— half a step —→←— half a step —→ +``` + +For an exact tie, the IEEE 754 rule is **round half to even**: choose the grid point whose integer is even. 5404319552844596 is even, so the result goes up. + +The result is therefore one grid step above the stored 0.3. The two doubles differ, so `==` returns `False`. + +## 5. Why it prints as 0.30000000000000004 + +This step explains a common confusion: if 0.1 is not stored exactly, why does `print(0.1)` show `0.1`? + +Languages print the **shortest decimal string that rounds back to the same double**. For the stored 0.1, the string `0.1` is enough, because no other double is nearer to 0.1. This hides the error. + +For the result of the addition, `0.3` is not enough. The string `0.3` rounds to the other grid point, the stored 0.3. The shortest string that identifies the result is `0.30000000000000004`. The printed form reveals the error only because the error moved the value to a different grid point. + +## 6. When addition is exact + +The problem comes from denominators with factors other than 2. Fractions with power-of-2 denominators are stored exactly, and their sums are exact while they fit in 53 bits: + +```python +>>> 0.5 + 0.25 == 0.75 +True +``` + +Here 1/2, 1/4, and 3/4 are all grid points, so no rounding happens. + +## 7. What to do in code + +- **Compare with a tolerance**, not with `==`: `math.isclose(0.1 + 0.2, 0.3)` in Python, or `abs(a - b) < 1e-9` with a tolerance that suits your data. +- **Use integers for money**: store 10 cents as `10`, not `0.10`. +- **Use a decimal type** when the decimal digits must be exact: `decimal.Decimal("0.1")` in Python, `BigDecimal` in Java. These types store base-10 digits, so 0.1 is exact. They are slower than doubles. + +## Summary + +0.1, 0.2, and 0.3 each have a repeating binary expansion, so each is rounded to the nearest double. The stored 0.1 and 0.2 are slightly high and the stored 0.3 is slightly low. Their exact sum falls halfway between two doubles, and the tie rule rounds it up, one step above the stored 0.3. The values differ, so the comparison is false. + +Not covered: 32-bit floats, special values (NaN, infinity), and how errors grow across long chains of operations. diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/model.md b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/model.md new file mode 100644 index 0000000..7a20360 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/model.md @@ -0,0 +1,30 @@ +# Model: why 0.1 + 0.2 != 0.3 + +Reader: technical, school maths, basic programming; new to floating point. + +Central question: why does `0.1 + 0.2 == 0.3` give false? + +Answer in one line: the computer stores each number as the nearest binary fraction, +and the three roundings (0.1, 0.2, and the sum) do not land on the same stored value as 0.3. + +Entities +- double (IEEE 754 binary64): a 53-bit integer significand times a power of two. +- grid: the set of values a double can hold; near 0.3 the spacing is 2^-54. +- rounding: each literal and each operation result goes to the nearest grid point; ties go to the even significand. +- printing: languages print the shortest decimal that maps back to the same double. + +Causal chain +1. 1/10 has a factor 5 in its denominator, so its binary expansion repeats (0.000110011...). +2. Storage rounds 0.1 and 0.2 slightly up; 0.3 is rounded slightly down. +3. The exact sum of the stored 0.1 and 0.2 lies exactly halfway between two grid points. +4. Round-half-to-even picks the upper point, one grid step above stored 0.3. +5. Shortest-round-trip printing shows that point as 0.30000000000000004. + +Analogy: 4-digit decimal, 1/3 + 1/3 -> 0.3333 + 0.3333 = 0.6666, but 2/3 is stored as 0.6667. + +Holds / breaks: 0.5 + 0.25 == 0.75 is exact (denominators are powers of 2). + +Confusion points: "0.1 prints as 0.1, so it is stored exactly" (false: printing hides it); +"it is a bug in the language" (false: it is the hardware number format). + +Scope: no float32, no NaN/infinity, no error analysis of long sums. diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/model.yaml b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/model.yaml new file mode 100644 index 0000000..bad74c2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/explainers/float-sum/model.yaml @@ -0,0 +1,11 @@ +format: IEEE 754 binary64 +significand_bits: 53 +grid_step_near_0_3: "2^-54 ≈ 5.55e-17" +stored: + "0.1": {integer: 7205759403792794, scale: "2^-56", exact: "0.1000000000000000055511151231257827021181583404541015625"} + "0.2": {integer: 7205759403792794, scale: "2^-55", exact: "0.200000000000000011102230246251565404236316680908203125"} + "0.3": {integer: 5404319552844595, scale: "2^-54", exact: "0.299999999999999988897769753748434595763683319091796875"} +exact_sum_of_stored: {integer: 5404319552844595.5, scale: "2^-54", exact: "0.3000000000000000166533453693773481063544750213623046875"} +result: {integer: 5404319552844596, scale: "2^-54", exact: "0.3000000000000000444089209850062616169452667236328125", printed: "0.30000000000000004"} +tie_rule: round half to even +exact_example: "0.5 + 0.25 == 0.75" diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/read.json new file mode 100644 index 0000000..8378c95 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/read.json @@ -0,0 +1,41 @@ +{ + "answers": { + "1": { + "answer": "In binary, a fraction ends only if its denominator has no prime factor other than 2. 1/10 has a factor of 5, so its binary expansion repeats forever. The computer has to cut it off, which means rounding it to the nearest value it can hold.", + "quote": "In base 2, the only allowed prime factor is 2. The denominator of 1/10 is 2 × 5. The factor 5 makes the binary expansion repeat forever:" + }, + "2": { + "answer": "The stored 0.1 and 0.2 are both a little too high, and the stored 0.3 is a little too low. Their exact sum falls exactly halfway between two grid points. The round-half-to-even rule rounds it up, so the result is one grid step above the stored 0.3, and == returns False.", + "quote": "Their exact sum falls halfway between two doubles, and the tie rule rounds it up, one step above the stored 0.3. The values differ, so the comparison is false." + }, + "3": { + "answer": "Compare with a tolerance, for example math.isclose(a, 0.3) or abs(a - b) < 1e-9 with a tolerance that suits the data. For money, use integers (cents). Where decimal digits must be exact, use a decimal type.", + "quote": "**Compare with a tolerance**, not with `==`: `math.isclose(0.1 + 0.2, 0.3)` in Python, or `abs(a - b) < 1e-9` with a tolerance that suits your data." + }, + "4": { + "answer": "Yes, it is True. 1/2, 1/4 and 3/4 have power-of-2 denominators, so all three are exact grid points and no rounding happens. The explanation shows this in Python only. It extends to the other languages through its claim that most languages use the same double format.", + "quote": "Here 1/2, 1/4, and 3/4 are all grid points, so no rounding happens." + } + }, + "stops": [ + "Term never explained: \"IEEE 754 binary64\"", + "No reason given for this grid step: \"Between 0.25 and 0.5, the grid step is 2⁻⁵⁴\"", + "Unclear why the integer is limited to 53 bits or what happens past that: \"(an integer of at most 53 bits) × (a power of 2)\"", + "A count of grid units that is not a whole number, measured on 0.1's scale using 0.3's grid step: \"stored 0.1 = 1801439850948198.5 units\"", + "No reason given for this tie rule or why even is preferred: \"the IEEE 754 rule is **round half to even**\"", + "The limit is never explained: \"their sums are exact while they fit in 53 bits\"", + "The table values have to be taken on trust: \"7205759403792794 × 2⁻⁵⁶\"" + ], + "excess": [ + "Explains printing, which is beside the main question: \"Languages print the **shortest decimal string that rounds back to the same double**.\"", + "Lists topics left out: \"Not covered: 32-bit floats, special values (NaN, infinity), and how errors grow across long chains of operations.\"", + "Repeats the short answer and sections 3–4: \"0.1, 0.2, and 0.3 each have a repeating binary expansion, so each is rounded to the nearest double.\"", + "Restates the error directions just shown in the table: \"The errors already point in different directions. 0.1 and 0.2 are a little too high. 0.3 is a little too low.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/run.json new file mode 100644 index 0000000..942bfb3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/float-sum/explanation.md`, along with `model.md` and `model.yaml` to hold the facts and numbers it relies on. I chose prose because the cause is one short chain: round each number to binary, add, round once more. A worked table and one small number-line sketch carry the numbers, so a diagram or interactive page would add little. I checked the binary arithmetic by hand, but I couldn't run `explainer check` because I have no shell here.", + "error": null, + "cost": 0.2104636, + "seconds": 55, + "turns": 4, + "files": [ + "explainers/float-sum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/explanation.md b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/explanation.md new file mode 100644 index 0000000..eaa51b2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/explanation.md @@ -0,0 +1,67 @@ +# Why does 0.1 + 0.2 not equal 0.3? + +In Python, JavaScript, and most other languages, `0.1 + 0.2 == 0.3` is `false`, and `0.1 + 0.2` prints `0.30000000000000004`. + +**The computer never holds 0.1, 0.2, or 0.3. It holds the nearest numbers it can store in binary. The addition is correct, but its result rounds to the next storable number above the one that 0.3 rounds to.** + +## What the computer holds + +A literal such as `0.1` becomes a **double**: the 64-bit binary format that most languages use for numbers with a decimal point. A double is a whole number of at most 53 binary digits, times a power of two. The other 11 bits hold the sign and the power. For example, 0.75 = 3 × 2^-2, so a double holds 0.75 exactly. + +## Why 0.1 must be rounded + +A whole number times a power of two is a fraction whose denominator is a power of two. A fraction like that ends in binary: 3/8 = 0.011. The denominator of 1/10 has a factor 5. No power of two is divisible by 5, so 1/10 never ends in binary: + + 1/10 = 0.000110011001100110011… (0011 repeats forever) + +This is the same as 1/3 = 0.3333… in decimal. A machine that keeps only a fixed number of digits must cut the expansion, so it rounds. With 4 decimal digits, 1/3 becomes 0.3333, and 0.3333 + 0.3333 = 0.6666, but 2/3 becomes 0.6667. The same thing happens with 0.1, 0.2, and 0.3 in binary. + +## The stored values + +The **stored value** of a literal is the double nearest to it. The **gap** is the distance between two neighboring doubles. + +The gap depends on the size of the number. Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54. The ends check: 2^52 × 2^-54 = 0.25, and 2^53 × 2^-54 = 0.5. Neighboring whole numbers differ by 1, so neighboring doubles here differ by u = 2^-54 ≈ 5.55e-17. Between 0.125 and 0.25 the multiplier is 2^-55, so the gap is u/2. Between 0.0625 and 0.125 the gap is u/4. + +To see the rounding, count each number in units of u. For example, 0.3 × 2^54 = 5404319552844595.2, so 0.3 is 5404319552844595.2 u. The nearest doubles are at 5404319552844595 u and 5404319552844596 u. The nearer one is …595 (short for 5404319552844595), so 0.3 rounds down. 0.1 is in the range where the gap is u/4, so its stored value is a multiple of u/4, such as …198.5. + +The **error** is the stored value minus the decimal: 0.1 and 0.2 round up, and 0.3 rounds down. + +| Number | Exact, in u | Gap here | Stored, in u | Stored, in decimal | Error | +|---|---|---|---|---|---| +| 0.1 | 1801439850948198.4 | u/4 | 1801439850948198.5 | 0.10000000000000000555 | +5.55e-18 | +| 0.2 | 3602879701896396.8 | u/2 | 3602879701896397 | 0.20000000000000001110 | +1.11e-17 | +| 0.3 | 5404319552844595.2 | u | 5404319552844595 | 0.29999999999999998890 | −1.11e-17 | +## What `+` does + +The `+` operation first finds the **exact sum** of the two stored values. Then it rounds that exact sum to a double, as every arithmetic operation does. The result is the **rounded sum**. + + exact sum = 1801439850948198.5 + 3602879701896397 = 5404319552844595.5 u + +The exact sum is exactly halfway between the doubles at …595 u and …596 u. This case is a **tie**. The IEEE 754 standard, which sets these rules, sends a tie to the neighbor whose count of u is even. 5404319552844596 is even, so the rounded sum is 5404319552844596 u, which is 0.30000000000000004441. + +Stored 0.3 is at …595 u. The rounded sum and stored 0.3 are neighboring doubles, one gap (5.55e-17) apart. They are different numbers, so `==` is false. + +The addition did not make a mistake. The rounded sum is the nearest double to the exact sum of its inputs. The inputs were not 0.1 and 0.2. + +## Why `print(0.1)` shows `0.1` + + +Languages print the shortest decimal that rounds back to the same double. The text "0.1" rounds back to stored 0.1, so one digit is enough. The text "0.3" rounds back to …595 u, not to the rounded sum at …596 u. The shortest decimal that rounds back to …596 u is 0.30000000000000004, with 17 significant digits. + +## When it works + + +`0.5 + 0.25 == 0.75` is true. All three are fractions with power-of-two denominators (1/2, 1/4, 3/4), so each is stored with error 0, and the exact sum is itself a double. The failure needs a decimal whose denominator has a factor 5, such as 0.1. + +A decimal type also avoids the problem. Python's `Decimal("0.1") + Decimal("0.2") == Decimal("0.3")` is true, because `Decimal` stores base-10 digits, and 1/10 ends in base 10. + +## What to do + + +Do not compare doubles with `==` when they come from decimal literals or from arithmetic. Compare with a tolerance instead: `abs((0.1 + 0.2) - 0.3) < 1e-9` is true, because the rounded sum and stored 0.3 are one gap (5.55e-17) apart. Choose a tolerance much larger than the gap, but smaller than any difference that matters to you; for values near 1, 1e-9 meets both conditions. For money, count whole cents as integers, or use a decimal type. + +## Not covered here + +- The sign and exponent bits of a double: see IEEE 754, or Goldberg, "What Every Computer Scientist Should Know About Floating-Point Arithmetic" (1991). +- How errors grow over many operations, such as adding 0.1 ten times: see a numerical analysis text. +- How to choose between an absolute and a relative tolerance: see the documentation for Python's `math.isclose`. diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/model.md b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/model.md new file mode 100644 index 0000000..db99cce --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/model.md @@ -0,0 +1,130 @@ +# Semantic model: why 0.1 + 0.2 != 0.3 in floating point + +> Source of truth. Every rendering derives from this file. + +## Central question + +Why does `0.1 + 0.2 == 0.3` evaluate to false in most programming languages? + +## Audience and prior knowledge + +A technical reader: school mathematics and science, basic programming. Knows decimals, fractions, powers, that computers store numbers in binary, and what `==` does. Does not know how a float is stored, which fractions end in binary, what rounding to a fixed number of bits does, or how languages print floats. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| double | The 64-bit binary floating-point format (IEEE 754 binary64) that most languages use for a number with a decimal point. | fact | +| significant bits | A double keeps 53 binary digits: every double is a whole number below 2^53 times a power of two. | fact | +| gap | The distance between two neighboring doubles. Because the 53 bits are fixed, the gap halves each time the size of the number halves. | fact | +| stored value | The double nearest to the decimal you wrote; what the program holds for `0.1`. | fact | +| error | stored value − written decimal; + means the stored value is above the decimal. | definition | +| exact sum | The true sum of two stored values, before rounding. | definition | +| rounded sum | The exact sum after rounding to the nearest double; this is the result of `+`. | fact | +| rounding | Replacing an exact number with the nearest double. A tie (exactly halfway) goes to the neighbor whose whole number is even. | fact | +| printed value | The shortest decimal that rounds back to the same double; what `print` shows. | fact (Python, JavaScript, and most modern languages) | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| decimal 0.1 | is rounded to | stored value | 1/10 does not end in binary | +| stored 0.1, stored 0.2 | add to | exact sum | | +| exact sum | is rounded to | rounded sum | every arithmetic operation rounds its result | +| rounded sum | is one gap above | stored 0.3 | so `==` is false | +| printed value | hides | the error of stored 0.1 | | + +## Dependencies (learning order) + +1. A fixed number of digits cannot hold every fraction (decimal: 1/3 → 0.3333). +2. A fraction ends in binary only if its denominator (in lowest terms) is a power of two; 1/10 has a factor 5. +3. So 0.1, 0.2, 0.3 each become the nearest double, with their own error. +4. Measured in the gap near 0.3, the exact sum falls at a half; it rounds up, while 0.3 rounded down. +5. Printing shows the shortest decimal that picks out the double, so 0.1 looks exact and the sum does not. + +## Causal chain + +```text +1/10 has a factor 5 in its denominator → no finite binary expansion +→ 0.1 and 0.2 round up, 0.3 rounds down (errors +5.55e-18, +1.11e-17, −1.11e-17) +→ exact sum = stored 0.3 + half a gap → tie → rounds to the even neighbor, the double above +→ rounded sum and stored 0.3 are neighboring, different doubles → == is false +``` + +## Quantities + +Unit u = 2^-54 ≈ 5.55e-17, the gap between doubles from 0.25 to 0.5. From 0.125 to 0.25 the gap is u/2; from 0.0625 to 0.125 it is u/4. + +| Number | Exact, in u | Allowed steps | Stored, in u | Stored, in decimal | Error | +|---|---|---|---|---|---| +| 0.1 | 1801439850948198.4 | u/4 | 1801439850948198.5 | 0.10000000000000000555 | +5.55e-18 | +| 0.2 | 3602879701896396.8 | u/2 | 3602879701896397 | 0.20000000000000001110 | +1.11e-17 | +| exact sum | 5404319552844595.5 | — | — | 0.30000000000000001665 | +1.67e-17 | +| rounded sum | — | u | 5404319552844596 | 0.30000000000000004441 | +4.44e-17 | +| 0.3 | 5404319552844595.2 | u | 5404319552844595 | 0.29999999999999998890 | −1.11e-17 | + +- Check: 0.3 × 2^54 = 5404319552844595.2; 0.1 × 2^56 = 7205759403792793.6 → 7205759403792794 quarter-units = 1801439850948198.5 u. +- Exact sum: 1801439850948198.5 + 3602879701896397 = 5404319552844595.5. A tie between …595 and …596; …596 is even, so the rounded sum is …596. (Round half up would also give …596.) +- Difference rounded sum − stored 0.3 = 1 u ≈ 5.55e-17. Relative: 5.55e-17 / 0.3 ≈ 1.9e-16, so the first 16 significant digits agree and the difference shows in the 17th: 0.3000000000000000|4. +- Printing: "0.1" rounds to stored 0.1, so one digit is enough. For the rounded sum, "0.3" and every shorter decimal round to stored 0.3, the wrong double; the first that works is 0.30000000000000004 (17 significant digits). +- Binary of 1/10: 0.000110011001100110011… (0011 repeats). +- Gap meaning: near 0.3 the gap is 5.55e-17 (fine); near 10^16 it is 2, so 10^16 + 1 cannot be stored. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| 0.5 + 0.25 == 0.75 | true | 1/2, 1/4, 3/4 have power-of-two denominators: errors 0, exact sum is a double | +| decimal type (Python `Decimal("0.1")`) | 0.1 + 0.2 == 0.3 is true | digits stored in base 10, where 1/10 ends | +| 4-digit decimal calculator | 1/3 → 0.3333; 0.3333 + 0.3333 = 0.6666, but 2/3 → 0.6667 | the same effect in base 10 with 1/3 | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Fixed 53-bit binary storage | Store exact decimal digits | Exact storage of every fraction needs unbounded memory (1/3 has no finite decimal either); fixed-size binary is what the processor adds in one step. Decimal types exist and are slower. | +| Compare with a tolerance: abs(a − b) < 1e-9 | `==` | `0.1 + 0.2 == 0.3` is false though the difference is 5.55e-17. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here (numbers) | +|---|---|---| +| A sum of stored values equals the stored value of the decimal sum | 0.5 + 0.25 == 0.75: all errors 0 | 0.1 + 0.2: rounded sum …596 u, stored 0.3 …595 u | +| Only power-of-two denominators end in binary | 3/8 = 0.011 in binary | 1/10: 2^k / 10 is never whole, because no power of 2 is divisible by 5 | +| Fixed digits lose information (decimal analogy) | 1/4 = 0.25 | 1/3 → 0.3333; 0.3333 + 0.3333 = 0.6666 ≠ 0.6667 | + +## Terms + +- **double**: the 64-bit binary format for numbers with a decimal point. +- **gap**: the distance between neighboring doubles; u = 5.55e-17 near 0.3. Small gap = fine resolution; near 10^16 the gap is 2. +- **stored value**: the double the program holds for a literal like `0.1`. Never call it "0.1" alone. +- **error**: stored value − written decimal. +- **exact sum** vs **rounded sum**: before and after rounding. Never "the sum" alone. +- **tie**: an exact number exactly halfway between two doubles. +- **printed value**: the shortest decimal that rounds back to the same double. + +## Epistemic status + +- **Established:** IEEE 754 binary64, round half to even, the stored values above. +- **Mathematical consequence:** 1/10 has no finite binary expansion. +- **Observed:** Python and JavaScript print `0.30000000000000004` for `0.1 + 0.2`. + +## Confusion points + +- "The addition is buggy" → the addition is correct to the last bit; the inputs were never exactly 0.1 and 0.2. +- "0.1 is stored exactly, because it prints as 0.1" → printing shows the shortest decimal that picks out the double. +- "The result is random noise" → it is deterministic: the same on every IEEE 754 machine. +- "Floats are always inexact" → 0.5 + 0.25 == 0.75 is exact. + +## Scope + +- **Out of scope:** sign and exponent bits — IEEE 754; Goldberg, "What Every Computer Scientist Should Know About Floating-Point Arithmetic" (1991). +- **Out of scope:** error growth in long sums (adding 0.1 ten times) — numerical analysis texts. +- **Out of scope:** how to choose a tolerance (relative vs absolute) — `math.isclose` documentation. +- **Out of scope:** languages whose literals are not doubles by default (Raku uses exact fractions) and fused or extended-precision hardware. + +## Representation decision + +- **Stage:** 1 (prose) +- **Reason:** one sequential causal chain with worked numbers; nothing for the reader to explore or to see move. A table carries the numbers. +- **Levels:** L1 the answer / L2 stored values and the rounded sum / L3 why 1/10 does not end in binary / L4 bit layout (Scope). diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/model.yaml b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/model.yaml new file mode 100644 index 0000000..8cb1cf5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/model.yaml @@ -0,0 +1,99 @@ +# Machine-readable half of model.md. `explainer check float-sum` holds every rendering to it. + +values: + bits: 53 + printed_sum: 0.30000000000000004 + gap_u: 5.55e-17 + stored: + p1: 0.10000000000000000555 + p2: 0.20000000000000001110 + p3: 0.29999999999999998890 + exact_sum: 0.30000000000000001665 + rounded_sum: 0.30000000000000004441 + units: + p1_exact: 1801439850948198.4 + p1_stored: 1801439850948198.5 + p2_exact: 3602879701896396.8 + p2_stored: 3602879701896397 + p3_exact: 5404319552844595.2 + p3_stored: 5404319552844595 + exact_sum: 5404319552844595.5 + rounded_sum: 5404319552844596 + errors: + p1: 5.55e-18 + p2: 1.11e-17 + p3: -1.11e-17 + binary_tenth: 0.000110011001100110011 + exponent_u: -54 + sum_exact_u: 5404319552844595.5 + sum_rounded_u: 5404319552844596 + p3_stored_u: 5404319552844595 + +allow: ["0011", -2, 52, 54, -55, 198.5, 595, 596, 15, 0.25, 0.5, 0.75, 0.125, 0.0625, 1, 2, 3, 4, 5, 8, 10, 16, 17, 64, 754, 0.3333, 0.6666, 0.6667, 1e-9, 0.011] + +require: + - path: printed_sum + in: [prose] + - path: sum_exact_u + in: [prose] + - path: sum_rounded_u + in: [prose] + - path: p3_stored_u + in: [prose] + - path: gap_u + in: [prose] + +claims: + - id: tenth-not-binary + kind: mechanism + statement: "A fraction ends in binary only if its denominator is a power of two; 1/10 has a factor 5, so 0.1 has no finite binary expansion." + example: "3/8 = 0.011 in binary ends; 1/10 = 0.000110011… repeats, because 2^k / 10 is never a whole number." + ask: "Why can a double hold 0.75 exactly but not 0.1?" + - id: stored-values + kind: mechanism + statement: "Each literal becomes the nearest double; 0.1 and 0.2 round up, 0.3 rounds down." + example: "In units of u = 2^-54: 0.1 = 1801439850948198.4 → .5; 0.2 = 3602879701896396.8 → 3602879701896397; 0.3 = 5404319552844595.2 → 5404319552844595." + - id: sum-rounds-up + kind: mechanism + statement: "The exact sum of the stored values lands exactly halfway between two doubles and rounds to the upper one, one gap above stored 0.3." + example: "1801439850948198.5 + 3602879701896397 = 5404319552844595.5 → tie → 5404319552844596 (even); stored 0.3 is 5404319552844595." + ask: "The stored 0.1 and stored 0.2 are each very close to 0.1 and 0.2. Why is their sum not the same double as stored 0.3?" + - id: exact-when-binary + kind: guarantee + statement: "When every number is a binary fraction, float addition is exact and == behaves as in school arithmetic." + example: "0.5 + 0.25 == 0.75 is true." + counterexample: "0.1 + 0.2 == 0.3 is false: 1/10 is not a binary fraction." + - id: printing-hides + kind: misconception + correction: "The stored value of 0.1 is 0.10000000000000000555; printing hides the error." + statement: "print(0.1) shows 0.1 because languages print the shortest decimal that rounds back to the same double, not the exact stored value." + example: "\"0.1\" picks out stored 0.1, so one digit suffices; for the rounded sum, \"0.3\" picks out the wrong double, so 17 digits are needed: 0.30000000000000004." + ask: "If 0.1 is not stored exactly, why does print(0.1) show 0.1?" + - id: why-tolerance + kind: why + statement: "Compare floats with a tolerance, or use a decimal type, instead of ==." + alternative: "Compare with ==" + counterexample: "0.1 + 0.2 == 0.3 is false although the difference is only 5.55e-17; abs((0.1 + 0.2) - 0.3) < 1e-9 is true." + +accept_unjustified: {} + +terms: + - term: stored value + means: "the double held for a literal; never plain '0.1 is stored'" + avoid: [stored exactly as 0.1] + - term: rounded sum + means: "the result of +, after rounding; 'exact sum' is before rounding" + avoid: [the true result of +] + +budget: {prose: 900, reason: "the mechanism needs a five-row number table, the binary argument, and the printing step"} + +quiz: + - q: "Does 0.5 + 0.25 == 0.75 evaluate to true or false with doubles, and why?" + expect: "True: 1/2, 1/4, 3/4 have power-of-two denominators, so all are stored exactly and the sum is exact." + misconception: "False, because floats are always inexact." + - q: "Is the + operation itself making an error that the hardware could avoid? Where does the difference come from?" + expect: "No; + gives the nearest double to the exact sum of its inputs. The inputs were already not 0.1 and 0.2 (1/10 does not end in binary), and their small errors push the exact sum to a point that rounds to a different double than 0.3 does." + misconception: "The addition circuit is imprecise / it is a bug." + - q: "Would 0.1 + 0.2 == 0.3 be true using Python's Decimal type with Decimal('0.1') etc.? Why?" + expect: "Yes: Decimal stores base-10 digits, where 1/10 ends, so no rounding occurs." + misconception: "No, computers can never represent 0.1." diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/narrative.md b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/narrative.md new file mode 100644 index 0000000..4024d15 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/narrative.md @@ -0,0 +1,53 @@ +# Narrative: float-sum + +## 1. Reader before and after + +- **Before:** A technical reader: school mathematics, basic programming. Knows binary integers, decimals, fractions, powers, and `==`. Has seen `0.1 + 0.2` print `0.30000000000000004` or heard of it. Does not know how a float is stored. +- **After:** "0.1 and 0.2 cannot be stored exactly in binary, so the computer adds two nearby numbers; their sum rounds to a different number than 0.3 does." + +## 2. Question and motive + +- **Question:** Why is `0.1 + 0.2 == 0.3` false? Result in words: the computer never holds 0.1, 0.2, or 0.3. It holds the nearest binary numbers. The addition is correct, but the nearest number to the sum is one step above the nearest number to 0.3. +- **Why care:** `==` on decimals silently fails in prices, loops, and tests. +- **Why this approach:** ask what number the computer actually holds for `0.1`; everything follows from that. + +## 3. Introduction ledger + +| Reference | Means | Grounded by | Beat | +|---|---|---|---| +| double | the 64-bit binary format for decimal-point numbers | `x = 0.1` in Python or JavaScript | 2 | +| 53 bits | a double is a whole number below 2^53 times a power of two | 0.75 = 3 × 2^-2 | 2 | +| binary fraction / ends in binary | denominator is a power of two | 3/8 = 0.011 | 3 | +| stored value | the nearest double to the literal | stored 0.1 = 0.10000000000000000555 | 4 | +| error | stored value − written decimal | +5.55e-18 for 0.1 | 4 | +| gap, u | distance between neighboring doubles; u = 2^-54 near 0.3 | 0.3 sits between …595 u and …596 u | 4 | +| exact sum / rounded sum | before / after rounding the result of + | …595.5 u / …596 u | 5 | +| tie, round half to even | exactly halfway → the even whole number | …595.5 → …596 | 5 | +| printed value | shortest decimal that rounds back to the same double | "0.1"; "0.30000000000000004" | 6 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | Why is it false? | — | `0.1 + 0.2 == 0.3` → false; prints 0.30000000000000004 | "The computer never holds 0.1, 0.2, or 0.3. It holds the nearest binary numbers. The addition is correct, but the sum rounds to the number one step above the one that 0.3 rounds to." | the result in words | +| 2 | instance | What does the computer hold for `0.1`? | so | 0.75 = 3 × 2^-2 | "In Python, JavaScript, and most languages, a literal like 0.1 becomes a double: a whole number below 2^53 times a power of two. The whole number has at most 53 binary digits." | the format | +| 3 | mechanism | Can 0.1 be written that way? | but | 1/3 = 0.3333… in decimal; 3/8 = 0.011; 1/10 = 0.000110011… in binary | "A whole number times a power of two is a fraction whose denominator is a power of two. 1/10 has a factor 5 in its denominator, so it repeats forever in binary, as 1/3 does in decimal. The computer must round it." | 0.1 must be rounded | +| 4 | mechanism | How far off is it, and which way? | therefore | table: exact and stored values counted in u; errors | "Neighboring doubles are one gap apart. Between 0.25 and 0.5 the gap is u = 2^-54 ≈ 5.55e-17; for numbers half as big the gap halves (u/2 for 0.2, u/4 for 0.1). Count each number in u: 0.3 is 5404319552844595.2 u, between doubles at …595 and …596; nearest is …595, so 0.3 rounds down. 0.1 and 0.2 round up. The error is stored value minus decimal." | the three stored values | +| 5 | mechanism | Then what does + do? | so | 1801439850948198.5 + 3602879701896397 = 5404319552844595.5 → 5404319552844596 | "+ first finds the exact sum of the two stored values, then rounds it to a double, like every arithmetic operation. The exact sum is exactly halfway between …595 and …596: a tie. The rule sends ties to the even whole number, …596. Stored 0.3 is …595, one gap below." | why == is false | +| 6 | objection | Why does print(0.1) show 0.1? | but | "0.1" vs "0.30000000000000004" | "Printing shows the shortest decimal that rounds back to the same double. '0.1' rounds back to stored 0.1. '0.3' rounds back to …595, not …596, so the rounded sum needs 17 digits." | printing hides the error | +| 7 | test | Does it always fail? | so | 0.5 + 0.25 == 0.75 → true; Python Decimal('0.1') + Decimal('0.2') == Decimal('0.3') → true | "Doubles are exact when every denominator is a power of two. Decimal types store base-10 digits, where 1/10 ends." | when it holds | +| 8 | close | What do I do? | therefore | `abs(a - b) < 1e-9`; money in whole cents | "Compare with a tolerance much larger than the gap but smaller than any difference you care about. For money, count whole cents as integers, or use a decimal type." | the fix | + +## 5. Concrete to symbol + +u is introduced as "the gap near 0.3, 2^-54 ≈ 5.55e-17" with the instance 0.3 × 2^54 = 5404319552844595.2. + +## 6. Links between representations + +Each row of the table pairs the value in u with the decimal value; the prose names the row. + +## 7. Close the loop + +- **Answer:** The computer adds the nearest doubles to 0.1 and 0.2, not 0.1 and 0.2. Their exact sum rounds to the double above 0.3, while 0.3 itself rounded to the double below. +- **General argument:** any decimal whose denominator has a factor 5 is rounded; sums of rounded numbers can round to a different neighbor than the rounded decimal sum. +- **Payoff:** predict when `==` is safe; use tolerance or decimal types. Scope pointers. diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/decisions.yaml new file mode 100644 index 0000000..ff9a8b8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/decisions.yaml @@ -0,0 +1,189 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T151909-probe#1: + decision: adopted +2026-10-04T151909-probe#10: + decision: adopted +2026-10-04T151909-probe#11: + decision: adopted +2026-10-04T151909-probe#12: + decision: adopted +2026-10-04T151909-probe#13: + decision: adopted +2026-10-04T151909-probe#14: + decision: adopted +2026-10-04T151909-probe#15: + decision: adopted +2026-10-04T151909-probe#2: + decision: adopted +2026-10-04T151909-probe#3: + decision: adopted +2026-10-04T151909-probe#4: + decision: adopted +2026-10-04T151909-probe#5: + decision: declined +2026-10-04T151909-probe#6: + decision: adopted +2026-10-04T151909-probe#7: + decision: declined +2026-10-04T151909-probe#8: + decision: declined +2026-10-04T151909-probe#9: + decision: adopted +2026-10-04T152128-coldread-narrative#1: + decision: adopted +2026-10-04T152128-coldread-narrative#10: + decision: adopted +2026-10-04T152128-coldread-narrative#11: + decision: adopted +2026-10-04T152128-coldread-narrative#12: + decision: adopted +2026-10-04T152128-coldread-narrative#13: + decision: adopted +2026-10-04T152128-coldread-narrative#14: + decision: adopted +2026-10-04T152128-coldread-narrative#15: + decision: adopted +2026-10-04T152128-coldread-narrative#16: + decision: adopted +2026-10-04T152128-coldread-narrative#17: + decision: adopted +2026-10-04T152128-coldread-narrative#18: + decision: adopted +2026-10-04T152128-coldread-narrative#19: + decision: adopted +2026-10-04T152128-coldread-narrative#2: + decision: adopted +2026-10-04T152128-coldread-narrative#20: + decision: adopted +2026-10-04T152128-coldread-narrative#21: + decision: adopted +2026-10-04T152128-coldread-narrative#22: + decision: adopted +2026-10-04T152128-coldread-narrative#23: + decision: adopted +2026-10-04T152128-coldread-narrative#24: + decision: declined +2026-10-04T152128-coldread-narrative#25: + decision: declined +2026-10-04T152128-coldread-narrative#26: + decision: adopted +2026-10-04T152128-coldread-narrative#27: + decision: adopted +2026-10-04T152128-coldread-narrative#28: + decision: adopted +2026-10-04T152128-coldread-narrative#29: + decision: adopted +2026-10-04T152128-coldread-narrative#3: + decision: adopted +2026-10-04T152128-coldread-narrative#30: + decision: adopted +2026-10-04T152128-coldread-narrative#31: + decision: declined +2026-10-04T152128-coldread-narrative#32: + decision: adopted +2026-10-04T152128-coldread-narrative#33: + decision: adopted +2026-10-04T152128-coldread-narrative#34: + decision: adopted +2026-10-04T152128-coldread-narrative#35: + decision: adopted +2026-10-04T152128-coldread-narrative#36: + decision: adopted +2026-10-04T152128-coldread-narrative#37: + decision: adopted +2026-10-04T152128-coldread-narrative#38: + decision: adopted +2026-10-04T152128-coldread-narrative#39: + decision: adopted +2026-10-04T152128-coldread-narrative#4: + decision: adopted +2026-10-04T152128-coldread-narrative#40: + decision: adopted +2026-10-04T152128-coldread-narrative#5: + decision: adopted +2026-10-04T152128-coldread-narrative#6: + decision: adopted +2026-10-04T152128-coldread-narrative#7: + decision: adopted +2026-10-04T152128-coldread-narrative#8: + decision: adopted +2026-10-04T152128-coldread-narrative#9: + decision: adopted +2026-10-04T152332-coldread-prose#1: + decision: adopted +2026-10-04T152332-coldread-prose#10: + decision: declined +2026-10-04T152332-coldread-prose#11: + decision: adopted +2026-10-04T152332-coldread-prose#12: + decision: adopted +2026-10-04T152332-coldread-prose#13: + decision: adopted +2026-10-04T152332-coldread-prose#14: + decision: adopted +2026-10-04T152332-coldread-prose#2: + decision: adopted +2026-10-04T152332-coldread-prose#3: + decision: adopted +2026-10-04T152332-coldread-prose#4: + decision: adopted +2026-10-04T152332-coldread-prose#5: + decision: adopted +2026-10-04T152332-coldread-prose#6: + decision: adopted +2026-10-04T152332-coldread-prose#7: + decision: adopted +2026-10-04T152332-coldread-prose#8: + decision: adopted +2026-10-04T152332-coldread-prose#9: + decision: adopted +2026-10-04T152417-quiz-prose#1: + decision: adopted +2026-10-04T152417-quiz-prose#10: + decision: adopted +2026-10-04T152417-quiz-prose#11: + decision: declined +2026-10-04T152417-quiz-prose#12: + decision: adopted +2026-10-04T152417-quiz-prose#13: + decision: declined +2026-10-04T152417-quiz-prose#14: + decision: adopted +2026-10-04T152417-quiz-prose#15: + decision: adopted +2026-10-04T152417-quiz-prose#16: + decision: adopted +2026-10-04T152417-quiz-prose#17: + decision: adopted +2026-10-04T152417-quiz-prose#2: + decision: adopted +2026-10-04T152417-quiz-prose#3: + decision: adopted +2026-10-04T152417-quiz-prose#4: + decision: declined +2026-10-04T152417-quiz-prose#5: + decision: adopted +2026-10-04T152417-quiz-prose#6: + decision: adopted +2026-10-04T152417-quiz-prose#7: + decision: adopted +2026-10-04T152417-quiz-prose#8: + decision: adopted +2026-10-04T152417-quiz-prose#9: + decision: adopted +2026-10-04T152559-coldread-prose#1: + decision: adopted +2026-10-04T152559-coldread-prose#2: + decision: adopted +2026-10-04T152559-coldread-prose#3: + decision: adopted +2026-10-04T152559-coldread-prose#4: + decision: declined +2026-10-04T152559-coldread-prose#5: + decision: adopted +2026-10-04T152559-coldread-prose#6: + decision: adopted +2026-10-04T152559-coldread-prose#7: + decision: adopted +2026-10-04T152559-coldread-prose#8: + decision: adopted diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T151909-probe.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T151909-probe.json new file mode 100644 index 0000000..71f2a93 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T151909-probe.json @@ -0,0 +1,218 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "float-sum", + "at": "2026-10-04T15:19:09", + "model": "default", + "prompt_sha": "22368ea1ea35", + "reply": { + "gaps": [ + { + "rule": 4, + "severity": "main", + "about": "The model gives the result of rounding 1/10 to 53 bits but never works through how the rounding is done.", + "why_it_matters": "The causal chain says '0.1, 0.2, 0.3 rounded to 53 bits' but never shows that step. Without it the long decimal expansions look like magic, and the reader cannot check where the +5.55e-18 error comes from.", + "suggested_addition": "Write each value as a whole number times a power of two. 0.1 × 2^55 = 3602879701896396.8, which rounds up to 3602879701896397, so stored 0.1 = 3602879701896397 / 2^55 with error +0.2/2^55 ≈ +5.55e-18. Stored 0.2 = 3602879701896397 / 2^54 (error +1.11e-17). 0.3 × 2^54 = 5404319552844595.2, which rounds down to 5404319552844595, so the error is −0.2/2^54 ≈ −1.11e-17." + }, + { + "rule": 2, + "severity": "main", + "about": "The claim 'the errors do not cancel to the error of 0.3' (Dependency 4) leaves its key step implicit: why 0.1 and 0.2 round up while 0.3 rounds down.", + "why_it_matters": "This sign difference is the whole reason the sum and stored 0.3 end up on different doubles. The file lists the signed errors but never adds them up or explains the signs.", + "suggested_addition": "Add the errors explicitly: +5.55e-18 + 1.11e-17 = +1.665e-17, while stored 0.3 has error −1.11e-17. The fractional parts .8 (0.1) and .2 (0.3) from the scaling above explain the signs. Contrast: for 0.5 + 0.25 all three errors are 0." + }, + { + "rule": 2, + "severity": "main", + "about": "The causal chain says 'exact sum slightly above 0.3 → rounded to the double above 0.3', which hides the tie.", + "why_it_matters": "Being above 0.3 does not decide the result. The exact sum 0.30000000000000001665 is exactly halfway between the two neighboring doubles, and only the tie rule picks the upper one. As written, the chain names the wrong reason.", + "suggested_addition": "Rewrite the chain step as: exact sum = lower neighbor + 2.776e-17 = exactly half a gap (5.55e-17 / 2) → tie → round half to even picks 0.30000000000000004440 (the significand ending ...334, which is even, over ...333)." + }, + { + "rule": 4, + "severity": "main", + "about": "The printed value ('shortest decimal that rounds back') has no worked example.", + "why_it_matters": "Confusion point 2 depends on it. The reader needs to see why stored 0.1 prints as '0.1' but the sum needs 17 digits.", + "suggested_addition": "'0.1' rounds to the stored 0.1, so 1 digit is enough. For the sum, '0.3' rounds to 0.29999999999999998889 (the wrong double), and so does every shorter candidate up to 0.3000000000000000. The first decimal that maps to 0.30000000000000004440 is 0.30000000000000004 (17 significant digits)." + }, + { + "rule": 4, + "severity": "edge", + "about": "The binary expansion of 1/10 is stated, not derived.", + "why_it_matters": "The audience does not know what a binary fraction is. A repeated-doubling example shows why the block 0011 repeats.", + "suggested_addition": "Double and take the integer part: 0.1→0.2 (0), 0.4 (0), 0.8 (0), 1.6 (1), 1.2 (1), 0.4 (0) ... The value 0.4 has come back, so the digits 0011 repeat forever: 0.0001100110011..." + }, + { + "rule": 2, + "severity": "edge", + "about": "The claim 'only power-of-two denominators end in binary' has no instance where it holds and no argument.", + "why_it_matters": "It is the mathematical core of the explanation, and the reason (the factor 5) appears only in the causal chain.", + "suggested_addition": "Holds: 3/8 = 0.011 in binary (3 × 2^3 / 8 is a whole number). Breaks: 1/10 × 2^k = 2^k/10 = 2^(k−1)/5 is never a whole number, because no power of 2 is divisible by 5." + }, + { + "rule": 1, + "severity": "edge", + "about": "The truncation row in 'Why this form' says 'with numbers' but gives none, and round half to even has no stated alternative.", + "why_it_matters": "The table promises numbers. The reader may also ask whether the tie rule is what causes the result.", + "suggested_addition": "Truncation: stored 0.1 would be 0.09999999999999999167. The sum still differs from stored 0.3 (it truncates to about 0.29999999999999993339), so truncation does not fix it. Round half up would also pick 0.30000000000000004 here. Half-to-even exists to avoid an upward bias across many ties, not to cause this result." + }, + { + "rule": 2, + "severity": "edge", + "about": "'Whole numbers below 2^53 are exact' and 'same result on every IEEE 754 machine' have no instance where the assumption is removed.", + "why_it_matters": "Each guarantee needs a breaking case.", + "suggested_addition": "2^53 + 1 = 9007199254740993 is stored as 9007199254740992. Determinism can break with x87 80-bit intermediates or with compiler fast-math/FMA contraction." + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts have more than one name, and some terms are used without definition.", + "why_it_matters": "In the file 'sum' means the exact sum, the rounded sum, and 'a double'. 'Significant bits', '53 bits' and 'fixed number of digits' name one idea. 'Stored value' is also called 'the double'. 'Error' and 'relative error' are used but never defined. 'Significant bits' and 'tie' are missing from Terms.", + "suggested_addition": "Define error = stored value − written decimal (for example +5.55e-18 for 0.1). Fix the names 'exact sum' (0.30000000000000001665) and 'rounded sum' (0.30000000000000004440). Add 'significant bits (53)' and 'tie' to Terms." + }, + { + "rule": 5, + "severity": "edge", + "about": "The signed errors and the relative error 1.9e-16 are not interpreted for the reader.", + "why_it_matters": "The reader should know what a + or − sign means, and that the difference only shows up around the 17th significant digit.", + "suggested_addition": "A + sign means the stored value is above the decimal. 1.9e-16 relative means the first 15 digits agree: 0.300000000000000 | 04, so you only see a difference in about the 17th digit." + }, + { + "rule": 6, + "severity": "edge", + "about": "The file says 'most programming languages' and lists Java under printed value without qualification.", + "why_it_matters": "The reader will ask which languages differ. Java's Double.toString was not always shortest before JDK 19.", + "suggested_addition": "Note that languages with default decimal or rational literals (for example Raku: 0.1 + 0.2 == 0.3 is True) differ. Qualify the Java status." + } + ], + "excess": [ + "Hex bit patterns (0x3FB999999999999A etc.) belong to L4 and are out of scope for this audience.", + "The full 55-digit exact expansions could be shortened to about 20 significant digits.", + "The list of decimal types in three languages (one example is enough).", + "Mention of Kahan summation." + ], + "next_questions": [ + { + "q": "How exactly does 0.1 become 0.1000000000000000055511...? What are the 53 bits?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why do 0.1 and 0.2 round up while 0.3 rounds down?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If 0.1 isn't stored exactly, why does print(0.1) show 0.1?", + "answered_by_model": true, + "declared_out_of_scope": false + }, + { + "q": "How should I compare floats then, and what tolerance should I use?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Does adding 0.1 ten times give exactly 1.0, and do errors grow in long sums?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.1529652, + "seconds": 57 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 4", + "text": "The model gives the result of rounding 1/10 to 53 bits but never works through how the rounding is done. → Write each value as a whole number times a power of two. 0.1 × 2^55 = 3602879701896396.8, which rounds up to 3602879701896397, so stored 0.1 = 3602879701896397 / 2^55 with error +0.2/2^55 ≈ +5.55e-18. Stored 0.2 = 3602879701896397 / 2^54 (error +1.11e-17). 0.3 × 2^54 = 5404319552844595.2, which rounds down to 5404319552844595, so the error is −0.2/2^54 ≈ −1.11e-17." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "The claim 'the errors do not cancel to the error of 0.3' (Dependency 4) leaves its key step implicit: why 0.1 and 0.2 round up while 0.3 rounds down. → Add the errors explicitly: +5.55e-18 + 1.11e-17 = +1.665e-17, while stored 0.3 has error −1.11e-17. The fractional parts .8 (0.1) and .2 (0.3) from the scaling above explain the signs. Contrast: for 0.5 + 0.25 all three errors are 0." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 2", + "text": "The causal chain says 'exact sum slightly above 0.3 → rounded to the double above 0.3', which hides the tie. → Rewrite the chain step as: exact sum = lower neighbor + 2.776e-17 = exactly half a gap (5.55e-17 / 2) → tie → round half to even picks 0.30000000000000004440 (the significand ending ...334, which is even, over ...333)." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 4", + "text": "The printed value ('shortest decimal that rounds back') has no worked example. → '0.1' rounds to the stored 0.1, so 1 digit is enough. For the sum, '0.3' rounds to 0.29999999999999998889 (the wrong double), and so does every shorter candidate up to 0.3000000000000000. The first decimal that maps to 0.30000000000000004440 is 0.30000000000000004 (17 significant digits)." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 4", + "text": "The binary expansion of 1/10 is stated, not derived. → Double and take the integer part: 0.1→0.2 (0), 0.4 (0), 0.8 (0), 1.6 (1), 1.2 (1), 0.4 (0) ... The value 0.4 has come back, so the digits 0011 repeat forever: 0.0001100110011..." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 2", + "text": "The claim 'only power-of-two denominators end in binary' has no instance where it holds and no argument. → Holds: 3/8 = 0.011 in binary (3 × 2^3 / 8 is a whole number). Breaks: 1/10 × 2^k = 2^k/10 = 2^(k−1)/5 is never a whole number, because no power of 2 is divisible by 5." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 1", + "text": "The truncation row in 'Why this form' says 'with numbers' but gives none, and round half to even has no stated alternative. → Truncation: stored 0.1 would be 0.09999999999999999167. The sum still differs from stored 0.3 (it truncates to about 0.29999999999999993339), so truncation does not fix it. Round half up would also pick 0.30000000000000004 here. Half-to-even exists to avoid an upward bias across many ties, not to cause this result." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 2", + "text": "'Whole numbers below 2^53 are exact' and 'same result on every IEEE 754 machine' have no instance where the assumption is removed. → 2^53 + 1 = 9007199254740993 is stored as 9007199254740992. Determinism can break with x87 80-bit intermediates or with compiler fast-math/FMA contraction." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts have more than one name, and some terms are used without definition. → Define error = stored value − written decimal (for example +5.55e-18 for 0.1). Fix the names 'exact sum' (0.30000000000000001665) and 'rounded sum' (0.30000000000000004440). Add 'significant bits (53)' and 'tie' to Terms." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 5", + "text": "The signed errors and the relative error 1.9e-16 are not interpreted for the reader. → A + sign means the stored value is above the decimal. 1.9e-16 relative means the first 15 digits agree: 0.300000000000000 | 04, so you only see a difference in about the 17th digit." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 6", + "text": "The file says 'most programming languages' and lists Java under printed value without qualification. → Note that languages with default decimal or rational literals (for example Raku: 0.1 + 0.2 == 0.3 is True) differ. Qualify the Java status." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "Hex bit patterns (0x3FB999999999999A etc.) belong to L4 and are out of scope for this audience." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "The full 55-digit exact expansions could be shortened to about 20 significant digits." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "The list of decimal types in three languages (one example is enough)." + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Mention of Kahan summation." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152128-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152128-coldread-narrative.json new file mode 100644 index 0000000..984e99e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152128-coldread-narrative.json @@ -0,0 +1,414 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "float-sum", + "at": "2026-10-04T15:21:28", + "model": "default", + "prompt_sha": "cce588df23ed", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: `0.1 + 0.2 == 0.3` → false; prints 0.30000000000000004. Said: The answer in three sentences.", + "unresolved": [ + "The answer in three sentences." + ], + "unsaid": [], + "leap": [ + "The 'Said' cell describes the narration instead of giving it. The reader never hears the three sentences, so the promised result is not delivered here." + ], + "excess": [ + "prints 0.30000000000000004: the reader has already seen or heard of this. It works as a hook, but it tells them nothing new." + ], + "purpose_unclear": false + }, + { + "at": "Beat 2 (instance)", + "text": "Shown: 0.75 = 3 × 2^-2. Said: A double is a whole number (53 bits) times a power of two.", + "unresolved": [ + "double", + "(53 bits)" + ], + "unsaid": [ + "The example does not connect 0.75 to 'double'. The words do not say that 3 is the 'whole number' and 2^-2 is the 'power of two'." + ], + "leap": [ + "The beat does not say why a float is stored this way, or that a double is what the computer holds for `0.1`. 'Double' is never tied to the code in Beat 1.", + "The beat does not say what the 53-bit limit means in practice: that the whole number must be below 2^53." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Shown: 1/3 = 0.3333… in decimal; 1/10 = 0.000110011… in binary. Said: Only power-of-two denominators end; 10 has a factor 5.", + "unresolved": [ + "end (meaning: the binary expansion terminates)" + ], + "unsaid": [ + "The beat never says aloud that 1/10 repeats forever in binary, the way 1/3 does in decimal. The analogy is shown but not spoken." + ], + "leap": [ + "The beat does not explicitly link a non-ending binary fraction to 'cannot be a whole number times a power of two, so it must be rounded'. The words never say the conclusion, 'must be rounded'." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Shown: table: exact and stored values in u; errors. Said: 0.1 and 0.2 round up, 0.3 rounds down.", + "unresolved": [ + "u", + "stored values", + "errors (sign convention)" + ], + "unsaid": [ + "The table's unit u is never defined in the words: not what it is, and not that it equals 2^-54.", + "The table's columns (exact, stored, error) are not walked through in the words.", + "The words never say what 'round up/down' is measured against: the nearest double." + ], + "leap": [ + "The beat does not say why the stored value is the nearest one, or that neighboring doubles are spaced one gap apart.", + "The beat does not say why 0.1 rounds up and 0.3 rounds down. The reader has to take it from the table." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: …198.5 + …397 = …595.5 → …596. Said: Exact sum is a tie; it rounds to the even neighbor, one gap above stored 0.3.", + "unresolved": [ + "…198.5 / …397 / …595.5 / …596 (the truncated digits and their units)", + "tie", + "even neighbor" + ], + "unsaid": [ + "The words do not say that these numbers are counts of u, or that …595 is stored 0.3.", + "The beat does not say why stored 0.1 can be a half-u (…198.5) when doubles were described as whole numbers. The explanation never says that the gap is smaller near 0.1." + ], + "leap": [ + "The beat does not give a reason for the rule that ties go to the even neighbor.", + "The beat does not explain why the sum is rounded at all: that the result of + is itself rounded to a double." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (objection)", + "text": "Shown: \"0.1\" vs \"0.30000000000000004\". Said: Printing shows the shortest decimal that picks out the double.", + "unresolved": [ + "picks out the double" + ], + "unsaid": [], + "leap": [ + "The beat does not explain why 0.30000000000000004 is the shortest decimal for …596 u. The reader needs to know that '0.3' would pick out …595 instead, and the beat does not say so." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (test)", + "text": "Shown: 0.5 + 0.25 == 0.75 → true; Decimal('0.1') + Decimal('0.2') == Decimal('0.3') → true. Said: Exact when denominators are powers of two, or when the base is 10.", + "unresolved": [ + "Decimal('0.1')", + "the base is 10" + ], + "unsaid": [ + "The beat never says that Decimal is a type that stores numbers in base 10." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (close)", + "text": "Shown: `abs(a - b) < 1e-9`; integer cents; Decimal. Said: Compare with a tolerance, or avoid binary for decimal money.", + "unresolved": [ + "tolerance" + ], + "unsaid": [ + "The beat does not say why the threshold is 1e-9.", + "The words do not explain 'integer cents'." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 1: 'The answer in three sentences.' The beat describes the narration instead of giving it, so the reader never actually receives the result.", + "Beat 4: 'u' is used in the table but never defined in the words. Everything numeric after this depends on it.", + "Beat 5: '…198.5 + …397 = …595.5 → …596'. The truncated numbers in u are not explained, and the half-u for 0.1 contradicts 'whole number times a power of two' from Beat 2.", + "Beat 5: 'tie' and 'even neighbor' are not introduced. This is the step that decides the result.", + "Beat 2: 'double' is never tied to what Python or JavaScript stores for `0.1`." + ], + "cuts": [ + "Beat 1's re-display of 0.30000000000000004 could be shortened, since the reader already knows it. This is a minor saving.", + "Beat 8: the three fixes could be reduced to one sentence. The integer-cents detail is not needed for the argument." + ], + "question": { + "text": "Why does `0.1 + 0.2 == 0.3` evaluate to false?", + "known_at": "Beat 1", + "result_stated": "Not in words. Beat 1 says only 'The answer in three sentences.' Pieces show up later: Beat 4 says '0.1 and 0.2 round up, 0.3 rounds down', and Beat 5 says 'one gap above stored 0.3'. The result is never stated as a whole." + }, + "motive": "Reason to care: missing from the beats. Beat 8's 'avoid binary for decimal money' hints at it only at the very end. Reason for the approach: missing. Beat 2's question, 'What does the computer hold?', appears only in the Reader's-question column, not in what is Said.", + "close": "No. Beat 8 gives fixes ('Compare with a tolerance, or avoid binary for decimal money') but does not restate the answer to the opening question. The general argument, that any decimal whose denominator has a factor 5 gets rounded and that sums of rounded numbers can land on a different neighbor, is never said in the beats.", + "recap": "0.1, 0.2 and 0.3 can't be stored exactly in binary, so the computer adds two slightly-high approximations, and the result rounds to the double just above the one it uses for 0.3.", + "strongest": "Beat 3: the parallel between 1/3 = 0.3333… in decimal and 1/10 = 0.000110011… in binary, plus '10 has a factor 5'. It is clear and fits what this reader already knows.", + "weakest": "Beat 5: the elided '…' numbers in an undefined unit u, a half-unit value, and an unexplained tie rule all arrive in the single step that delivers the answer.", + "suggestions": [ + "Write the actual three sentences in Beat 1's Said column.", + "Define u in Beat 4's words: the gap between neighboring doubles near 0.3, equal to 2^-54.", + "Explain that the gap is smaller near 0.1, which is why it can sit on a half-u, or express each value in its own gap.", + "Say in Beat 5 that + rounds its exact result to the nearest double, and give a one-line reason for round-half-to-even.", + "Add a sentence early on about why to care, such as prices, loops, or tests.", + "End Beat 8 by restating the answer and the general rule about factors of 5 before giving the fixes." + ] + }, + "usage": { + "cost": 0.0989208, + "seconds": 33 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 1: 'The answer in three sentences.' The beat describes the narration instead of giving it, so the reader never actually receives the result." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'u' is used in the table but never defined in the words. Everything numeric after this depends on it." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: '…198.5 + …397 = …595.5 → …596'. The truncated numbers in u are not explained, and the half-u for 0.1 contradicts 'whole number times a power of two' from Beat 2." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: 'tie' and 'even neighbor' are not introduced. This is the step that decides the result." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'double' is never tied to what Python or JavaScript stores for `0.1`." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 1's re-display of 0.30000000000000004 could be shortened, since the reader already knows it. This is a minor saving." + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 8: the three fixes could be reduced to one sentence. The integer-cents detail is not needed for the argument." + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): The answer in three sentences." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): The 'Said' cell describes the narration instead of giving it. The reader never hears the three sentences, so the promised result is not delivered here." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): double" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): (53 bits)" + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): The example does not connect 0.75 to 'double'. The words do not say that 3 is the 'whole number' and 2^-2 is the 'power of two'." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (instance): The beat does not say why a float is stored this way, or that a double is what the computer holds for `0.1`. 'Double' is never tied to the code in Beat 1." + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (instance): The beat does not say what the 53-bit limit means in practice: that the whole number must be below 2^53." + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): end (meaning: the binary expansion terminates)" + }, + { + "n": 16, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): The beat never says aloud that 1/10 repeats forever in binary, the way 1/3 does in decimal. The analogy is shown but not spoken." + }, + { + "n": 17, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): The beat does not explicitly link a non-ending binary fraction to 'cannot be a whole number times a power of two, so it must be rounded'. The words never say the conclusion, 'must be rounded'." + }, + { + "n": 18, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): u" + }, + { + "n": 19, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): stored values" + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): errors (sign convention)" + }, + { + "n": 21, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The table's unit u is never defined in the words: not what it is, and not that it equals 2^-54." + }, + { + "n": 22, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The table's columns (exact, stored, error) are not walked through in the words." + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The words never say what 'round up/down' is measured against: the nearest double." + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): The beat does not say why the stored value is the nearest one, or that neighboring doubles are spaced one gap apart." + }, + { + "n": 25, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): The beat does not say why 0.1 rounds up and 0.3 rounds down. The reader has to take it from the table." + }, + { + "n": 26, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): …198.5 / …397 / …595.5 / …596 (the truncated digits and their units)" + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): tie" + }, + { + "n": 28, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): even neighbor" + }, + { + "n": 29, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The words do not say that these numbers are counts of u, or that …595 is stored 0.3." + }, + { + "n": 30, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The beat does not say why stored 0.1 can be a half-u (…198.5) when doubles were described as whole numbers. The explanation never says that the gap is smaller near 0.1." + }, + { + "n": 31, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): The beat does not give a reason for the rule that ties go to the even neighbor." + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): The beat does not explain why the sum is rounded at all: that the result of + is itself rounded to a double." + }, + { + "n": 33, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (objection): picks out the double" + }, + { + "n": 34, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (objection): The beat does not explain why 0.30000000000000004 is the shortest decimal for …596 u. The reader needs to know that '0.3' would pick out …595 instead, and the beat does not say so." + }, + { + "n": 35, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): Decimal('0.1')" + }, + { + "n": 36, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): the base is 10" + }, + { + "n": 37, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (test): The beat never says that Decimal is a type that stores numbers in base 10." + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): tolerance" + }, + { + "n": 39, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (close): The beat does not say why the threshold is 1e-9." + }, + { + "n": 40, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (close): The words do not explain 'integer cents'." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152332-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152332-coldread-prose.json new file mode 100644 index 0000000..6f298bc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152332-coldread-prose.json @@ -0,0 +1,308 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "float-sum", + "at": "2026-10-04T15:23:32", + "model": "default", + "prompt_sha": "5a19de7d0d5c", + "reply": { + "paragraphs": [ + { + "at": "L3 (opening)", + "text": "In Python, JavaScript, and most other languages, `0.1 + 0.2 == 0.3` is `false`...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L5 (bold summary)", + "text": "The computer never holds 0.1, 0.2, or 0.3...", + "unresolved": [ + "one step above" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L9 What the computer holds", + "text": "A literal such as `0.1` becomes a **double**...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L13 Why 0.1 must be rounded", + "text": "A whole number times a power of two is a fraction whose denominator is a power of two...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L15 code block", + "text": "1/10 = 0.000110011001100110011…", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L17", + "text": "This is the same as 1/3 = 0.3333… in decimal...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L21 The stored values", + "text": "The **stored value** of a literal is the double nearest to it. The **gap**...", + "unresolved": [], + "leap": [ + "Between 0.25 and 0.5 the gap is u = 2^-54: no reason is given for this number. The reader has to work out that 53 digits starting at 2^-2 end at 2^-54.", + "\"Because a double has a fixed 53 binary digits\" conflicts with \"at most 53 binary digits\" in L9, and it isn't explained why a fixed digit count makes the gap shrink." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L23", + "text": "To see the rounding, count each number in units of u...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L25 table", + "text": "| Number | Exact, in u | Steps allowed here | ...", + "unresolved": [ + "Steps allowed here", + "Error" + ], + "leap": [ + "The table never says why 0.1 gets u/4 and 0.2 gets u/2. The reader has to match 0.1 to the range 0.0625–0.125 and 0.2 to the range 0.125–0.25 from L21." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L31", + "text": "The error is the stored value minus the decimal. 0.1 and 0.2 round up...", + "unresolved": [], + "leap": [], + "excess": [ + "0.1 and 0.2 round up (positive error). 0.3 rounds down (negative error)." + ], + "purpose_unclear": false + }, + { + "at": "L35 What + does", + "text": "The `+` operation first finds the **exact sum**...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L37 code block", + "text": "exact sum = 1801439850948198.5 + 3602879701896397 = 5404319552844595.5 u", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L39", + "text": "The exact sum is exactly halfway between …595 and …596. This case is a **tie**...", + "unresolved": [ + "The rule" + ], + "leap": [ + "It never says that …596 is the even one, so the reader can't see why the tie goes up here.", + "\"even whole number\": a whole number of what? The text doesn't say the count of u is what must be even." + ], + "excess": [ + "so ties do not always push results upward" + ], + "purpose_unclear": false + }, + { + "at": "L41", + "text": "The stored value of 0.3 is 5404319552844595 u...", + "unresolved": [], + "leap": [], + "excess": [ + "The stored value of 0.3 is 5404319552844595 u." + ], + "purpose_unclear": false + }, + { + "at": "L43", + "text": "The addition did not make a mistake...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L48 Why print(0.1) shows 0.1", + "text": "Languages print the shortest decimal that rounds back to the same double...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L53 When it works", + "text": "`0.5 + 0.25 == 0.75` is true...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L55", + "text": "A decimal type also avoids the problem...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L60 What to do", + "text": "Do not compare doubles with `==`...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L64-66 Not covered here", + "text": "- The sign and exponent bits of a double...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "L39: \"The rule sends a tie to the even whole number\". This is the turning point of the argument, but the text names no rule and never says …596 is the even choice. The reader can't confirm why the sum rounds up.", + "L21: \"Between 0.25 and 0.5 the gap is u = 2^-54\". Every later number is counted in u, but the text gives no reason for this value." + ], + "cuts": [ + "L39: \"so ties do not always push results upward\". It is an aside that pulls attention away from the case being explained.", + "L31: \"0.1 and 0.2 round up (positive error). 0.3 rounds down (negative error).\" The Error column already shows this; keep only the definition of error.", + "L41: \"The stored value of 0.3 is 5404319552844595 u.\" This repeats the table." + ], + "question": { + "text": "Why is 0.1 + 0.2 == 0.3 false, and why does it print 0.30000000000000004?", + "known_at": "Title and L3", + "result_stated": "Yes. L5: \"The computer never holds 0.1, 0.2, or 0.3. It holds the nearest binary numbers to them. The addition is correct, but the result rounds to the number one step above the number that 0.3 rounds to.\"" + }, + "motive": "Reason to care: it is implied by L3 (a surprising `false`) and made concrete in L60, \"Do not compare doubles with `==`...\". Reason for the approach: L23, \"To see the rounding, count each number in units of u.\" No sentence says why this method was chosen over others.", + "close": "Yes. L43: \"The addition did not make a mistake. The result is the nearest double to the exact sum of its inputs. The inputs were not 0.1 and 0.2.\" L53 then states the general condition: \"The failure needs a decimal whose denominator has a factor 5.\"", + "recap": "0.1 and 0.2 get stored slightly too high and 0.3 slightly too low, so their sum lands on the neighboring double just above stored 0.3.", + "strongest": "L17: the 4-digit 1/3 example (0.3333 + 0.3333 = 0.6666, but 2/3 becomes 0.6667) shows the same failure in decimal, where the reader can check it.", + "weakest": "L39: the tie paragraph leaves the even-rounding rule unexplained at the exact step where the result is decided.", + "suggestions": [ + "In L39, say: \"Ties go to the neighbor whose count of u is even; …596 is even, so the sum rounds up.\"", + "In L21, give the reason: numbers from 0.25 to 0.5 are 53-digit whole numbers times 2^-54, so neighbors are 2^-54 apart. Halving the range halves the gap.", + "Add \"in the range\" next to the u/4 and u/2 values in the table, or link each row to its range from L21.", + "Make L9's \"at most 53\" and L21's \"fixed 53\" agree." + ] + }, + "usage": { + "cost": 0.10868280000000001, + "seconds": 34 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "L39: \"The rule sends a tie to the even whole number\". This is the turning point of the argument, but the text names no rule and never says …596 is the even choice. The reader can't confirm why the sum rounds up." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "L21: \"Between 0.25 and 0.5 the gap is u = 2^-54\". Every later number is counted in u, but the text gives no reason for this value." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "L39: \"so ties do not always push results upward\". It is an aside that pulls attention away from the case being explained." + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "L31: \"0.1 and 0.2 round up (positive error). 0.3 rounds down (negative error).\" The Error column already shows this; keep only the definition of error." + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "L41: \"The stored value of 0.3 is 5404319552844595 u.\" This repeats the table." + }, + { + "n": 6, + "severity": "edge", + "kind": "unresolved", + "text": "L5 (bold summary): one step above" + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "L21 The stored values: Between 0.25 and 0.5 the gap is u = 2^-54: no reason is given for this number. The reader has to work out that 53 digits starting at 2^-2 end at 2^-54." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "L21 The stored values: \"Because a double has a fixed 53 binary digits\" conflicts with \"at most 53 binary digits\" in L9, and it isn't explained why a fixed digit count makes the gap shrink." + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "L25 table: Steps allowed here" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "L25 table: Error" + }, + { + "n": 11, + "severity": "edge", + "kind": "leap", + "text": "L25 table: The table never says why 0.1 gets u/4 and 0.2 gets u/2. The reader has to match 0.1 to the range 0.0625–0.125 and 0.2 to the range 0.125–0.25 from L21." + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "L39: The rule" + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "L39: It never says that …596 is the even one, so the reader can't see why the tie goes up here." + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "L39: \"even whole number\": a whole number of what? The text doesn't say the count of u is what must be even." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152417-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152417-quiz-prose.json new file mode 100644 index 0000000..0bf498b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152417-quiz-prose.json @@ -0,0 +1,195 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "float-sum", + "at": "2026-10-04T15:24:05", + "model": "default", + "prompt_sha": "008e69251529", + "reply": { + "answers": { + "1": "The main objects are: decimal literals (0.1, 0.2, 0.3); the double, a 64-bit binary format that holds a whole number of at most 53 binary digits times a power of two; the stored value, which is the double nearest a literal; the gap, which is the distance between neighboring doubles (u = 2^-54 between 0.25 and 0.5); the error, which is the stored value minus the decimal; the + operation, giving an exact sum and then a rounded sum; the tie and the round-half-to-even rule; `==`; printing as the shortest decimal that rounds back to the double; and the Decimal type.", + "2": "Each literal is rounded to its stored value, a double on a grid. The grid's gap depends on the size of the number. `+` adds the stored values exactly, then rounds the result onto the grid. The rule for ties decides which neighbor it lands on. `==` compares the two doubles bit for bit. Printing shows the shortest decimal that rounds back to that double, which hides small errors unless two doubles need to be told apart.", + "3": "0.1 and 0.2 cannot end in binary because 10 has a factor of 5, so both are stored slightly too high, while 0.3 is stored slightly too low. Their exact sum falls halfway between two doubles, the tie rounds up to the double just above stored 0.3, and so `==` returns false.", + "4": "The key variable is whether the decimal's denominator contains a factor of 5, meaning it is not a power-of-two fraction. If every value is a power-of-two fraction (0.5, 0.25, 0.75), the stored error is zero, the exact sum is already a double, and `==` is true. A second variable is the number of digits kept. With more digits the gap gets smaller, but a repeating binary fraction still has to be cut. The text implies this but does not discuss changing the digit count.", + "5": "Assumption or simplification: the text ignores the sign and exponent bits and treats a double as 'a whole number of at most 53 binary digits times a power of two'. Established fact: 0.1 + 0.2 prints 0.30000000000000004, and in units of u the stored 0.3 is 5404319552844595 while the sum rounds to …596.", + "6": "A double can only hold a binary fraction with at most 53 significant bits. 1/10 has a factor 5 in its denominator, so its binary expansion never ends and it has to be rounded, much as 1/3 is in decimal. Stored 0.1 and 0.2 are each slightly above their true values, and stored 0.3 is slightly below. Adding the stored 0.1 and 0.2 exactly gives a value halfway between two doubles, and the round-to-even rule picks the one just above stored 0.3, so the two differ by one gap and `==` is false. Printing shows the shortest decimal that identifies a double, which is why 0.1 prints as 0.1 but the sum needs 17 digits.", + "7": "The hardest part was the table and the units of u. Fractional values in u, such as 1801439850948198.5, are confusing until you link them to 'steps allowed here = u/4'. The text also never says why the gap is exactly 2^-54 in [0.25, 0.5), which is 2^-2 × 2^-52 from the 53 digits. In the tie step, it never says outright that …596 is the even one. One worked line on the gap, and the sentence '…596 is even, …595 is odd', would have made it easier.", + "8": "It is true. 1/2, 1/4 and 3/4 have power-of-two denominators, so each is stored with zero error, and the exact sum is itself a double, so rounding changes nothing.", + "9": "No. The `+` returns the nearest double to the exact sum of its inputs, which is correct. The difference comes from the inputs, which were already rounded versions of 0.1 and 0.2, and from the fact that 0.3 was rounded independently (downward). The final rounding of the sum, a tie broken to even, is the only rounding the operation does, and it follows the rule.", + "10": "Yes, it would be true. Decimal stores base-10 digits, and 1/10 ends in base 10, so no rounding happens.", + "11": "0.75 = 3 × 2^-2 is a whole number times a power of two, so its denominator is a power of two. 1/10 has a factor 5 in its denominator, no power of two is divisible by 5, so its binary expansion repeats forever and has to be cut.", + "12": "Both stored values err upward (+5.55e-18 and +1.11e-17), so their exact sum is 5404319552844595.5 u. That is a tie between …595 and …596, and ties go to even, so it rounds to …596. Stored 0.3 was rounded down, to …595. The two results are neighboring doubles one gap (5.55e-17) apart.", + "13": "Languages print the shortest decimal that rounds back to the same double. The text '0.1' already rounds back to stored 0.1, so one digit is enough and the error stays hidden." + }, + "audit": { + "terms": [ + "The neighbor spacing is called 'gap' (§The stored values), 'Steps allowed here' (table header), 'one step above' (bold summary line 5) and 'u' (line 21), which in fact is the gap only in [0.25, 0.5).", + "The result of `+` is called 'rounded sum' (line 35), 'the result' (line 43) and '0.1 + 0.2' (§What to do).", + "The rounded values are called 'nearest binary numbers' (line 5), 'stored value' (line 21) and 'stored 0.1' (line 48).", + "The deviation is called 'Error' (table, line 31) and 'difference' (line 48 and line 60, where it means the gap between the sum and stored 0.3)." + ], + "unexplained": [ + "'Steps allowed here' column with values u/4, u/2, u (table): the text does not say that this is why stored 0.1 can be …198.5, a value with a fractional part in u.", + "'u = 2^-54 ≈ 5.55e-17' (line 21): no derivation from the 53 digits is given, so the reader cannot say what the gap would be in another range.", + "'1e-9' tolerance (line 60): the text does not say why this value was chosen or what would go wrong with 1e-15 or 1e-3.", + "'1 part in 5 × 10^15 ... 17th digit' (line 48): the step from this ratio to the 17th digit is not shown." + ], + "missing_why": [ + "'Because a double has a fixed 53 binary digits, the gap shrinks as the number shrinks. Between 0.25 and 0.5 the gap is u = 2^-54' (line 21): the text does not explain why the exponent is -54.", + "'The rule sends a tie to the even whole number' (line 39): the text does not say why this rule exists, and does not state that 596 is even and 595 is odd.", + "'Languages print the shortest decimal that rounds back to the same double' (line 48): the text does not say why languages use this rule." + ], + "excess": [ + "'Decimal types are slower, because the processor adds binary numbers directly' (line 55): a detour that no question or step of the argument uses.", + "'Not covered here' list (lines 62-66): reference pointers that this reader does not need." + ] + }, + "gaps": [ + "The table's fractional 'in u' values (…198.4 rounding to …198.5) depend on the 'u/4 steps' idea, which needs one explicit sentence.", + "The text never states that …596 is even, so the reader has to work out the tie direction alone.", + "The value u = 2^-54 is given without derivation, which makes the whole 'count in units of u' method feel arbitrary.", + "The text gives no guidance on why 1e-9 is a reasonable tolerance." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "1": "Full credit. Covers double, 53 significant bits, stored value, gap with u = 2^-54, error, exact vs rounded sum, tie with half-to-even, printed value, and the Decimal type.", + "2": "Full credit. Covers literal to stored value, the gap scaling with magnitude, the exact sum rounded onto the grid, tie-breaking, == comparing doubles, and printing hiding errors.", + "3": "Full credit. The chain is correct: factor 5, 0.1 and 0.2 stored high and 0.3 low, a tie, rounding to the double above stored 0.3, == false. 'Rounds up' matches the model's outcome for this case.", + "4": "Full credit. Names the power-of-two denominator as the key variable, with 0.5 + 0.25 as the case. Precision (bits kept) is noted as a secondary variable.", + "5": "Full credit. Separates the simplification (sign and exponent bits ignored, which the model scopes out) from established facts (IEEE stored values, the observed 0.30000000000000004).", + "6": "Full credit. Gives a complete, correct explanation that includes the printing step.", + "7": "Full credit. A specific difficulty report naming the units of u, where the gap comes from, and which neighbor is even. This is a subjective item and is not checked against model.md.", + "8": "Full credit. True; the power-of-two denominators give zero error and an exact sum.", + "9": "Full credit. + is correctly rounded. The difference comes from the inputs, which were already rounded, and from 0.3 being rounded independently.", + "10": "Full credit. True; Decimal stores base-10 digits, where 1/10 ends.", + "11": "Full credit. States the power-of-two denominator rule and the factor 5, and explains that no power of 2 is divisible by 5.", + "12": "Full credit. The exact sum 5404319552844595.5 u is a tie, goes to the even …596, and stored 0.3 is …595, one gap below.", + "13": "Full credit. Print shows the shortest decimal that rounds back to the same double, and '0.1' suffices." + }, + "pass": true + }, + "usage": { + "cost": 0.17389939999999998, + "seconds": 44 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The neighbor spacing is called 'gap' (§The stored values), 'Steps allowed here' (table header), 'one step above' (bold summary line 5) and 'u' (line 21), which in fact is the gap only in [0.25, 0.5)." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The result of `+` is called 'rounded sum' (line 35), 'the result' (line 43) and '0.1 + 0.2' (§What to do)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "The rounded values are called 'nearest binary numbers' (line 5), 'stored value' (line 21) and 'stored 0.1' (line 48)." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "The deviation is called 'Error' (table, line 31) and 'difference' (line 48 and line 60, where it means the gap between the sum and stored 0.3)." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'Steps allowed here' column with values u/4, u/2, u (table): the text does not say that this is why stored 0.1 can be …198.5, a value with a fractional part in u." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'u = 2^-54 ≈ 5.55e-17' (line 21): no derivation from the 53 digits is given, so the reader cannot say what the gap would be in another range." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'1e-9' tolerance (line 60): the text does not say why this value was chosen or what would go wrong with 1e-15 or 1e-3." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'1 part in 5 × 10^15 ... 17th digit' (line 48): the step from this ratio to the 17th digit is not shown." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "'Because a double has a fixed 53 binary digits, the gap shrinks as the number shrinks. Between 0.25 and 0.5 the gap is u = 2^-54' (line 21): the text does not explain why the exponent is -54." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'The rule sends a tie to the even whole number' (line 39): the text does not say why this rule exists, and does not state that 596 is even and 595 is odd." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'Languages print the shortest decimal that rounds back to the same double' (line 48): the text does not say why languages use this rule." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "'Decimal types are slower, because the processor adds binary numbers directly' (line 55): a detour that no question or step of the argument uses." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "'Not covered here' list (lines 62-66): reference pointers that this reader does not need." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The table's fractional 'in u' values (…198.4 rounding to …198.5) depend on the 'u/4 steps' idea, which needs one explicit sentence." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "The text never states that …596 is even, so the reader has to work out the tie direction alone." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "The value u = 2^-54 is given without derivation, which makes the whole 'count in units of u' method feel arbitrary." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The text gives no guidance on why 1e-9 is a reasonable tolerance." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152559-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152559-coldread-prose.json new file mode 100644 index 0000000..b46c7cf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152559-coldread-prose.json @@ -0,0 +1,252 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "float-sum", + "at": "2026-10-04T15:25:59", + "model": "default", + "prompt_sha": "5a19de7d0d5c", + "reply": { + "paragraphs": [ + { + "at": "Title + line 3", + "text": "In Python, JavaScript, and most other languages, `0.1 + 0.2 == 0.3` is `false`...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 5 (bold summary)", + "text": "The computer never holds 0.1, 0.2, or 0.3...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What the computer holds, line 9", + "text": "A literal such as `0.1` becomes a **double**...", + "unresolved": [], + "leap": [ + "\"64-bit binary format\" but \"at most 53 binary digits\": the reader isn't told what the other 11 bits do until the 'Not covered' list at the end. This is an edge issue, not a stopper." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why 0.1 must be rounded, line 13 + expansion", + "text": "A whole number times a power of two is a fraction whose denominator is a power of two...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 17", + "text": "This is the same as 1/3 = 0.3333… in decimal...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The stored values, line 21", + "text": "The **stored value** of a literal is the double nearest to it. The **gap** is...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 23", + "text": "The gap depends on the size of the number. Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54...", + "unresolved": [], + "leap": [ + "Why the multiplier must be 2^-54 in this range. The text never links it back to the 53-digit limit from line 9 (a bigger whole number with a smaller power would need more than 53 digits). The reader may reread line 9 to work this out." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 25", + "text": "To see the rounding, count each number in units of u...", + "unresolved": [ + "\"…595\" / \"…198.5\": the ellipsis shorthand for long counts is used without saying what it means. It can be inferred." + ], + "leap": [ + "\"its stored value can be a quarter-multiple of u, such as …198.5\": the example is a half, which makes the reader stop to check that .5 counts as a multiple of 1/4. The text also doesn't show why .4 rounds to .5 rather than .25." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "table, lines 27-31", + "text": "| Number | Exact, in u | Gap here | Stored, in u | ...", + "unresolved": [ + "\"Error\" column: its sign convention is only defined in the next paragraph." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 33", + "text": "The error is the stored value minus the decimal. 0.1 and 0.2 round up; 0.3 rounds down.", + "unresolved": [], + "leap": [], + "excess": [ + "\"0.3 rounds down\" was already said in line 25, and the table's signs show both directions." + ], + "purpose_unclear": false + }, + { + "at": "What `+` does, line 37 + exact sum", + "text": "The `+` operation first finds the **exact sum**...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 41", + "text": "The exact sum is exactly halfway... **tie**... IEEE 754...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 43", + "text": "Stored 0.3 is at …595 u. The rounded sum and stored 0.3 are neighboring doubles...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 45", + "text": "The addition did not make a mistake...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why print(0.1) shows 0.1, line 50", + "text": "Languages print the shortest decimal that rounds back to the same double...", + "unresolved": [], + "leap": [ + "\"One gap is about 1 part in 5 × 10^15 of 0.3, so the two doubles agree in their first 16 digits\": the reader has to do the arithmetic (5.55e-17 / 0.3) to check this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "When it works, line 55", + "text": "`0.5 + 0.25 == 0.75` is true...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 57", + "text": "A decimal type also avoids the problem...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What to do, line 62", + "text": "Do not compare doubles with `==`... Compare with a tolerance instead...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered here, lines 66-68", + "text": "- The sign and exponent bits of a double...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "line 33: cut \"0.1 and 0.2 round up; 0.3 rounds down.\" The table's Error column already shows it, and line 25 already said it for 0.3. Moving the error definition above the table would also stop the reader puzzling over the column.", + "line 50: cut the last sentence (\"One gap is about 1 part in 5 × 10^15...\"). It asks for arithmetic, and the claim before it (\"with 17 significant digits\") already makes the point." + ], + "question": { + "text": "Why does 0.1 + 0.2 == 0.3 come out false, and why does the sum print as 0.30000000000000004?", + "known_at": "Title and line 3", + "result_stated": "Yes, in words at line 5: \"The computer never holds 0.1, 0.2, or 0.3. It holds the nearest numbers it can store in binary. The addition is correct, but its result rounds to the next storable number above the one that 0.3 rounds to.\"" + }, + "motive": "Reason to care: only implied. Line 3 presents the surprise, and line 62 makes it practical: \"Do not compare doubles with `==`...\". No early sentence says why the reader should care. Reason for the approach: line 25, \"To see the rounding, count each number in units of u\", together with line 23's demonstration that the gap is constant within a range. That is enough.", + "close": "Yes. Line 45 gives the general argument in words: \"The addition did not make a mistake. The rounded sum is the nearest double to the exact sum of its inputs. The inputs were not 0.1 and 0.2.\" Line 55 states the general condition: \"The failure needs a decimal whose denominator has a factor 5.\"", + "recap": "0.1 and 0.2 are each stored slightly too high and 0.3 slightly too low, and the correctly rounded sum lands one storable step above stored 0.3, so == says false.", + "strongest": "Lines 41-43: the exact sum works out to …595.5 u, the tie rule sends it to …596 u, and that is one gap above stored 0.3 at …595 u. The whole result rests on concrete counts the reader can check.", + "weakest": "Line 23-25: why the multiplier is 2^-54 is not tied back to the 53-digit limit, and \"quarter-multiple of u, such as …198.5\" uses a half as its example of a quarter. It is the one place the reader rereads.", + "suggestions": [ + "In line 23, add one clause such as \"53 binary digits reach up to 2^53, so to cover 0.25–0.5 the power must be 2^-54\" to close the leap.", + "In line 25, show the rounding for 0.1 explicitly: the neighbors are …198.25 and …198.5, and .4 is nearer to .5.", + "Define the Error column before the table." + ] + }, + "usage": { + "cost": 0.1279186, + "seconds": 41 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "line 33: cut \"0.1 and 0.2 round up; 0.3 rounds down.\" The table's Error column already shows it, and line 25 already said it for 0.3. Moving the error definition above the table would also stop the reader puzzling over the column." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "line 50: cut the last sentence (\"One gap is about 1 part in 5 × 10^15...\"). It asks for arithmetic, and the claim before it (\"with 17 significant digits\") already makes the point." + }, + { + "n": 3, + "severity": "edge", + "kind": "leap", + "text": "What the computer holds, line 9: \"64-bit binary format\" but \"at most 53 binary digits\": the reader isn't told what the other 11 bits do until the 'Not covered' list at the end. This is an edge issue, not a stopper." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "line 23: Why the multiplier must be 2^-54 in this range. The text never links it back to the 53-digit limit from line 9 (a bigger whole number with a smaller power would need more than 53 digits). The reader may reread line 9 to work this out." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "line 25: \"…595\" / \"…198.5\": the ellipsis shorthand for long counts is used without saying what it means. It can be inferred." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "line 25: \"its stored value can be a quarter-multiple of u, such as …198.5\": the example is a half, which makes the reader stop to check that .5 counts as a multiple of 1/4. The text also doesn't show why .4 rounds to .5 rather than .25." + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "table, lines 27-31: \"Error\" column: its sign convention is only defined in the next paragraph." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Why print(0.1) shows 0.1, line 50: \"One gap is about 1 part in 5 × 10^15 of 0.3, so the two doubles agree in their first 16 digits\": the reader has to do the arithmetic (5.55e-17 / 0.3) to check this." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152657-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152657-quiz-prose.json new file mode 100644 index 0000000..fa6e1d9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/explainers/float-sum/review/runs/2026-10-04T152657-quiz-prose.json @@ -0,0 +1,162 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "float-sum", + "at": "2026-10-04T15:26:49", + "model": "default", + "prompt_sha": "008e69251529", + "reply": { + "answers": { + "1": "The main objects are decimal literals (0.1, 0.2, 0.3) and the double, a 64-bit binary format. A double is a whole number of at most 53 binary digits times a power of two. The other objects are: the stored value (the nearest double to a literal); the gap, meaning the distance between neighbouring doubles, with u = 2^-54 for doubles between 0.25 and 0.5; the error (stored value minus the decimal); the exact sum; the rounded sum; the tie-to-even rule from IEEE 754; the shortest-decimal printing rule; the == comparison; Python's Decimal type; and a tolerance.", + "2": "Each literal is rounded to its stored value, a nearby double. The gap between doubles depends on the size of the number: u/4 near 0.1, u/2 near 0.2 and u near 0.3. So each stored value has its own small error. The + operation adds the stored values exactly, then rounds the result to a double, breaking ties toward an even count of u. == compares the rounded sum with stored 0.3. Printing shows the shortest decimal that rounds back to the same double.", + "3": "0.1 and 0.2 cannot end in binary, so both are stored slightly too high, while 0.3 is stored slightly too low. Their exact sum lands halfway between two doubles, and the tie rounds up to the double just above stored 0.3, so == is false.", + "4": "The most important variable is whether the decimal's denominator has a factor of 5. If every value is a fraction with a power-of-two denominator, such as 0.5, 0.25 or 0.75, each one is stored with error 0. The exact sum is then itself a double, so == is true. With a factor of 5, the value has to be rounded, and the errors can push the sum onto a different double. The explanation also implies that a larger number has a larger gap, so its rounding error is bigger in absolute terms.", + "5": "Simplification: the sign and exponent bits are left out, and growth of errors over many operations is not covered. Established fact: IEEE 754 rounds a tie to the neighbour with an even count, and stored 0.1 = 0.10000000000000000555.", + "6": "Languages store numbers as doubles: a whole number of up to 53 binary digits times a power of two. 1/10 has a factor of 5 in its denominator, so it repeats forever in binary and must be rounded to the nearest double. As a result, stored 0.1 and 0.2 are slightly high, and stored 0.3 is slightly low. Adding the stored values exactly gives a result halfway between two doubles, and tie-to-even rounds it to the double one gap above stored 0.3, so == is false even though + did its job correctly. Printing shows the shortest decimal that maps back to each double, so 0.1 prints as 0.1 but the sum prints as 0.30000000000000004.", + "7": "The hardest part was counting in units of u (line 23). It does not say why doubles between 0.25 and 0.5 must use whole numbers from 2^52 to 2^53 times 2^-54, rather than a smaller whole number with a larger multiplier. One sentence would help: the whole number always uses all 53 digits. Mixing u/4 steps with values written in u (for example ...198.5) also takes effort.", + "8": "True. 1/2, 1/4 and 3/4 have power-of-two denominators, so each is stored exactly. The exact sum, 0.75, is itself a double, so no rounding happens.", + "9": "No. + computes the exact sum of its inputs and rounds it to the nearest double, which is correct. The difference comes from the inputs: they were the stored values, not 0.1 and 0.2. The tie-breaking rounding of the sum then moves the result to a different double from the one 0.3 rounds to.", + "10": "Yes, it is true. Decimal stores base-10 digits, and 1/10 ends in base 10, so all three values are exact.", + "11": "0.75 = 3 × 2^-2 is a whole number times a power of two. 1/10 has a factor of 5 in its denominator, and no power of two is divisible by 5. So in binary 0.1 repeats forever (0011…) and has to be cut off and rounded.", + "12": "Their errors do not cancel the way 0.3's error needs. Both are rounded up (+5.55e-18 and +1.11e-17), while 0.3 is rounded down (−1.11e-17). The exact sum, ...595.5 u, sits exactly halfway between ...595 and ...596. Tie-to-even sends it to ...596, one gap (5.55e-17) above stored 0.3 at ...595.", + "13": "Printing shows the shortest decimal that rounds back to the same double. The text \"0.1\" rounds to stored 0.1, so one digit is enough. The sum's double, ...596, needs 0.30000000000000004 instead." + }, + "audit": { + "terms": [ + "The same idea has two names: 'the power' (line 9: 'hold the sign and the power') and 'exponent' (line 65: 'sign and exponent bits').", + "Three words name the same integer part of a double: 'whole number' (line 9), 'multiplier' is used for the power-of-two factor (line 23: 'the multiplier is 2^-55'), and 'count of u' (line 40). This risks confusing which part is meant.", + "'gap' and 'u' are used together and overlap: 'gap ... u = 2^-54' (line 23), but elsewhere the gap is 'u/4' or 'u/2'. Later, 'one gap (5.55e-17)' (lines 42 and 61) assumes the gap means u." + ], + "unexplained": [ + "The 'Error' column (line 29 table: +5.55e-18, +1.11e-17, −1.11e-17) is given in decimal while every other column is in u. The text never says that these errors are what decide which double the sum lands on.", + "Line 49, 'with 17 significant digits': nothing says why 17 matters or what it implies.", + "Line 9, '53 binary digits ... other 11 bits': nothing says what more or fewer digits would change, for example a smaller gap." + ], + "missing_why": [ + "Line 23, 'Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54': it does not explain why the whole number must fill all 53 digits, which is what fixes the multiplier.", + "Line 40, tie-to-even: it does not explain why ties go to an even count instead of always rounding up or down.", + "Line 25, '0.1 ... its stored value is a multiple of u/4, such as …198.5': it does not show that 198.5 is the nearest multiple of 0.25 to 198.4." + ], + "excess": [ + "Lines 63–67, the 'Not covered here' list of references, is not needed by this reader.", + "Line 61, the money advice ('count whole cents as integers'), is a detour from the argument." + ] + }, + "gaps": [ + "The explanation never says why the integer part always has exactly 53 bits within a range. That is the step that makes the gap depend on the size of the number.", + "The tie-to-even rule is stated with no reason behind it.", + "It does not show that the decimal-errors table and the counts in u say the same thing." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "9": "Full marks, narrowly. The answer gives the right reason, that the inputs were stored values and not 0.1 and 0.2, but does not say why (1/10 does not end in binary)." + }, + "pass": true + }, + "usage": { + "cost": 0.16206720000000002, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same idea has two names: 'the power' (line 9: 'hold the sign and the power') and 'exponent' (line 65: 'sign and exponent bits')." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Three words name the same integer part of a double: 'whole number' (line 9), 'multiplier' is used for the power-of-two factor (line 23: 'the multiplier is 2^-55'), and 'count of u' (line 40). This risks confusing which part is meant." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'gap' and 'u' are used together and overlap: 'gap ... u = 2^-54' (line 23), but elsewhere the gap is 'u/4' or 'u/2'. Later, 'one gap (5.55e-17)' (lines 42 and 61) assumes the gap means u." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "The 'Error' column (line 29 table: +5.55e-18, +1.11e-17, −1.11e-17) is given in decimal while every other column is in u. The text never says that these errors are what decide which double the sum lands on." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 49, 'with 17 significant digits': nothing says why 17 matters or what it implies." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 9, '53 binary digits ... other 11 bits': nothing says what more or fewer digits would change, for example a smaller gap." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 23, 'Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54': it does not explain why the whole number must fill all 53 digits, which is what fixes the multiplier." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 40, tie-to-even: it does not explain why ties go to an even count instead of always rounding up or down." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 25, '0.1 ... its stored value is a multiple of u/4, such as …198.5': it does not show that 198.5 is the nearest multiple of 0.25 to 198.4." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "Lines 63–67, the 'Not covered here' list of references, is not needed by this reader." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "Line 61, the money advice ('count whole cents as integers'), is a detour from the argument." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "The explanation never says why the integer part always has exactly 53 bits within a range. That is the step that makes the gap depend on the size of the number." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "The tie-to-even rule is stated with no reason behind it." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "It does not show that the decimal-errors table and the counts in u say the same thing." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/read.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/read.json new file mode 100644 index 0000000..ea740ac --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/read.json @@ -0,0 +1,40 @@ +{ + "answers": { + "1": { + "answer": "A double is a whole number times a power of two, so it can only be a fraction whose denominator is a power of two. 1/10 has a factor 5 in its denominator, and no power of two is divisible by 5. So in binary, 1/10 repeats forever and has to be cut off and rounded.", + "quote": "The denominator of 1/10 has a factor 5. No power of two is divisible by 5, so 1/10 never ends in binary:" + }, + "2": { + "answer": "Stored 0.1 and stored 0.2 are both slightly too big, and stored 0.3 is slightly too small. The exact sum of the stored inputs lands exactly halfway between two doubles (…595.5 u). The tie rule rounds to the even one, …596 u. Stored 0.3 is …595 u, one gap away, so the two values are different and == is false.", + "quote": "The exact sum is exactly halfway between the doubles at …595 u and …596 u. This case is a **tie**." + }, + "3": { + "answer": "Compare with a tolerance, such as abs((0.1 + 0.2) - 0.3) < 1e-9. The tolerance should be much bigger than the gap but smaller than any difference that matters to you. For money, count whole cents as integers or use a decimal type.", + "quote": "Compare with a tolerance instead: `abs((0.1 + 0.2) - 0.3) < 1e-9` is true" + }, + "4": { + "answer": "Yes, it is true. 1/2, 1/4 and 3/4 have power-of-two denominators, so each is stored with no error, and the exact sum is itself a double.", + "quote": "All three are fractions with power-of-two denominators (1/2, 1/4, 3/4), so each is stored with error 0, and the exact sum is itself a double." + } + }, + "stops": [ + "Where the sizes come from: the text says the range 0.25–0.5 uses 'whole numbers from 2^52 up to 2^53, times 2^-54' but never says why the multiplier is 2^-54: \"Doubles between 0.25 and 0.5 are whole numbers from 2^52 up to 2^53, times 2^-54.\"", + "Why 0.1's stored value is …198.5 and not …198.25 is only clear after checking the table: \"its stored value is a multiple of u/4, such as …198.5.\"", + "The tie rule is stated but not justified, so the reader may wonder why 'even' is the choice: \"sends a tie to the neighbor whose count of u is even.\"", + "The rule for printing is new and needs a reread: \"Languages print the shortest decimal that rounds back to the same double.\"", + "Why 1e-9 is the right size 'for values near 1' is not explained: \"for values near 1, 1e-9 meets both conditions.\"", + "The table has no blank line before the next heading, which may break how the page renders: \"| 0.3 | 5404319552844595.2 | u | 5404319552844595 | 0.29999999999999998890 | −1.11e-17 |\"" + ], + "excess": [ + "The Decimal type is off the main question: \"Python's `Decimal(\"0.1\") + Decimal(\"0.2\") == Decimal(\"0.3\")` is true\"", + "The list of topics not covered: \"How errors grow over many operations, such as adding 0.1 ten times: see a numerical analysis text.\"", + "Giving the gap for the 0.0625–0.125 range again, which the table already shows: \"Between 0.0625 and 0.125 the gap is u/4.\"", + "The idea that the addition is correct repeats the summary at the top: \"The addition did not make a mistake.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/published/run.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/run.json new file mode 100644 index 0000000..88d5ad3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/float-sum/explanation.md`, plain prose with one table of numbers, because the answer is a single chain of cause and effect with worked values (\"1/10 never ends in binary → 0.1, 0.2 and 0.3 get rounded → their sum lands one step above where 0.3 lands\"), with nothing to click through or watch move; it passes `explainer check`, the probe, two cold reads with no blocking findings, and the blind test (13/13), though code execution wasn't allowed here so I worked out all the numbers by hand.", + "error": null, + "cost": 1.552655, + "seconds": 607, + "turns": 60, + "files": [ + "explainers/float-sum/explanation.md" + ], + "review_cost": 0.8245, + "reviews": 6 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/explanation.md b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/explanation.md new file mode 100644 index 0000000..ec350e0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/explanation.md @@ -0,0 +1,55 @@ +# Why does 0.1 + 0.2 not equal 0.3? + + + +**The computer never holds 0.1, 0.2, or 0.3. It holds the nearest binary fractions, and those small errors add up to a result that is a different number from the stored 0.3.** In most languages, `0.1 + 0.2` prints `0.30000000000000004`, so `0.1 + 0.2 == 0.3` is false. + +## The same problem in base 10 + +Suppose a calculator keeps 3 decimal digits. It cannot hold 1/3, so it stores 0.333. Add two of them: 0.333 + 0.333 = 0.666. But the calculator stores 2/3 as 0.667. So "1/3 + 1/3 == 2/3" is false on this calculator. + +The calculator adds correctly. The inputs were already wrong, because 1/3 has no finite decimal form. + +## Why 0.1 has the same problem in base 2 + +Most languages store a decimal number as a **double**: a binary fraction with 53 significant bits (the IEEE 754 standard). A binary fraction is a sum of powers of 2: 1/2, 1/4, 1/8, and so on. + + +In base 2, 0.1 is like 1/3 in base 10. Its binary form, 0.000110011001100…, repeats forever. A double keeps only 53 bits, so it stores the nearest binary fraction. Call this the **stored value**: + +| Written | Stored value (first 34 digits) | +|---|---| +| 0.1 | 0.1000000000000000055511151231257827 | +| 0.2 | 0.2000000000000000111022302462515654 | +| 0.3 | 0.2999999999999999888977697537484345 | + +The stored 0.1 and 0.2 are slightly too high. The stored 0.3 is slightly too low. + +## What happens in the addition + + +1. The computer adds the two stored values. The exact sum is 0.3000000000000000166533453693773481. +2. That sum is not a double either, so the computer rounds it to the nearest double. The sum is exactly halfway between two doubles; the tie rule picks the upper one: 0.3000000000000000444089209850062616. +3. The stored 0.3 is the double just below that result. The two differ by 5.551115123125783e-17, the gap between neighbouring doubles near 0.3. +4. `==` compares the doubles bit for bit. They differ, so the answer is false. + +## When addition is exact + + +Before you read on, predict: is `0.5 + 0.25 == 0.75` true? + +It is true. 0.5 is 1/2 and 0.25 is 1/4, so both are short binary fractions. Their sum, 0.75, is also one. No step rounds, so no error appears. The problem comes only from numbers that have no short binary form, such as 0.1, 0.2, and 0.3. + +## "But print(0.1) shows 0.1" + + +Languages print the shortest decimal that maps back to the same double. For the stored 0.1, that decimal is "0.1". For the result of 0.1 + 0.2, the shortest such decimal is 0.30000000000000004, because "0.3" maps to a different double. The print hides the error; it does not remove it. + +## What to do + + +- Compare with a tolerance, not `==`. `abs(0.1 + 0.2 - 0.3) < 1e-9` is true. +- For money, count in integer cents: 10 + 20 == 30 is always true. +- If you need exact decimal arithmetic, use a decimal type, for example Python `Decimal` or Java `BigDecimal`. These are slower than doubles. + +This behaviour is not a bug and not random. Every language that uses IEEE 754 doubles (JavaScript, Python, Java, C, and most others) gives the same result every time. diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/model.md b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/model.md new file mode 100644 index 0000000..892e7d2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/model.md @@ -0,0 +1,80 @@ +# Semantic model: why 0.1 + 0.2 != 0.3 + +> Source of truth. Every rendering derives from this file. + +## Central question + +Why does `0.1 + 0.2 == 0.3` evaluate to false in most programming languages? + +## Audience and prior knowledge + +A technical reader: school mathematics (fractions, powers, repeating decimals), basic programming (variables, `==`, printing). New to binary fractions and IEEE 754. Does not know that a float stores a binary fraction with a fixed number of bits. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| double | The default number type for decimals in most languages: a binary fraction with 53 significant bits (IEEE 754 binary64). | fact | +| stored value | The double nearest to the number written in the source code. | fact | +| rounding error | The difference between a written number and its stored value. | fact | +| spacing | The distance between two neighbouring doubles; 2^-54 ≈ 5.55e-17 for numbers between 0.25 and 0.5. | fact | + +## Causal chain + +```text +0.1 has an infinite binary expansion → the double keeps 53 bits → stored value ≠ 0.1 +→ the errors of 0.1 and 0.2 add → the exact sum is rounded to a double +→ that double is one spacing above the stored value of 0.3 → == is false +``` + +## Quantities + +- stored 0.1 = 0.1000000000000000055511151231257827… (error +5.55e-18) +- stored 0.2 = 0.2000000000000000111022302462515654… (error +1.11e-17) +- stored 0.3 = 0.2999999999999999888977697537484345… (error −1.11e-17) +- exact sum of stored 0.1 and 0.2 = 0.3000000000000000166533453693773481… +- result of 0.1 + 0.2 = 0.3000000000000000444089209850062616… (prints as 0.30000000000000004) +- difference result − stored 0.3 = 2^-54 ≈ 5.551115123125783e-17 = one spacing +- The exact sum lies exactly halfway between the two neighbouring doubles; round-half-to-even chooses the upper one. + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Store a binary fraction | Store decimal digits | Binary hardware is faster and simpler; decimal types exist (Python `Decimal`, Java `BigDecimal`) but are slower. | +| Fixed 53 bits | Unlimited bits | 0.1 in binary is 0.000110011001100… and never ends, so no finite memory holds it exactly. | +| Compare with a tolerance | `==` | `0.1 + 0.2 == 0.3` is false; `abs(0.1 + 0.2 - 0.3) < 1e-9` is true. | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| Sums of doubles are exact when every value is a sum of powers of 2 that fits in 53 bits | 0.5 + 0.25 == 0.75 is true | 0.1 + 0.2 == 0.3 is false | +| Decimal analogy: 3 decimal digits | 0.5 + 0.25 = 0.75 exact | 1/3 → 0.333; 0.333 + 0.333 = 0.666, but 2/3 → 0.667 | + +## Terms + +- **double**: as above. +- **stored value**: the double the computer actually holds. Never call it "the number 0.1". +- **rounding error**: written number minus stored value. +- **spacing**: gap between neighbouring doubles; small near 0, large for large numbers. + +## Epistemic status + +- **Established:** IEEE 754 binary64 values above; they follow mathematically from the standard. + +## Confusion points + +- "The computer adds badly." → The addition is correctly rounded. The inputs are already not 0.1 and 0.2. +- "But `print(0.1)` shows 0.1." → Languages print the shortest decimal that maps back to the same double. The stored value is still not 0.1. +- "The error is random." → It is deterministic: the same code gives the same result every time. + +## Scope + +- **Out of scope:** exponent field and bit layout — see IEEE 754; error accumulation in long sums (Kahan summation); NaN and infinities. + +## Representation decision + +- **Stage:** 1 +- **Reason:** A short linear causal chain with worked numbers; a table carries the numbers. No parameters to explore. +- **Levels:** L1 the answer / L2 the stored values / L3 why binary cannot hold 0.1 / L4 what to do. diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/model.yaml b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/model.yaml new file mode 100644 index 0000000..66b3a57 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/model.yaml @@ -0,0 +1,60 @@ +# Machine-readable half of model.md. `explainer check float-sum` holds every rendering to it. + +values: + bits: 53 + stored_01: "0.1000000000000000055511151231257827" + stored_02: "0.2000000000000000111022302462515654" + stored_03: "0.2999999999999999888977697537484345" + exact_sum: "0.3000000000000000166533453693773481" + result: "0.3000000000000000444089209850062616" + result_printed: "0.30000000000000004" + gap: "5.551115123125783e-17" + +# Written inputs, the binary expansion, the standard's name, and small illustrative integers. +allow: [0.1, 0.2, 0.3, 0.5, 0.25, 0.75, 0.333, 0.666, 0.667, 1e-9, 2, 3, 4, 8, 10, 20, 30, 34, 754, + "0.000110011001100", + "0.1000000000000000055511151231257827", "0.2000000000000000111022302462515654", + "0.2999999999999999888977697537484345", "0.3000000000000000166533453693773481", + "0.3000000000000000444089209850062616", "0.30000000000000004", "5.551115123125783e-17"] + +require: + - path: result_printed + in: [prose] + - path: stored_01 + in: [prose] + - path: stored_03 + in: [prose] + +claims: + - id: why-binary-inexact + kind: why + statement: "0.1 has an infinite binary expansion, so a double with 53 bits stores only the nearest binary fraction." + alternative: "Store 0.1 exactly." + counterexample: "0.1 = 0.000110011001100… in binary and never ends; stored 0.1 = 0.1000000000000000055511…" + ask: "Why can a double not hold 0.1 exactly when it can hold 0.75 exactly?" + - id: mech-sum + kind: mechanism + statement: "The rounding errors of the stored 0.1 and 0.2 add, and the sum is rounded to the double one spacing above the stored 0.3." + example: "stored 0.1 + stored 0.2 = 0.30000000000000001665…; rounded to 0.30000000000000004441…; stored 0.3 = 0.29999999999999998889…" + - id: guarantee-exact + kind: guarantee + statement: "Addition gives the exact answer when the inputs and the answer are sums of powers of 2 that fit in 53 bits." + example: "0.5 + 0.25 == 0.75 is true." + counterexample: "0.1 + 0.2 == 0.3 is false, because 0.1, 0.2, 0.3 are not sums of powers of 2." + - id: misc-print + kind: misconception + correction: "The stored value of 0.1 is 0.1000000000000000055511…; printing chooses a short decimal for display only." + statement: "print(0.1) shows 0.1 because the language prints the shortest decimal that maps back to the same double, not because the double equals 0.1." + ask: "If print(0.1) shows 0.1, why is the stored value not 0.1?" + - id: why-tolerance + kind: why + statement: "Compare doubles with a tolerance, or use integers or a decimal type, not ==." + alternative: "Compare with ==" + counterexample: "0.1 + 0.2 == 0.3 is false; abs(0.1 + 0.2 - 0.3) < 1e-9 is true." + +accept_unjustified: {} + +terms: + - term: stored value + means: "the double the computer holds for a written number" + avoid: [imprecise number, fuzzy] diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/numbers.py b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/numbers.py new file mode 100644 index 0000000..03f6bc5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/explainers/float-sum/numbers.py @@ -0,0 +1,10 @@ +from decimal import Decimal as D +import math + +for x in [0.1, 0.2, 0.3, 0.1 + 0.2]: + print(repr(x), D(x), x.hex()) +print("errors", D(0.1) - D("0.1"), D(0.2) - D("0.2"), D(0.3) - D("0.3")) +print("exact sum", D(0.1) + D(0.2)) +print("gap", D(0.1 + 0.2) - D(0.3)) +print("ulp(0.3)", D(math.ulp(0.3))) +print(0.5 + 0.25 == 0.75, abs(0.1 + 0.2 - 0.3) < 1e-9, 0.1 + 0.2 - 0.3) diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/read.json new file mode 100644 index 0000000..616f474 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/read.json @@ -0,0 +1,41 @@ +{ + "answers": { + "1": { + "answer": "In base 2, 0.1 is like 1/3 in base 10: its binary form repeats forever. A double keeps only 53 bits, so the computer stores the nearest binary fraction, which is slightly higher than 0.1.", + "quote": "In base 2, 0.1 is like 1/3 in base 10. Its binary form, 0.000110011001100…, repeats forever. A double keeps only 53 bits, so it stores the nearest binary fraction." + }, + "2": { + "answer": "The stored 0.1 and 0.2 are both slightly too high. Their exact sum is not a double, so it gets rounded. It is exactly halfway between two doubles, and the tie rule picks the upper one. The stored 0.3 is slightly too low and is the double just below that result. == compares the bits, they differ, so the answer is false.", + "quote": "The stored 0.3 is the double just below that result. The two differ by 5.551115123125783e-17, the gap between neighbouring doubles near 0.3." + }, + "3": { + "answer": "Compare with a tolerance, for example abs(a - 0.3) < 1e-9. For money, count in integer cents. If exact decimal arithmetic is needed, use a decimal type such as Python Decimal or Java BigDecimal.", + "quote": "Compare with a tolerance, not `==`. `abs(0.1 + 0.2 - 0.3) < 1e-9` is true." + }, + "4": { + "answer": "Yes, it is true. 0.5 (1/2), 0.25 (1/4) and 0.75 are all short binary fractions, so no step rounds and no error appears.", + "quote": "It is true. 0.5 is 1/2 and 0.25 is 1/4, so both are short binary fractions. Their sum, 0.75, is also one. No step rounds, so no error appears." + } + }, + "stops": [ + "\"53 significant bits\": the term 'significant bits' is not defined, and it is unclear how 53 bits relate to the 34 digits shown in the table.", + "\"(the IEEE 754 standard)\": the standard is named but never explained.", + "\"the tie rule picks the upper one\": the tie rule is not explained, so a reader may wonder whether it always rounds up.", + "\"The sum is exactly halfway between two doubles\": no reason is given for why it lands exactly halfway; a reader has to take this on trust.", + "\"short binary fractions\": 'short' is never defined, for example how many bits counts as short.", + "\"`abs(0.1 + 0.2 - 0.3) < 1e-9`\": there is no reason for choosing 1e-9 or guidance on how to pick a tolerance.", + "\"Languages print the shortest decimal that maps back to the same double\": 'maps back' needs a reread to follow." + ], + "excess": [ + "\"\": an internal note that is visible in the source and means nothing to this reader.", + "\"Stored value (first 34 digits)\": the full 34-digit expansions are more detail than needed; 'slightly too high/low' carries the point.", + "\"These are slower than doubles.\": a side remark that does not help answer the question.", + "\"Every language that uses IEEE 754 doubles (JavaScript, Python, Java, C, and most others) gives the same result every time.\": partly repeats the opening 'In most languages' line." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/run.json new file mode 100644 index 0000000..b1120da --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/float-sum/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/float-sum/explanation.md` as prose. The cause is a short chain of steps with exact numbers, and text plus one table explains that better than a diagram or interactive page; it passes `explainer check`, and the files also include `model.md`/`model.yaml` and a leftover `numbers.py` I couldn't run or delete, so the decimal values come from the IEEE 754 standard and I didn't recompute them.", + "error": null, + "cost": 0.4691284, + "seconds": 83, + "turns": 24, + "files": [ + "explainers/float-sum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/explainers/gps-relativity/explanation.md b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/explainers/gps-relativity/explanation.md new file mode 100644 index 0000000..e2a4958 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/explainers/gps-relativity/explanation.md @@ -0,0 +1,150 @@ +# How GPS finds your position, and why it needs relativity + +The short version: GPS is a clock system. Each satellite says what time it is. Your receiver works out how long each message took to arrive, and those delays tell it where it is. Light covers about 30 cm in a nanosecond, so the clocks have to agree to within a few billionths of a second. At that level of precision, Einstein's two theories of relativity both change how fast the satellite clocks tick, and the change is large enough that it has to be designed out. + +The rest of this file explains that in four steps: + +1. Distance from time +2. Why you need four satellites, not three +3. How the receiver actually solves for position (with code) +4. Relativity: what changes the satellite clocks' rate, by how much, and what the system does about it + +--- + +## 1. Distance from time + +About 30 GPS satellites orbit roughly 20,200 km above the ground, and each goes round about twice a day. Each one carries atomic clocks and constantly broadcasts a radio signal containing, in effect: + +> "I am satellite #12. This part of the signal left me at exactly time *t*ₛ. At that moment I was at position (*x*, *y*, *z*)." + +The satellite knows its own position because ground stations track every satellite and regularly upload a precise description of its orbit, called the *ephemeris*. + +Your receiver notes the time *t*ᵣ when that part of the signal arrives. Radio waves travel at the speed of light, *c* ≈ 299,792,458 m/s, so + +``` +distance to satellite = c × (t_r − t_s) +``` + +The signal takes about 67–86 milliseconds to arrive. Getting a distance accurate to a metre means measuring that delay to about 3 nanoseconds. **Every nanosecond of timing error is about 30 cm of position error.** Keep that conversion in mind, because the rest of the story depends on it. + +One distance puts you somewhere on a sphere centred on that satellite. With a second satellite you are on the circle where two spheres meet. A third sphere cuts that circle at two points. One of those points is usually far out in space or deep underground, so in principle three satellites give you a position. + +In practice, three is not enough. + +## 2. Why four satellites: your receiver's clock is bad + +The satellites have atomic clocks that cost a lot and are steered against a master time scale on the ground. Your phone has a cheap quartz oscillator. It may easily be off by a millisecond, and a millisecond is 300 km. + +So *t*ᵣ in the formula above isn't the true arrival time. It is the true arrival time plus an unknown receiver clock error δ*t*. That error is the **same for every satellite**, because all the measurements are made with the same clock. Each measured distance is therefore wrong by the same amount, *b* = *c*·δ*t*. These measured distances are called **pseudoranges**: + +``` +ρᵢ = |sᵢ − p| + b +``` + +Here `sᵢ` is satellite *i*'s known position, `p` = (*x*, *y*, *z*) is your unknown position, and `b` is your unknown clock error expressed in metres. + +That makes **four unknowns**: *x*, *y*, *z* and *b*. Four unknowns need four equations, so you need four satellites. A useful side effect is that once the receiver has solved for *b*, it knows the time to within tens of nanoseconds. That is why GPS is also one of the world's main sources of precise time, used by phone networks, power grids and financial systems. + +Usually 8–12 satellites are visible. With more than four, the receiver does a least-squares fit, which averages out noise. + +## 3. Solving the equations + +The equations are non-linear because of the square root inside |sᵢ − p|. The standard method is to guess, linearise and repeat: + +1. Start with a rough guess, such as the centre of the Earth with *b* = 0. +2. For each satellite, predict the pseudorange from the current guess and compare it with the measured one. +3. Work out how much each prediction would change if you nudged *x*, *y*, *z* or *b*. For the position this is the unit vector pointing from the satellite to you. For *b* it is simply 1. +4. Solve the resulting *linear* least-squares problem for the correction, apply it, and go back to step 2. + +This usually converges in 4–6 iterations, even from the centre of the Earth. + +```python +import numpy as np + +def solve_position(sat_pos, pseudoranges, iters=10): + """sat_pos: (N,3) array of satellite positions in metres (N >= 4). + pseudoranges: (N,) measured pseudoranges in metres. + Returns (receiver position, receiver clock bias in metres).""" + x = np.zeros(4) # [x, y, z, b], starting at Earth's centre + for _ in range(iters): + diff = sat_pos - x[:3] + ranges = np.linalg.norm(diff, axis=1) + predicted = ranges + x[3] + residual = pseudoranges - predicted + # Jacobian: d(predicted)/d(x,y,z) = -unit vector to satellite; d/db = 1 + J = np.hstack([-diff / ranges[:, None], np.ones((len(ranges), 1))]) + dx, *_ = np.linalg.lstsq(J, residual, rcond=None) + x += dx + if np.linalg.norm(dx) < 1e-4: + break + return x[:3], x[3] +``` + +Real receivers also correct for several known sources of error: the signal slowing down in the ionosphere and the troposphere, reflections off buildings, small errors in the broadcast orbits, and the effects described below. All of this together gives a typical phone accuracy of a few metres. + +Note what the whole method assumes: **the satellites' clocks agree with each other and with GPS system time.** The receiver can absorb an error in its own clock, because that error is the same for every satellite. It cannot absorb errors that differ from satellite to satellite. The receiver has no way to tell one satellite's clock being 10 ns fast from that satellite being 3 m farther away. + +## 4. Where relativity comes in + +Relativity says clocks don't all run at the same rate. Two effects matter for GPS, and they push in opposite directions. + +### Effect 1: moving clocks run slow (special relativity) + +A clock moving at speed *v* relative to you ticks slower by a fraction of roughly + +``` +v² / (2c²) +``` + +GPS satellites move at about 3.87 km/s: + +``` +(3874 m/s)² / (2 × (3.0×10⁸ m/s)²) ≈ 8.3 × 10⁻¹¹ +``` + +Over one day (86,400 s) that adds up to about **7 microseconds slow**. + +### Effect 2: clocks higher up run fast (general relativity) + +Clocks deeper in a gravitational field run slower. A clock at the satellite's altitude feels weaker gravity than one on the ground, so it runs faster. The fractional difference is + +``` +(GM/c²) × (1/R_earth − 1/r_orbit) +``` + +Here *GM* = 3.986×10¹⁴ m³/s² is Earth's gravitational parameter, *R*_earth ≈ 6,371 km and *r*_orbit ≈ 26,560 km, measured from Earth's centre: + +``` +3.986×10¹⁴ × (1/6.371×10⁶ − 1/2.656×10⁷) / (3.0×10⁸)² ≈ 5.3 × 10⁻¹⁰ +``` + +Over one day that is about **45 microseconds fast**. + +### The net effect + +``` ++45 μs/day (higher up → faster) + −7 μs/day (moving → slower) +─────────── ++38 μs/day satellite clocks gain on ground clocks +``` + +38 microseconds is 38,000 nanoseconds. At 30 cm per nanosecond, that much timing error is about **11 km of light travel per day**. The system needs nanosecond agreement, and relativity puts tens of thousands of nanoseconds of disagreement into it every day. + +A note on the popular claim that "without relativity GPS would drift 11 km a day". That number shows the *scale* of the effect, but it isn't literally what would happen. If every satellite clock gained exactly the same amount, part of the error would end up in the receiver's *b*. The ground stations also already estimate each satellite's clock drift and broadcast corrections. A more accurate statement is this: if nobody had accounted for relativity, the satellite clocks would steadily drift away from ground time, and the parts of the effect that differ from satellite to satellite and moment to moment (next section) would put metre-level errors directly into your position. Relativity isn't an optional refinement. You have to account for it for the system to work. + +### What GPS actually does about it + +**1. Detune the clocks at the factory.** Each satellite's reference oscillator is meant to run at 10.23 MHz. It is deliberately set slightly slow, to 10.22999999543 MHz, a fractional offset of about 4.465×10⁻¹⁰. Once the satellite is in orbit, the relativistic speed-up brings it to exactly 10.23 MHz as seen from the ground. This one adjustment removes almost all of the 38 μs/day. + +**2. Correct for the slightly elliptical orbit.** The orbits are not perfect circles. When a satellite is nearest Earth it moves faster and sits deeper in gravity, so its clock runs slower. When it is farthest away, its clock runs faster. The resulting periodic clock error reaches a few tens of nanoseconds, which means up to about 10 m of range error. A fixed factory offset can't remove a varying error, so the receiver calculates it from the broadcast orbit and subtracts it. In the GPS specification it appears as `Δt_r = F · e · √a · sin(E)`, where *e* is how elliptical the orbit is, *a* is the size of the orbit, *E* is where the satellite is in its orbit, and *F* = −4.4428×10⁻¹⁰ s/√m is a constant built from *G*, Earth's mass and *c*. + +**3. Account for Earth turning while the signal travels (the Sagnac effect).** Positions are computed in a frame that rotates with Earth. During the roughly 70 ms the signal is in flight, the ground under you moves by up to about 30 m. The receiver handles this by rotating each satellite's position back by the angle Earth turns during the signal's flight. Leave this out and you get errors of tens of metres. + +## Summary + +- GPS measures **time**, not distance. A delay multiplied by the speed of light gives a range, and 1 ns of error gives 30 cm of range error. +- Your receiver's clock is unknown, so it solves for **four unknowns** (*x*, *y*, *z*, clock error) using **at least four satellites**, by repeatedly linearising and fitting. +- That only works if the **satellite clocks keep nanosecond agreement** with ground time. +- Relativity breaks that agreement by a lot. Orbital speed makes satellite clocks lose about 7 μs/day, and weaker gravity makes them gain about 45 μs/day, for a net gain of about **38 μs/day**, thousands of times the allowed error. +- GPS deals with this by **detuning the satellite clocks before launch**, having **receivers correct for elliptical orbits**, and **accounting for Earth's rotation** during signal travel. Each of these is engineering that relativity requires. diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/read.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/read.json new file mode 100644 index 0000000..fc05702 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "The receiver records the time t_r when a timestamped part of the satellite's signal arrives. It subtracts the send time t_s that the signal carries, then multiplies that delay by the speed of light: distance = c × (t_r − t_s). Each nanosecond of timing error gives about 30 cm of distance error.", + "quote": "Your receiver notes the time *t*ᵣ when that part of the signal arrives. Radio waves travel at the speed of light, *c* ≈ 299,792,458 m/s, so" + }, + "2": { + "answer": "The receiver's cheap clock has an unknown error, and that error is the same for every satellite. It adds the same offset b to every measured distance. So there are four unknowns (x, y, z and b), and four unknowns need four equations, which means four satellites.", + "quote": "That makes **four unknowns**: *x*, *y*, *z* and *b*. Four unknowns need four equations, so you need four satellites." + }, + "3": { + "answer": "Relativity makes satellite clocks run at a different rate from ground clocks. They gain about 45 μs/day from weaker gravity and lose about 7 μs/day from their speed, a net gain of about 38 μs/day. At 30 cm per ns, that is about 11 km of light travel per day. The explanation warns that the system would not literally drift 11 km a day: the parts of the effect that differ between satellites would cause metre-level position errors, which is still enough to break it.", + "quote": "38 microseconds is 38,000 nanoseconds. At 30 cm per nanosecond, that much timing error is about **11 km of light travel per day**." + }, + "4": { + "answer": "1 µs is 1,000 ns, and each ns is about 30 cm, so every computed distance is off by about 300 m. All three are off by the same amount, because one clock makes all the measurements. With only three satellites the receiver can't solve for that shared offset, so the 300 m error goes straight into its position.", + "quote": "Each measured distance is therefore wrong by the same amount, *b* = *c*·δ*t*." + } + }, + "stops": [ + "Ephemeris is defined, but it's not clear how often the upload happens or how accurate it is: \"regularly upload a precise description of its orbit, called the *ephemeris*\"", + "Why one of the two intersection points can be thrown out is only briefly justified: \"One of those points is usually far out in space or deep underground\"", + "The Jacobian/unit-vector step needs calculus the reader may not have, and its sign is confusing: \"For the position this is the unit vector pointing from the satellite to you\" while the code comment says \"-unit vector to satellite\"", + "Least-squares is used without being explained: \"the receiver does a least-squares fit, which averages out noise\"", + "The 11 km caveat is hard to follow, especially why a shared error would partly end up in b: \"part of the error would end up in the receiver's *b*\"", + "It points to \"next section\" for effects that vary between satellites, which isn't an obvious link: \"(next section) would put metre-level errors directly into your position\"", + "The receiver-correction formula uses terms that aren't really defined: \"*E* is where the satellite is in its orbit\"", + "The Sagnac explanation is brief, and it's unclear why the rotating frame matters: \"rotating each satellite's position back by the angle Earth turns during the signal's flight\"", + "The factory offset (4.465×10⁻¹⁰) doesn't obviously match the net of 5.3×10⁻¹⁰ and 8.3×10⁻¹¹: \"a fractional offset of about 4.465×10⁻¹⁰\"" + ], + "excess": [ + "The full Python code isn't needed for the conceptual questions: \"def solve_position(sat_pos, pseudoranges, iters=10):\"", + "The aside about time services goes beyond the question: \"used by phone networks, power grids and financial systems\"", + "The summary repeats points made earlier: \"GPS measures **time**, not distance. A delay multiplied by the speed of light gives a range, and 1 ns of error gives 30 cm of range error.\"", + "The list of other error sources is off-topic: \"the signal slowing down in the ionosphere and the troposphere, reflections off buildings\"", + "The GPS spec formula and constant are more detail than this reader needs: \"*F* = −4.4428×10⁻¹⁰ s/√m is a constant built from *G*, Earth's mass and *c*\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/run.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/run.json new file mode 100644 index 0000000..256ef70 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/gps-relativity/explanation.md`, a prose explanation, because this topic is a chain of reasoning with worked numbers and a short Python position solver, and prose with inline calculations and code follows that chain best for a technical reader.", + "error": null, + "cost": 0.17838859999999995, + "seconds": 57, + "turns": 2, + "files": [ + "explainers/gps-relativity/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/explanation.md b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/explanation.md new file mode 100644 index 0000000..a95d00a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/explanation.md @@ -0,0 +1,104 @@ +# How GPS finds your position, and why it needs relativity + +**The short answer.** A GPS receiver does not send anything. It only listens. Each satellite broadcasts its position and the exact time it sent its signal. The receiver measures how long each signal took to arrive. Then it multiplies that time by the speed of light to get a distance. With distances to four satellites, it can solve for where it is. + +This works only if the satellite clocks are correct to a few billionths of a second (nanoseconds). Relativity says a clock in orbit ticks at a different rate from a clock on the ground: about 38 microseconds per day faster. GPS builds that correction into every satellite and every receiver. + +Below, each step follows from the one before it. + +--- + +## 1. Time becomes distance + +Radio signals travel at the speed of light, *c* ≈ 300,000 km/s. The distance to a satellite is: + +``` +distance = c × (time received − time sent) +``` + +**Example.** GPS satellites orbit about 20,200 km above the ground. A signal from a satellite directly overhead takes about 0.067 s (67 ms) to arrive. A satellite near the horizon is farther away, so its signal takes up to about 86 ms. + +This conversion is unforgiving. Light travels **30 cm in one nanosecond**. A timing error of 10 ns gives a distance error of 3 m. A timing error of 1 microsecond (1,000 ns) gives 300 m. So GPS is really a system for distributing very precise time. + +## 2. Distances become a position + +One distance tells you that you are somewhere on a sphere around that satellite. + +- Two spheres intersect in a circle. +- A third sphere cuts that circle at two points. One point is near the Earth. The other is far out in space, so the receiver discards it. + +So three exact distances give a position. A flat version of the same idea: if you are 3 km from tower A, 4 km from tower B, and 5 km from tower C, only one spot on the map fits all three. + +## 3. The receiver's clock is the weak link, so use a fourth satellite + +Each satellite carries atomic clocks. Your phone has a cheap quartz clock. Suppose the quartz clock is wrong by 1 millisecond. Then every distance is wrong by 300 km, and the three spheres do not meet at a sensible point. + +The simplest fix is an atomic clock in every receiver. That fails on cost, size, and power. GPS uses a better trick: treat the receiver clock error *b* as a fourth unknown. The receiver's error adds the same amount, *c·b*, to every measured distance. For each satellite *i* at known position (*xᵢ, yᵢ, zᵢ*): + +``` +measured_distanceᵢ = √((x − xᵢ)² + (y − yᵢ)² + (z − zᵢ)²) + c·b +``` + +There are four unknowns (*x, y, z, b*), so the receiver needs four satellites. It solves the equations numerically, usually with more than four satellites to average out noise. A bonus: once it knows *b*, the receiver knows the true time to within tens of nanoseconds. + +**What this trick cannot fix.** The term *c·b* removes only an error that is **the same for every signal**. If one satellite's clock is off by 30 ns and the others are not, that satellite's distance is off by 9 m, and the solve cannot detect it. So every satellite clock must agree with one shared time scale, called **GPS time**, to within a few nanoseconds. Relativity enters here. + +## 4. Clocks in orbit do not tick at ground rate + +*Before you read on, predict: does a satellite clock run fast or slow compared with a clock on the ground?* + +Two established effects act on the satellite clock. Both are measured in experiments, not only predicted. They act in opposite directions. + +**Effect 1: speed (special relativity).** A moving clock ticks slower than a clock at rest, as seen by the clock at rest. For speeds much less than *c*, the clock loses a fraction *v²/2c²* of its time. A GPS satellite moves at about 3.87 km/s: + +``` +v² / 2c² = (3,870 m/s)² / (2 × (3.0×10⁸ m/s)²) ≈ 8.3 × 10⁻¹¹ +× 86,400 s per day ≈ 7.2 µs per day slower +``` + +**Effect 2: gravity (general relativity).** A clock deeper in a gravitational field ticks slower. The ground clock sits closer to the Earth's centre (6,371 km) than the satellite (26,560 km). So the satellite clock ticks *faster*. The fraction is the difference in gravitational potential divided by *c²*: + +``` +(GM/c²) × (1/R_earth − 1/r_orbit) += 4.43 mm × (1/6,371 km − 1/26,560 km) ≈ 5.29 × 10⁻¹⁰ +× 86,400 s per day ≈ 45.7 µs per day faster +``` + +(*GM/c²* for the Earth is a length, about 4.43 mm. It sets how strong the effect is.) + +**Net result.** Gravity wins: + +``` ++45.7 µs/day − 7.2 µs/day ≈ +38.5 µs/day +``` + +**What that number means.** 38.5 µs × 30 cm/ns ≈ **11.5 km**. That is the distance error that one day of uncorrected drift represents. Compare it with the few-nanosecond budget from section 3. The drift is about 0.45 ns per second, so it uses up that budget in about 10 seconds. + +## 5. How GPS corrects it + +**Before launch.** Engineers set each satellite clock slightly slow on the ground. The nominal clock frequency is 10.23 MHz. The factory setting is **10.22999999543 MHz**, which is lower by the net fraction 4.46 × 10⁻¹⁰. In orbit, the two relativistic effects speed the clock up by exactly that fraction. Seen from the ground, it then ticks at 10.23 MHz. + +**In the receiver.** The fixed offset assumes a perfectly circular orbit. Real orbits are slightly elliptical. As a satellite moves closer to the Earth and then farther away, its speed and gravity change, so its clock rate changes too. For an orbit with eccentricity 0.02, this swings the clock by up to about ±46 ns (±14 m). The amount differs for each satellite, so the fourth-satellite trick cannot absorb it. The receiver calculates this term from the broadcast orbit data and subtracts it. + +**From the ground.** Control stations track every satellite clock and upload small corrections for the remaining drift. Relativity is built into the model that these corrections assume. + +## 6. What if relativity were ignored? + +You often read that GPS would "drift 10 km per day" without relativity. That figure assumes the receiver trusts the satellite clocks. It is a fair measure of the size of the effect. It is not an exact forecast: + +- **The part common to all satellites** would partly cancel. It shifts every distance equally, so the receiver would absorb it into *b*, like its own clock error. The *time* the receiver reports would still drift by 38.5 µs per day. Power grids, phone networks, and financial exchanges use GPS time and would fail. +- **The parts that differ between satellites** would not cancel. Each satellite's eccentricity term alone produces errors of up to about 14 m. Satellite clocks would also drift away from GPS time, and from the ground stations that track them. + +Either way, GPS cannot meet its metre-level accuracy without relativity. + +--- + +## Summary + +1. The receiver measures signal travel time; time × *c* = distance. 1 ns = 30 cm. +2. Three distances fix a point. A fourth satellite removes the receiver's own clock error. +3. That trick removes only a shared error, so satellite clocks must agree to nanoseconds. +4. In orbit, speed slows the clock by 7.2 µs/day and weaker gravity speeds it up by 45.7 µs/day: net +38.5 µs/day, or 11.5 km of light travel. +5. GPS sets the clocks slow before launch and corrects the per-satellite orbit effect in the receiver. + +**Not covered here:** signal delays in the atmosphere, the Sagnac effect (the Earth rotates while the signal travels), how orbits are predicted, and centimetre-level methods such as RTK. diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/model.md b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/model.md new file mode 100644 index 0000000..bbc70a0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/model.md @@ -0,0 +1,21 @@ +# Model: GPS and relativity + +**Central question.** How does a receiver find its position from satellite signals, and why must relativity be included? + +**Reader.** Technical, school physics and maths, basic programming. New to GPS and to relativity. + +**Entities.** Satellite (atomic clock, known position), signal (carries send time and orbit data), receiver (cheap quartz clock), GPS time (the shared time scale). + +**Causal chain.** +1. Distance = c × travel time. 1 ns of timing error = 30 cm of distance error. +2. One distance → sphere. Three spheres → a point (plus one point far in space). +3. Receiver clock is unknown → add unknown b → need four satellites. +4. Step 3 removes only an error common to all signals. Each satellite clock must agree with GPS time to ~ns. +5. Satellite clocks run at a different rate from ground clocks: −7.2 µs/day (speed, special relativity) + 45.7 µs/day (weaker gravity, general relativity) = +38.5 µs/day. +6. Fix: set the clock to 10.22999999543 MHz before launch; receiver adds a per-satellite eccentricity term (up to ~46 ns ≈ 14 m for e = 0.02). + +**Epistemic notes.** Both effects are established and measured. The popular "10 km/day of position error" assumes the receiver trusts the satellite clocks; a drift common to all satellites partly cancels in the four-unknown solve. Per-satellite differences and the time output do not cancel. + +**Confusion points.** "Doesn't the 4th satellite fix all clock errors?" — only the shared one. "Fast or slow?" — two effects with opposite sign; gravity wins. + +**Scope (out).** Atmospheric delay, Sagnac effect, ephemeris fitting, carrier-phase / RTK, other GNSS. diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/model.yaml b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/model.yaml new file mode 100644 index 0000000..84b921f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/explainers/gps-relativity/model.yaml @@ -0,0 +1,22 @@ +c_m_per_s: 299792458 +light_per_ns_cm: 30 +earth_radius_km: 6371 +orbit_radius_km: 26560 +orbit_altitude_km: 20200 +orbit_speed_km_s: 3.87 +orbit_period_h: 11.97 +signal_travel_ms: [67, 86] +receiver_clock_error_example_ms: 1 +receiver_clock_error_example_km: 300 +special_relativity_fraction: 8.3e-11 +special_relativity_us_per_day: -7.2 +general_relativity_fraction: 5.29e-10 +general_relativity_us_per_day: 45.7 +net_fraction: 4.46e-10 +net_us_per_day: 38.5 +net_range_km_per_day: 11.5 +nominal_clock_mhz: 10.23 +factory_clock_mhz: 10.22999999543 +eccentricity_example: 0.02 +eccentricity_term_ns: 46 +eccentricity_term_m: 14 diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/read.json new file mode 100644 index 0000000..d9d35e7 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "The receiver measures how long each satellite's signal took to arrive: the time it was received minus the send time, which the satellite broadcasts. It multiplies that travel time by the speed of light (c ≈ 300,000 km/s) to get the distance. Each nanosecond is 30 cm.", + "quote": "The receiver measures how long each signal took to arrive. Then it multiplies that time by the speed of light to get a distance." + }, + "2": { + "answer": "Three exact distances would be enough to fix a position. But the receiver's cheap quartz clock is wrong by some unknown amount b, and that adds the same c·b to every distance. That makes four unknowns (x, y, z, b), so it needs four equations, which means four satellites.", + "quote": "There are four unknowns (*x, y, z, b*), so the receiver needs four satellites." + }, + "3": { + "answer": "Orbiting clocks gain about 38.5 µs per day: about 45.7 µs faster from weaker gravity, minus about 7.2 µs slower from speed. That equals about 11.5 km of light travel per day. The drift uses up the few-nanosecond budget in about 10 seconds. The explanation adds that 'drift ~10 km/day' measures the size of the effect, not an exact forecast. The part shared by all satellites would be absorbed into b, but reported time would still drift 38.5 µs/day. The per-satellite parts (eccentricity up to about 14 m, and clocks drifting away from GPS time) would still break metre-level accuracy.", + "quote": "38.5 µs × 30 cm/ns ≈ **11.5 km**. That is the distance error that one day of uncorrected drift represents." + }, + "4": { + "answer": "Each distance would be wrong by about 300 m (1 µs = 1,000 ns × 30 cm). The error is the same for every distance, so the three spheres would not meet at the right point. Reasoning from the explanation: a 1 µs fast clock makes each arrival time look about 1 µs later, so each distance would probably come out about 300 m too long. The explanation does not state the direction outright.", + "quote": "A timing error of 1 microsecond (1,000 ns) gives 300 m." + } + }, + "stops": [ + "'difference in gravitational potential' is used without being defined: \"The fraction is the difference in gravitational potential divided by *c²*\"", + "The units are hard to follow when mm is multiplied by 1/km: \"= 4.43 mm × (1/6,371 km − 1/26,560 km) ≈ 5.29 × 10⁻¹⁰\"", + "The term 'eccentricity' is not defined: \"For an orbit with eccentricity 0.02, this swings the clock by up to about ±46 ns (±14 m).\"", + "The net fraction 4.46 × 10⁻¹⁰ appears without being linked to the 38.5 µs/day: \"which is lower by the net fraction 4.46 × 10⁻¹⁰\"", + "'partly cancel' sounds like it conflicts with 'absorb it into b', which sounds complete: \"The part common to all satellites** would partly cancel.\"", + "No method is given for how the equations are solved: \"It solves the equations numerically, usually with more than four satellites to average out noise.\"", + "It's unclear why the observer matters here: \"as seen by the clock at rest\"", + "The direction of the error (sign of b) is not explained: \"The receiver's error adds the same amount, *c·b*, to every measured distance.\"" + ], + "excess": [ + "\"*Before you read on, predict: does a satellite clock run fast or slow compared with a clock on the ground?*\"", + "\"A flat version of the same idea: if you are 3 km from tower A, 4 km from tower B, and 5 km from tower C\"", + "\"**Not covered here:** signal delays in the atmosphere, the Sagnac effect\"", + "\"Below, each step follows from the one before it.\"", + "The Summary repeats the earlier sections: \"The receiver measures signal travel time; time × *c* = distance. 1 ns = 30 cm.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/run.json new file mode 100644 index 0000000..a891a46 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I wrote `explainers/gps-relativity/explanation.md` as prose with worked numbers, because the subject is one causal chain (time → distance → position → clock agreement → relativity) and needs no diagram or interactive page; I also wrote `model.md` and `model.yaml` with the values, but I had no shell, so `explainer check` was not run and the numbers are hand-checked only.", + "error": null, + "cost": 0.28801780000000005, + "seconds": 89, + "turns": 6, + "files": [ + "explainers/gps-relativity/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/explanation.md b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/explanation.md new file mode 100644 index 0000000..ff7aaeb --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/explanation.md @@ -0,0 +1,75 @@ +# How does GPS find your position, and why does it need relativity? + +**A GPS receiver times radio signals from four satellites and solves for its position and its own clock error. This works only if the satellite clocks agree to a few billionths of a second. Relativity makes those clocks gain 38 µs a day on ground clocks. Light covers 11.4 km in 38 µs, so engineers build the clocks to tick slow.** + +## 1. Time gives distance + +About 30 GPS satellites orbit 20,200 km above Earth. Each one carries atomic clocks and broadcasts a signal that says: "I am satellite k, I am at position P, and I sent this at time t." Your receiver (the GPS chip in a phone) only listens. + +The receiver notes when each signal arrives. Then: + +- travel time = receive time − send time +- range = travel time × c, where c is light speed and the range is the true distance to the satellite. + +From a satellite directly overhead, the travel time is about 67 ms. Light covers 300 m in one microsecond (1 µs = 10⁻⁶ s) and 0.3 m in one nanosecond (1 ns = 10⁻⁹ s). So every timing error becomes a range error at that rate. + +One range puts you somewhere on a sphere around the satellite: every point at that distance. Three spheres meet in two points. The receiver rejects the one far out in space. Three satellites would be enough, if every clock were perfect. + +## 2. The receiver's clock is wrong, so it uses a fourth satellite + + +Your phone's clock is quartz, not atomic. It can easily be off by 1 ms, and 1 ms × c = 300 km. + + +So the receiver does not compute a range. It computes a **pseudorange**: light speed × (receive time on the receiver clock − send time on the satellite clock). A pseudorange equals the range plus **b**, the receiver clock error expressed in metres. One clock makes every measurement, so b is the same in every pseudorange. That makes b one more unknown to solve for, next to the position. + + +**Worked example, on a line.** Put beacon A at 0 km and beacon B at 30 km. The receiver is at 12 km, so the true travel times are 40 µs and 60 µs. The receiver clock is 5 µs fast, so it measures 45 µs and 65 µs. Those give pseudoranges of 13.5 km and 19.5 km. + +The receiver lies between the beacons, so the two ranges must add up to 30 km. The pseudoranges add up to 33 km. The excess, 3 km, is b counted twice, so b = 1.5 km (5 µs × c). Subtract b from each: 12 km and 18 km. The receiver is at 12 km. With only beacon A, the receiver could not detect the 1.5 km error. + +In three dimensions the unknowns are x, y, z, and b. Four unknowns need four equations, so the receiver needs four satellites. +## 3. The satellite clocks must be almost perfect + + +The receiver can solve for its own clock error because every pseudorange shares it. A satellite clock error sits in only one pseudorange, so no unknown absorbs it. Each satellite clock must therefore match **GPS time**, the time kept by atomic clocks at ground stations. + +How close? Each nanosecond of satellite clock error is 0.3 m of range error. For a fix within about 5 m, the satellite clocks must agree with GPS time to about 17 ns. + +## 4. Relativity moves the satellite clocks by microseconds a day + + +**Time dilation** means that clocks in different motion, or at different heights in gravity, tick at different rates. It is measured, not only predicted. Two effects act on a satellite clock, compared with a clock on the ground: + +| Cause | Theory | Effect on the satellite clock | +|---|---|---| +| Speed: 3.9 km/s in orbit | special relativity: a moving clock ticks slower | loses 7 µs per day | +| Height: gravity is weaker 20,200 km up | general relativity: a clock higher above Earth ticks faster | gains 45 µs per day | +| **Net** | | **gains 38 µs per day** | + +The gravity effect is larger, so the clock runs fast. 38 µs is only 4.5 × 10⁻¹⁰ of a day, but light covers 11.4 km in 38 µs. Each uncorrected satellite clock would move 11.4 km of range further from GPS time every day. + +## 5. Why the receiver's b does not cancel the drift + + +Every satellite clock drifts at the same rate. Why does b not absorb the drift? + +Only if every clock had the same error at the same moment. If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right. + +But each satellite clock is set to GPS time at a different moment. Suppose satellite A is set on day 0 and satellite B on day 3. On day 5, A is 190 µs fast (57 km) and B is 76 µs fast (22.8 km). The 34 km difference is not shared, so no single b can absorb it, and the position fix fails. Every clock set on a different day carries a different error. + +## 6. The remedy: make the clocks tick slow before launch + + +Each satellite clock counts the ticks of a 10.23 MHz signal. Before launch, engineers set it to 10.22999999543 MHz: slower by 4.5 × 10⁻¹⁰, the fraction the clock gains in orbit. On the ground the clock runs slow. In orbit, relativity speeds it up to the rate of GPS time. Ground stations upload small corrections for what remains; the built-in offset keeps those corrections small instead of growing by 38 µs a day. + +## Summary + +Four pseudoranges give x, y, z, and the receiver's clock error, but only if satellite clocks stay within about 17 ns of GPS time. Relativity makes them gain 38 µs a day, so each clock is built to tick slow. + +## Not covered here + +- **Atmosphere:** charged and humid layers of the atmosphere slow the signal. These delays are the largest remaining errors. +- **Earth's rotation during the 67 ms flight (Sagnac effect)** and the **orbit-shape correction** (orbits are slightly elliptical, so each satellite clock error swings by up to about 23 ns): receivers correct both. See N. Ashby, "Relativity in the Global Positioning System", *Living Reviews in Relativity* (2003). +- **Why a moving clock ticks slow:** see the "light clock" argument in special relativity. +- **How the receiver solves the equations:** iteration from a guess, with a best fit when it hears more than four satellites. diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/model.md b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/model.md new file mode 100644 index 0000000..98a06cc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/model.md @@ -0,0 +1,141 @@ +# Semantic model: GPS positioning and relativity + +> Source of truth. Every rendering derives from this file. + +## Central question + +How does a GPS receiver find its position from satellite signals, and why must the system correct for relativity? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (speed = distance / time, the speed of light, gravity, solving simultaneous equations), basic programming. New to GPS and to relativity. Knows that light speed is about 300,000 km/s. Does not know what a pseudorange is, why four satellites are needed, or what time dilation is in numbers. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Satellite | One of about 30 GPS satellites, orbiting at about 20,200 km altitude, each carrying atomic clocks. | fact | +| Satellite clock | An atomic clock on a satellite; it stamps each signal with its send time. | fact | +| Signal | A radio message from a satellite that says "I am satellite k, at position P, and I sent this at time t". | fact (simplified) | +| Receiver | The GPS chip in a phone or car; it only listens, it never transmits. | fact | +| Receiver clock | A cheap quartz clock in the receiver; its error is not known in advance. | fact | +| Range | The true distance from a satellite to the receiver. | definition | +| Pseudorange | Light speed × (receive time on the receiver clock − send time on the satellite clock): a range plus an unknown error from the receiver clock. | definition | +| Clock bias | b: how far the receiver clock is ahead of GPS time, times light speed (in metres). | definition | +| GPS time | The reference time kept by atomic clocks at ground stations; satellite clocks are steered to it. | definition | +| Time dilation | Clocks in different motion or at different heights in gravity tick at different rates. A moving clock ticks slower than a clock at rest, by a fraction ≈ v²/2c². A clock higher above Earth ticks faster than one on the ground. | established fact (measured: Pound–Rebka 1959, Hafele–Keating 1971, GPS clocks) | +| Rate offset | The built-in setting that makes each satellite clock tick slow before launch: 10.23 MHz → 10.22999999543 MHz. 10.23 MHz is the nominal tick frequency of the satellite clock; the signal is derived from it. | fact | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| Signal travel time | times light speed gives | range | 1 µs ↔ 300 m | +| Receiver clock bias | adds the same error b to | every pseudorange | same instant, same clock | +| Four pseudoranges | determine | x, y, z, b | four equations, four unknowns | +| Orbit speed | slows | satellite clock | special relativity, −7 µs/day | +| Weaker gravity at altitude | speeds up | satellite clock | general relativity, +45 µs/day | +| Satellite clock rate error | grows into | range error | 38 µs/day ↔ 11 km/day | + +## Dependencies (learning order) + +1. Time of flight gives distance (range). +2. Several ranges fix a point (intersection of spheres). +3. The receiver clock is wrong, so each range is a pseudorange; add b as a fourth unknown, use four satellites. +4. The method needs satellite clocks that agree with ground time to nanoseconds. +5. Relativity makes satellite clocks run 38 µs/day fast; the system corrects it. + +## Causal chain + +```text +satellite sends time stamp → receiver measures travel time → × c → pseudorange +4 pseudoranges → solve x, y, z, b → position (and exact time) +satellite clock error of 1 ns → range error of 0.3 m +orbit: speed (−7 µs/day) + weaker gravity (+45 µs/day) → +38 µs/day → 11 km/day of range error if not corrected +→ rate offset: clocks set to tick slow before launch (10.23 MHz → 10.22999999543 MHz) +``` + +Why the drift would not cancel: b absorbs only an error that is the same in every pseudorange. All satellite clocks drift at the same rate, but each one was last set to GPS time on a different day. Satellite A set on day 0 and satellite B set on day 3: on day 5, A is 190 µs (57 km) off and B is 76 µs (22.8 km) off. The 34 km difference cannot go into one b. If all clocks were off by exactly the same 38 µs, b would absorb it: the position would be right and only the reported time would be wrong. + +## Temporal sequence + +1. Before launch: engineers set each satellite clock slightly slow. +2. In orbit: each satellite broadcasts its position and send time continuously. +3. The receiver records the receive time of signals from 4+ satellites. +4. The receiver computes pseudoranges and solves for x, y, z, b. + +## Quantities + +- Light speed c = 299,792,458 m/s ≈ 0.3 m per nanosecond ≈ 300 m per microsecond. +- Orbit radius ≈ 26,560 km (altitude ≈ 20,200 km); orbit speed ≈ 3.9 km/s. +- Accuracy target: a position error of about 5 m needs satellite clocks within about 17 ns of GPS time (5 m / 0.3 m per ns). 1 µs gives 300 m. +- Signal travel time ≈ 67 ms from directly overhead. +- Special relativity (speed): satellite clock loses ≈ 7 µs/day (v²/2c² ≈ 8.3 × 10⁻¹¹). +- General relativity (gravity): satellite clock gains ≈ 45 µs/day (GM(1/R_earth − 1/r)/c² ≈ 5.3 × 10⁻¹⁰). +- Net: gains ≈ 38 µs/day (fractional rate ≈ 4.465 × 10⁻¹⁰). 38 µs × c ≈ 11.4 km. +- Factory offset: 10.23 MHz × (1 − 4.465 × 10⁻¹⁰) = 10.22999999543 MHz. +- Receiver quartz clock: easily off by 1 ms → 300 km. + +## Alternative states + +| State | What changes | Why | +|---|---|---| +| Receiver clock perfect | 3 satellites suffice | no b to solve | +| Receiver clock off by 5 µs, 3D, only 3 satellites | position wrong by km, no way to detect | 4 unknowns, 3 equations | +| No relativity correction | ranges drift ≈ 11 km per day | clocks gain 38 µs/day | +| Only the speed effect corrected | clocks gain 45 µs/day → 13.5 km/day | the speed correction cancels −7 µs/day; the +45 µs/day gravity gain remains | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Solve for receiver clock bias b as a fourth unknown | Trust the receiver clock | Quartz clock off by 1 ms → every range off by 300 km. Even 1 µs → 300 m. | +| Use 4 satellites | Use 3 (three spheres meet at two points; the one far from Earth is rejected) | With b unknown, 3 equations, 4 unknowns: any b gives a different point. | +| Pre-set satellite clocks slow | Let them run at the nominal 10.23 MHz | They gain 38 µs/day; ranges drift 11.4 km per day. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here (numbers) | +|---|---|---| +| Inconsistent pseudoranges reveal the clock bias (1D worked example) | Beacons at 0 and 30 km, receiver at 12 km. True travel times 40 µs and 60 µs. Receiver clock 5 µs fast: measured 45 and 65 µs → pseudoranges 13.5 and 19.5 km. Sum 33 km ≠ 30 km → excess 3 km = 2b → b = 1.5 km = 5 µs. Subtract: 12 and 18 km → x = 12 km. (Assumes the receiver lies between the beacons, as satellites surround a receiver.) | With one beacon only: 13.5 km, no check → position off by 1.5 km, undetected. | +| Satellite clock error becomes range error | 1 ns → 0.3 m | 38 µs → 11.4 km | +| b absorbs a shared error only | all clocks 38 µs off → b absorbs 11.4 km, position right | clocks set on day 0 and day 3, read on day 5: 57 km vs 22.8 km, 34 km difference survives | +| The two relativity effects oppose | at GPS altitude: −7 + 45 = +38 µs/day | — | + +## Terms + +- **Range:** true distance satellite–receiver. +- **Pseudorange:** measured travel time × c, using the receiver clock; equals range + b. Never call it "distance". +- **Clock bias (b):** receiver clock error expressed in metres (error in seconds × c). 5 µs ↔ 1.5 km. Low b (1 ns) → 0.3 m; high (1 ms) → 300 km. +- **GPS time:** the reference time kept by ground stations; satellite clocks are steered to it. +- **Time dilation:** the rate difference between clocks. 38 µs/day is a fraction 4.5 × 10⁻¹⁰: tiny as a fraction, large as distance. + +## Epistemic status + +- **Established:** both relativistic effects are measured on GPS clocks and match theory; the clock offset has been built in since the first satellites. +- **Mathematical consequence:** 11.4 km/day = 38 µs/day × c. +- **Simplification:** the "11 km/day position error" is the growth of the range error if satellite clocks were left uncorrected against GPS time. b absorbs the part shared equally by all satellites, so the position error comes from how the clock errors differ between satellites; that difference grows by tens of km within days. The ground stations also upload small clock corrections. + +## Confusion points + +- "Relativity is about the satellite's speed, so the clock runs slow" → gravity effect is larger and opposite; net is fast. +- "GPS needs 3 satellites" → 3 is enough only with a perfect receiver clock; real receivers need 4. +- "The phone sends a signal to the satellite" → receiver only listens. +- "The receiver clock bias would absorb the relativity drift" → only the part shared equally by all satellites; clocks set on different days carry different errors. +- "38 µs is negligible" → light goes 11.4 km in 38 µs. + +## Scope + +- **Out of scope:** atmospheric delays (ionosphere, troposphere) — the largest remaining error sources; dual-frequency receivers remove most of the ionospheric part. +- **Out of scope:** Sagnac effect (Earth rotates during the 67 ms flight) — receivers correct it; see Ashby, "Relativity in the Global Positioning System", Living Reviews in Relativity (2003). +- **Out of scope:** the eccentricity correction (the receiver removes a periodic clock error up to about 23 ns, 7 m, from slightly elliptical orbits) and the forms of the time-dilation formulas — Ashby (2003). +- **Out of scope:** more than four satellites (a least-squares fit) and satellite geometry (bunched satellites amplify errors). +- **Out of scope:** why one-way ranging (unlimited users, receivers never transmit) and why atomic clocks on satellites. +- **Out of scope:** how the receiver solves the nonlinear equations (iterative least squares from a guess). +- **Out of scope:** satellite orbit data (ephemeris) and how the ground segment computes it. + +## Representation decision + +- **Stage:** 1 +- **Reason:** the subject is one sequential causal argument with worked numbers; there is no parameter space a reader must explore, and the 1D worked example fits in text. +- **Levels:** L1 answer / L2 positioning with pseudoranges / L3 why clocks must agree / L4 relativity numbers and the fix. diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/model.yaml b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/model.yaml new file mode 100644 index 0000000..a86253f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/model.yaml @@ -0,0 +1,117 @@ +# Machine-readable half of model.md. `explainer check gps-relativity` holds every rendering to it. + +values: + light: + m_per_ns: 0.3 + m_per_us: 300 + km_per_us: 0.3 + orbit: + altitude_km: 20200 + speed_km_s: 3.9 + travel_ms: 67 + satellites: 30 + accuracy: + position_m: 5 + clock_ns: 17 + quartz: + error_ms: 1 + error_km: 300 + example1d: + beacon_b_km: 30 + receiver_km: 12 + other_km: 18 + t_a_us: 40 + t_b_us: 60 + bias_us: 5 + meas_a_us: 45 + meas_b_us: 65 + pr_a_km: 13.5 + pr_b_km: 19.5 + sum_km: 33 + excess_km: 3 + bias_km: 1.5 + relativity: + speed_us_day: 7 + gravity_us_day: 45 + net_us_day: 38 + net_km_day: 11.4 + speed_only_km_day: 13.5 + fraction: 4.5e-10 + ecc_ns: 23 + nominal_mhz: 10.23 + offset_mhz: 10.22999999543 + reset: + day_a: 0 + day_b: 3 + day_read: 5 + err_a_us: 190 + err_a_km: 57 + err_b_us: 76 + err_b_km: 22.8 + diff_km: 34 + +allow: [1, 2, 3, 4, 10] + +require: + - path: relativity/net_us_day + in: [prose] + - path: relativity/net_km_day + in: [prose] + - path: relativity/speed_us_day + in: [prose] + - path: relativity/gravity_us_day + in: [prose] + - path: example1d/bias_km + in: [prose] + +claims: + - id: def-pseudorange + kind: definition + statement: "A pseudorange is light speed times (receive time on the receiver clock minus send time on the satellite clock): the range plus the receiver clock bias b." + - id: why-fourth-unknown + kind: why + statement: "The receiver treats its own clock error b as a fourth unknown, so it needs four satellites." + alternative: "Trust the receiver's quartz clock and use three satellites." + counterexample: "A quartz clock off by 1 ms puts every range off by 300 km; even 1 µs gives 300 m." + ask: "Why does a GPS receiver need signals from four satellites and not three?" + - id: mech-bias-1d + kind: mechanism + statement: "Pseudoranges that do not fit together reveal the clock bias." + example: "Beacons 30 km apart, receiver at 12 km, clock 5 µs fast: pseudoranges 13.5 and 19.5 km sum to 33 km, 3 km too much, so b = 1.5 km and x = 12 km." + - id: mech-clock-to-range + kind: mechanism + statement: "Every nanosecond of satellite clock error becomes 0.3 m of range error." + example: "5 m accuracy needs satellite clocks within about 17 ns; 38 µs gives 11.4 km." + - id: mech-two-effects + kind: mechanism + statement: "The orbit speed slows the satellite clock by 7 µs/day; the weaker gravity at altitude speeds it up by 45 µs/day; net, the clock gains 38 µs/day." + example: "−7 + 45 = +38 µs/day; 38 µs × 300 m/µs = 11.4 km per day." + ask: "Do GPS satellite clocks run fast or slow compared with clocks on the ground, and why?" + - id: guar-bias-absorbs-shared + kind: guarantee + statement: "b absorbs only an error shared equally by all satellites, so uncorrected relativistic drift still ruins positions." + example: "If every clock were off by exactly 38 µs, b would absorb 11.4 km and the position would be right." + counterexample: "Clocks set on day 0 and day 3, read on day 5: 57 km and 22.8 km off; the 34 km difference cannot go into one b." + ask: "The receiver already solves for a clock error. Why can that not soak up the relativity drift of the satellite clocks?" + - id: fix-rate-offset + kind: mechanism + statement: "Engineers set each satellite clock to tick slow before launch, so that in orbit it matches GPS time." + example: "10.23 MHz is set to 10.22999999543 MHz, a fraction 4.5 × 10⁻¹⁰ slower." + +terms: + - term: pseudorange + means: "measured travel time × c using the receiver clock; 'range' means only the true distance" + avoid: [measured distance, true pseudorange] + +budget: {prose: 1000, reason: "two mechanisms (positioning with clock bias, relativity) and one guarantee whose failing case needs its own numbers"} + +quiz: + - q: "A receiver's quartz clock is 2 µs fast. Roughly how wrong is each pseudorange, and does that make the final position wrong by that much?" + expect: "Each pseudorange is about 600 m too long, but the same error is in every pseudorange, so the receiver solves for it as b and the position comes out right." + misconception: "The position is 600 m wrong." + - q: "If engineers had corrected only for the satellites' speed and not for gravity, would the clocks run fast or slow, and by how much per day?" + expect: "Fast, by about 45 µs per day (about 13.5 km of range per day), because the gravity effect remains." + misconception: "Slow, because moving clocks run slow." + - q: "Why does a 38 µs per day clock drift matter so much for positioning?" + expect: "Signals travel at light speed, so 38 µs of clock error is about 11.4 km of range error, growing every day." + misconception: "38 µs is too small to matter." diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/narrative.md b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/narrative.md new file mode 100644 index 0000000..eb5db07 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/narrative.md @@ -0,0 +1,52 @@ +# Narrative: gps-relativity + +## 1. Reader before and after + +- **Before:** a technical reader: school mathematics and science (distance = speed × time, light speed ≈ 300,000 km/s, gravity, simultaneous equations), basic programming. New to GPS and to relativity. +- **After:** "A GPS receiver times signals from four satellites, turns each time into a distance, and solves for its position and its own clock error. This works only if the satellite clocks agree to nanoseconds, and relativity makes them gain 38 µs a day — 11 km of range — so they are set to tick slow before launch." + +## 2. Question and motive + +- **Question:** How does GPS find your position, and why does it need relativity? Result in words: the receiver measures how long signals take to arrive from four satellites; relativity makes the satellite clocks run fast by 38 µs a day, which would add about 11 km of range error per day, so the clocks are built to run slow. +- **Why care:** a phone with a cheap clock finds itself to a few metres, and the everyday device depends on Einstein's theory. +- **Why this approach:** distance = speed × time; the signal speed is known exactly (light speed), so measure time. + +## 3. Introduction ledger + +| Reference | Means | Grounded by | Beat | +|---|---|---|---| +| range | true distance from a satellite to the receiver | 67 ms of travel × light speed ≈ 20,200 km | 2 | +| pseudorange | travel time measured with the receiver clock × light speed; range + b | 1D example: 13.5 km where the range is 12 km | 4 | +| clock bias b | receiver clock error, in metres | 5 µs → 1.5 km | 4 | +| GPS time | reference time from atomic clocks at ground stations | satellites steered to it | 5 | +| time dilation | clocks in different motion or height tick at different rates | −7 and +45 µs/day | 6 | +| rate offset | satellite clock set to tick slow before launch | 10.23 → 10.22999999543 MHz | 8 | +| µs, ns | microsecond (10⁻⁶ s), nanosecond (10⁻⁹ s) | 1 µs ↔ 300 m, 1 ns ↔ 0.3 m | 2 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How does GPS work, and where does relativity come in? | — | answer paragraph (the result of §2 in words) | "The receiver times signals from four satellites and solves for its position and its own clock error. Relativity makes satellite clocks gain 38 µs a day, about 11 km of range, so they are built to tick slow." | the whole answer in outline | +| 2 | mechanism | How can a satellite tell you how far away it is? | so | the signal carries the satellite's position and its send time (the time stamp) | "Travel time = receive time − send time. Range = travel time × light speed c. From overhead, 67 ms × c ≈ 20,200 km. Light goes 300 m in 1 µs and 0.3 m in 1 ns, so a timing error becomes a range error at that rate." | timing gives range; timing must be precise | +| 3 | mechanism | How many ranges fix a point? | therefore | spheres | "One range puts the receiver somewhere on a sphere around the satellite (every point at that distance). Three spheres meet in two points; one is far from Earth and is rejected." | 3 satellites if all clocks were perfect | +| 4 | objection + instance | But my phone's clock is quartz, not atomic. | but | 1D worked example: beacons 0 and 30 km, receiver at 12 km, clock 5 µs fast; 13.5 + 19.5 = 33 km ≠ 30 km | "The receiver measures travel time with its own clock, so each result is a pseudorange: range + b, where b is the receiver clock error in metres (5 µs → 1.5 km). One clock makes every measurement, so b is the same in all of them. The excess 3 km = 2b gives b = 1.5 km. In 3D, b is a fourth unknown next to x, y, z, so the receiver needs a fourth satellite: four equations, four unknowns." | pseudorange, b, why four satellites | +| 5 | mechanism | So what accuracy do the satellite clocks need? | but | 17 ns ↔ 5 m | "The receiver can solve for its own clock error, because it is shared. Each satellite clock error is different and goes into only one pseudorange, so the receiver cannot solve for it. Satellite clocks must match GPS time — the time of atomic clocks at ground stations. For a 5 m fix (a typical phone), the error must be under about 17 ns." | satellite clocks must agree to ~10 ns | +| 6 | mechanism | What does relativity do to those clocks? | so | −7, +45, +38 µs/day vs ground clocks | "Time dilation: clocks in different motion or height tick at different rates (measured, not only predicted). The speed of 3.9 km/s makes a satellite clock lose 7 µs a day. The weaker gravity 20,200 km up makes it gain 45 µs a day. Net: it gains 38 µs a day, which is 11.4 km of range per day." | net fast 38 µs/day = 11.4 km/day | +| 7 | objection | Doesn't b absorb it, if every satellite drifts the same? | but | satellite A set to GPS time on day 0, B on day 3, read on day 5: 190 µs (57 km) vs 76 µs (22.8 km) | "b absorbs only an error that is equal in every pseudorange. The drift rate is equal, but each clock was set on a different day, so the accumulated errors differ: 34 km that no single b can absorb." | why the drift still ruins fixes | +| 8 | close | So what does GPS do? | therefore | 10.23 MHz → 10.22999999543 MHz | "Each satellite clock counts ticks of a 10.23 MHz signal. Before launch, engineers set it 4.5 × 10⁻¹⁰ slower — exactly the 38 µs/day gain as a fraction — so in orbit it ticks at the ground rate. Ground stations upload small remaining corrections." | the fix | +| 9 | recap | — | — | — | one-sentence answer; Scope list: atmosphere, Earth's rotation during flight, orbit-shape correction, more than four satellites | after-sentence | + +## 5. Concrete to symbol + +Pseudorange = range + b shown in the 1D example (13.5 = 12 + 1.5) before the general equation. + +## 6. Links between representations + +Times and distances always paired: µs with km via 0.3 km per µs. + +## 7. Close the loop + +- **Answer:** four time measurements give position and receiver clock error; relativity's 38 µs/day would add 11 km/day of range error, so satellite clocks tick slow by design. +- **Payoff:** the reader can turn any timing error into a range error, and predict which effect wins. +- **Scope:** atmosphere, Sagnac, eccentricity correction, least squares with more satellites. diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/decisions.yaml new file mode 100644 index 0000000..3c9df8d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/decisions.yaml @@ -0,0 +1,185 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T145628-probe#1: + decision: adopted +2026-10-04T145628-probe#10: + decision: declined +2026-10-04T145628-probe#11: + decision: declined +2026-10-04T145628-probe#12: + decision: declined +2026-10-04T145628-probe#13: + decision: adopted +2026-10-04T145628-probe#14: + decision: adopted +2026-10-04T145628-probe#15: + decision: adopted +2026-10-04T145628-probe#16: + decision: adopted +2026-10-04T145628-probe#2: + decision: adopted +2026-10-04T145628-probe#3: + decision: adopted +2026-10-04T145628-probe#4: + decision: adopted +2026-10-04T145628-probe#5: + decision: declined +2026-10-04T145628-probe#6: + decision: adopted +2026-10-04T145628-probe#7: + decision: adopted +2026-10-04T145628-probe#8: + decision: adopted +2026-10-04T145628-probe#9: + decision: adopted +2026-10-04T145834-coldread-narrative#1: + decision: adopted +2026-10-04T145834-coldread-narrative#10: + decision: adopted +2026-10-04T145834-coldread-narrative#11: + decision: adopted +2026-10-04T145834-coldread-narrative#12: + decision: adopted +2026-10-04T145834-coldread-narrative#13: + decision: adopted +2026-10-04T145834-coldread-narrative#14: + decision: adopted +2026-10-04T145834-coldread-narrative#15: + decision: adopted +2026-10-04T145834-coldread-narrative#16: + decision: adopted +2026-10-04T145834-coldread-narrative#17: + decision: adopted +2026-10-04T145834-coldread-narrative#18: + decision: adopted +2026-10-04T145834-coldread-narrative#19: + decision: adopted +2026-10-04T145834-coldread-narrative#2: + decision: adopted +2026-10-04T145834-coldread-narrative#20: + decision: adopted +2026-10-04T145834-coldread-narrative#21: + decision: adopted +2026-10-04T145834-coldread-narrative#22: + decision: adopted +2026-10-04T145834-coldread-narrative#23: + decision: adopted +2026-10-04T145834-coldread-narrative#24: + decision: adopted +2026-10-04T145834-coldread-narrative#25: + decision: adopted +2026-10-04T145834-coldread-narrative#26: + decision: adopted +2026-10-04T145834-coldread-narrative#27: + decision: adopted +2026-10-04T145834-coldread-narrative#28: + decision: declined +2026-10-04T145834-coldread-narrative#29: + decision: adopted +2026-10-04T145834-coldread-narrative#3: + decision: adopted +2026-10-04T145834-coldread-narrative#30: + decision: adopted +2026-10-04T145834-coldread-narrative#31: + decision: adopted +2026-10-04T145834-coldread-narrative#32: + decision: adopted +2026-10-04T145834-coldread-narrative#33: + decision: declined +2026-10-04T145834-coldread-narrative#34: + decision: adopted +2026-10-04T145834-coldread-narrative#35: + decision: adopted +2026-10-04T145834-coldread-narrative#36: + decision: adopted +2026-10-04T145834-coldread-narrative#37: + decision: adopted +2026-10-04T145834-coldread-narrative#4: + decision: adopted +2026-10-04T145834-coldread-narrative#5: + decision: declined +2026-10-04T145834-coldread-narrative#6: + decision: adopted +2026-10-04T145834-coldread-narrative#7: + decision: adopted +2026-10-04T145834-coldread-narrative#8: + decision: adopted +2026-10-04T145834-coldread-narrative#9: + decision: adopted +2026-10-04T150100-coldread-prose#1: + decision: adopted +2026-10-04T150100-coldread-prose#10: + decision: declined +2026-10-04T150100-coldread-prose#11: + decision: adopted +2026-10-04T150100-coldread-prose#12: + decision: adopted +2026-10-04T150100-coldread-prose#13: + decision: adopted +2026-10-04T150100-coldread-prose#14: + decision: adopted +2026-10-04T150100-coldread-prose#15: + decision: declined +2026-10-04T150100-coldread-prose#16: + decision: adopted +2026-10-04T150100-coldread-prose#2: + decision: adopted +2026-10-04T150100-coldread-prose#3: + decision: declined +2026-10-04T150100-coldread-prose#4: + decision: declined +2026-10-04T150100-coldread-prose#5: + decision: adopted +2026-10-04T150100-coldread-prose#6: + decision: adopted +2026-10-04T150100-coldread-prose#7: + decision: declined +2026-10-04T150100-coldread-prose#8: + decision: declined +2026-10-04T150100-coldread-prose#9: + decision: adopted +2026-10-04T150300-quiz-prose#1: + decision: declined +2026-10-04T150300-quiz-prose#10: + decision: declined +2026-10-04T150300-quiz-prose#11: + decision: declined +2026-10-04T150300-quiz-prose#12: + decision: declined +2026-10-04T150300-quiz-prose#13: + decision: declined +2026-10-04T150300-quiz-prose#14: + decision: declined +2026-10-04T150300-quiz-prose#15: + decision: declined +2026-10-04T150300-quiz-prose#16: + decision: declined +2026-10-04T150300-quiz-prose#17: + decision: declined +2026-10-04T150300-quiz-prose#18: + decision: declined +2026-10-04T150300-quiz-prose#19: + decision: adopted +2026-10-04T150300-quiz-prose#2: + decision: declined +2026-10-04T150300-quiz-prose#20: + decision: declined +2026-10-04T150300-quiz-prose#21: + decision: declined +2026-10-04T150300-quiz-prose#22: + decision: declined +2026-10-04T150300-quiz-prose#23: + decision: declined +2026-10-04T150300-quiz-prose#3: + decision: adopted +2026-10-04T150300-quiz-prose#4: + decision: adopted +2026-10-04T150300-quiz-prose#5: + decision: declined +2026-10-04T150300-quiz-prose#6: + decision: adopted +2026-10-04T150300-quiz-prose#7: + decision: declined +2026-10-04T150300-quiz-prose#8: + decision: declined +2026-10-04T150300-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145628-probe.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145628-probe.json new file mode 100644 index 0000000..00da2a5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145628-probe.json @@ -0,0 +1,231 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "gps-relativity", + "at": "2026-10-04T14:56:28", + "model": "default", + "prompt_sha": "4e9abe4986eb", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "Why an uncorrected 38 µs/day drift would actually ruin positions, given that b absorbs any error shared by all satellites", + "why_it_matters": "The central 'why must it correct' answer depends on this. The file grants that b absorbs common error, then says the position error 'would still grow without limit' without saying why. All GPS satellites share roughly the same orbit radius, so the drift rate is the same for each, and a sharp reader will conclude that b cancels it. The step that makes per-satellite errors differ is missing.", + "suggested_addition": "Add a concrete instance where the drift is NOT common. Satellite clocks are synchronized and adjusted at different times. If satellite A was last set to GPS time on day 0 and satellite B on day 3, then on day 5 A is off by 5×38 = 190 µs (57 km) and B by 2×38 = 76 µs (22.8 km). The 34 km difference cannot be absorbed by one b. Also give the instance where it holds: if all clocks were off by exactly 38 µs, b would absorb 11.4 km, the position would come out right, and the time the receiver reports would be wrong by 38 µs." + }, + { + "rule": 4, + "severity": "main", + "about": "Dependency step 2 ('several ranges fix a point / intersection of spheres') has no worked example", + "why_it_matters": "The only worked example is 1D. The reader never sees ranges intersect in 2D or 3D, or why the count of unknowns sets the number of satellites.", + "suggested_addition": "2D example: beacons at (0,0) and (10,0) km with true ranges 5 and √65 ≈ 8.06 km. The two circles meet at (3,4) and (3,−4): two points. A third beacon at (0,10) with range √45 ≈ 6.71 km picks (3,4). State the 3D analogue: three spheres meet at two points, and one of them is usually far from Earth's surface and gets rejected." + }, + { + "rule": 2, + "severity": "main", + "about": "The claim 'four pseudoranges determine x, y, z, b' has no numeric instance and no breaking instance", + "why_it_matters": "This is the core positioning guarantee. Its assumptions are never stated: the geometry must not be degenerate, and the solution must be unique. The text also says 'three spheres meet at a point', but they meet at two.", + "suggested_addition": "Breaking cases: four satellites lying in one plane, or bunched close together, give a nearly singular system. For example, a timing error of 3 m that turns into 30 m of position error (geometry amplification ×10). Also fix 'three spheres meet at a point' to 'at two points'." + }, + { + "rule": 2, + "severity": "edge", + "about": "The 1D worked example silently assumes the receiver lies between the two beacons", + "why_it_matters": "If the receiver is outside the segment, e.g. at x = 40 km, the pseudoranges are 40+b and 10+b. Their difference is always 30 km, whatever x is, so x and b can't be separated. The claim 'inconsistent pseudoranges reveal the bias' breaks under an assumption the file never states.", + "suggested_addition": "State the assumption, and give the breaking case: receiver at 40 km with a 5 µs fast clock gives 41.5 and 11.5 km. Every (x, b) with x − b = 38.5 fits, so the position is undetermined. This links to why satellite geometry matters." + }, + { + "rule": 1, + "severity": "main", + "about": "Why the time-dilation formulas have these forms: v²/2c² and GM(1/R_earth − 1/r)/c²", + "why_it_matters": "The audience is new to relativity. Without the why, the formulas are unexplained magic. The natural school-physics alternative for the gravity term is g·h/c², and that gives the wrong answer.", + "suggested_addition": "Say that v²/2c² is the small-speed approximation of 1 − √(1 − v²/c²). At 3.87 km/s the two agree to about 10⁻²⁰. The natural alternative g·h/c² = 9.8 × 2.02×10⁷ / 8.99×10¹⁶ ≈ 2.2×10⁻⁹ (≈190 µs/day) overestimates about 4× because gravity weakens with height. The 1/r form accounts for that. Note too that the reference is a clock on the ground at R_earth, which is why R_earth appears." + }, + { + "rule": 3, + "severity": "main", + "about": "'Time dilation' is used with no intuition for why motion slows a clock and weaker gravity speeds it up", + "why_it_matters": "The audience 'does not know what time dilation is'. The file gives numbers but no reason for the signs. That is the very confusion point it lists, and readers may take the result on faith.", + "suggested_addition": "Give one sentence per effect. A moving clock ticks slower as seen from the ground, by a fraction of about v²/2c² (8.3×10⁻¹¹ at 3.9 km/s). A clock higher in a gravity well ticks faster than one lower down, by a fraction of about the potential difference/c² (5.3×10⁻¹⁰). Add one sentence noting that both effects have been confirmed independently, e.g. Hafele–Keating (1971) and Pound–Rebka (1959)." + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts are given two names, and some terms are used before they are defined", + "why_it_matters": "Readers may think these are different things.", + "suggested_addition": "Pick one name each: 'orbit-shape correction' vs 'eccentricity correction'; 'factory offset' vs 'pre-set slow' vs 'clock offset built in'; 'ground time' vs 'GPS time' (GPS time is used in the Clock bias entity before Terms defines it, and it is not in Entities); 'range error' vs 'position error' vs 'ranges drift' for the 11 km figure; 'time of flight' vs 'signal travel time'; 'time dilation' as both the phenomenon (Entities) and the rate difference (Terms). Also define '10.23 MHz' as the satellite clock's nominal tick frequency, from which the signal is derived." + }, + { + "rule": 5, + "severity": "edge", + "about": "'Nanoseconds' clock agreement (dependency 4) is not tied to an accuracy target", + "why_it_matters": "The reader can't tell why nanoseconds matter rather than microseconds.", + "suggested_addition": "State that a typical position error of about 5 m needs satellite clock errors under about 17 ns (5 m / 0.3 m per ns). Contrast with 1 µs, which gives 300 m: useless for driving directions." + }, + { + "rule": 2, + "severity": "edge", + "about": "The 'Only special relativity applied' row explains itself as 'wrong sign of the bigger effect', which is confusing", + "why_it_matters": "The reasoning step is implicit. If the SR correction makes the clock run faster to cancel −7 µs/day, the remaining error is +45 µs/day because GR goes uncorrected. 'Wrong sign' does not explain that.", + "suggested_addition": "Rewrite as: the SR correction speeds the clock by 7 µs/day, which cancels the −7. The +45 µs/day GR gain remains uncorrected, which gives 13.5 km/day of range drift." + }, + { + "rule": 1, + "severity": "edge", + "about": "Why one-way listening ranging rather than two-way (radar-like) ranging", + "why_it_matters": "Two-way ranging is the natural alternative, and it would remove the receiver clock bias altogether. The file says the receiver only listens but never says why that design was chosen.", + "suggested_addition": "With two-way ranging, every user transmits, so the number of users is limited and receivers become costly and detectable. One-way ranging serves unlimited users, and the price is the unknown b plus a fourth satellite." + }, + { + "rule": 1, + "severity": "edge", + "about": "Why atomic clocks on the satellites rather than quartz", + "why_it_matters": "The receiver gets away with quartz because b is solved for; the reader may ask why satellites can't do the same.", + "suggested_addition": "A good quartz clock drifts about 10⁻⁹ per day, roughly 86 µs/day, or 26 km. Satellite clocks can't be solved for by the user, so they need atomic stability of about 10⁻¹³, roughly 9 ns/day, or 2.6 m." + }, + { + "rule": 6, + "severity": "edge", + "about": "What happens with more than four satellites", + "why_it_matters": "Real receivers track 8–12 satellites. Having 5 or more equations for 4 unknowns turns the problem into a best fit and gives an error check. The file never mentions this.", + "suggested_addition": "With 8 satellites there are 8 equations and 4 unknowns. A least-squares fit averages down noise, and the leftover mismatch (e.g. residuals of a few metres) shows how much to trust the fix." + } + ], + "excess": [ + "Orbit period ≈ 12 h: not used in any argument", + "Eccentricity correction (23 ns ≈ 7 m): a secondary detail, not needed to answer why relativity must be corrected", + "'GPS time itself must stay right for the system's timing users': a side point unless it is used to justify the correction", + "ISS low-orbit counterexample: useful for rule 2 but marked as not rendered" + ], + "next_questions": [ + { + "q": "If every satellite clock gains the same 38 µs/day, why doesn't the receiver's b just absorb it?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why does moving fast slow a clock, and why does weaker gravity speed it up?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How accurate is GPS in practice, and what limits it once relativity is corrected?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "How does the receiver know where each satellite is at the moment it sent its signal?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "What does the receiver do with more than four satellites, and how does it actually solve the nonlinear equations?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.18053100000000002, + "seconds": 70 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "Why an uncorrected 38 µs/day drift would actually ruin positions, given that b absorbs any error shared by all satellites → Add a concrete instance where the drift is NOT common. Satellite clocks are synchronized and adjusted at different times. If satellite A was last set to GPS time on day 0 and satellite B on day 3, then on day 5 A is off by 5×38 = 190 µs (57 km) and B by 2×38 = 76 µs (22.8 km). The 34 km difference cannot be absorbed by one b. Also give the instance where it holds: if all clocks were off by exactly 38 µs, b would absorb 11.4 km, the position would come out right, and the time the receiver reports would be wrong by 38 µs." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 4", + "text": "Dependency step 2 ('several ranges fix a point / intersection of spheres') has no worked example → 2D example: beacons at (0,0) and (10,0) km with true ranges 5 and √65 ≈ 8.06 km. The two circles meet at (3,4) and (3,−4): two points. A third beacon at (0,10) with range √45 ≈ 6.71 km picks (3,4). State the 3D analogue: three spheres meet at two points, and one of them is usually far from Earth's surface and gets rejected." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 2", + "text": "The claim 'four pseudoranges determine x, y, z, b' has no numeric instance and no breaking instance → Breaking cases: four satellites lying in one plane, or bunched close together, give a nearly singular system. For example, a timing error of 3 m that turns into 30 m of position error (geometry amplification ×10). Also fix 'three spheres meet at a point' to 'at two points'." + }, + { + "n": 4, + "severity": "edge", + "kind": "rule 2", + "text": "The 1D worked example silently assumes the receiver lies between the two beacons → State the assumption, and give the breaking case: receiver at 40 km with a 5 µs fast clock gives 41.5 and 11.5 km. Every (x, b) with x − b = 38.5 fits, so the position is undetermined. This links to why satellite geometry matters." + }, + { + "n": 5, + "severity": "main", + "kind": "rule 1", + "text": "Why the time-dilation formulas have these forms: v²/2c² and GM(1/R_earth − 1/r)/c² → Say that v²/2c² is the small-speed approximation of 1 − √(1 − v²/c²). At 3.87 km/s the two agree to about 10⁻²⁰. The natural alternative g·h/c² = 9.8 × 2.02×10⁷ / 8.99×10¹⁶ ≈ 2.2×10⁻⁹ (≈190 µs/day) overestimates about 4× because gravity weakens with height. The 1/r form accounts for that. Note too that the reference is a clock on the ground at R_earth, which is why R_earth appears." + }, + { + "n": 6, + "severity": "main", + "kind": "rule 3", + "text": "'Time dilation' is used with no intuition for why motion slows a clock and weaker gravity speeds it up → Give one sentence per effect. A moving clock ticks slower as seen from the ground, by a fraction of about v²/2c² (8.3×10⁻¹¹ at 3.9 km/s). A clock higher in a gravity well ticks faster than one lower down, by a fraction of about the potential difference/c² (5.3×10⁻¹⁰). Add one sentence noting that both effects have been confirmed independently, e.g. Hafele–Keating (1971) and Pound–Rebka (1959)." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts are given two names, and some terms are used before they are defined → Pick one name each: 'orbit-shape correction' vs 'eccentricity correction'; 'factory offset' vs 'pre-set slow' vs 'clock offset built in'; 'ground time' vs 'GPS time' (GPS time is used in the Clock bias entity before Terms defines it, and it is not in Entities); 'range error' vs 'position error' vs 'ranges drift' for the 11 km figure; 'time of flight' vs 'signal travel time'; 'time dilation' as both the phenomenon (Entities) and the rate difference (Terms). Also define '10.23 MHz' as the satellite clock's nominal tick frequency, from which the signal is derived." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 5", + "text": "'Nanoseconds' clock agreement (dependency 4) is not tied to an accuracy target → State that a typical position error of about 5 m needs satellite clock errors under about 17 ns (5 m / 0.3 m per ns). Contrast with 1 µs, which gives 300 m: useless for driving directions." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 2", + "text": "The 'Only special relativity applied' row explains itself as 'wrong sign of the bigger effect', which is confusing → Rewrite as: the SR correction speeds the clock by 7 µs/day, which cancels the −7. The +45 µs/day GR gain remains uncorrected, which gives 13.5 km/day of range drift." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 1", + "text": "Why one-way listening ranging rather than two-way (radar-like) ranging → With two-way ranging, every user transmits, so the number of users is limited and receivers become costly and detectable. One-way ranging serves unlimited users, and the price is the unknown b plus a fourth satellite." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 1", + "text": "Why atomic clocks on the satellites rather than quartz → A good quartz clock drifts about 10⁻⁹ per day, roughly 86 µs/day, or 26 km. Satellite clocks can't be solved for by the user, so they need atomic stability of about 10⁻¹³, roughly 9 ns/day, or 2.6 m." + }, + { + "n": 12, + "severity": "edge", + "kind": "rule 6", + "text": "What happens with more than four satellites → With 8 satellites there are 8 equations and 4 unknowns. A least-squares fit averages down noise, and the leftover mismatch (e.g. residuals of a few metres) shows how much to trust the fix." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Orbit period ≈ 12 h: not used in any argument" + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Eccentricity correction (23 ns ≈ 7 m): a secondary detail, not needed to answer why relativity must be corrected" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "'GPS time itself must stay right for the system's timing users': a side point unless it is used to justify the correction" + }, + { + "n": 16, + "severity": "excess", + "kind": "model entry", + "text": "ISS low-orbit counterexample: useful for rule 2 but marked as not rendered" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145834-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145834-coldread-narrative.json new file mode 100644 index 0000000..c02ef3a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T145834-coldread-narrative.json @@ -0,0 +1,401 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "gps-relativity", + "at": "2026-10-04T14:58:34", + "model": "default", + "prompt_sha": "c53fdd66ab92", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: answer paragraph. Said: the result in two sentences.", + "unresolved": [], + "unsaid": [ + "The table never gives the 'answer paragraph' or the 'two sentences', so the reader can't check what the result actually is." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (mechanism)", + "text": "Shown: signal content; 67 ms; 1 µs ↔ 300 m. Said: time stamp × c = range.", + "unresolved": [ + "time stamp", + "c" + ], + "unsaid": [ + "'signal content': the words never say what the signal carries.", + "'67 ms': no one says this is the travel time, or that it comes to about 20,200 km.", + "'1 µs ↔ 300 m': the words never say what this conversion is for." + ], + "leap": [ + "'time stamp × c = range': a time stamp is a clock reading, not a travel time. The step 'travel time = arrival time − time stamp' is never said, so the formula as written doesn't follow." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Shown: spheres. Said: 3 spheres meet in 2 points; one is far from Earth.", + "unresolved": [], + "unsaid": [], + "leap": [ + "Nobody says why a single range is a sphere (every point at that distance from the satellite)." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (objection + instance)", + "text": "Shown: 1D worked example. Said: bias b is the same for all; extra measurement reveals it.", + "unresolved": [ + "bias b", + "pseudorange (listed as something the reader now knows, but it is never said or shown)" + ], + "unsaid": [ + "The numbers in the 1D example (13.5 = 12 + 1.5) and the step 5 µs → 1.5 km never appear in Shown or Said.", + "'four satellites' is never said; the reader has to work out 3 + 1 alone." + ], + "leap": [ + "'the same for all': the reason (one receiver clock is used for every measurement) is not given.", + "'extra measurement reveals it': it isn't said that this works by adding a fourth unknown and a fourth equation (simultaneous equations)." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: 17 ns ↔ 5 m. Said: satellite clock errors are not solved; they must match GPS time.", + "unresolved": [ + "GPS time", + "ns" + ], + "unsaid": [ + "Nothing says why 5 m is the target, or where the ~10 ns requirement comes from." + ], + "leap": [ + "'satellite clock errors are not solved': the reader has just learned that a clock error can be solved for, and the reason the satellites' errors can't be (each satellite has its own error, not a shared one) only comes in beat 7." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Shown: −7, +45, +38 µs/day. Said: two effects, opposite signs.", + "unresolved": [ + "which 'two effects' (speed and gravity are never named)", + "time dilation (never said)" + ], + "unsaid": [ + "Nothing says which number belongs to which cause, or what the signs mean (slow or fast compared with ground clocks).", + "The 11.4 km/day figure (listed as something the reader now knows) is neither shown nor said." + ], + "leap": [ + "There is no reason at all for why a moving clock runs slow or a higher clock runs fast. Relativity, the topic named in the title, arrives as bare numbers." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (objection)", + "text": "Shown: day 0/3/5 example. Said: only shared error is absorbed.", + "unresolved": [], + "unsaid": [ + "The day 0/3/5 example isn't described, so the reader can't tell what it shows." + ], + "leap": [ + "If every satellite gains the same 38 µs/day, that looks like a shared error that b should absorb. The beat doesn't say why the drift is not shared." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (close)", + "text": "Shown: 10.22999999543 MHz. Said: rate offset; ground corrections.", + "unresolved": [ + "MHz", + "rate offset", + "ground corrections", + "what 10.23 MHz is (the clock's nominal frequency is never given)" + ], + "unsaid": [ + "The shown frequency is never linked to 38 µs/day. The fractional offset (about 4.5 × 10⁻¹⁰) is not explained." + ], + "leap": [ + "The table doesn't say how a slower tick rate cancels the relativistic gain." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (recap)", + "text": "Said: answer in the opening words; scope.", + "unresolved": [ + "'scope': the items left out are not listed in the beat" + ], + "unsaid": [], + "leap": [], + "excess": [ + "'answer in the opening words': this repeats beat 1 and gives no sign of saying anything new." + ], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 2: 'time stamp × c = range'. Travel time is never defined as arrival time minus the stamp, and 'c' is not introduced.", + "Beat 4: 'bias b is the same for all; extra measurement reveals it'. The bias b and 'pseudorange' are never defined, and the move to four satellites and four unknowns is not said.", + "Beat 6: 'two effects, opposite signs'. The effects are not named or explained. The relativity step, which is the core of the explanation, is only numbers.", + "Beat 7: 'only shared error is absorbed'. It doesn't say why the satellites' relativistic drift counts as unshared." + ], + "cuts": [ + "Beat 9: 'answer in the opening words' repeats beat 1. Merge it into beat 8's close or cut it to one sentence." + ], + "question": { + "text": "How does a GPS receiver find its position from satellite signals, and why does relativity have to be corrected for?", + "known_at": "Beat 1 (reader's question: 'How does GPS work, and where does relativity come in?')", + "result_stated": "Only as a reference. Beat 1 says 'the result in two sentences', but the beats never show the sentences themselves (they appear only in the author's plan, section 2)." + }, + "motive": "Missing from the beats. The plan's reasons to care ('a phone with a cheap clock finds itself to a few metres', and that an everyday device depends on Einstein's theory) never appear in Said or Shown. The reason for the approach is only implied in beat 2 ('time stamp × c = range'); the line 'signal speed is known exactly, so measure time' appears only in the plan.", + "close": "Partial. Beat 8 says 'rate offset; ground corrections' and beat 9 says 'answer in the opening words'. Neither states the general argument in words: four timings give position plus clock error, a 38 µs/day drift means about 11 km/day, so the clocks are set slow.", + "recap": "GPS finds you by timing signals from four satellites (the fourth one cancels your phone's clock error), and because relativity makes satellite clocks gain about 38 µs a day, they are built to tick slightly slow.", + "strongest": "Beat 4: answering 'my phone clock is not atomic' with a worked 1D example that shows the extra unknown being solved for.", + "weakest": "Beat 6: relativity, the topic named in the title, arrives as '−7, +45, +38 µs/day' and 'two effects, opposite signs', with no names and no reasons.", + "suggestions": [ + "In beat 2, say 'travel time = arrival time − time stamp' before multiplying by c, and define c as light speed.", + "In beat 6, name the effects: speed (special relativity, −7 µs/day) and weaker gravity at altitude (general relativity, +45 µs/day). Say the 11.4 km/day figure aloud.", + "In beat 7, explain why b can't absorb the drift. The satellite clocks drift away from GPS time, so the time stamps and the satellite positions computed from them become inconsistent, and the satellites' clock states differ.", + "In beat 8, link 10.23 → 10.22999999543 MHz to a fractional change of about 4.5 × 10⁻¹⁰, which is 38 µs per 86,400 s.", + "Put the 'why care' line (cheap phone clock, metre-level accuracy) into beat 1." + ] + }, + "usage": { + "cost": 0.11163419999999998, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'time stamp × c = range'. Travel time is never defined as arrival time minus the stamp, and 'c' is not introduced." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'bias b is the same for all; extra measurement reveals it'. The bias b and 'pseudorange' are never defined, and the move to four satellites and four unknowns is not said." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 6: 'two effects, opposite signs'. The effects are not named or explained. The relativity step, which is the core of the explanation, is only numbers." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 7: 'only shared error is absorbed'. It doesn't say why the satellites' relativistic drift counts as unshared." + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 9: 'answer in the opening words' repeats beat 1. Merge it into beat 8's close or cut it to one sentence." + }, + { + "n": 6, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): The table never gives the 'answer paragraph' or the 'two sentences', so the reader can't check what the result actually is." + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): time stamp" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (mechanism): c" + }, + { + "n": 9, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): 'signal content': the words never say what the signal carries." + }, + { + "n": 10, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): '67 ms': no one says this is the travel time, or that it comes to about 20,200 km." + }, + { + "n": 11, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (mechanism): '1 µs ↔ 300 m': the words never say what this conversion is for." + }, + { + "n": 12, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (mechanism): 'time stamp × c = range': a time stamp is a clock reading, not a travel time. The step 'travel time = arrival time − time stamp' is never said, so the formula as written doesn't follow." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): Nobody says why a single range is a sphere (every point at that distance from the satellite)." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (objection + instance): bias b" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (objection + instance): pseudorange (listed as something the reader now knows, but it is never said or shown)" + }, + { + "n": 16, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (objection + instance): The numbers in the 1D example (13.5 = 12 + 1.5) and the step 5 µs → 1.5 km never appear in Shown or Said." + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (objection + instance): 'four satellites' is never said; the reader has to work out 3 + 1 alone." + }, + { + "n": 18, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (objection + instance): 'the same for all': the reason (one receiver clock is used for every measurement) is not given." + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (objection + instance): 'extra measurement reveals it': it isn't said that this works by adding a fourth unknown and a fourth equation (simultaneous equations)." + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): GPS time" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): ns" + }, + { + "n": 22, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): Nothing says why 5 m is the target, or where the ~10 ns requirement comes from." + }, + { + "n": 23, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 'satellite clock errors are not solved': the reader has just learned that a clock error can be solved for, and the reason the satellites' errors can't be (each satellite has its own error, not a shared one) only comes in beat 7." + }, + { + "n": 24, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): which 'two effects' (speed and gravity are never named)" + }, + { + "n": 25, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): time dilation (never said)" + }, + { + "n": 26, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): Nothing says which number belongs to which cause, or what the signs mean (slow or fast compared with ground clocks)." + }, + { + "n": 27, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): The 11.4 km/day figure (listed as something the reader now knows) is neither shown nor said." + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): There is no reason at all for why a moving clock runs slow or a higher clock runs fast. Relativity, the topic named in the title, arrives as bare numbers." + }, + { + "n": 29, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (objection): The day 0/3/5 example isn't described, so the reader can't tell what it shows." + }, + { + "n": 30, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (objection): If every satellite gains the same 38 µs/day, that looks like a shared error that b should absorb. The beat doesn't say why the drift is not shared." + }, + { + "n": 31, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): MHz" + }, + { + "n": 32, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): rate offset" + }, + { + "n": 33, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): ground corrections" + }, + { + "n": 34, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (close): what 10.23 MHz is (the clock's nominal frequency is never given)" + }, + { + "n": 35, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (close): The shown frequency is never linked to 38 µs/day. The fractional offset (about 4.5 × 10⁻¹⁰) is not explained." + }, + { + "n": 36, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (close): The table doesn't say how a slower tick rate cancels the relativistic gain." + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (recap): 'scope': the items left out are not listed in the beat" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150100-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150100-coldread-prose.json new file mode 100644 index 0000000..9d33cd5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150100-coldread-prose.json @@ -0,0 +1,375 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "gps-relativity", + "at": "2026-10-04T15:01:00", + "model": "default", + "prompt_sha": "05a1f51134c7", + "reply": { + "paragraphs": [ + { + "at": "Title", + "text": "How does GPS find your position, and why does it need relativity?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Lead (bold paragraph)", + "text": "A GPS receiver times radio signals from four satellites and solves for its position and its own clock error. ... engineers build them to tick slow.", + "unresolved": [], + "leap": [ + "\"Relativity makes those clocks gain 38 µs a day\": the reader is told this here and has to wait until §4 for the reason. That is fine in a summary.", + "\"which would add 11.4 km of range error per day\": this reads as if the error is the clock drift itself. §5 later says a drift shared by all clocks would be absorbed, so this framing needs revising once the reader gets there." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 para 1", + "text": "About 30 GPS satellites orbit 20,200 km above Earth. Each one carries atomic clocks and broadcasts ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 para 2 + list", + "text": "The receiver notes when each signal arrives. Then: travel time = receive time − send time; range = travel time × c ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 para 3", + "text": "From a satellite directly overhead, the travel time is about 67 ms. Light covers 300 m in one microsecond ...", + "unresolved": [], + "leap": [], + "excess": [ + "\"(1 µs = 10⁻⁶ s)\": this definition comes after µs was already used in the lead, and this reader probably knows the unit anyway. It costs very little." + ], + "purpose_unclear": false + }, + { + "at": "§1 para 4", + "text": "One range puts you somewhere on a sphere around the satellite ... Three satellites would be enough, if every clock were perfect.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 para 1", + "text": "Your phone's clock is quartz, not atomic. It can easily be off by 1 ms, and 1 ms × c = 300 km.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 para 2", + "text": "So the receiver does not compute a range. It computes a pseudorange ... That makes b one more unknown to solve for, next to the position.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 worked example", + "text": "Worked example, on a line. Put beacon A at 0 km and beacon B at 30 km ... Those give pseudoranges of 13.5 km and 19.5 km.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 para 4", + "text": "The two pseudoranges add up to 33 km, but the beacons are only 30 km apart. The excess, 3 km, is b counted twice ...", + "unresolved": [], + "leap": [ + "\"The two pseudoranges add up to 33 km, but the beacons are only 30 km apart\": the shortcut of adding the ranges only works because the receiver lies between the beacons. The text does not say so, but it is a small point." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 para 5", + "text": "In three dimensions the unknowns are x, y, z, and b. Four unknowns need four equations ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 para 1", + "text": "The receiver can solve for its own clock error because that error is shared by all the pseudoranges. A satellite clock error is different ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 para 2", + "text": "How close? Each nanosecond of satellite clock error is 0.3 m of range error. For a fix within about 5 m ... about 17 ns.", + "unresolved": [], + "leap": [ + "\"For a fix within about 5 m\": the 5 m target is given without a reason. It works as an example." + ], + "excess": [ + "\"Each nanosecond of satellite clock error is 0.3 m of range error\": this repeats the conversion from §1. The repeat is short and useful here." + ], + "purpose_unclear": false + }, + { + "at": "§4 para 1", + "text": "Time dilation means that clocks in different motion, or at different heights in gravity, tick at different rates ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 table", + "text": "Speed: 3.9 km/s ... loses 7 µs per day | Height ... gains 45 µs per day | Net gains 38 µs per day", + "unresolved": [], + "leap": [ + "\"a clock higher above Earth ticks faster\": the text gives no reason, only a pointer to experiment. The 'Not covered' section later admits this, and the main line does not need the reason." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 para 2", + "text": "The gravity effect is larger, so the clock runs fast. A correction for speed alone would leave the full 45 µs a day, about 13.5 km of range per day.", + "unresolved": [], + "leap": [], + "excess": [ + "\"A correction for speed alone would leave the full 45 µs a day, about 13.5 km of range per day.\": nobody has suggested correcting for speed alone, so the reader stops to work out why this hypothetical is here. Nothing later uses it." + ], + "purpose_unclear": false + }, + { + "at": "§4 para 3", + "text": "38 µs is only 4.5 × 10⁻¹⁰ of a day, but light covers 11.4 km in 38 µs.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 para 1", + "text": "Every satellite clock drifts at the same rate. So does b not absorb the drift, as it absorbs the receiver clock error?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 para 2", + "text": "Only if every clock had the same error at the same moment. If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right.", + "unresolved": [], + "leap": [ + "\"b would absorb 11.4 km, and the position would come out right\": this contradicts the lead's \"would add 11.4 km of range error per day\". The reader has to go back and work out that the real damage comes from differences between clocks, not from the 11.4 km itself." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 para 3", + "text": "But each satellite clock is set to GPS time at a different moment. Suppose satellite A is set on day 0 and satellite B on day 3 ...", + "unresolved": [], + "leap": [ + "\"each satellite clock is set to GPS time at a different moment\": the obvious question is why not reset them often, or all at once. The text never answers it." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§6", + "text": "Each satellite clock counts the ticks of a signal at a nominal 10.23 MHz ... Ground stations upload small corrections for what remains.", + "unresolved": [ + "\"nominal\"" + ], + "leap": [ + "\"Ground stations upload small corrections for what remains\": if uploads can correct the clocks at all, why not upload corrections for the whole 38 µs a day? The text does not say why a rate offset built in before launch is better." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Summary", + "text": "Four pseudoranges give x, y, z, and the receiver's clock error, but only if satellite clocks stay within about 17 ns of GPS time; relativity's net 38 µs a day would break that by 11.4 km a day, so each clock is built to tick slow.", + "unresolved": [], + "leap": [ + "\"would break that by 11.4 km a day\": this repeats the framing that §5 qualified. The damage is the unshared part of the drift, not 11.4 km as such." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered: Atmosphere", + "text": "the ionosphere and troposphere delay the signal ...", + "unresolved": [ + "\"ionosphere\"", + "\"troposphere\"" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered: Sagnac / orbit-shape", + "text": "Earth's rotation during the 67 ms flight (Sagnac effect) and the orbit-shape correction ...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered: moving clock", + "text": "any introduction to special relativity (the \"light clock\" argument).", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered: solving", + "text": "iteration from a guess, with least squares when it hears more than four satellites.", + "unresolved": [ + "\"least squares\"" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Lead (\"would add 11.4 km of range error per day\") and Summary (\"would break that by 11.4 km a day\") vs §5 para 2 (\"If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right\"): the headline cost of relativity is stated in a way that the argument itself then says is not how the failure happens. A careful reader stops at §5 and has to rebuild the claim as 'the clocks drift apart by up to 11.4 km a day'." + ], + "cuts": [ + "§4 para 2: cut \"A correction for speed alone would leave the full 45 µs a day, about 13.5 km of range per day.\" It answers a question nobody asked.", + "§1 para 3: the \"(1 µs = 10⁻⁶ s)\" and \"(1 ns = 10⁻⁹ s)\" definitions are probably unneeded for this reader. They are very short, so cutting them saves little." + ], + "question": { + "text": "How does a GPS receiver work out its position from satellite signals, and why does that depend on correcting for relativity?", + "known_at": "Title, with the answer sketched in the bold lead.", + "result_stated": "Yes. The lead says: \"Relativity makes those clocks gain 38 µs a day, which would add 11.4 km of range error per day, so engineers build them to tick slow.\" The Summary restates it." + }, + "motive": "Reason to care: this is weak and only implied, through \"Your receiver (the GPS chip in a phone or car)\". It is an everyday device with a surprising dependence on relativity. Reason for the approach: present. §3 says \"A satellite clock error is different: it sits in only one pseudorange, so no unknown can absorb it\", and §5 explains why b cannot cancel the drift. Together these justify building in the rate offset. Missing: why a fixed slow rate is better than ground uploads alone.", + "close": "Yes. The Summary says: \"Four pseudoranges give x, y, z, and the receiver's clock error, but only if satellite clocks stay within about 17 ns of GPS time; relativity's net 38 µs a day would break that by 11.4 km a day, so each clock is built to tick slow.\" It answers the opening question in words, though the 11.4 km framing ignores the shared-versus-unshared point from §5.", + "recap": "GPS solves for your position plus your phone's clock error from four satellites, which only works if the satellite clocks agree almost perfectly, and since relativity makes orbiting clocks run 38 µs a day fast, they are built to tick slightly slow.", + "strongest": "§2 worked example on a line (the 12 km receiver with a clock 5 µs fast): it makes the clock-bias unknown b concrete and easy to check.", + "weakest": "§5 para 2 together with the Lead and Summary: they disagree about whether 11.4 km a day is the actual error. Second weakest is the unexplained \"Ground stations upload small corrections\" in §6.", + "suggestions": [ + "Rephrase the lead and summary as something like 'clocks set at different times would drift apart, reaching tens of km of disagreement within days', so they agree with §5.", + "Add one sentence to §6 on why a built-in rate offset is preferred over large, frequent uploaded corrections.", + "Drop the speed-only hypothetical in §4, or say why it matters.", + "Optionally note that adding the two pseudoranges in the 1-D example works because the receiver is between the beacons." + ] + }, + "usage": { + "cost": 0.15960499999999997, + "seconds": 58 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Lead (\"would add 11.4 km of range error per day\") and Summary (\"would break that by 11.4 km a day\") vs §5 para 2 (\"If every satellite clock were exactly 38 µs fast, b would absorb 11.4 km, and the position would come out right\"): the headline cost of relativity is stated in a way that the argument itself then says is not how the failure happens. A careful reader stops at §5 and has to rebuild the claim as 'the clocks drift apart by up to 11.4 km a day'." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "§4 para 2: cut \"A correction for speed alone would leave the full 45 µs a day, about 13.5 km of range per day.\" It answers a question nobody asked." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "§1 para 3: the \"(1 µs = 10⁻⁶ s)\" and \"(1 ns = 10⁻⁹ s)\" definitions are probably unneeded for this reader. They are very short, so cutting them saves little." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Lead (bold paragraph): \"Relativity makes those clocks gain 38 µs a day\": the reader is told this here and has to wait until §4 for the reason. That is fine in a summary." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Lead (bold paragraph): \"which would add 11.4 km of range error per day\": this reads as if the error is the clock drift itself. §5 later says a drift shared by all clocks would be absorbed, so this framing needs revising once the reader gets there." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "§2 para 4: \"The two pseudoranges add up to 33 km, but the beacons are only 30 km apart\": the shortcut of adding the ranges only works because the receiver lies between the beacons. The text does not say so, but it is a small point." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "§3 para 2: \"For a fix within about 5 m\": the 5 m target is given without a reason. It works as an example." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "§4 table: \"a clock higher above Earth ticks faster\": the text gives no reason, only a pointer to experiment. The 'Not covered' section later admits this, and the main line does not need the reason." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "§5 para 2: \"b would absorb 11.4 km, and the position would come out right\": this contradicts the lead's \"would add 11.4 km of range error per day\". The reader has to go back and work out that the real damage comes from differences between clocks, not from the 11.4 km itself." + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "§5 para 3: \"each satellite clock is set to GPS time at a different moment\": the obvious question is why not reset them often, or all at once. The text never answers it." + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "§6: \"nominal\"" + }, + { + "n": 12, + "severity": "edge", + "kind": "leap", + "text": "§6: \"Ground stations upload small corrections for what remains\": if uploads can correct the clocks at all, why not upload corrections for the whole 38 µs a day? The text does not say why a rate offset built in before launch is better." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Summary: \"would break that by 11.4 km a day\": this repeats the framing that §5 qualified. The damage is the unshared part of the drift, not 11.4 km as such." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: Atmosphere: \"ionosphere\"" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: Atmosphere: \"troposphere\"" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Not covered: solving: \"least squares\"" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150300-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150300-quiz-prose.json new file mode 100644 index 0000000..8ae8363 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/explainers/gps-relativity/review/runs/2026-10-04T150300-quiz-prose.json @@ -0,0 +1,237 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "gps-relativity", + "at": "2026-10-04T15:02:47", + "model": "default", + "prompt_sha": "8989a402f1c2", + "reply": { + "answers": { + "1": "The GPS satellites (about 30, 20,200 km up) and their atomic clocks; the receiver (for example a phone chip) and its quartz clock; the broadcast signal (satellite ID, position P, send time t); range and pseudorange; b, the receiver's clock error in metres; GPS time, kept by atomic clocks at ground stations; the two relativity effects (special relativity from speed, general relativity from gravity/height); and the 10.23 MHz clock frequency offset set before launch.", + "2": "Each satellite broadcasts its position and send time. The receiver times when each signal arrives. Travel time × c gives a range. Because the receiver clock is wrong, each measurement is a pseudorange = range + b, and b is the same in every pseudorange. Four pseudoranges let it solve for x, y, z and b. Satellite clocks must match GPS time because an error in one satellite clock sits in only one pseudorange, so nothing absorbs it. Relativity makes satellite clocks run fast compared with GPS time, so engineers set their frequency slightly low before launch. Ground stations upload small corrections for what is left.", + "3": "Position comes from signal travel times × light speed, so every nanosecond of satellite clock error is 0.3 m of range error. Relativity makes satellite clocks gain 38 µs/day (11.4 km of range per day), so the clocks are built to tick slow by 4.5×10⁻¹⁰ to cancel this.", + "4": "Satellite clock error is the key variable. Each extra ns adds 0.3 m of range error to that satellite's pseudorange only, so the fix gets worse, and past about 17 ns it is worse than about 5 m. If the relativistic drift were left uncorrected, each clock would move away from GPS time by 11.4 km of range per day, and the fix would fail. A change in the receiver clock error instead does not hurt accuracy, because b absorbs it.", + "5": "Assumption/simplification: the 1D worked example with two beacons on a line (and leaving out atmosphere, Sagnac and orbit shape). Established fact: time dilation 'is measured, not only predicted'. Net, satellite clocks gain 38 µs/day (lose 7 from speed, gain 45 from gravity).", + "6": "A GPS receiver turns signal travel times into distances by multiplying by light speed. Its cheap clock adds the same unknown offset to every distance, so it uses four satellites to solve for x, y, z and that offset. A satellite clock error affects only one distance, so satellite clocks must stay within about 17 ns of GPS time. Relativity makes them gain a net 38 µs per day (gravity +45, speed −7), which is 11.4 km of range per day. Engineers therefore set the clocks slightly slow before launch so that they run at the right rate in orbit.", + "7": "Section 5 was hardest. It does not say why satellites are set to GPS time at different moments. It also says the difference 'grows every day', but in its own example both clocks gain 38 µs/day, so the A−B gap stays at 114 µs (34.2 km). An explicit day-by-day table, and a reason why per-satellite drift becomes uneven (different set times, different clocks), would help. Also the text says 34 km, but 57 − 22.8 = 34.2.", + "8": "2 µs × 300 m/µs ≈ 600 m added to every pseudorange. It does not make the position wrong by 600 m. The error is the same in all pseudoranges, so the receiver solves for it as b (about 600 m) and subtracts it.", + "9": "They would run fast by about 45 µs per day, the gravity gain with nothing to cancel it. That is about 13.5 km of range per day.", + "10": "Range = time × c, and light covers 11.4 km in 38 µs. A 5 m fix needs satellite clocks within about 17 ns, so 38 µs/day is thousands of times over that budget after one day. Each satellite's error is its own and is not absorbed.", + "11": "The receiver's quartz clock error b is a fourth unknown alongside x, y, z. Four unknowns need four equations (pseudoranges). With perfect clocks, three would do: three spheres meet in two points and one is rejected as being in space.", + "12": "Fast, by a net 38 µs/day. Weaker gravity at 20,200 km makes them gain 45 µs/day (general relativity). Their 3.9 km/s speed makes them lose 7 µs/day (special relativity). Gravity wins. (Once built slow, they match GPS time in orbit.)", + "13": "b can only absorb an error that is the same in every pseudorange at the same moment. Satellites are set to GPS time at different moments, so their accumulated errors differ (for example 57 km vs 22.8 km on day 5). The difference is not shared, so no single b removes it. The explanation does not say why setting times differ." + }, + "audit": { + "terms": [ + "The ground reference has three names: 'ground clocks' (lead), 'GPS time' (§3, 'atomic clocks at ground stations'), and 'a clock on the ground' (§4). It is unclear whether these are the same thing.", + "Satellite clock error is called 'drift' (§5 title, §5), 'gain' (§4 table, lead), and 'error' (§3). For example: 'Relativity makes those clocks gain 38 µs a day' (lead) vs 'relativity drift' / 'cancel the drift' (§5).", + "'fix' means two things: 'For a fix within about 5 m' (§3) and 'position fix fails' (§5) mean a computed position, but '## 6. The fix: make the clocks tick slow' means the engineering correction.", + "b is given in two units without the link being stated: 'the receiver clock error expressed in metres' (§2) vs 'b = 1.5 km (5 µs)' (§2 example).", + "'range' / 'true distance' / 'distance' (§1) all mean the same thing, and pseudorange is introduced right after. This is minor but should stay consistent." + ], + "unexplained": [ + "'This difference grows every day' (§5): in the example both clocks gain 38 µs/day, so the 114 µs gap between A and B stays constant. The text does not say what makes the difference grow.", + "'10.23 MHz' and '10.22999999543 MHz' (§6): no reason is given for this base frequency, or for what the many decimal places mean for the reader.", + "'For a fix within about 5 m' (§3): the 5 m target is given without saying why 5 m, or what a looser or tighter target would do to the 17 ns budget.", + "'swings by up to about 23 ns' (Not covered): there is no comparison with the 17 ns budget, although 23 ns exceeds it. That is the key implication.", + "'3.9 km/s', 'loses 7 µs', 'gains 45 µs' (§4 table) are given with no way to see how they scale (for example, how the loss would change at higher speed or lower orbit)." + ], + "missing_why": [ + "The §4 table gives −7 and +45 µs/day with no formula or reasoning for their size. The text does not say why higher altitude makes a clock tick faster, only that it does.", + "§5: 'each satellite clock is set to GPS time at a different moment' is given without saying why the satellites are not all set together, or why a common rate still leaves unequal errors.", + "§6: why a rate (frequency) change is the right fix rather than periodic resets. Ground corrections exist, so why not just reset? This is only partly answered by 'keeps those corrections small'.", + "§1: 'Three spheres meet in two points': no reason is given why it is exactly two points.", + "§2: why the receiver clock error adds (pseudorange = range + b) rather than scaling the range. The 1D example shows it but does not state why." + ], + "excess": [ + "The Summary (lines 66–68) repeats the lead paragraph almost word for word.", + "'Light covers 300 m in one microsecond … 0.3 m in one nanosecond' with unit definitions (§1): this reader knows c ≈ 300,000 km/s, though the conversion rates are mildly useful.", + "The 'Not covered here' bullets on atmosphere and solving by iteration are detours the questions do not use." + ] + }, + "gaps": [ + "Section 5 says the satellite-to-satellite difference 'grows every day', but its own example (same drift rate, different set days) gives a constant 114 µs gap. The explanation never says what makes satellite errors diverge over time.", + "No derivation or scaling of the 7 µs and 45 µs figures, and no explanation of why higher gravitational potential means faster ticking.", + "There is no explanation of why satellites are synchronised at different moments.", + "The orbit-shape swing of 23 ns exceeds the stated 17 ns budget, but the effect is only noted under 'Not covered'.", + "Small arithmetic rounding: 57 − 22.8 = 34.2 km, stated as 34 km." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "1": "Full marks: lists every entity in model.md with correct details (30 satellites at 20,200 km, atomic vs quartz clocks, signal contents, range/pseudorange, b in metres, GPS time, both relativity effects, 10.23 MHz offset).", + "2": "Full marks: the causal chain is complete and in order. It explains why a satellite clock error is not absorbed by b and mentions the ground corrections.", + "3": "Full marks: gives the central answer with 0.3 m/ns, +38 µs/day, 11.4 km/day and the 4.5×10⁻¹⁰ slow-tick fix.", + "4": "Full marks: identifies satellite clock error as the key variable, with the 17 ns to 5 m budget, and contrasts it with receiver clock error, which b absorbs.", + "5": "Full marks: correctly separates simplifications (1D example; atmosphere, Sagnac and eccentricity out of scope) from measured fact (time dilation).", + "6": "Full marks: accurate summary with correct numbers (+45, −7, net 38 µs/day, 11.4 km/day, 17 ns).", + "7": "Full marks: specific and valid critique. If both clocks drift at the same rate, the A−B gap stays constant at 114 µs (34.2 km), which conflicts with 'grows every day'. It also notes the rounding (34 vs 34.2) and that the rendering never says why setting times differ. This is a candidate audit finding for the explainer to decide.", + "8": "Full marks: 600 m per pseudorange, absorbed as b, so the position is not off by 600 m.", + "9": "Full marks: fast by 45 µs/day, about 13.5 km/day, because the gravity gain remains.", + "10": "Full marks: 38 µs × c ≈ 11.4 km per day, set against the 17 ns budget.", + "11": "Full marks: b is a fourth unknown, so four equations are needed. Also covers the perfect-clock three-satellite case.", + "12": "Full marks: net fast by 38 µs/day; gravity +45 (GR) and speed −7 (SR).", + "13": "Full marks: b absorbs only a shared error. Gives the day-5 example (57 vs 22.8 km) showing the difference cannot go into one b." + }, + "pass": true + }, + "usage": { + "cost": 0.1966034, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The ground reference has three names: 'ground clocks' (lead), 'GPS time' (§3, 'atomic clocks at ground stations'), and 'a clock on the ground' (§4). It is unclear whether these are the same thing." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Satellite clock error is called 'drift' (§5 title, §5), 'gain' (§4 table, lead), and 'error' (§3). For example: 'Relativity makes those clocks gain 38 µs a day' (lead) vs 'relativity drift' / 'cancel the drift' (§5)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'fix' means two things: 'For a fix within about 5 m' (§3) and 'position fix fails' (§5) mean a computed position, but '## 6. The fix: make the clocks tick slow' means the engineering correction." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "b is given in two units without the link being stated: 'the receiver clock error expressed in metres' (§2) vs 'b = 1.5 km (5 µs)' (§2 example)." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "'range' / 'true distance' / 'distance' (§1) all mean the same thing, and pseudorange is introduced right after. This is minor but should stay consistent." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'This difference grows every day' (§5): in the example both clocks gain 38 µs/day, so the 114 µs gap between A and B stays constant. The text does not say what makes the difference grow." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'10.23 MHz' and '10.22999999543 MHz' (§6): no reason is given for this base frequency, or for what the many decimal places mean for the reader." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'For a fix within about 5 m' (§3): the 5 m target is given without saying why 5 m, or what a looser or tighter target would do to the 17 ns budget." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'swings by up to about 23 ns' (Not covered): there is no comparison with the 17 ns budget, although 23 ns exceeds it. That is the key implication." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "'3.9 km/s', 'loses 7 µs', 'gains 45 µs' (§4 table) are given with no way to see how they scale (for example, how the loss would change at higher speed or lower orbit)." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "The §4 table gives −7 and +45 µs/day with no formula or reasoning for their size. The text does not say why higher altitude makes a clock tick faster, only that it does." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "§5: 'each satellite clock is set to GPS time at a different moment' is given without saying why the satellites are not all set together, or why a common rate still leaves unequal errors." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "§6: why a rate (frequency) change is the right fix rather than periodic resets. Ground corrections exist, so why not just reset? This is only partly answered by 'keeps those corrections small'." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "§1: 'Three spheres meet in two points': no reason is given why it is exactly two points." + }, + { + "n": 15, + "severity": "gap", + "kind": "missing_why", + "text": "§2: why the receiver clock error adds (pseudorange = range + b) rather than scaling the range. The 1D example shows it but does not state why." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "The Summary (lines 66–68) repeats the lead paragraph almost word for word." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "'Light covers 300 m in one microsecond … 0.3 m in one nanosecond' with unit definitions (§1): this reader knows c ≈ 300,000 km/s, though the conversion rates are mildly useful." + }, + { + "n": 18, + "severity": "excess", + "kind": "excess", + "text": "The 'Not covered here' bullets on atmosphere and solving by iteration are detours the questions do not use." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "Section 5 says the satellite-to-satellite difference 'grows every day', but its own example (same drift rate, different set days) gives a constant 114 µs gap. The explanation never says what makes satellite errors diverge over time." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "No derivation or scaling of the 7 µs and 45 µs figures, and no explanation of why higher gravitational potential means faster ticking." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "There is no explanation of why satellites are synchronised at different moments." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "The orbit-shape swing of 23 ns exceeds the stated 17 ns budget, but the effect is only noted under 'Not covered'." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "Small arithmetic rounding: 57 − 22.8 = 34.2 km, stated as 34 km." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/read.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/read.json new file mode 100644 index 0000000..0a9ed1b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/read.json @@ -0,0 +1,45 @@ +{ + "answers": { + "1": { + "answer": "The receiver records when each satellite's signal arrives. Each signal says when it was sent, so travel time = receive time − send time. Multiplying the travel time by the speed of light c gives the range, the distance to that satellite. Because the receiver's own clock is wrong, what it actually computes is a pseudorange: the true range plus b, its clock error expressed in metres.", + "quote": "travel time = receive time − send time\n- range = travel time × c, where c is light speed and the range is the true distance to the satellite." + }, + "2": { + "answer": "The receiver's quartz clock is off by an unknown amount. That error, b, appears equally in every pseudorange, so it becomes a fourth unknown alongside x, y and z. Four unknowns need four equations, which means four satellites.", + "quote": "In three dimensions the unknowns are x, y, z, and b. Four unknowns need four equations, so the receiver needs four satellites." + }, + "3": { + "answer": "Relativity makes each satellite clock gain a net 38 µs a day: it gains 45 µs from weaker gravity and loses 7 µs from its speed. That is about 11.4 km of range error per day. A fix needs the satellite clocks within about 17 ns, roughly 5 m. Because each satellite clock is set at a different moment, the clocks carry different errors that the shared b cannot absorb. The explanation does not say how quickly within a day the fix breaks down.", + "quote": "38 µs is only 4.5 × 10⁻¹⁰ of a day, but light covers 11.4 km in 38 µs. Each uncorrected satellite clock would move 11.4 km of range further from GPS time every day." + }, + "4": { + "answer": "Every computed distance would be about 300 m too long, because 1 µs × c = 300 m. If the receiver treats these as true ranges with only three satellites, it has no fourth equation to detect or remove this shared error. The explanation does not say how large the resulting position error would be.", + "quote": "Light covers 300 m in one microsecond (1 µs = 10⁻⁶ s) and 0.3 m in one nanosecond (1 ns = 10⁻⁹ s). So every timing error becomes a range error at that rate." + } + }, + "stops": [ + "Section 5 says the clocks drift at the same rate, then says b absorbs the drift only if the errors are identical. It's unclear at first why a shared rate isn't enough: \"Every satellite clock drifts at the same rate. Why does b not absorb the drift?\"", + "It never says why satellites are set to GPS time at different moments, or why they aren't simply reset often: \"But each satellite clock is set to GPS time at a different moment.\"", + "Ground stations upload corrections, which makes it unclear why drift would make GPS fail at all: \"Ground stations upload small corrections for what remains\"", + "No reason is given for why three spheres meet in exactly two points: \"Three spheres meet in two points.\"", + "It doesn't say how a 5 m fix turns into a 17 ns budget (5 m / 0.3 m per ns), or why the full budget goes to each clock: \"For a fix within about 5 m, the satellite clocks must agree with GPS time to about 17 ns.\"", + "Why a higher clock ticks faster is stated but never explained: \"general relativity: a clock higher above Earth ticks faster\"", + "In the 1D example, the step that the ranges must add to 30 km needs a moment's thought: \"The receiver lies between the beacons, so the two ranges must add up to 30 km.\"", + "The clock adjustment is given as a frequency, while the drift is in µs per day; the link between them is implicit: \"set it to 10.22999999543 MHz: slower by 4.5 × 10⁻¹⁰\"" + ], + "excess": [ + "\"It is measured, not only predicted.\"", + "\"Your receiver (the GPS chip in a phone) only listens.\"", + "\"About 30 GPS satellites orbit 20,200 km above Earth.\"", + "\"Earth's rotation during the 67 ms flight (Sagnac effect)\"", + "\"Why a moving clock ticks slow: see the \\\"light clock\\\" argument in special relativity.\"", + "\"How the receiver solves the equations: iteration from a guess, with a best fit when it hears more than four satellites.\"", + "\"Four pseudoranges give x, y, z, and the receiver's clock error, but only if satellite clocks stay within about 17 ns of GPS time.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/run.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/run.json new file mode 100644 index 0000000..9bea378 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/gps-relativity/explanation.md` as prose only, because the subject is one chain of reasoning with worked numbers at each step, which needs no diagram or interactive page. The probe, the cold read and the blind test have all run, I fixed what they found, and `explainer check` passes.", + "error": null, + "cost": 1.5428533999999998, + "seconds": 567, + "turns": 79, + "files": [ + "explainers/gps-relativity/explanation.md" + ], + "review_cost": 0.6484, + "reviews": 4 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/explanation.md b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/explanation.md new file mode 100644 index 0000000..6705031 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/explanation.md @@ -0,0 +1,51 @@ +# How does GPS find your position, and why does it need relativity? + + + +**GPS finds your position by timing radio signals. Each satellite sends the time it transmitted. Your receiver measures how long each signal took to arrive, and multiplies by the speed of light to get a distance. Four distances fix your position. Relativity matters because the satellite clocks tick faster than ground clocks by 38 microseconds per day. At the speed of light, that error is about 11 km of distance per day.** + +## Step 1: time becomes distance + +About 30 GPS satellites orbit at 20,200 km altitude. Each satellite carries an atomic clock. Each satellite continuously broadcasts a **signal**: a radio message that says "I sent this at time t, and I was at position (x, y, z)". + +Your receiver notes when the signal arrives. The **travel time** is receive time minus send time. For a satellite straight overhead, the travel time is about 67 ms (20,200 km ÷ 299,792,458 m/s). + +Distance = travel time × c. Light travels about 30 cm per nanosecond, so the clocks must agree to within a few nanoseconds for metre-level accuracy. + +## Step 2: why four satellites, not three + +One distance puts you somewhere on a sphere around one satellite. Three spheres meet at a point. So three satellites should be enough. + +But your phone has a cheap quartz clock, not an atomic clock. Suppose the phone clock is 1 ms wrong. Then every distance is wrong by 1 ms × c = 300 km. The three spheres no longer meet at your true position. + +GPS solves this by treating the clock error as a fourth unknown. The **receiver clock bias** b is the offset of your clock from GPS time. Each measured distance is a **pseudorange** ρ: the true distance plus the same error c·b. For satellite i at (xᵢ, yᵢ, zᵢ): + +```text +ρᵢ = √((x − xᵢ)² + (y − yᵢ)² + (z − zᵢ)²) + c·b +``` + +There are four unknowns (x, y, z, b), so the receiver needs four equations: four satellites. It solves them by iteration: guess a position, compute the predicted pseudoranges, and correct the guess until the predicted values match the measured ones. A side effect: your phone gets the time accurate to nanoseconds. + +This method works only if the satellite clocks all keep GPS time. Relativity says that they do not, if nobody corrects them. + +## Step 3: two effects change the satellite clock rate + +**Predict first:** a satellite moves fast and sits high above Earth. Does its clock run fast or slow compared with a clock on the ground? + +Two effects pull in opposite directions. + +1. **Speed slows the clock (special relativity).** A moving clock ticks slower by a fraction of about v²/(2c²). The orbit speed is 3.87 km/s, which gives 8.3 × 10⁻¹¹. Over one day (86,400 s), the satellite clock loses about 7 µs. +2. **Weaker gravity speeds the clock up (general relativity).** A clock deeper in a gravity well ticks slower. A ground clock sits at 6,371 km from Earth's center. A satellite sits at 26,560 km, where gravity is weaker. The fraction is about (GM/c²)(1/R_Earth − 1/r) = 5.3 × 10⁻¹⁰. Over one day, the satellite clock gains about 45 µs. + +The gravity effect wins. Net: the satellite clock gains about 38 µs per day. + +## Step 4: why 38 µs matters + +38 µs looks small. Multiply by c: 38 µs × 300 m per µs ≈ 11 km. The system needs agreement to a few nanoseconds. The drift is 0.44 ns per second, so after one minute it is 26 ns, or 8 m. After one day, the satellite time stamps would be off by the light-travel time of 11 km. + +The fix is simple. Engineers set each satellite clock to tick slow before launch. The nominal frequency is 10.23 MHz; the clock is set to 10.22999999543 MHz. In orbit, relativity speeds the clock up to exactly 10.23 MHz as seen from the ground. + +## Limits + +- **Estimate:** "11 km per day" is the light-travel distance of 38 µs. The actual position error would grow in a more complex way, because the bias b absorbs part of an error that all satellites share. The drift would still break the system within hours. +- **Out of scope:** Earth's rotation during signal travel (the Sagnac effect), a small extra correction for slightly elliptical orbits (tens of ns, computed by the receiver), and signal delay in the ionosphere. diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/model.md b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/model.md new file mode 100644 index 0000000..c2bef06 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/model.md @@ -0,0 +1,99 @@ +# Semantic model: GPS positioning and relativity + +> Source of truth. Every rendering derives from this file. + +## Central question + +How does a GPS receiver find its position from satellite signals, and why must the system correct for relativity? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (speed = distance / time, Pythagoras, simultaneous equations, gravity), basic programming. New to GPS and to relativity. Does not know: pseudorange, receiver clock bias, why four satellites, that clocks tick at different rates with speed and gravity. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Satellite | One of about 30 GPS satellites at 20,200 km altitude, each with an atomic clock. | fact | +| Signal | A radio message that carries the satellite's send time and its position. | fact | +| Receiver | The device on the ground (phone) with a cheap quartz clock. | fact | +| Travel time | Receive time minus send time; about 67 ms for a satellite overhead. | fact | +| Pseudorange | Travel time × c; a distance that still contains the receiver clock error. | definition | +| Receiver clock bias | The unknown offset of the receiver's clock from GPS time. | definition | +| Clock rate shift | The change in how fast a satellite clock ticks compared with a ground clock. | fact (relativity) | + +## Relationships + +| From | Relationship | To | +|---|---|---| +| Satellite clock | stamps | Signal (send time) | +| Travel time × c | gives | Pseudorange | +| 4 pseudoranges | solve for | x, y, z, receiver clock bias | +| Orbit speed | slows | Satellite clock (special relativity) | +| Weaker gravity at altitude | speeds up | Satellite clock (general relativity) | +| Clock rate shift | becomes | Range error (× c) | + +## Causal chain + +```text +time stamp → travel time → × c → pseudorange → 4 equations → position + clock bias +satellite speed (−7 µs/day) + weaker gravity (+45 µs/day) → net +38 µs/day → ×c → ≈11 km/day range drift → so: tune satellite clocks slow before launch +``` + +## Quantities + +- c = 299,792,458 m/s; light travels about 30 cm per ns, 300 m per µs. +- Altitude 20,200 km; orbit radius 26,560 km; Earth radius 6,371 km; orbit speed 3.87 km/s. +- Travel time overhead ≈ 20,200 km / c ≈ 67 ms. +- Special relativity: fractional rate ≈ −v²/(2c²) = −8.3 × 10⁻¹¹ → × 86,400 s ≈ −7 µs/day. +- General relativity: fractional rate ≈ (GM/c²)(1/R_E − 1/r) = +5.3 × 10⁻¹⁰ → ≈ +45 µs/day. +- Net ≈ +38 µs/day → 38 µs × 300 m/µs ≈ 11 km of light travel per day. +- Fix: nominal 10.23 MHz clock set to 10.22999999543 MHz before launch. +- Receiver clock 1 ms wrong → 300 km range error. + +## Alternative states + +| Scenario | What changes | Why | +|---|---|---| +| No relativity correction | Satellite clocks gain 38 µs/day on ground clocks | ranges drift by about 11 km per day | +| Only 3 satellites | cannot solve for clock bias | 4 unknowns, 3 equations | +| Perfect receiver clock | 3 satellites would suffice | bias is no longer unknown | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Use 4 satellites | 3 spheres meet at a point; use 3 | receiver quartz clock may be 1 ms off → every range wrong by 300 km | +| Measure time, then multiply by c | measure distance directly | no way to measure 20,200 km directly; time stamps are what a radio can carry | +| Correct the satellite clock rate | ignore relativity; effects are tiny (5 parts in 10¹⁰) | ×c turns 38 µs/day into ≈11 km/day | + +## Concrete cases + +| Claim | Holds | Breaks | +|---|---|---| +| Position from 4 pseudoranges | all clocks agree with GPS time to ns → metres | satellite clocks drift 38 µs/day → ≈11 km/day | + +## Terms + +- travel time, pseudorange, receiver clock bias, clock rate shift. "GPS time" = the common time the satellite clocks are kept to. + +## Epistemic status + +- **Established:** both rate shifts are measured on GPS clocks and match relativity. +- **Estimate:** "11 km per day" is the light-travel equivalent of 38 µs. Real position error would grow in a more complex way, because the receiver's bias term absorbs part of a shift common to all satellites. The drift would still break the system. + +## Confusion points + +- "Relativity makes satellite clocks slow" → speed slows them (7 µs/day), but gravity speeds them up more (45 µs/day). Net: fast. +- "GPS measures distance" → it measures time; distance is time × c. +- "Third satellite is enough in 3D" → only if the receiver clock is perfect. + +## Scope + +- **Out of scope:** Sagnac effect from Earth's rotation; orbit eccentricity correction (tens of ns, computed by the receiver); ionosphere delay; least-squares with more than 4 satellites. + +## Representation decision + +- **Stage:** 1 +- **Reason:** a sequential argument with worked numbers; no state change or parameter space a reader must explore. +- **Levels:** L1 answer / L2 time → distance → 4 equations / L3 two clock effects with numbers / L4 the factory frequency offset. diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/model.yaml b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/model.yaml new file mode 100644 index 0000000..1033f73 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/explainers/gps-relativity/model.yaml @@ -0,0 +1,77 @@ +# Machine-readable half of model.md. `explainer check gps-relativity` holds every rendering to it. + +values: + satellites_in_orbit: 30 + altitude_km: 20200 + c_m_per_s: 299792458 + travel_time_ms: 67 + light_cm_per_ns: 30 + light_m_per_us: 300 + receiver_clock_error_ms: 1 + receiver_range_error_km: 300 + unknowns: 4 + orbit_speed_km_s: 3.87 + sr_fraction: 8.3e-11 + seconds_per_day: 86400 + sr_us_per_day: 7 + earth_radius_km: 6371 + orbit_radius_km: 26560 + gr_fraction: 5.3e-10 + gr_us_per_day: 45 + net_us_per_day: 38 + drift_km_per_day: 11 + drift_ns_per_s: 0.44 + drift_ns_per_min: 26 + drift_m_per_min: 8 + nominal_mhz: 10.23 + set_mhz: 10.22999999543 + +allow: [1, 2, 3, 4, 10] + +require: + - path: net_us_per_day + in: [prose] + - path: drift_km_per_day + in: [prose] + - path: set_mhz + in: [prose] + +claims: + - id: time-to-distance + kind: mechanism + statement: "The receiver turns travel time into distance by multiplying by c." + example: "Satellite overhead at 20,200 km: travel time about 67 ms. 1 ns of clock error is 30 cm." + - id: why-four + kind: why + statement: "Four satellites are needed because the receiver clock bias is a fourth unknown." + alternative: "Use three satellites; three spheres meet at a point." + counterexample: "A phone clock 1 ms wrong makes every distance wrong by 300 km." + ask: "Why does a receiver need four satellites, not three?" + - id: two-effects + kind: mechanism + statement: "Speed slows the satellite clock; weaker gravity speeds it up more." + example: "-7 µs/day from speed, +45 µs/day from gravity, net +38 µs/day." + ask: "Does a GPS satellite clock run fast or slow compared with a ground clock, and why?" + - id: drift-size + kind: why + statement: "A tiny clock drift matters because c converts it into a large distance." + alternative: "Ignore relativity; the rate shift is only a few parts in 10^10." + counterexample: "38 µs/day × 300 m/µs ≈ 11 km/day; 8 m after one minute." + - id: the-fix + kind: mechanism + statement: "Satellite clocks are set slow before launch so they tick at the nominal rate in orbit." + example: "10.23 MHz nominal, set to 10.22999999543 MHz." + +accept_unjustified: {} + +budget: {prose: 750, reason: "two mechanisms (positioning, then two opposing clock effects), each with worked numbers"} + +terms: + - term: pseudorange + means: "travel time × c, which still contains the receiver clock bias" + avoid: [true range] + +quiz: + - q: "Does a GPS satellite clock run fast or slow compared with a ground clock?" + expect: "Fast, by about 38 µs/day: gravity's +45 beats speed's -7." + misconception: "Slow, because moving clocks run slow." diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/read.json new file mode 100644 index 0000000..f9e17ba --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/read.json @@ -0,0 +1,41 @@ +{ + "answers": { + "1": { + "answer": "The receiver records when each signal arrives. Each signal carries its send time, so the travel time is the receive time minus the send time. Multiplying the travel time by the speed of light c gives the distance to that satellite.", + "quote": "Your receiver notes when the signal arrives. The **travel time** is receive time minus send time." + }, + "2": { + "answer": "A cheap receiver clock is wrong by some unknown amount, and that error adds the same amount (c·b) to every distance. The clock bias b becomes a fourth unknown alongside x, y and z. Four unknowns need four equations, so the receiver needs four satellites.", + "quote": "There are four unknowns (x, y, z, b), so the receiver needs four equations: four satellites." + }, + "3": { + "answer": "Two effects change the satellite clock rate. Speed slows it by about 7 µs per day, and weaker gravity speeds it up by about 45 µs per day. The net result is that satellite clocks gain about 38 µs per day. Multiplied by c, that is about 11 km of distance per day. The system needs its clocks to agree within a few nanoseconds, and the drift reaches 26 ns (8 m) after just one minute.", + "quote": "38 µs looks small. Multiply by c: 38 µs × 300 m per µs ≈ 11 km." + }, + "4": { + "answer": "Each distance would be off by about 300 m (1 µs × c). Since the clock is fast, the distances would come out too long. With only three satellites, nothing absorbs this error, so the spheres would not meet at the true position.", + "quote": "Then every distance is wrong by 1 ms × c = 300 km. The three spheres no longer meet at your true position." + } + }, + "stops": [ + "\"the bias b absorbs part of an error that all satellites share. The drift would still break the system within hours.\": If b can absorb a shared error, why does the shared drift still break the system? The reason is not given.", + "\"(GM/c²)(1/R_Earth − 1/r)\": GM is never defined.", + "\"The drift would still break the system within hours.\": this seems to conflict with the earlier framing of an error of 11 km per day.", + "\"the clock is set to 10.22999999543 MHz\": the explanation does not show how this number follows from the 38 µs per day drift.", + "\"A clock deeper in a gravity well ticks slower.\": this is stated without any reason.", + "\"It solves them by iteration: guess a position...\": it is unclear why the equations are not solved directly." + ], + "excess": [ + "\"\": internal build note.", + "\"\": internal markup that does not help the reader.", + "\"A side effect: your phone gets the time accurate to nanoseconds.\": not needed for the question.", + "\"Out of scope: Earth's rotation during signal travel (the Sagnac effect)...\": lists topics that are never covered.", + "\"About 30 GPS satellites orbit at 20,200 km altitude.\": the count of satellites is not used anywhere." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/run.json new file mode 100644 index 0000000..78ebe80 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/gps-relativity/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/gps-relativity/explanation.md` (prose, with `model.md` and `model.yaml` behind it; `explainer check` passes) because the subject is one chain of reasoning with worked numbers: timing gives distance, four satellites solve the clock error, and relativity makes the satellite clocks gain 38 µs/day (about 11 km/day) — there is nothing to explore or animate.", + "error": null, + "cost": 0.44009420000000005, + "seconds": 91, + "turns": 18, + "files": [ + "explainers/gps-relativity/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/explainers/greenhouse/explanation.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/explainers/greenhouse/explanation.md new file mode 100644 index 0000000..2096434 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/explainers/greenhouse/explanation.md @@ -0,0 +1,167 @@ +# How greenhouse gases warm the Earth + +**The short answer:** The Earth gets energy from sunlight and loses it as infrared radiation to space. A few gases in the air, mainly water vapour and carbon dioxide, absorb that infrared. Because of them, the radiation that reaches space comes mostly from high, cold layers of the atmosphere rather than from the warm surface. Cold things radiate less. To get rid of all the energy it receives, the planet has to be warmer than it would be without those gases. Adding more of them pushes the radiating layer higher, where it is colder. The planet then loses less energy than it takes in, and it warms until the two match again. + +The rest of this document builds that answer one step at a time. Each step is a small piece of physics you can check with a calculator. The companion page `index.html` lets you change the inputs of the two models below and see what happens. + +--- + +## 1. The planet is a budget: energy in = energy out + +Leave the atmosphere out at first. The Earth sits in sunlight and gains energy. It also radiates energy away into space. If it gains more than it loses, it warms up. A warmer body radiates more, so the warming continues until the losses equal the gains. That balanced state sets the planet's temperature. + +**Energy in.** At Earth's distance from the Sun, sunlight carries about **S = 1361 W/m²** through a surface facing it. The Earth catches this on a disc of area πR², but it is a sphere of area 4πR². Averaged over the whole surface, day and night, that gives S/4 ≈ 340 W/m². About 30% of it is reflected straight back by clouds, ice and bright surfaces (this fraction is the *albedo*, α ≈ 0.3). What is actually absorbed is + +``` +absorbed = S (1 − α) / 4 ≈ 1361 × 0.7 / 4 ≈ 238 W/m² +``` + +**Energy out.** Every object radiates according to its temperature. For an ideal emitter (a "black body") the **Stefan–Boltzmann law** gives + +``` +emitted = σ T⁴, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴ +``` + +Note the fourth power. A small rise in temperature makes a body radiate quite a lot more. + +**Balance.** Set the two equal and solve for T: + +``` +σ T⁴ = 238 → T = (238 / 5.67e-8)^(1/4) ≈ 255 K ≈ −18 °C +``` + +The measured average surface temperature is about **288 K (+15 °C)**. That leaves a gap of roughly **33 K**. This gap is the greenhouse effect, and the rest of the document explains where it comes from. + +## 2. Sunlight and earthlight are different colours + +The radiation a body gives off peaks at a wavelength set by its temperature (**Wien's law**: peak ≈ 2898 µm·K / T). + +| Emitter | Temperature | Peak wavelength | Kind of light | +|---|---|---|---| +| Sun's surface | ~5800 K | ~0.5 µm | visible | +| Earth's surface | ~288 K | ~10 µm | thermal infrared | + +The two spectra hardly overlap. This is what makes the greenhouse effect possible. Air can be almost transparent to incoming sunlight and still be largely opaque to outgoing infrared. The air acts on the energy going out and does little to the energy coming in. + +## 3. Why only some gases absorb infrared + +About 99% of dry air is nitrogen (N₂) and oxygen (O₂), and these barely absorb thermal infrared. The reason is molecular. + +A molecule absorbs or emits infrared by changing how it vibrates or rotates. It can only do this if the motion **changes the molecule's electric dipole**, because the dipole is what the light's oscillating electric field can push on. N₂ and O₂ are two identical atoms. However they stretch, the molecule stays symmetric and has no dipole, so infrared light has almost nothing to act on. + +Molecules with three or more atoms, or with two different atoms, behave differently: + +- **CO₂** (O=C=O) is straight and symmetric at rest. When it **bends**, or when it stretches asymmetrically, a dipole appears. The bending mode absorbs strongly near **15 µm**, close to the peak of Earth's emission. +- **H₂O** is bent to begin with and absorbs across wide ranges of the infrared. +- **CH₄** (methane), **N₂O** (nitrous oxide) and **O₃** (ozone) each have their own absorption bands. + +Together these absorb most outgoing infrared at most wavelengths. The main exception is a gap between about 8 and 13 µm, the **atmospheric window**, where clear air is fairly transparent. Through that window the surface radiates fairly directly to space. + +One detail matters for later. In the lower atmosphere, a CO₂ molecule that absorbs a photon nearly always collides with neighbouring molecules (mostly N₂ and O₂) before it can re-emit. Collisions happen roughly every nanosecond, and the excited state would take much longer than that to radiate. So the absorbed energy turns into ordinary heat in the surrounding air. Separately, CO₂ is constantly excited by collisions and **emits** at a rate set by the *local air temperature*. Remember this point: **what a layer of air emits depends on how warm that layer is.** + +## 4. The simplest model: one absorbing layer + +Model the atmosphere as a single layer. It is transparent to sunlight and absorbs a fraction ε of the surface's infrared, with 0 ≤ ε ≤ 1. A layer that absorbs a fraction ε also emits ε times the black-body amount (Kirchhoff's law). It emits from both faces, so the same amount goes up and down. + +Write E_s = σT_s⁴ for the surface emission and E_a = σT_a⁴ for the black-body emission at the layer's temperature. At equilibrium every part must balance: + +``` +Atmosphere layer: absorbs ε·E_s = emits ε·E_a up + ε·E_a down + → E_a = E_s / 2 + +Top of atmosphere: 238 (in) = (1 − ε)·E_s + ε·E_a (out) + = E_s (1 − ε/2) + +So: T_s = [ 238 / (σ (1 − ε/2)) ]^(1/4) +``` + +Some values: + +| ε | T_s | +|---|---| +| 0 (no greenhouse gases) | 255 K | +| 0.78 | ≈ 288 K, today's value | +| 1 (fully opaque) | 255 × 2^(1/4) ≈ 303 K | + +This shows the mechanism directly. The surface now gets sunlight **plus** downward infrared from the warm air, so it has to warm up until its own emission gets rid of both. The downward infrared is real and measurable: instruments on the ground record about 340 W/m² of it on average, which is more than the surface receives from the Sun. + +The model has a problem, though. In it, the effect stops growing once the layer is fully opaque (ε = 1). The real atmosphere is already opaque at the centre of the CO₂ band, yet adding CO₂ still causes warming. Explaining that needs a better picture. + +## 5. The better picture: the height at which Earth radiates to space + +Two facts about the real atmosphere matter here. + +1. **It gets colder with height.** In the lowest ~10–15 km (the troposphere), air rising and expanding cools by about **6.5 K per km** on average. This rate of cooling with height is called the *lapse rate*. Convection keeps this profile in place. +2. **It is thick in infrared.** At a wavelength the gases absorb strongly, a photon emitted low down is absorbed again almost immediately. Only photons emitted from fairly **high up**, where there is little absorbing gas above them, get out to space. + +So at each wavelength there is an **effective emission height**, roughly the level above which the air is too thin to stop the radiation. Space receives radiation at the *temperature of that height*, not the temperature of the surface. + +Averaged over all wavelengths, the Earth radiates to space like a black body at 255 K. In the real atmosphere, 255 K is found at about **5 km** up. Starting from the emission height and going down at the lapse rate: + +``` +T_surface ≈ T_emission + Γ × H ≈ 255 K + 6.5 K/km × 5 km ≈ 288 K +``` + +The same 33 K appears again, this time explained through altitude. + +**Now add CO₂.** With more absorber, the air becomes opaque higher up, so the emission height rises. A higher level is colder, and colder air radiates less (T⁴ again). For a while the planet sends out **less** than the 238 W/m² it absorbs. This shortfall is called **radiative forcing**. The planet keeps gaining energy and warms until the whole temperature profile has shifted up enough that the new, higher emission level is back at 255 K. Because the lapse rate holds the profile together, the surface ends up warmer by about Γ × (rise in emission height). + +This also deals with the "CO₂ is already saturated" objection. Being saturated at the band centre means that **surface** radiation at those wavelengths cannot escape. That was never the source of the effect. The emission level can always move higher. At the edges of the band, where absorption is weaker, extra CO₂ also moves those wavelengths from "escapes from low and warm" to "escapes from high and cold". As a result the effect grows roughly with the **logarithm** of concentration, not linearly, but it does not stop. + +## 6. Putting numbers on it + +A widely used fit from detailed radiation calculations gives the forcing from CO₂ as + +``` +ΔF ≈ 5.35 × ln(C / C₀) W/m² +``` + +Doubling CO₂ (C/C₀ = 2) gives ΔF ≈ **3.7 W/m²**. Before industrialisation C₀ was about 280 ppm. By the mid-2020s it was above 420 ppm. + +How much warming does that cause? Differentiate σT⁴ at 255 K: + +``` +d(σT⁴)/dT = 4σT³ ≈ 3.8 W/m² per K +``` + +That is how much more a black body at 255 K radiates for each degree of warming. Detailed models give about 3.2 W/m² per K for the real Earth (this is called the *Planck response*). On its own, then, doubling CO₂ warms the planet by about + +``` +ΔT ≈ 3.7 / 3.2 ≈ 1.2 K +``` + +Using the lapse rate, that corresponds to the emission height rising by only about 1.2 / 6.5 ≈ 0.18 km, under 200 metres. + +## 7. Feedbacks: the planet responds to its own warming + +The 1.2 K figure assumes nothing else changes. Several other things do change: + +- **Water vapour (strongly amplifies).** Warmer air can hold about 7% more water vapour per kelvin (the Clausius–Clapeyron relation). Water vapour is itself a greenhouse gas, so this adds to the warming. +- **Lapse rate (partly offsets).** In the tropics, warming is greater aloft than at the surface. That lets the planet radiate a little more easily, which damps the warming somewhat. +- **Ice and snow (amplifies).** Melting replaces bright surfaces with darker ones, so the albedo falls and more sunlight is absorbed. +- **Clouds (uncertain, probably amplifying overall).** Clouds both reflect sunlight, which cools, and absorb infrared, which warms. How they change is the largest single source of uncertainty. + +Feedbacks can be included by replacing the 3.2 with a smaller net "restoring" value λ. Then ΔT = ΔF / λ. Current assessments put the long-term warming from doubled CO₂ (the *equilibrium climate sensitivity*) **likely between about 2.5 and 4 K, with about 3 K as the best estimate**. That is roughly 2–3 times the no-feedback value. + +**Why CO₂ and not water vapour?** Water vapour contributes the largest share of today's greenhouse effect, around half. Clouds contribute about a quarter, and CO₂ about a fifth. But the amount of water vapour is set by temperature: if there is too much, it rains out within days. CO₂ does not condense, and it stays in the air for centuries. CO₂ therefore acts as the control setting, and water vapour amplifies whatever change CO₂ makes. + +## 8. Common misconceptions + +- **"It works like a glass greenhouse."** Not really. A real greenhouse stays warm mainly because the glass stops warm air from rising away by convection. The name stuck anyway. +- **"The gases reflect heat back down."** They absorb it and then emit according to their own temperature. The emission-height picture explains the effect better than "trapping" does. +- **"It's to do with the ozone hole."** The ozone hole is a separate problem. Ozone is a greenhouse gas, but the ozone hole does not drive global warming. +- **"Extra CO₂ can't matter if the band is saturated."** See Section 5: the emission height keeps rising. + +## 9. A fingerprint you can check + +If the Sun were causing the warming, the whole atmosphere would warm. Greenhouse warming predicts something different: the **troposphere warms while the stratosphere cools**. Above about 15 km, extra CO₂ mainly helps the air radiate its energy to space, so that layer cools. Observations since the 1960s show this pattern: a warming troposphere under a cooling stratosphere. + +--- + +### Summary + +1. In equilibrium, absorbed sunlight (~238 W/m²) equals emitted infrared (σT⁴), which gives 255 K for a planet with no greenhouse gases. +2. Sunlight is visible and earthlight is infrared, so a gas can affect one and not the other. +3. Only molecules whose vibrations change their dipole (CO₂, H₂O, CH₄, …) absorb infrared. N₂ and O₂ do not. +4. These gases make the planet radiate to space from about 5 km up, where it is 255 K. The lapse rate then puts the surface 33 K warmer. +5. More CO₂ raises the emission height, into colder air. Less energy leaves than arrives, and the planet warms until balance returns. Doubling CO₂ gives about 3.7 W/m² of forcing and 1.2 K of warming directly, and about 3 K once feedbacks are included. diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/explainers/greenhouse/index.html b/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/explainers/greenhouse/index.html new file mode 100644 index 0000000..9337922 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/explainers/greenhouse/index.html @@ -0,0 +1,142 @@ + + + + +Greenhouse effect: two models you can play with + + + + +

How greenhouse gases warm the Earth: interactive models

+

This page goes with explanation.md. Each model below is a few lines of arithmetic. Move the sliders and the numbers and diagrams update. All fluxes are global averages in W/m².

+ +

Model 1: one absorbing layer

+

The atmosphere is modelled as one layer. Sunlight passes straight through it. It absorbs a fraction ε of the infrared coming up from the surface, and it emits ε·σTa⁴ both up and down. Start at ε = 0, then increase it, and watch the surface temperature and the downward infrared.

+
+ + + +
+

Things to notice: (1) at ε = 0 the surface is 255 K, the temperature set by energy balance alone. (2) Around ε ≈ 0.78 it reaches the observed 288 K. (3) The model cannot go above ε = 1. The real atmosphere is already nearly opaque at the centre of the CO₂ band and still warms when CO₂ is added, which is why Model 2 is needed.

+
+ +

Model 2: emission height and lapse rate

+

Temperature falls by about Γ = 6.5 K/km with height. On average the Earth radiates to space from the level where the air is 255 K, about 5 km up. Adding CO₂ lifts that level into colder air, so less energy leaves the planet (the forcing, ΔF = 5.35·ln(C/280)). The planet warms until the outgoing radiation balances the incoming again.

+
+ + + + + +
+

Gray line: pre-industrial temperature profile. Red line: the profile after the planet has warmed by ΔT. Dots: the effective emission height. With no feedbacks (λ = 3.2), the emission level has risen by ΔT/Γ but is back at 255 K, so outgoing radiation balances again. A doubling raises it by under 200 m. The effect is small on this scale, but it accounts for all of the warming.

+
+ + + + diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/read.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/read.json new file mode 100644 index 0000000..79a339b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/read.json @@ -0,0 +1,46 @@ +{ + "answers": { + "1": { + "answer": "Sunlight and the Earth's own radiation have different wavelengths. The Sun, at about 5800 K, peaks in visible light near 0.5 µm. The Earth, at about 288 K, peaks in thermal infrared near 10 µm. The two spectra hardly overlap, so the air can let visible sunlight through and still absorb much of the outgoing infrared.", + "quote": "The two spectra hardly overlap. This is what makes the greenhouse effect possible. Air can be almost transparent to incoming sunlight and still be largely opaque to outgoing infrared." + }, + "2": { + "answer": "Radiation reaches space from an effective emission height, on average about 5 km up, where the air is 255 K. More CO2 makes the air opaque higher up, so this height rises into colder air. Colder air radiates less, so the planet sends out less than it absorbs. It keeps warming until the new, higher level is back at 255 K. Because temperature falls with height at a fixed rate (the lapse rate), the surface ends up warmer by about the lapse rate times the rise in height.", + "quote": "With more absorber, the air becomes opaque higher up, so the emission height rises. A higher level is colder, and colder air radiates less (T⁴ again). For a while the planet sends out **less** than the 238 W/m² it absorbs." + }, + "3": { + "answer": "A molecule can only absorb or emit infrared if its vibration changes its electric dipole. N2 and O2 are each made of two identical atoms, so they stay symmetric however they stretch and never have a dipole. Infrared light has almost nothing to act on.", + "quote": "N₂ and O₂ are two identical atoms. However they stretch, the molecule stays symmetric and has no dipole, so infrared light has almost nothing to act on." + }, + "4": { + "answer": "Reasoning from the text: with nothing to absorb the infrared, it would radiate to space straight from the surface, not from a high, cold layer. The surface temperature would then just be the energy-balance temperature, about 255 K (−18 °C), with no extra warming from altitude and the lapse rate.", + "quote": "In equilibrium, absorbed sunlight (~238 W/m²) equals emitted infrared (σT⁴), which gives 255 K for a planet with no greenhouse gases." + } + }, + "stops": [ + "The symbol Γ appears in a formula, but the text only names the 'lapse rate' and never says Γ stands for it: \"T_surface ≈ T_emission + Γ × H\"", + "The rule behind 'absorbs ε, so also emits ε' is just named, with no reason given: \"A layer that absorbs a fraction ε also emits ε times the black-body amount (Kirchhoff's law).\"", + "'Electric dipole' is used as the key idea but never explained: \"It can only do this if the motion **changes the molecule's electric dipole**\"", + "Two different numbers are given without saying why they differ: \"Detailed models give about 3.2 W/m² per K for the real Earth (this is called the *Planck response*)\" versus 3.8", + "No reason is given for why the effect follows a logarithm: \"the effect grows roughly with the **logarithm** of concentration, not linearly\"", + "The lapse-rate feedback is hard to follow: \"In the tropics, warming is greater aloft than at the surface. That lets the planet radiate a little more easily\"", + "No reason is given for why the stratosphere cools: \"Above about 15 km, extra CO₂ mainly helps the air radiate its energy to space, so that layer cools.\"", + "The λ value appears without being derived: \"replacing the 3.2 with a smaller net \"restoring\" value λ\"", + "Convection is mentioned as the cause without explanation: \"Convection keeps this profile in place.\"" + ], + "excess": [ + "\"**\"It's to do with the ozone hole.\"** The ozone hole is a separate problem.\"", + "\"A real greenhouse stays warm mainly because the glass stops warm air from rising away by convection.\"", + "\"If the Sun were causing the warming, the whole atmosphere would warm.\"", + "\"Before industrialisation C₀ was about 280 ppm. By the mid-2020s it was above 420 ppm.\"", + "\"Current assessments put the long-term warming from doubled CO₂ (the *equilibrium climate sensitivity*) **likely between about 2.5 and 4 K\"", + "\"**Clouds (uncertain, probably amplifying overall).**\"", + "\"**CH₄** (methane), **N₂O** (nitrous oxide) and **O₃** (ozone) each have their own absorption bands.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/run.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/run.json new file mode 100644 index 0000000..20eff7a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/none/run.json @@ -0,0 +1,13 @@ +{ + "result": "I made `explainers/greenhouse/explanation.md` and `explainers/greenhouse/index.html`. The prose builds the explanation step by step, using equations a technical reader can check with a calculator: energy balance, infrared absorption, one-layer and emission-height models, forcing and feedbacks. The page adds two models whose sliders change the inputs, so the reader can see how surface temperature responds to absorptivity, CO₂ and feedback strength; I haven't opened it in a browser yet.", + "error": null, + "cost": 0.3816954, + "seconds": 124, + "turns": 3, + "files": [ + "explainers/greenhouse/explanation.md", + "explainers/greenhouse/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/explanation.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/explanation.md new file mode 100644 index 0000000..0bb9f11 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/explanation.md @@ -0,0 +1,127 @@ +# How greenhouse gases warm the Earth + +**Short answer.** Earth loses heat to space only as infrared light. Greenhouse gases absorb infrared light, so most of the infrared that reaches space comes from high in the atmosphere, not from the surface. High air is cold, and cold air emits less infrared. To send out as much energy as it gets from the Sun, the whole lower atmosphere and the surface must become warmer. More CO2 lifts the emitting altitude higher, into colder air, so the surface must warm more. + +The sections below build this answer step by step, with numbers. + +## 1. The puzzle: Earth is 33 K too warm + +Energy that enters a planet must leave it, or the planet keeps warming. Start with the energy that enters. + +- Sunlight at Earth's distance delivers **1361 W/m²** to a surface that faces the Sun. +- Earth intercepts sunlight with a disc of area πR², but it has a surface of area 4πR². Spread over the whole sphere, the mean input is 1361 / 4 ≈ **340 W/m²**. +- Clouds, ice, and other bright surfaces reflect about 30% of it. Earth absorbs 340 × 0.70 ≈ **238 W/m²**. + +Now the energy that leaves. Every object emits thermal radiation. The **Stefan–Boltzmann law** gives the power per square metre from an object at absolute temperature T: + + P = σT⁴, with σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴. + +The T⁴ makes emission very sensitive to temperature: 10% warmer means about 46% more emission. + +Set emission equal to absorption and solve for T: + + T = (238 / 5.67 × 10⁻⁸)^(1/4) ≈ **255 K** (−18 °C). + +The measured mean surface temperature is **288 K** (15 °C). The surface is 33 K warmer than the simple balance predicts. A surface at 288 K emits σ × 288⁴ ≈ **390 W/m²**, but only about 238 W/m² leaves the top of the atmosphere. The atmosphere stops about **150 W/m²** of the surface emission from reaching space. The rest of this explanation shows how. + +## 2. Sunlight and Earth-light are different colours + +The wavelength at which an object emits most strongly falls as the object gets hotter (**Wien's law**: peak wavelength ≈ 2898 µm·K / T). + +| Emitter | Temperature | Peak wavelength | Type | +|---|---|---|---| +| Sun | 5800 K | 2898 / 5800 ≈ 0.5 µm | visible light | +| Earth | 288 K | 2898 / 288 ≈ 10 µm | infrared | + +The two bands hardly overlap. So the atmosphere can be nearly transparent to the incoming energy and nearly opaque to the outgoing energy. The gases that do this are the greenhouse gases. + +## 3. Why CO2 and water absorb infrared, but N2 and O2 do not + +A molecule absorbs light when the light's electric field can push its charges into a new motion. For infrared, that motion is a vibration of the atoms. The push works only if the vibration changes the molecule's **dipole**, the separation of its positive and negative charge. + +- **N2 and O2** (99% of dry air) have two identical atoms. When they stretch, the charge stays symmetric, so the dipole stays zero. They are almost transparent to infrared. +- **CO2** is O=C=O. When it bends, the carbon moves off the line of the oxygens and a dipole appears. This bend absorbs strongly near **15 µm**, close to the peak of Earth's emission. +- **H2O** is already bent and has many vibration and rotation modes. It absorbs across much of the infrared. CH4 and N2O act in the same way. + +A gas that absorbs at a wavelength also emits at that wavelength, at a rate set by its own temperature. Greenhouse gases do not reflect infrared. They absorb it, share the energy with the surrounding air through collisions, and emit new infrared in all directions. + +## 4. A first model: one absorbing layer + +Take the simplest version. Put one layer of air above the surface. It is transparent to sunlight and absorbs all infrared. + +*Predict before you read on: will the surface be warmer than 255 K, colder, or the same?* + +- Space sees only the layer. The layer must emit 238 W/m² upward, so the layer is at 255 K. +- The layer emits equally up and down. It sends 238 W/m² down to the surface as well. +- The surface receives 238 (Sun) + 238 (layer) = 476 W/m². It must emit that much: T = (476 / σ)^(1/4) ≈ **303 K**. + +So the absorbing layer warms the surface. But 303 K is 15 K too warm. The model fails for two reasons: + +1. Real air does not absorb all infrared. Between about 8 and 13 µm, few gases absorb, and some surface emission escapes directly through this **window**. +2. Warm surface air rises, so **convection** carries heat upward. Radiation is not the only path. + +The one-layer model shows *that* absorption warms the surface. It cannot show why *more* CO2 warms it further. If the layer already absorbs everything, extra CO2 would change nothing. The next model fixes this. + +## 5. The real mechanism: the emission level and the lapse rate + +Two facts replace the single layer. + +**Fact 1: infrared escapes from a height, not from the surface.** An infrared photon emitted low in the air is soon absorbed again. A photon reaches space only if little absorbing gas lies above it. So, at each wavelength, the infrared that reaches space comes mostly from one altitude, the **emission level**. Averaged over all wavelengths, Earth's emission level is about **5 km** up. + +**Fact 2: air gets colder with height.** Rising air expands as pressure falls, and expansion cools it. Convection keeps mixing the lower atmosphere (the **troposphere**), so its temperature falls at a nearly fixed rate, the **lapse rate**. The mean lapse rate is about **6.5 K per km**. + +Combine them: + +```text + altitude temperature + ~5 km ─── emission level ─── 255 K → emits 238 W/m² to space + │ air cools 6.5 K/km + │ as you go up + 0 km ─── surface ────────── 255 + 6.5 × 5 ≈ 288 K +``` + +The emission level must be at 255 K, because it must emit 238 W/m². The surface sits 5 km lower, on the same temperature slope, so it is about 33 K warmer. This matches the measured 288 K. + +## 6. Why adding CO2 warms the surface + +*Predict: the centre of the CO2 band already absorbs almost all 15 µm light within the lowest few hundred metres. Does adding more CO2 still matter?* + +It does. More CO2 means more absorbing gas above each altitude. So a photon must start higher before it can escape, and the emission level rises. The chain is: + +1. More CO2 → emission level rises. +2. Higher air is colder (lapse rate) → the emission level emits less (σT⁴). +3. Less energy leaves than arrives → the planet gains energy. +4. The troposphere and surface warm until the new, higher emission level reaches 255 K again. +5. The surface is now further below the emission level on the same slope → it is warmer. + +The imbalance in step 3 is the **radiative forcing**. A fit to detailed radiation calculations gives: + + ΔF ≈ 5.35 × ln(C / C₀) W/m², where C is the CO2 concentration and C₀ the starting value. + +The logarithm appears because the band centre is already opaque. Extra CO2 acts mostly at the edges of the band, where absorption is weaker, and each doubling widens the band by a similar amount. + +**Worked example: double CO2.** + +- Forcing: 5.35 × ln 2 ≈ **3.7 W/m²** less energy leaving. +- How much warming removes this? Differentiate P = σT⁴: dP/dT = 4σT³ = 4 × 5.67 × 10⁻⁸ × 255³ ≈ **3.8 W/m² per K**. +- Warming needed: 3.7 / 3.8 ≈ **1 K**, with nothing else changed. (Detailed calculations give about 1.2 K.) +- Equivalent rise of the emission level: 1 K / 6.5 K per km ≈ **150 m**. + +**Feedbacks amplify this.** Warmer air holds about 7% more water vapour per K, and water vapour is a greenhouse gas, so warming adds more absorption. Ice loss and clouds also respond. Water vapour cannot start a warming by itself, because extra water condenses and rains out within days; its amount follows temperature. CO2 stays in the air for centuries, so CO2 sets the level and water vapour amplifies it. Including feedbacks, the warming for doubled CO2 is *estimated* at **2.5 to 4 K** (IPCC AR6 likely range). + +## 7. When the mechanism does not warm: the stratosphere + +The argument in section 6 needs one assumption: the emission level must be colder than the air below it. Remove that assumption and the result breaks. + +- **Holds (troposphere):** temperature falls 6.5 K per km. Raising the emission level by 150 m makes it about 1 K colder, so it emits less and the surface must warm. +- **Breaks (isothermal air):** if temperature were the same at all heights, raising the emission level would not change its temperature. Emission would not change, and there would be no warming. +- **Reverses (stratosphere):** above about 12 km, ozone absorbs sunlight and temperature *rises* with height. At the very centre of the CO2 band, the emission level is already in the stratosphere. Here, more CO2 raises the emission level into warmer air, and the stratosphere cools. Satellites observe this stratospheric cooling at the same time as surface warming. A brighter Sun would warm both, so this pattern points to greenhouse gases as the cause. + +## 8. What the numbers mean + +- **238 W/m²** — the energy Earth must shed. It is fixed by sunlight and reflectivity, not by greenhouse gases. +- **255 K** — the temperature of the layer that sheds it. Greenhouse gases decide *where* that layer is, not its temperature. +- **5 km** — the emission height. Near 0 km, there would be almost no greenhouse effect (surface near 255 K). At 10 km, with the same lapse rate, the surface would be near 320 K. +- **3.7 W/m²** — about 1.5% of 238 W/m². A small change in the energy balance, held for decades, is enough to warm the surface by several kelvin. + +**Not covered here:** cloud effects in detail, how fast the oceans take up heat (the delay between forcing and warming), and the carbon cycle that sets CO2 levels. A glass greenhouse also works differently: it warms mostly because the glass stops convection, not because of infrared absorption. diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/model.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/model.md new file mode 100644 index 0000000..eb2714a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/model.md @@ -0,0 +1,35 @@ +# Model: how greenhouse gases warm the Earth + +**Central question:** Why is the surface warmer than sunlight alone predicts, and why does more CO2 make it warmer? + +**Reader:** technical, school physics, new to climate. Knows energy conservation and what a molecule is. Does not know Stefan–Boltzmann, Wien, lapse rate, emission height, forcing. + +**Representation:** Stage 1, controlled prose with one text figure. The argument is a sequential causal chain; one vertical profile figure carries the only spatial idea. + +## Entities +- Sunlight (short-wave, peak ~0.5 µm). Infrared from Earth (long-wave, peak ~10 µm). +- Greenhouse gases: CO2, H2O, CH4 (3+ atoms, vibrations change dipole). N2, O2: transparent to IR. +- Emission level: the altitude from which infrared escapes to space. +- Lapse rate: temperature fall with height in the troposphere, set by convection. + +## Causal chain +1. Absorbed sunlight 238 W/m² must be balanced by 238 W/m² of IR out. +2. A body that emits 238 W/m² has T = 255 K (Stefan–Boltzmann). +3. Surface is 288 K and emits 390 W/m². The atmosphere stops ~150 W/m² of it. +4. Greenhouse gases absorb and emit IR; IR reaches space mainly from the emission level (~5 km). +5. Air cools with height (6.5 K/km), so emission level at 255 K implies surface ≈ 288 K. +6. More CO2 raises the emission level → colder emitter → less IR out → imbalance → column warms until balance returns. + +## Confusion points +- "Gases reflect heat" — they absorb and re-emit. +- "CO2 is saturated" — the emission level still rises; band wings widen. +- One-layer model overshoots (303 K). +- Mechanism needs a lapse rate; with an isothermal atmosphere raising the emission level changes nothing (stratosphere case). + +## Epistemic status +- Steps 1–5: established physics and observation. +- 5.35 ln(C/C0): fit to radiative-transfer calculations. +- Climate sensitivity 2.5–4 K: estimate (IPCC AR6 likely range). + +## Scope +Out: clouds in detail, ocean heat uptake and time lags, carbon cycle, history of measurements. diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/model.yaml b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/model.yaml new file mode 100644 index 0000000..50d871d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/explainers/greenhouse/model.yaml @@ -0,0 +1,25 @@ +solar_constant_w_m2: 1361 +mean_incoming_w_m2: 340 +albedo: 0.30 +absorbed_solar_w_m2: 238 +sigma: 5.67e-8 +effective_temp_k: 255 +surface_temp_k: 288 +greenhouse_warming_k: 33 +surface_emission_w_m2: 390 +atmosphere_net_stop_w_m2: 150 +sun_temp_k: 5800 +sun_peak_um: 0.5 +earth_peak_um: 10 +wien_constant_um_k: 2898 +one_layer_surface_k: 303 +lapse_rate_k_per_km: 6.5 +emission_height_km: 5 +co2_band_um: 15 +forcing_coefficient: 5.35 +doubling_forcing_w_m2: 3.7 +planck_response_w_m2_k: 3.8 +no_feedback_warming_k: 1.0 +emission_rise_doubling_m: 150 +water_vapour_per_k_percent: 7 +sensitivity_likely_range_k: [2.5, 4] diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/read.json new file mode 100644 index 0000000..9046887 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "Sunlight comes in mostly as visible light near 0.5 µm. Earth gives off infrared near 10 µm. The two bands barely overlap, so greenhouse gases can let the incoming visible light through while absorbing the outgoing infrared.", + "quote": "The two bands hardly overlap. So the atmosphere can be nearly transparent to the incoming energy and nearly opaque to the outgoing energy." + }, + "2": { + "answer": "More CO2 puts more absorbing gas above each height, so infrared has to start higher before it can escape. The emission level therefore rises into colder air, which gives off less energy. The planet then gains energy until that higher level is back at 255 K. Because the surface now sits further below it on the same temperature slope, the surface ends up warmer.", + "quote": "More CO2 means more absorbing gas above each altitude. So a photon must start higher before it can escape, and the emission level rises." + }, + "3": { + "answer": "Each is made of two identical atoms. When they stretch, their charge stays symmetric, so no dipole forms and infrared light can't push them into vibrating. They are almost transparent to infrared.", + "quote": "When they stretch, the charge stays symmetric, so the dipole stays zero. They are almost transparent to infrared." + }, + "4": { + "answer": "Reasoning from the text: with nothing absorbing infrared, heat would escape straight from the surface, so the emission level would be near 0 km. The surface would then sit at about the plain balance temperature, around 255 K, with almost no greenhouse warming.", + "quote": "Near 0 km, there would be almost no greenhouse effect (surface near 255 K)." + } + }, + "stops": [ + "Sudden 'emission level' term needs reread: \"at each wavelength, the infrared that reaches space comes mostly from one altitude, the **emission level**\" — why one altitude, and how is 5 km averaged?", + "Why the layer emits equally both ways: \"The layer emits equally up and down.\"", + "Lapse rate reason is compressed: \"Rising air expands as pressure falls, and expansion cools it.\"", + "Puzzling claim: \"the centre of the CO2 band already absorbs almost all 15 µm light within the lowest few hundred metres\" — 'band' not defined.", + "Using 255 K rather than the surface temperature here is unexplained: \"dP/dT = 4σT³ = 4 × 5.67 × 10⁻⁸ × 255³\"", + "Logarithm reason is hard to follow: \"each doubling widens the band by a similar amount.\"", + "Unexplained jump: \"Equivalent rise of the emission level: 1 K / 6.5 K per km ≈ **150 m**.\"", + "No reason given for why the stratosphere cools when its emission rises into warmer air: \"more CO2 raises the emission level into warmer air, and the stratosphere cools.\"" + ], + "excess": [ + "\"Predict before you read on: will the surface be warmer than 255 K, colder, or the same?\"", + "\"Including feedbacks, the warming for doubled CO2 is *estimated* at **2.5 to 4 K** (IPCC AR6 likely range).\"", + "\"A glass greenhouse also works differently: it warms mostly because the glass stops convection\"", + "\"The T⁴ makes emission very sensitive to temperature: 10% warmer means about 46% more emission.\"", + "\"Not covered here:** cloud effects in detail, how fast the oceans take up heat\"", + "\"Satellites observe this stratospheric cooling at the same time as surface warming.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/run.json new file mode 100644 index 0000000..4fb140c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I wrote `explainers/greenhouse/explanation.md`, plus `model.md` and `model.yaml` as the source facts. I chose prose with one small text figure because the answer is one chain of causes with numbers, which prose handles best, and the only spatial idea, how air cools with height, fits in that figure. I couldn't run `explainer check` because I had no shell, so the numbers are hand-checked only.", + "error": null, + "cost": 0.29882359999999997, + "seconds": 89, + "turns": 4, + "files": [ + "explainers/greenhouse/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/explanation.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/explanation.md new file mode 100644 index 0000000..81b640b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/explanation.md @@ -0,0 +1,91 @@ +# How do greenhouse gases warm the Earth? + + + +**Greenhouse gases make the Earth radiate its energy to space from high, cold air instead of from the warm surface. Cold air emits less, so the whole lower atmosphere, and the surface with it, must be warmer before enough energy leaves.** + +The rest of this page puts numbers on each step. + +## 1. Without greenhouse gases: 255 K + + +Start from conservation of energy. Over years, the Earth must emit as much power to space as it absorbs from the Sun. This is the **energy balance**. + +- Sunlight delivers 1361 W/m² to a panel that faces the Sun. +- The Earth catches sunlight on a disc of area πR², but it emits from its whole sphere, of area 4πR². Per square metre of surface, the average input is 1361 / 4 = 340 W/m². +- The **albedo** is the fraction of sunlight that clouds, ice, and ground reflect straight back to space: 0.30. The Earth absorbs 340 × 0.70 ≈ 240 W/m². + +A body at temperature T emits σT⁴ watts per square metre (the Stefan–Boltzmann law, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴). Emission grows fast with temperature: 255 K emits 240 W/m², and 288 K emits 390 W/m². + +Set σT⁴ = 240 W/m² and solve: T = 255 K (−18 °C). This is the **emission temperature**: whatever emits the Earth's energy to space must be at 255 K. If the surface emitted directly to space, the surface would be at 255 K. + + +The division by 4 matters: with 1361 × 0.70 instead, T = 360 K (87 °C), as if every point on Earth faced the noon Sun all day. + +The measured average surface temperature is 288 K (15 °C). The surface is 33 K warmer than the emission temperature. The greenhouse gases cause that 33 K. + +## 2. What greenhouse gases do + + +Sunlight peaks near 0.5 µm, in the visible. The Earth is far cooler than the Sun, so it emits at longer wavelengths, near 10 µm. This is **infrared**: invisible radiation from the ground and the air. + +A **greenhouse gas** lets most sunlight pass but absorbs and emits infrared. Water vapour, CO₂, and methane are greenhouse gases. Nitrogen and oxygen, 99% of dry air, are not: they barely absorb infrared at all. + +So sunlight reaches the ground. But infrared from the ground is absorbed by the air above it and emitted again, in all directions, at the temperature of that air. + +## 3. Where the infrared escapes: 5 km up + + +Infrared reaches space only from air that has little greenhouse gas above it. Lower down, the air above absorbs it again. Infrared really escapes from a spread of heights, which differ by wavelength. Call their average the **emission height**. + +The air at the emission height must emit 240 W/m², so it must be at the emission temperature, 255 K. + +The air gets colder with height. Rising air expands and cools, and convection keeps mixing the lower atmosphere. The measured fall is about 6.5 K per km, the **lapse rate**. At this rate, the air reaches 255 K about 5 km up. That is the emission height. + +| Height | Temperature | +|---|---| +| 0 km (surface) | 288 K | +| 5 km (emission height) | 255 K | + +Now the 33 K has a cause: 255 K + 6.5 K/km × 5 km ≈ 288 K. The surface is warmer than the emitting air because it sits 5 km below it. + + +The surface at 288 K emits 390 W/m², more than the 240 W/m² that leaves the top. The surface does not lose that much, because the warm air just above it emits about 330 W/m² of infrared back down (a measured value). + +## 4. Adding CO₂ + +**Predict first:** CO₂ doubles, from its pre-industrial 280 ppm to 560 ppm. What happens to the emission height, and then to the surface? + + +1. More gas means the infrared must start higher before little gas remains above it. The emission height rises, on average by about 150 m (a result of radiation calculations). +2. The air 150 m higher is about 1 K colder (6.5 K/km × 0.15 km). +3. Colder air emits less: σ·254⁴ ≈ 236 W/m², against 240 W/m² at 255 K. The Earth now emits about 3.7 W/m² less than it absorbs. This imbalance, just after the change and before any warming, is the **radiative forcing**. It is small next to 240 W/m², but it persists every second until something changes. +4. The Earth gains energy and warms. Convection still sets the lapse rate, so the temperature at every height rises by the same amount. (This assumes the lapse rate stays at 6.5 K/km; in reality it changes a little, one of the feedbacks below.) +5. Warming stops when the air at the new emission height is back at 255 K. The surface is then about 1 K warmer, because it now sits 150 m further below the emitting air. + +This 1 K counts only this mechanism. Warmer air holds more water vapour, which is itself a greenhouse gas; with this and other feedbacks, the estimated warming per doubling is about 3 K, likely 2.5–4 K (assessment of the Intergovernmental Panel on Climate Change, IPCC, from models and observations). + +## 5. The condition: colder air above + + +The usual picture, "greenhouse gases trap heat like a blanket", says more gas always warms. A test case shows that it does not. + +Suppose the atmosphere had the same temperature at every height (a lapse rate of 0). Add CO₂. The emission height still rises, but the air there is still 255 K. It emits the same 240 W/m². There is no imbalance and no warming. + +So the warming depends on two things together: the gas lifts the emission height, and the air is colder higher up. + +## 6. "But CO₂ is already saturated" + + +Near the ground, CO₂ already absorbs almost all infrared at its main wavelength (15 µm). People conclude that more CO₂ can do nothing. But what reaches space is decided high up, not near the ground. High up, the air is thin, and infrared emitted there still escapes. Add CO₂, and that thin air absorbs more, so the escape moves up again, to colder air. Each doubling adds about the same forcing: 280 → 560 ppm gives 3.7 W/m², and 560 → 1120 ppm gives 3.7 W/m² again. Today's 420 ppm gives about 2.2 W/m². + +## Summary + +Energy balance requires the Earth to emit 240 W/m² to space, which holds the emitting air at 255 K. Greenhouse gases move that emitting air up to about 5 km. The lapse rate of 6.5 K/km makes the surface 33 K warmer, at 288 K. More greenhouse gas lifts the emission height and warms the surface. + +## Not covered here + +- Feedbacks in detail (water vapour, clouds, ice): IPCC AR6 Working Group I, Chapter 7. +- Why only some molecules absorb infrared: a molecular spectroscopy text. +- Why each doubling gives the same forcing: Pierrehumbert, *Principles of Planetary Climate*. +- Where added CO₂ goes (the carbon cycle), and how fast the ocean lets the surface warm. diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/model.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/model.md new file mode 100644 index 0000000..7a827c1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/model.md @@ -0,0 +1,146 @@ +# Semantic model: the greenhouse effect + +> Source of truth. Every rendering derives from this file. + +## Central question + +How do greenhouse gases make the Earth's surface warmer than it would be without them? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (energy, power in watts, temperature in kelvin, molecules, light as a spectrum of wavelengths, gases cool when they expand), basic programming. New to this subject. Knows that hot objects glow and that hotter objects radiate more. Does not know the Stefan–Boltzmann law, albedo, emission height, the lapse rate, or radiative forcing. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Sunlight | Radiation from the Sun, mostly visible light, wavelengths near 0.5 µm. | established | +| Infrared | Radiation that the Earth (ground and air) emits, wavelengths near 10 µm, invisible to the eye. | established | +| Albedo | The fraction of sunlight the Earth reflects straight back to space: 0.30. | observed | +| Energy balance | In a steady state, the power the Earth absorbs from sunlight equals the power it emits to space as infrared. | established (conservation of energy) | +| Stefan–Boltzmann law | A surface at temperature T emits power σT⁴ per square metre (σ = 5.67×10⁻⁸ W m⁻² K⁻⁴). | established | +| Emission temperature | The temperature that emits 240 W/m² by σT⁴: 255 K. Energy balance fixes it. | mathematical consequence | +| Greenhouse gas | A gas that absorbs and emits infrared: water vapour, CO₂, methane. N₂ and O₂ (99% of dry air) do not. | established | +| Emission height | The average altitude from which infrared escapes to space; above it, too little greenhouse gas remains to absorb it again. ≈ 5 km today. | established (a simplification of a spread of heights across wavelengths) | +| Lapse rate | The fall of air temperature with height in the lower atmosphere: about 6.5 K per km. | observed average | +| Radiative forcing | The imbalance (W/m²) at the top of the atmosphere just after a change, such as added CO₂, before the Earth warms. | established; value is model output | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Sunlight | passes through | atmosphere | greenhouse gases absorb little of it | +| Surface | emits | infrared | σT⁴ | +| Greenhouse gas | absorbs and re-emits in all directions | infrared | part goes back down to the surface | +| Greenhouse gas amount | sets | emission height | more gas → higher emission height | +| Convection | holds | lapse rate at ≈ 6.5 K/km | rising air expands and cools | +| Energy balance | fixes | air at emission height at 255 K | | +| Lapse rate × emission height | adds to 255 K to give | surface temperature | 255 + 6.5 × 5.08 ≈ 288 K | + +## Dependencies (learning order) + +1. Energy balance and Stefan–Boltzmann → emission temperature 255 K; surface would be 255 K with no greenhouse gas. +2. Sunlight vs infrared; which gases absorb infrared. +3. Infrared escapes from the emission height, not the surface. +4. Lapse rate ties the surface to the air at the emission height. +5. Adding gas raises emission height → imbalance → warming. + +## Causal chain + +```text +more greenhouse gas → infrared escapes from higher up → the air there is colder → it emits less to space +→ Earth absorbs more than it emits (radiative forcing) → Earth warms +→ warming stops when the air at the new emission height is back at 255 K +→ the lapse rate holds, so the surface is warmer by (6.5 K/km × rise in emission height) +``` + +## Temporal sequence + +1. CO₂ is added. Emission height rises at once. +2. Outgoing infrared drops; the Earth gains energy (forcing ≈ 3.7 W/m² for doubled CO₂). +3. Surface and air warm over years to decades (the ocean slows it). The imbalance shrinks as they warm. +4. Balance returns when the air at the emission height is at 255 K again. + +## Quantities + +- Solar constant: 1361 W/m² on a surface facing the Sun. +- Average over the sphere: 1361 / 4 = 340 W/m². +- Albedo 0.30. Absorbed: 340 × 0.70 ≈ 240 W/m². +- Emission temperature: σT⁴ = 240 → T ≈ 255 K (−18 °C). +- Observed mean surface temperature: 288 K (15 °C). Difference: 33 K. +- Surface at 288 K emits σ·288⁴ ≈ 390 W/m²; only 240 W/m² leaves to space. +- Surface budget (rounded, observed estimates): gains 160 W/m² sunlight + 330 W/m² infrared from the air = 490; loses 390 W/m² infrared + 100 W/m² convection and evaporation = 490. +- Peak wavelengths: Sun (5800 K) ≈ 0.5 µm; Earth (288 K) ≈ 10 µm. +- Lapse rate 6.5 K/km (dry rising air cools 9.8 K/km; condensing water releases heat and reduces it). 33 K / 6.5 K/km ≈ 5 km: average emission height. +- Doubling CO₂: forcing ≈ 5.35 × ln 2 ≈ 3.7 W/m². 280 → 420 ppm: 5.35 × ln 1.5 ≈ 2.2 W/m². +- Air 1 K colder (254 K vs 255 K) emits about 3.8 W/m² less. 1 K at 6.5 K/km is about 150 m of rise. +- Warming that restores balance, no feedbacks: 3.7 / 3.76 ≈ 1 K, where 3.76 W/m² per K = 4σ·255³ is the extra emission per 1 K of warming. +- With feedbacks (mainly water vapour): about 3 K per doubling (IPCC AR6 best estimate; likely 2.5–4 K). +- CO₂: about 280 ppm in 1750, about 420 ppm in 2024. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| No greenhouse gas | surface 255 K | surface emits straight to space | +| Present atmosphere | surface 288 K | emission height ≈ 5 km, 33 K colder than surface | +| Doubled CO₂, no feedbacks | ≈ +1 K | emission height rises ≈ 150 m | +| Isothermal atmosphere (no lapse rate) | added gas causes no warming | emission height rises but the air there is not colder | +| Stratosphere (above ≈ 12 km, temperature rises with height) | added CO₂ cools the stratosphere | observed | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Divide solar constant by 4 | Use 1361 W/m² directly | σT⁴ = 1361 × 0.7 → T ≈ 360 K (87 °C). The Earth intercepts sunlight on a disc of area πR² but emits from a sphere of area 4πR². | +| Explain by emission height + lapse rate | "Gases trap heat like a blanket" | A blanket picture says more gas always warms and gives no number. Isothermal air is a counterexample: emission height rises, emitting air is still 255 K, outgoing stays 240 W/m², no warming. The trapping picture also suggests saturation ends the effect; it does not. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| More greenhouse gas warms the surface (assumes temperature falls with height) | Emission height 5.08 → 5.23 km; air there 1 K colder emits ≈ 3.7 W/m² less; Earth warms until that air is 255 K; surface 288 → 289 K | Isothermal air: emitting air is 255 K at any height → no imbalance, no warming. Stratosphere, where temperature rises with height: added CO₂ cools it. | +| Mechanism: energy balance sets 255 K | 340 × 0.7 = 238 ≈ 240 W/m²; σ·255⁴ ≈ 240 W/m² | — | +| Mechanism: more gas raises emission height | infrared escapes where little gas remains above; with twice the gas, that amount of gas above lies higher | at wavelengths no gas absorbs (8–13 µm "window", mostly) the surface emits straight to space and nothing changes; averaged over all wavelengths the rise is ≈ 150 m | +| Saturation does not end the effect | 280 → 560 ppm: 3.7 W/m²; 560 → 1120 ppm: again 3.7 W/m² | — | + +## Terms + +- **Sunlight:** short-wave radiation from the Sun (near 0.5 µm). +- **Infrared:** long-wave radiation the Earth emits (near 10 µm). Here, "infrared" always means the Earth's. +- **Albedo:** fraction of sunlight reflected. 0.30 → 240 W/m² absorbed; at 0, 340 W/m² would be absorbed. +- **Greenhouse gas:** a gas that absorbs and emits infrared. +- **Energy balance:** absorbed power = emitted power. +- **Emission temperature:** 255 K, the temperature that emits 240 W/m². One name only; not "balance temperature", not "no-atmosphere temperature". +- **Emission height:** the average altitude from which infrared escapes to space; ≈ 5 km today. 0 km: no greenhouse gas, surface 255 K. Each extra km: surface 6.5 K warmer. +- **Lapse rate:** 6.5 K drop per km. If 0, added gas does nothing; if larger, the same rise warms more. +- **Radiative forcing:** the imbalance just after a change, before the Earth warms. 3.7 W/m² for doubled CO₂ (compare 240 W/m² absorbed). It shrinks to 0 as the Earth warms. + +## Epistemic status + +- **Observed / established:** solar constant, albedo, surface 288 K, CO₂ concentrations, absorption spectra, lapse rate average, stratospheric cooling, surface budget (rounded). +- **Mathematical consequence:** 255 K from σT⁴ = 240; ≈ 1 K no-feedback warming. +- **Assumed:** a single emission height (really a spread across wavelengths); a fixed lapse rate. +- **Estimated / model output:** 3.7 W/m² forcing (radiation models); 3 K with feedbacks (IPCC assessment, likely 2.5–4 K). +- **Disputed:** that the effect exists is not disputed; the size of cloud feedback is the main uncertainty. + +## Confusion points + +- "CO₂ already absorbs all the infrared at its main wavelength, so more does nothing" → more gas still raises the emission height; each doubling adds about the same 3.7 W/m². +- "Greenhouse gases trap heat like a blanket" → the gases absorb and re-emit; warming comes from escape at a colder height. (A real glass greenhouse works mainly by stopping warm air from rising: a different mechanism.) +- "N₂ and O₂ are most of the air, so they matter most" → they do not absorb infrared. +- "390 W/m² leaves the surface but only 240 reaches space — energy is lost" → the air sends 330 W/m² back down; the surface budget balances. + +## Scope + +- **Out of scope:** feedbacks in detail (water vapour, clouds, ice) — IPCC AR6 WG1 Chapter 7. +- **Out of scope:** why a molecule must change its charge separation when it vibrates to absorb infrared — a molecular spectroscopy text. +- **Out of scope:** the carbon cycle and where added CO₂ goes. +- **Out of scope:** why forcing is logarithmic in CO₂ — Pierrehumbert, *Principles of Planetary Climate*. +- **Out of scope:** how fast warming happens (ocean heat uptake). + +## Representation decision + +- **Stage:** 1 +- **Reason:** The argument is a short causal chain with arithmetic. One table of numbers carries the comparison. No parameter needs exploration that a worked example with numbers does not give. +- **Levels:** L1 gases make the surface 33 K warmer / L2 emission from a colder height / L3 energy balance and lapse rate / L4 forcing numbers. diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/model.yaml b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/model.yaml new file mode 100644 index 0000000..a77b8ef --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/model.yaml @@ -0,0 +1,119 @@ +# Machine-readable half of model.md. `explainer check greenhouse` holds every rendering to it. + +values: + solar_constant_wm2: 1361 + sunlight_avg_wm2: 340 + albedo: 0.30 + absorbed_wm2: 240 + emission_temp_k: 255 + emission_temp_c: -18 + surface_temp_k: 288 + surface_temp_c: 15 + greenhouse_warming_k: 33 + surface_emits_wm2: 390 + surface_sun_gain_wm2: 160 + back_radiation_wm2: 330 + convection_wm2: 100 + surface_budget_wm2: 490 + sunlight_um: 0.5 + infrared_um: 10 + lapse_rate_k_per_km: 6.5 + emission_height_km: 5 + forcing_2xco2_wm2: 3.7 + forcing_now_wm2: 2.2 + height_rise_m: 150 + warming_no_feedback_k: 1 + warming_with_feedback_k: 3 + warming_likely_range_k: [2.5, 4] + sigma: 5.67e-8 + absorbed_fraction: 0.70 + sun_temp_k: 5800 + temp_at_2km_k: 275 + height_rise_km: 0.15 + emission_at_254k_wm2: 236 + wrong_temp_k: 360 + wrong_temp_c: 87 + co2_1750_ppm: 280 + co2_2024_ppm: 420 + co2_double_ppm: 560 + n2_o2_share_pct: 99 + +allow: [4, 2, 1750, 2024, 254, 12, 1120] + +require: + - path: emission_temp_k + in: [prose] + - path: surface_temp_k + in: [prose] + - path: absorbed_wm2 + in: [prose] + - path: lapse_rate_k_per_km + in: [prose] + - path: emission_height_km + in: [prose] + - path: forcing_2xco2_wm2 + in: [prose] + +claims: + - id: balance-255 + kind: mechanism + statement: "Energy balance fixes the temperature of whatever emits to space at 255 K: absorbed 240 W/m² = σT⁴." + example: "1361/4 = 340 W/m²; ×0.70 = 240 W/m²; σ·255⁴ ≈ 240 W/m². Observed surface is 288 K, 33 K warmer." + - id: why-divide-4 + kind: why + about: [divide-by-4] + statement: "Sunlight is averaged over the sphere: the Earth intercepts it on a disc πR² but emits from a sphere 4πR²." + alternative: "Use 1361 W/m² directly." + counterexample: "σT⁴ = 1361 × 0.7 gives T ≈ 360 K (87 °C), as if every point faced the noon Sun all day." + - id: which-gases + kind: definition + statement: "Greenhouse gases absorb and emit infrared (~10 µm) but let sunlight (~0.5 µm) through; N₂ and O₂ do not absorb infrared." + - id: emission-height + kind: mechanism + statement: "Infrared escapes to space from the emission height, about 5 km up, where too little greenhouse gas remains above to absorb it again; the air there must be 255 K." + example: "Surface 288 K = 255 K + 6.5 K/km × 5.08 km." + ask: "Where does the infrared that reaches space mostly come from, what is the temperature there, and why that height and that temperature?" + - id: more-gas-warms + kind: guarantee + statement: "Adding greenhouse gas raises the emission height; because temperature falls with height, the air there is colder and emits less, so the Earth warms until that air is back at 255 K." + example: "Doubled CO₂: emission height rises ~150 m; that air is 1 K colder and emits ~3.7 W/m² less; the surface warms ~1 K before feedbacks." + counterexample: "In isothermal air the higher emitting air is still 255 K: outgoing stays 240 W/m², no warming. In the stratosphere, where temperature rises with height, added CO₂ cools." + ask: "Explain, step by step, how doubling CO₂ in today's atmosphere leads to a warmer surface, and when the warming stops." + - id: why-not-blanket + kind: why + about: [blanket] + statement: "The 'blanket that traps heat' picture is replaced by emission from a colder height." + alternative: "Gases trap heat like a blanket." + counterexample: "The blanket picture predicts warming in isothermal air (there is none) and predicts that saturated absorption ends the effect (each doubling still adds 3.7 W/m²)." + - id: surface-budget + kind: mechanism + statement: "The surface emits 390 W/m², more than the 240 W/m² that reaches space, because the air sends 330 W/m² of infrared back down." + example: "Gains 160 sunlight + 330 from the air = 490; loses 390 infrared + 100 convection and evaporation = 490." + - id: saturation + kind: misconception + statement: "Saturation does not stop the effect: more gas still raises the emission height; each doubling adds about 3.7 W/m²." + correction: "Escape happens high up where the gas is thin; more gas still raises the emission height. 280→560 ppm and 560→1120 ppm each give about 3.7 W/m²." + ask: "CO₂ already absorbs nearly all infrared at its main wavelength near the ground. Does adding more CO₂ still warm the Earth? Why?" + +accept_unjustified: {ln: "forcing formula 5.35 ln(C/C0) only computes numbers; why it is logarithmic is out of scope"} + +terms: + - term: emission temperature + means: "255 K, the temperature that emits 240 W/m²" + avoid: [balance temperature, no-atmosphere temperature, effective temperature] + - term: infrared + means: "the long-wave radiation the Earth emits" + avoid: [heat radiation, longwave radiation] + +budget: {prose: 1200, reason: "the mechanism needs four linked steps (balance, emission height, lapse rate, adding gas), each with numbers, plus the counterexample and the saturation misconception"} + +quiz: + - q: "Without greenhouse gases, what would the Earth's average surface temperature be, roughly, and what sets that number?" + expect: "About 255 K (−18 °C): absorbed sunlight (240 W/m²) must equal emitted infrared σT⁴." + misconception: "Absolute zero, or the same as today." + - q: "Adding CO₂ makes the Earth emit less infrared to space at first. Why?" + expect: "Infrared escapes from a higher altitude, where the air is colder, and colder air emits less (σT⁴)." + misconception: "CO₂ traps heat / reflects heat back like a mirror, with no role for altitude or temperature falling with height." + - q: "If the atmosphere had the same temperature at every height, would adding CO₂ warm the surface?" + expect: "No: the emission height rises but the air there is at the same temperature, so outgoing infrared does not change." + misconception: "Yes, because more CO₂ always traps more heat." diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/narrative.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/narrative.md new file mode 100644 index 0000000..1988c44 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/narrative.md @@ -0,0 +1,57 @@ +# Narrative: greenhouse + +> Pass 3. Every rendering follows these beats. + +## 1. Reader before and after + +- **Before:** a technical reader (school maths and science, basic programming), new to climate physics. Knows watts, kelvin, molecules, wavelengths, that hotter things radiate more, that a gas cools when it expands. +- **After:** "The Earth must emit 240 W/m² to space, which fixes the temperature of the air that emits it at 255 K. Greenhouse gases move that emitting air up to about 5 km, and the surface is 6.5 K warmer per km below it. More gas raises the emitting air, so the surface warms." + +## 2. Question and motive + +- **Question:** How do greenhouse gases warm the Earth? Result in words: they make the Earth emit its infrared to space from high, cold air instead of from the warm surface; to emit enough from cold air, the whole column (and the surface) must be warmer. +- **Why care:** the familiar "trapping heat" picture cannot say how much, and it suggests the effect stops once CO₂ absorbs everything. The correct picture gives numbers and answers that objection. +- **Why this approach:** start from energy conservation, the one thing no reader doubts. It gives a temperature (255 K) that disagrees with the measured 288 K. The 33 K gap is the thing to explain. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| W/m² | power per square metre of the Earth's surface | 1361 W/m² of sunlight on a panel facing the Sun | 2 | +| albedo | fraction of sunlight reflected | 0.30 → 30% of 340 W/m² reflected | 2 | +| energy balance | absorbed = emitted | 240 in, 240 out | 2 | +| σT⁴ (Stefan–Boltzmann) | power per m² a body emits at temperature T | 255 K → 240 W/m²; 288 K → 390 W/m² | 2 | +| emission temperature | 255 K, the temperature that emits 240 W/m² | σ·255⁴ ≈ 240 | 2 | +| infrared | radiation the Earth emits, near 10 µm | sunlight peaks near 0.5 µm, 20× shorter | 3 | +| greenhouse gas | gas that absorbs and emits infrared | CO₂, water vapour, methane; not N₂, O₂ | 3 | +| emission height | average height from which infrared escapes to space | ≈ 5 km | 4 | +| lapse rate | temperature drop per km of height, 6.5 K/km | 288 − 6.5 × 5 ≈ 255 | 4 | +| radiative forcing | imbalance right after a change, before warming | 3.7 W/m² for doubled CO₂ | 5 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How do greenhouse gases warm the Earth? | — | answer in two sentences | gases move the emission of infrared up into cold air; the surface must be warmer so that cold air emits enough | the result in words | +| 2 | instance | What temperature would the Earth have without them? | start from energy | sunlight 1361 W/m² on a panel facing the Sun; ÷4 because a disc πR² catches it and a sphere 4πR² emits (without ÷4: 360 K); albedo 0.30 reflected → 240 W/m² absorbed; σT⁴ = power per m² a body emits at temperature T; σT⁴ = 240 → 255 K. Measured surface: 288 K | energy balance fixes 255 K for whatever emits to space; the measured surface is 288 K | 33 K to explain | +| 3 | name | What is different about greenhouse gases? | so the atmosphere must change where the emission comes from | sunlight 0.5 µm vs infrared 10 µm (a cooler body emits at longer wavelengths); CO₂, water vapour, methane absorb infrared; N₂, O₂ do not | the gases pass sunlight, absorb and re-emit infrared | the gases act on outgoing infrared only | +| 4 | mechanism | Then where does the 240 W/m² leave from? | but absorbed infrared is re-emitted in all directions; only infrared emitted where little gas remains above gets out | table: height vs temperature | the air that emits to space must emit 240 W/m², so it must be 255 K. Measured air reaches 255 K about 5 km up: the emission height. Air cools 6.5 K/km because rising air expands and cools, and convection keeps this profile. 255 + 6.5 × 5 ≈ 288 | the 33 K is the lapse rate times the emission height | +| 5 | objection | The surface emits σ·288⁴ ≈ 390 W/m², but only 240 leaves the top. Where does the rest go? | but | surface budget, each term named | the air emits 330 W/m² of infrared down. Surface gains 160 sunlight + 330 from the air; loses 390 infrared + 100 carried up by rising warm air and evaporation | the numbers add up | +| 6 | test (predict) | What happens when CO₂ doubles (280 → 560 ppm)? | therefore | prediction prompt, then: emission height rises ≈ 150 m; that air is 1 K colder; it emits 3.7 W/m² less, the radiative forcing; the Earth warms until the emitting air is 255 K: surface ≈ 1 K warmer. One line: water vapour and other feedbacks raise this to ≈ 3 K (out of scope) | more gas → higher → colder → less out → warm | the causal chain with numbers | +| 7 | test | Does more gas always warm? | but the mechanism needs colder air above | isothermal case | if air had the same temperature at every height, the higher emitting air would still be 255 K: outgoing unchanged, no warming | the assumption that matters | +| 8 | objection | Isn't CO₂ already saturated? | so | 280 → 560 ppm: 3.7 W/m²; 560 → 1120 ppm: 3.7 W/m² again | saturation near the ground does not matter, because escape happens high up, where the gas is thin; more gas still lifts the emission height. Each doubling adds about the same forcing | the saturation answer | +| 9 | close | — | so | recap | "Greenhouse gases move the Earth's emission to space up to about 5 km, where the air is 33 K colder than the surface. Energy balance holds that air at 255 K, so the surface sits at 288 K. More gas lifts the emission height and warms the surface." Scope: feedbacks, carbon cycle, timing | the "after" sentence | + +## 5. Concrete to symbol + +σT⁴: first 255 K → 240 W/m² and 288 K → 390 W/m², then the symbol with σ = 5.67×10⁻⁸. Surface = 255 + 6.5 × h: first with h = 5 km (288 K), then h = 5.15 km (289 K). + +## 6. Links between representations + +The height-temperature table and the sentence "255 K at 5 km" use the same numbers; the doubled-CO₂ row repeats the table with the emission height moved. + +## 7. Close the loop + +- **Answer:** Greenhouse gases make the Earth emit to space from high, cold air. Energy balance holds that air at 255 K, and the lapse rate makes the surface warmer by 6.5 K per km below it. +- **General argument:** any change that raises the emission height, while temperature falls with height, warms the surface. +- **Payoff:** the reader can predict the effect of more CO₂, of no lapse rate, and answer the saturation objection. Scope: feedbacks, carbon cycle, timing. diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/decisions.yaml new file mode 100644 index 0000000..b8d38d7 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/decisions.yaml @@ -0,0 +1,532 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T152224-probe#1: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#10: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#11: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#12: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#13: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#14: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#15: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#2: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#3: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#4: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#5: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#6: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#7: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#8: + decision: declined + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152224-probe#9: + decision: adopted + note: '8: T^4 is a stated physical law the reader accepts; row removed. 6,10: scope. + 12,13,15 kept: motive, scope pointer, breaking case of the guarantee.' +2026-10-04T152459-coldread-narrative#1: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#10: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#11: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#12: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#13: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#14: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#15: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#16: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#17: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#18: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#19: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#2: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#20: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#21: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#22: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#23: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#24: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#25: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#26: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#27: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#28: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#29: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#3: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#30: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#31: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#32: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#33: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#34: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#35: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#36: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#37: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#38: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#39: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#4: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#40: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#41: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#42: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#43: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#44: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#45: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#46: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#47: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#48: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#49: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#5: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#50: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#51: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#52: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#53: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#54: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#6: + decision: declined + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#7: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#8: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152459-coldread-narrative#9: + decision: adopted + note: '6 declined: one-line feedback pointer answers the reader''s obvious ''how + much really'' question. Remaining: tabular shorthand; rendering defines each term.' +2026-10-04T152723-coldread-prose#1: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#10: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#2: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#3: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#4: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#5: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#6: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#7: + decision: declined + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#8: + decision: adopted + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152723-coldread-prose#9: + decision: declined + note: '1: budget table replaced by one sentence. 7,9 resolved by the same edits.' +2026-10-04T152840-quiz-prose#1: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#10: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#11: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#12: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#13: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#14: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#15: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#16: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#17: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#18: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#19: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#2: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#20: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#21: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#22: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#23: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#3: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#4: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#5: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#6: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#7: + decision: adopted + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#8: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T152840-quiz-prose#9: + decision: declined + note: '12 scored 1 because the ask did not request a reason: asks rewritten, duplicates + removed. 6,9,11,13,18,21,22: out of scope (logarithm, feedbacks). 14,15: carry + the why-divide-4 claim and the probe''s main surface-budget finding.' +2026-10-04T153030-quiz-prose#1: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#10: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#11: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#12: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#13: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#14: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#15: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#16: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#17: + decision: adopted + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#18: + decision: adopted + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#19: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#2: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#20: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#21: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#22: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#3: + decision: adopted + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#4: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#5: + decision: adopted + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#6: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#7: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#8: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' +2026-10-04T153030-quiz-prose#9: + decision: declined + note: 'Item 5 partial: labeled emission height as an average of a spread, 150 m + and 3 K as calculation/assessment, removed unmodeled satellite clause. Rest: scope + (logarithm, feedbacks, timing) or already declined.' diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152224-probe.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152224-probe.json new file mode 100644 index 0000000..959d08f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152224-probe.json @@ -0,0 +1,212 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "greenhouse", + "at": "2026-10-04T15:22:24", + "model": "default", + "prompt_sha": "5539eb6fbccb", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "Why the surface warms by exactly 'lapse rate × rise in emission height'. Nothing says why air gets colder with height, or what links the surface temperature to the temperature at the emission height.", + "why_it_matters": "The causal chain jumps from 'the emission layer is back at 255 K' to 'the surface is warmer'. That only follows if convection holds the temperature profile at a fixed 6.5 K/km, so the whole profile shifts together. The lapse rate is listed only as an 'observed average', and the reader is never told why higher air is colder. The step the central answer depends on is left unstated.", + "suggested_addition": "Add: rising air expands at lower pressure and cools as it does (dry air cools about 9.8 K/km; moist air about 6.5 K/km on average because condensing water releases heat). Convection keeps this profile in place, so surface temperature = emission temperature + 6.5 K/km × emission height. Today that is 255 + 6.5×5.08 ≈ 288 K. After doubling: 255 + 6.5×5.23 ≈ 289 K, so the surface is about 1 K warmer. State that this assumes the lapse rate stays fixed." + }, + { + "rule": 2, + "severity": "main", + "about": "Why more greenhouse gas raises the emission height, and why the rise is only about 150 m", + "why_it_matters": "The relationship 'more gas → higher emission height' is the key step of the mechanism, but the model only asserts it. It gives no reason and no worked example. The 150 m figure is worked backwards from the forcing, not from the gas itself.", + "suggested_addition": "Add: infrared escapes from roughly the level where the gas above it can absorb it only about once (optical depth ≈ 1). Air thins by a factor e every ~8 km (the scale height). So doubling a well-mixed gas moves that level up by about 8 km × ln 2 ≈ 5.5 km, but only at wavelengths where the gas absorbs and the air at that level is still colder than below. At most wavelengths nothing changes. Averaged over the whole spectrum, the effective rise is ≈ 150 m (3.7 W/m² ÷ 3.76 W/m² per K ÷ 6.5 K/km ≈ 0.15 km)." + }, + { + "rule": 4, + "severity": "main", + "about": "The surface emits 390 W/m² but only 240 W/m² leaves to space. The model never says where the other 150 W/m² goes.", + "why_it_matters": "A reader who sees these two numbers will ask where the difference went. Without back-radiation, the numbers look like a violation of the energy balance the model itself sets up.", + "suggested_addition": "Add a surface budget with numbers: the surface absorbs ≈160 W/m² of sunlight plus ≈340 W/m² of infrared emitted downward by the greenhouse gases. It loses ≈390 W/m² as infrared plus ≈100 W/m² carried upward by convection and evaporation. Only ≈240 W/m² leaves the top of the atmosphere, from ≈5 km up." + }, + { + "rule": 3, + "severity": "main", + "about": "One quantity, 255 K, has several names: 'no-atmosphere temperature', 'balance temperature', 'emission temperature', and 'the new emission height is back at 255 K'", + "why_it_matters": "The reader may take these for different quantities. 'Emission temperature' appears in Relationships, but it is defined only in the last Terms section and is missing from Entities.", + "suggested_addition": "Use one name, 'emission temperature (255 K)'. Define it in Entities before Relationships: the temperature that radiates 240 W/m² by σT⁴. Change 'emission height is back at 255 K' to 'the air at the emission height is back at 255 K'." + }, + { + "rule": 3, + "severity": "edge", + "about": "Two pairs of names, and terms used without definition", + "why_it_matters": "Two concepts each have two names: 'imbalance' vs 'radiative forcing', and 'emission height' vs 'emitting layer'/'layer'/'that air'. Albedo, Wien's law, feedbacks, 'saturated' and 'band' are used but never defined. The Audience section says the reader does not know Wien's law.", + "suggested_addition": "Say that forcing is the imbalance at the moment CO₂ is added (3.7 W/m²), and that the imbalance shrinks toward 0 as the Earth warms. Define albedo as the fraction of sunlight reflected (0.30 → 240 W/m² absorbed; 0.0 would give 340 W/m² and ≈278 K). Define feedback, band and saturation in one line each, or drop Wien's formula." + }, + { + "rule": 2, + "severity": "edge", + "about": "The guarantee 'energy balance sets 255 K' has no example where it breaks", + "why_it_matters": "Rule 2 asks for a case where removing the assumption breaks the guarantee. The table leaves that cell as '—'. The two assumptions are steady state and fixed albedo.", + "suggested_addition": "Breaks without steady state: today the Earth absorbs ≈1 W/m² more than it emits, so emission is about 239 W/m² and the effective temperature is ≈254.7 K, not 255 K. Breaks without fixed albedo: at albedo 0.25, absorbed is 255 W/m² and T ≈ 259 K." + }, + { + "rule": 2, + "severity": "edge", + "about": "The claim that 'each doubling adds about the same 3.7 W/m²' (logarithmic) has no worked example and no case where it fails", + "why_it_matters": "This is an always-type claim, used in the saturation confusion point.", + "suggested_addition": "280→560 ppm: 5.35 ln 2 ≈ 3.7 W/m². 560→1120 ppm: also 3.7 W/m². 280→420 ppm: 5.35 ln 1.5 ≈ 2.2 W/m². It breaks at very low concentrations, where absorption grows roughly in proportion to the amount, so from 0→280 ppm the log formula would give an infinite forcing." + }, + { + "rule": 1, + "severity": "edge", + "about": "The 'why T⁴' argument is circular", + "why_it_matters": "The 390 W/m² figure is itself computed with σT⁴, so it cannot show that T⁴ is needed rather than linear T.", + "suggested_addition": "Say that 390 W/m² is a measured surface emission (≈ 396 W/m² in observed budgets). Show that a linear law fitted to 240 W/m² at 255 K would predict 240 × 288/255 ≈ 271 W/m²." + }, + { + "rule": 5, + "severity": "edge", + "about": "The meaning of 3.76 W/m² per K (the Planck response) and of the 3 K climate sensitivity", + "why_it_matters": "These numbers appear in the Quantities section, but the model never says what a low or high value would mean for the reader.", + "suggested_addition": "3.76 W/m² per K is the extra emission for each 1 K of warming. A larger value means less warming per W/m² of forcing. Sensitivity of 2.5 K vs 4 K means 2.5 K vs 4 K of eventual warming per doubling of CO₂." + }, + { + "rule": 4, + "severity": "edge", + "about": "The temporal sequence step 'warm over years to decades (the ocean slows it)' has no numbers", + "why_it_matters": "It is a mechanism step with no worked example.", + "suggested_addition": "With 3.7 W/m² heating a 100 m ocean mixed layer (heat capacity ≈ 4.2×10⁸ J/m²/K), the first 1 K takes about 4.2×10⁸ / 3.7 ≈ 1.1×10⁸ s ≈ 3.6 years if no extra heat were emitted. In reality it takes longer, because emission grows as the Earth warms and heat mixes into the deep ocean." + } + ], + "excess": [ + "Wien's law constant 2898 µm·K (only the two peak wavelengths are needed)", + "CO₂ history 280 ppm (1750) / 420 ppm (2024)", + "IPCC 2.5–4 K likely range and feedback-inclusive 3 K (feedbacks are declared out of scope)", + "Ozone-hole confusion point", + "Stratospheric cooling as a fingerprint (useful but not needed for the central question)" + ], + "next_questions": [ + { + "q": "Why is the air colder higher up, and why does the surface stay tied to the emission-height temperature?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Where do the other 150 W/m² that the surface emits go? Does the atmosphere radiate back down?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why does adding gas raise the emission height, and why by only about 150 m?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If no-feedback warming is about 1 K, why is the expected warming 3 K, and how much warming should 280→420 ppm already have caused?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "If CO₂ absorption is saturated, why does each doubling still add about 3.7 W/m²?", + "answered_by_model": true, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.1644216, + "seconds": 59 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "Why the surface warms by exactly 'lapse rate × rise in emission height'. Nothing says why air gets colder with height, or what links the surface temperature to the temperature at the emission height. → Add: rising air expands at lower pressure and cools as it does (dry air cools about 9.8 K/km; moist air about 6.5 K/km on average because condensing water releases heat). Convection keeps this profile in place, so surface temperature = emission temperature + 6.5 K/km × emission height. Today that is 255 + 6.5×5.08 ≈ 288 K. After doubling: 255 + 6.5×5.23 ≈ 289 K, so the surface is about 1 K warmer. State that this assumes the lapse rate stays fixed." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "Why more greenhouse gas raises the emission height, and why the rise is only about 150 m → Add: infrared escapes from roughly the level where the gas above it can absorb it only about once (optical depth ≈ 1). Air thins by a factor e every ~8 km (the scale height). So doubling a well-mixed gas moves that level up by about 8 km × ln 2 ≈ 5.5 km, but only at wavelengths where the gas absorbs and the air at that level is still colder than below. At most wavelengths nothing changes. Averaged over the whole spectrum, the effective rise is ≈ 150 m (3.7 W/m² ÷ 3.76 W/m² per K ÷ 6.5 K/km ≈ 0.15 km)." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 4", + "text": "The surface emits 390 W/m² but only 240 W/m² leaves to space. The model never says where the other 150 W/m² goes. → Add a surface budget with numbers: the surface absorbs ≈160 W/m² of sunlight plus ≈340 W/m² of infrared emitted downward by the greenhouse gases. It loses ≈390 W/m² as infrared plus ≈100 W/m² carried upward by convection and evaporation. Only ≈240 W/m² leaves the top of the atmosphere, from ≈5 km up." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 3", + "text": "One quantity, 255 K, has several names: 'no-atmosphere temperature', 'balance temperature', 'emission temperature', and 'the new emission height is back at 255 K' → Use one name, 'emission temperature (255 K)'. Define it in Entities before Relationships: the temperature that radiates 240 W/m² by σT⁴. Change 'emission height is back at 255 K' to 'the air at the emission height is back at 255 K'." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 3", + "text": "Two pairs of names, and terms used without definition → Say that forcing is the imbalance at the moment CO₂ is added (3.7 W/m²), and that the imbalance shrinks toward 0 as the Earth warms. Define albedo as the fraction of sunlight reflected (0.30 → 240 W/m² absorbed; 0.0 would give 340 W/m² and ≈278 K). Define feedback, band and saturation in one line each, or drop Wien's formula." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 2", + "text": "The guarantee 'energy balance sets 255 K' has no example where it breaks → Breaks without steady state: today the Earth absorbs ≈1 W/m² more than it emits, so emission is about 239 W/m² and the effective temperature is ≈254.7 K, not 255 K. Breaks without fixed albedo: at albedo 0.25, absorbed is 255 W/m² and T ≈ 259 K." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 2", + "text": "The claim that 'each doubling adds about the same 3.7 W/m²' (logarithmic) has no worked example and no case where it fails → 280→560 ppm: 5.35 ln 2 ≈ 3.7 W/m². 560→1120 ppm: also 3.7 W/m². 280→420 ppm: 5.35 ln 1.5 ≈ 2.2 W/m². It breaks at very low concentrations, where absorption grows roughly in proportion to the amount, so from 0→280 ppm the log formula would give an infinite forcing." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 1", + "text": "The 'why T⁴' argument is circular → Say that 390 W/m² is a measured surface emission (≈ 396 W/m² in observed budgets). Show that a linear law fitted to 240 W/m² at 255 K would predict 240 × 288/255 ≈ 271 W/m²." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 5", + "text": "The meaning of 3.76 W/m² per K (the Planck response) and of the 3 K climate sensitivity → 3.76 W/m² per K is the extra emission for each 1 K of warming. A larger value means less warming per W/m² of forcing. Sensitivity of 2.5 K vs 4 K means 2.5 K vs 4 K of eventual warming per doubling of CO₂." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 4", + "text": "The temporal sequence step 'warm over years to decades (the ocean slows it)' has no numbers → With 3.7 W/m² heating a 100 m ocean mixed layer (heat capacity ≈ 4.2×10⁸ J/m²/K), the first 1 K takes about 4.2×10⁸ / 3.7 ≈ 1.1×10⁸ s ≈ 3.6 years if no extra heat were emitted. In reality it takes longer, because emission grows as the Earth warms and heat mixes into the deep ocean." + }, + { + "n": 11, + "severity": "excess", + "kind": "model entry", + "text": "Wien's law constant 2898 µm·K (only the two peak wavelengths are needed)" + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "CO₂ history 280 ppm (1750) / 420 ppm (2024)" + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "IPCC 2.5–4 K likely range and feedback-inclusive 3 K (feedbacks are declared out of scope)" + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Ozone-hole confusion point" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Stratospheric cooling as a fingerprint (useful but not needed for the central question)" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152459-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152459-coldread-narrative.json new file mode 100644 index 0000000..4ec6281 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152459-coldread-narrative.json @@ -0,0 +1,526 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "greenhouse", + "at": "2026-10-04T15:24:59", + "model": "default", + "prompt_sha": "46e8105a72c2", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: answer in two sentences. Said: gases move the emission of infrared up into cold air; the surface must be warmer so that cold air emits enough", + "unresolved": [ + "emits enough" + ], + "unsaid": [], + "leap": [ + "\"the surface must be warmer so that cold air emits enough\": nothing yet says why the surface has to warm when the air higher up emits. The link between them (temperature falling with height) only comes in beat 4. That's fine for a hook, but the reader holds it as a puzzle." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (instance)", + "text": "Shown: 1361/4 = 340; ×0.70 = 240; σT⁴ = 240 → 255 K; why ÷4 (360 K otherwise). Said: energy balance fixes 255 K", + "unresolved": [ + "1361", + "×0.70", + "σT⁴", + "σ", + "energy balance", + "360 K otherwise" + ], + "unsaid": [ + "1361 is never said to be the sunlight arriving per m², and its units are not given", + "0.70 (the 30% reflected, i.e. albedo) is shown but never named or explained", + "the reason for ÷4 is announced ('why ÷4') but the words don't give it", + "σT⁴ appears as a formula, but the words never say it is the power a body emits per m² at temperature T", + "288 K observed and the 33 K gap: these are listed as what the reader now knows, but they appear in neither Shown nor Said" + ], + "leap": [ + "Why is 255 K a puzzle? Without the measured 288 K, the reader has no gap to explain." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (name)", + "text": "Shown: sunlight 0.5 µm vs infrared 10 µm; which gases absorb. Said: the gases pass sunlight, absorb and re-emit infrared; N₂, O₂ do not", + "unresolved": [], + "unsaid": [ + "the 0.5 µm vs 10 µm contrast is shown, but the words never say that the Earth emits at 10 µm because it is cold, or why the difference matters", + "'which gases absorb': the gases that do absorb (CO₂, water vapour) are never named in the words. Only the ones that don't (N₂, O₂) are." + ], + "leap": [ + "Why does the Earth's own emission sit in the infrared? This reader knows hotter things radiate more, but not that a cooler body emits at longer wavelengths." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Shown: table: height vs temperature; emission height ≈ 5 km at 255 K. Said: infrared escapes where little gas remains above; that air must be 255 K; air cools 6.5 K/km (convection: rising air expands and cools); 255 + 6.5 × 5 ≈ 288", + "unresolved": [], + "unsaid": [ + "'emission height ≈ 5 km' is shown, but the words never say where 5 km comes from" + ], + "leap": [ + "\"that air must be 255 K\": beat 2 presented 255 K as the Earth's temperature without the gases. Reusing it as the temperature of the air that emits to space needs a sentence like 'whatever emits the 240 W/m² must be at 255 K'.", + "Why does re-emission move the escape point upward? Re-emitted infrared goes in every direction, and the step from that to 'escape happens where little gas remains above' is compressed.", + "288 is treated as the target here, but the reader was never given it as the measured value" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (objection)", + "text": "Shown: surface budget. Said: the air sends 330 W/m² back down; 160 + 330 = 390 + 100", + "unresolved": [ + "160", + "100", + "390" + ], + "unsaid": [ + "the objection itself ('the surface emits 390 W/m² but only 240 leaves') appears in neither Said nor Shown", + "160 (sunlight absorbed at the surface) and 100 (heat carried up by convection and evaporation) are never identified", + "390 W/m² for the surface is never derived in the beats (σ·288⁴)" + ], + "leap": [ + "The equation balances, but the reader can't tell what each term is, so it can't be checked" + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 6 (test, predict)", + "text": "Shown: prediction prompt, then: rise ≈ 150 m, 1 K colder, −3.7 W/m², warm until balance, ≈ 1 K (≈ 3 K with feedbacks). Said: more gas → higher → colder → less out → warm", + "unresolved": [ + "feedbacks", + "−3.7 W/m²" + ], + "unsaid": [ + "how much CO₂ is added (a doubling) is never said", + "the 150 m rise is shown without saying where it comes from", + "−3.7 W/m² is shown but never named as the imbalance before warming" + ], + "leap": [ + "≈ 1 K becoming ≈ 3 K 'with feedbacks' is given without explanation" + ], + "excess": [ + "\"(≈ 3 K with feedbacks)\": the argument doesn't need it, and it brings in an idea it never explains" + ], + "purpose_unclear": false + }, + { + "at": "Beat 7 (test)", + "text": "Shown: isothermal case; stratosphere cools. Said: no lapse rate, no warming", + "unresolved": [ + "lapse rate", + "stratosphere", + "isothermal" + ], + "unsaid": [ + "'stratosphere cools' is shown and never mentioned in the words" + ], + "leap": [ + "Why would the stratosphere cool? Nothing says that temperature rises with height there." + ], + "excess": [ + "\"stratosphere cools\": not needed for the main line, and unexplained" + ], + "purpose_unclear": false + }, + { + "at": "Beat 8 (objection)", + "text": "Shown: 280→560: 3.7; 560→1120: 3.7. Said: saturation near the ground does not stop the rise of the emission height", + "unresolved": [ + "saturation", + "280→560", + "3.7" + ], + "unsaid": [ + "the units (ppm of CO₂, W/m²) are not given", + "that the figures show the same forcing for each doubling is not said in words" + ], + "leap": [ + "Why does each doubling give the same 3.7? Not explained." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (close)", + "text": "Shown: recap sentence. Said: answer in the opening's words; scope pointers", + "unresolved": [ + "scope pointers" + ], + "unsaid": [ + "neither the recap sentence nor the general argument is actually written in the beat. It only describes them." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 2: 288 K observed / 33 K gap is in neither Shown nor Said. Without it there is no puzzle, and beat 4's '≈ 288' has nothing to match.", + "Beat 2: 'σT⁴' and '×0.70' are never explained, so the 255 K that everything after depends on rests on symbols the reader hasn't been given.", + "Beat 4: 'that air must be 255 K' goes unexplained (it reuses beat 2's result for the emitting layer). This is the key step of the argument.", + "Beat 4: 'emission height ≈ 5 km' is shown with no source, so 255 + 6.5 × 5 ≈ 288 looks tuned to fit." + ], + "cuts": [ + "Beat 7: 'stratosphere cools' (unexplained and off the main line)", + "Beat 6: '(≈ 3 K with feedbacks)'", + "Beat 5: cut the whole surface budget, or state the objection and name the terms; as written it costs a reread and gives nothing back" + ], + "question": { + "text": "How do greenhouse gases warm the Earth's surface? The reader knows this at beat 1.", + "known_at": "Beat 1", + "result_stated": "Yes, at beat 1: \"gases move the emission of infrared up into cold air; the surface must be warmer so that cold air emits enough\"" + }, + "motive": "Missing in the beats. The reason to care (the 'trapping heat' picture can't give amounts and suggests saturation) appears only in the author's plan. The reason for the approach (start from energy conservation and show the 33 K gap) is only partly there: beat 2's bridge says 'start from energy', but the gap is never shown.", + "close": "Can't be confirmed. Beat 9 says only 'answer in the opening's words; scope pointers' and 'recap sentence', with no actual wording. The general argument (any rise in emission height warms the surface, as long as temperature falls with height) is not said in the beats. Beat 7's 'no lapse rate, no warming' comes closest.", + "recap": "Greenhouse gases make the Earth radiate to space from high, cold air that has to sit at 255 K, and because air warms by about 6.5 K per km going down, the surface 5 km below ends up about 33 K warmer.", + "strongest": "Beat 6: the said causal chain 'more gas → higher → colder → less out → warm', with a prediction prompt and numbers.", + "weakest": "Beat 5: an objection that is never stated, with unnamed numbers '160 + 330 = 390 + 100'. Runner-up: beat 2 leaves out 288 K.", + "suggestions": [ + "In beat 2, say '1361 W/m² of sunlight arrives; 30% is reflected; σT⁴ is the power per m² emitted at temperature T', and state 'measured: 288 K, so 33 K is missing'.", + "In beat 4, add 'whatever emits the 240 W/m² to space must be at 255 K, and now that is air at about 5 km, not the surface'.", + "In beat 1, voice the motive: 'trapping heat' can't say how much, and suggests the effect stops once CO₂ absorbs everything.", + "In beat 5, state the objection and name the 160 (absorbed sunlight) and 100 (convection and evaporation) terms.", + "In beat 8, give units (ppm, W/m²) and say 'each doubling adds the same 3.7 W/m²'.", + "Write the closing sentence and the general argument into beat 9 itself." + ] + }, + "usage": { + "cost": 0.15388179999999999, + "seconds": 57 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 288 K observed / 33 K gap is in neither Shown nor Said. Without it there is no puzzle, and beat 4's '≈ 288' has nothing to match." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'σT⁴' and '×0.70' are never explained, so the 255 K that everything after depends on rests on symbols the reader hasn't been given." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'that air must be 255 K' goes unexplained (it reuses beat 2's result for the emitting layer). This is the key step of the argument." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'emission height ≈ 5 km' is shown with no source, so 255 + 6.5 × 5 ≈ 288 looks tuned to fit." + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 7: 'stratosphere cools' (unexplained and off the main line)" + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 6: '(≈ 3 K with feedbacks)'" + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 5: cut the whole surface budget, or state the objection and name the terms; as written it costs a reread and gives nothing back" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): emits enough" + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): \"the surface must be warmer so that cold air emits enough\": nothing yet says why the surface has to warm when the air higher up emits. The link between them (temperature falling with height) only comes in beat 4. That's fine for a hook, but the reader holds it as a puzzle." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): 1361" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): ×0.70" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): σT⁴" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): σ" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): energy balance" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (instance): 360 K otherwise" + }, + { + "n": 16, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): 1361 is never said to be the sunlight arriving per m², and its units are not given" + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): 0.70 (the 30% reflected, i.e. albedo) is shown but never named or explained" + }, + { + "n": 18, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): the reason for ÷4 is announced ('why ÷4') but the words don't give it" + }, + { + "n": 19, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): σT⁴ appears as a formula, but the words never say it is the power a body emits per m² at temperature T" + }, + { + "n": 20, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): 288 K observed and the 33 K gap: these are listed as what the reader now knows, but they appear in neither Shown nor Said" + }, + { + "n": 21, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (instance): Why is 255 K a puzzle? Without the measured 288 K, the reader has no gap to explain." + }, + { + "n": 22, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (name): the 0.5 µm vs 10 µm contrast is shown, but the words never say that the Earth emits at 10 µm because it is cold, or why the difference matters" + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (name): 'which gases absorb': the gases that do absorb (CO₂, water vapour) are never named in the words. Only the ones that don't (N₂, O₂) are." + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (name): Why does the Earth's own emission sit in the infrared? This reader knows hotter things radiate more, but not that a cooler body emits at longer wavelengths." + }, + { + "n": 25, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): 'emission height ≈ 5 km' is shown, but the words never say where 5 km comes from" + }, + { + "n": 26, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): \"that air must be 255 K\": beat 2 presented 255 K as the Earth's temperature without the gases. Reusing it as the temperature of the air that emits to space needs a sentence like 'whatever emits the 240 W/m² must be at 255 K'." + }, + { + "n": 27, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): Why does re-emission move the escape point upward? Re-emitted infrared goes in every direction, and the step from that to 'escape happens where little gas remains above' is compressed." + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): 288 is treated as the target here, but the reader was never given it as the measured value" + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (objection): 160" + }, + { + "n": 30, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (objection): 100" + }, + { + "n": 31, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (objection): 390" + }, + { + "n": 32, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (objection): the objection itself ('the surface emits 390 W/m² but only 240 leaves') appears in neither Said nor Shown" + }, + { + "n": 33, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (objection): 160 (sunlight absorbed at the surface) and 100 (heat carried up by convection and evaporation) are never identified" + }, + { + "n": 34, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (objection): 390 W/m² for the surface is never derived in the beats (σ·288⁴)" + }, + { + "n": 35, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (objection): The equation balances, but the reader can't tell what each term is, so it can't be checked" + }, + { + "n": 36, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (test, predict): feedbacks" + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (test, predict): −3.7 W/m²" + }, + { + "n": 38, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (test, predict): how much CO₂ is added (a doubling) is never said" + }, + { + "n": 39, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (test, predict): the 150 m rise is shown without saying where it comes from" + }, + { + "n": 40, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (test, predict): −3.7 W/m² is shown but never named as the imbalance before warming" + }, + { + "n": 41, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (test, predict): ≈ 1 K becoming ≈ 3 K 'with feedbacks' is given without explanation" + }, + { + "n": 42, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): lapse rate" + }, + { + "n": 43, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): stratosphere" + }, + { + "n": 44, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): isothermal" + }, + { + "n": 45, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (test): 'stratosphere cools' is shown and never mentioned in the words" + }, + { + "n": 46, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test): Why would the stratosphere cool? Nothing says that temperature rises with height there." + }, + { + "n": 47, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (objection): saturation" + }, + { + "n": 48, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (objection): 280→560" + }, + { + "n": 49, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (objection): 3.7" + }, + { + "n": 50, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (objection): the units (ppm of CO₂, W/m²) are not given" + }, + { + "n": 51, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (objection): that the figures show the same forcing for each doubling is not said in words" + }, + { + "n": 52, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (objection): Why does each doubling give the same 3.7? Not explained." + }, + { + "n": 53, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (close): scope pointers" + }, + { + "n": 54, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 9 (close): neither the recap sentence nor the general argument is actually written in the beat. It only describes them." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152723-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152723-coldread-prose.json new file mode 100644 index 0000000..2f659d8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152723-coldread-prose.json @@ -0,0 +1,381 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "greenhouse", + "at": "2026-10-04T15:27:23", + "model": "default", + "prompt_sha": "363f5ff7238d", + "reply": { + "paragraphs": [ + { + "at": "Line 5 (bold lead)", + "text": "Greenhouse gases make the Earth send its heat to space from high, cold air instead of from the warm surface...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 7", + "text": "The rest of this page puts numbers on each step.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 12", + "text": "Start from conservation of energy... energy balance.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 list, line 14", + "text": "Sunlight delivers 1361 W/m² to a panel that faces the Sun.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 list, line 15", + "text": "The Earth catches sunlight on a disc of area πR², but it emits from its whole sphere...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 list, line 16", + "text": "The albedo is the fraction of sunlight... 240 W/m².", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 19", + "text": "The division by 4 matters. Without it, the calculation below gives 360 K...", + "unresolved": [], + "leap": [ + "\"the calculation below gives 360 K\": this points ahead to a calculation the reader has not seen yet, so they cannot check the number at this point" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 21", + "text": "A body at temperature T emits σT⁴ watts per square metre...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 23", + "text": "Set σT⁴ = 240 W/m² and solve: T = 255 K... emission temperature", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§1 line 25", + "text": "The measured average surface temperature is 288 K... The greenhouse gases cause that 33 K.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 line 30", + "text": "The Sun is hot (about 5800 K)... This is infrared", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 line 32", + "text": "A greenhouse gas lets most sunlight pass but absorbs and emits infrared...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§2 line 34", + "text": "So sunlight reaches the ground. But infrared from the ground is absorbed by the air...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 line 39", + "text": "Infrared reaches space only from air that has little greenhouse gas above it... emission height.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 line 41", + "text": "The air at the emission height must emit 240 W/m², so it must be at the emission temperature, 255 K.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 line 43", + "text": "The air gets colder with height... the lapse rate.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 table, lines 45-49", + "text": "Height / Temperature: 0 km 288 K, 2 km 275 K, 5 km 255 K", + "unresolved": [], + "leap": [], + "excess": [ + "\"| 2 km | 275 K |\": this row is never used in the argument" + ], + "purpose_unclear": false + }, + { + "at": "§3 line 51", + "text": "Now the 33 K has a cause: 255 K + 6.5 K/km × 5 km ≈ 288 K.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 line 54", + "text": "Where does the surface's extra emission go? ... The air emits 330 W/m² of infrared down to the surface.", + "unresolved": [], + "leap": [ + "\"The air emits 330 W/m² of infrared down to the surface\": the figure is given with no reason why it is so large, or why it is larger than the 240 W/m² that leaves the top" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§3 surface budget table, lines 56-60", + "text": "sunlight 160 | infrared 390; infrared from the air 330 | rising warm air and evaporation 100", + "unresolved": [], + "leap": [ + "\"sunlight | 160\": §1 said the Earth absorbs 240 W/m², and the text never says where the other 80 W/m² goes (into the air?). The reader stops to reconcile the two figures", + "The table explains the surface's warmth by downward infrared, but it is not tied back to the emission-height account. The reader may wonder which of the two is the real explanation" + ], + "excess": [ + "The whole surface-budget aside: the main argument (emission height + lapse rate) does not need it" + ], + "purpose_unclear": false + }, + { + "at": "§4 line 64", + "text": "Predict first: CO₂ doubles, from its pre-industrial 280 ppm to 560 ppm...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 1, line 67", + "text": "More gas means the infrared must start higher... about 150 m.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 2, line 68", + "text": "The air 150 m higher is about 1 K colder", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 3, line 69", + "text": "Colder air emits less... radiative forcing.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 4, line 70", + "text": "The Earth gains energy and warms. The lapse rate holds, so the whole profile warms together.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 list item 5, line 71", + "text": "Warming stops when the air at the new emission height is back at 255 K...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§4 line 73", + "text": "This 1 K counts only this mechanism... about 3 K (IPCC best estimate). Today's 420 ppm already gives a forcing of about 2.2 W/m².", + "unresolved": [ + "IPCC" + ], + "leap": [ + "\"Today's 420 ppm already gives a forcing of about 2.2 W/m²\": the reader cannot get 2.2 from 3.7 per doubling, because the rule that each doubling adds the same forcing does not come until §6" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 line 78", + "text": "The usual picture, \"greenhouse gases trap heat like a blanket\", says more gas always warms...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 line 80", + "text": "Suppose the atmosphere had the same temperature at every height (a lapse rate of 0)...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§5 line 82", + "text": "So the warming depends on two things together...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "§6 line 87", + "text": "Near the ground, CO₂ already absorbs almost all infrared at its main wavelength (15 µm)...", + "unresolved": [], + "leap": [ + "\"the escape happens high up, where the air is thin\": the text does not say why thin air makes the near-ground saturation irrelevant (thin air absorbs less, so it is not saturated there)" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Summary line 91", + "text": "Energy balance requires the Earth to emit 240 W/m² to space...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered, lines 95-98", + "text": "Feedbacks in detail...; Why only some molecules absorb infrared...; Why each doubling gives the same forcing...; Where added CO₂ goes...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "The surface-budget aside in §3 (line 54 and the table at lines 56-60) is the biggest cut. It is off the main line, and its 160 W/m² sunlight figure clashes with the 240 W/m² from §1 without explanation. Cut it, or account for the missing 80 W/m² in one clause.", + "§4 line 73: \"Today's 420 ppm already gives a forcing of about 2.2 W/m².\" Cut it, or move it to §6 after the per-doubling rule.", + "§3 table: the 2 km row, which is never used." + ], + "question": { + "text": "Why does adding gases that absorb infrared make the surface warmer, and by how much?", + "known_at": "The title (line 1), and the bold lead (line 5) right after it", + "result_stated": "Yes, in words, at line 5: \"Greenhouse gases make the Earth send its heat to space from high, cold air instead of from the warm surface. Cold air emits less, so the whole lower atmosphere, and the surface with it, must be warmer before enough energy leaves.\"" + }, + "motive": "There is a reason for the approach: line 78, \"The usual picture, 'greenhouse gases trap heat like a blanket', says more gas always warms. A test case shows that it does not.\" It comes late, in §5. There is no reason to care about the question near the start. The stakes (CO₂ doubling, \"Today's 420 ppm\") only appear in §4.", + "close": "Yes. Line 91: \"Greenhouse gases move that emitting air up to about 5 km. The lapse rate of 6.5 K/km makes the surface 33 K warmer, at 288 K. More greenhouse gas lifts the emission height and warms the surface.\" Line 82 also states the general condition: \"the gas lifts the emission height, and the air is colder higher up.\"", + "recap": "Greenhouse gases push the level where Earth's heat escapes to space higher up, into colder air, and because air gets colder with height, the surface has to sit warmer below that level; more CO₂ pushes the level higher and warms the surface more.", + "strongest": "§5 (lines 78-82): the test with a lapse rate of 0 shows that warming needs both a rising emission height and colder air above, which also explains why the blanket picture falls short. §4's numbered chain (lines 67-71) is a close second.", + "weakest": "The §3 surface budget (lines 54-60). Its 160 W/m² sunlight figure contradicts the 240 W/m² from §1 with no explanation, and the section brings in a back-radiation picture that is never tied to the emission-height argument.", + "suggestions": [ + "In the surface budget, say that the air and clouds absorb about 80 W/m² of sunlight directly, so the 160 and 240 figures fit together.", + "Add one sentence near the top on why this matters, for example: it is how scientists predict the warming from CO₂ that people add.", + "In §6, say that thin air high up absorbs too little to be saturated, which is why adding gas there still matters.", + "Spell out IPCC (Intergovernmental Panel on Climate Change) at first use." + ] + }, + "usage": { + "cost": 0.1570202, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "The surface-budget aside in §3 (line 54 and the table at lines 56-60) is the biggest cut. It is off the main line, and its 160 W/m² sunlight figure clashes with the 240 W/m² from §1 without explanation. Cut it, or account for the missing 80 W/m² in one clause." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "§4 line 73: \"Today's 420 ppm already gives a forcing of about 2.2 W/m².\" Cut it, or move it to §6 after the per-doubling rule." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "§3 table: the 2 km row, which is never used." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "§1 line 19: \"the calculation below gives 360 K\": this points ahead to a calculation the reader has not seen yet, so they cannot check the number at this point" + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "§3 line 54: \"The air emits 330 W/m² of infrared down to the surface\": the figure is given with no reason why it is so large, or why it is larger than the 240 W/m² that leaves the top" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "§3 surface budget table, lines 56-60: \"sunlight | 160\": §1 said the Earth absorbs 240 W/m², and the text never says where the other 80 W/m² goes (into the air?). The reader stops to reconcile the two figures" + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "§3 surface budget table, lines 56-60: The table explains the surface's warmth by downward infrared, but it is not tied back to the emission-height account. The reader may wonder which of the two is the real explanation" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "§4 line 73: IPCC" + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "§4 line 73: \"Today's 420 ppm already gives a forcing of about 2.2 W/m²\": the reader cannot get 2.2 from 3.7 per doubling, because the rule that each doubling adds the same forcing does not come until §6" + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "§6 line 87: \"the escape happens high up, where the air is thin\": the text does not say why thin air makes the near-ground saturation irrelevant (thin air absorbs less, so it is not saturated there)" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152840-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152840-quiz-prose.json new file mode 100644 index 0000000..f5e5cc5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T152840-quiz-prose.json @@ -0,0 +1,227 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "greenhouse", + "at": "2026-10-04T15:28:28", + "model": "default", + "prompt_sha": "709adbb037ff", + "reply": { + "answers": { + "1": "The main objects are the Sun (sunlight, 1361 W/m²), the Earth's surface, and the atmosphere. Within the atmosphere they are the greenhouse gases (water vapour, CO₂, methane) and the non-absorbing gases (nitrogen and oxygen). The key quantities are albedo (0.30), infrared radiation, the energy balance, the emission temperature (255 K), the emission height (about 5 km), the lapse rate (6.5 K/km) and radiative forcing.", + "2": "Sunlight passes through the greenhouse gases and is absorbed by the ground, apart from the 30% that is reflected. The ground emits infrared. Greenhouse gases absorb it and emit it again at the temperature of the air around them. Infrared escapes to space only from air that has little greenhouse gas above it, which on average is about 5 km up. Energy balance requires 240 W/m² out, so the air at that height must be at 255 K. The lapse rate then sets every lower layer, including the surface, warmer than that.", + "3": "Greenhouse gases move the place where Earth's infrared escapes to space up into cold air. Because the emitting air must be at 255 K to balance the incoming sunlight, and air warms by 6.5 K per km going down, the surface ends up at about 288 K. Adding gas raises the emission height and so warms the surface further.", + "4": "The most important variable is the amount of CO₂. Doubling it raises the emission height by about 150 m. Air there is about 1 K colder, so it emits about 3.7 W/m² less than the 240 W/m² absorbed. That imbalance is the radiative forcing. The Earth then gains energy, and the whole temperature profile warms until the new emission height is back at 255 K. The surface ends up about 1 K warmer from this mechanism alone, or about 3 K with feedbacks such as water vapour (IPCC best estimate).", + "5": "Assumption: the lapse rate stays fixed at 6.5 K/km while the planet warms, and the whole profile shifts together. The 5 km emission height is also an average that stands in for many wavelengths. Observation: the measured average surface temperature is 288 K, and the downward infrared from the air at the surface is about 330 W/m², which the text calls a measured value.", + "6": "The Earth must emit as much power as it absorbs, about 240 W/m², and that fixes the temperature of whatever emits to space at 255 K. Greenhouse gases absorb and re-emit infrared, so the radiation that escapes comes from about 5 km up rather than from the ground. Air cools by about 6.5 K per km with height, so the surface 5 km below is about 33 K warmer, at 288 K. Adding CO₂ lifts the emission height into colder air, which briefly cuts the outgoing infrared, so the Earth warms until the emitting air is back at 255 K. This works only because the air is colder higher up, and it still works when the lower air is 'saturated', because what escapes is decided high up.", + "7": "The hardest part was the step from '1 K colder' to '3.7 W/m² less'. The text never shows how that number comes from σT⁴; a single line using the Stefan–Boltzmann law at 255 K would have closed the gap. It is also unclear why 'the lapse rate holds' during warming, and what exactly sets 'little greenhouse gas above it', and therefore the 5 km height.", + "8": "About 255 K (−18 °C). It is set by energy balance: absorbed sunlight is 1361/4 × 0.70 ≈ 240 W/m², and σT⁴ = 240 W/m² gives T = 255 K. The division by 4 comes from catching sunlight on a disc but emitting from a whole sphere.", + "9": "The extra gas pushes the emission height up by about 150 m, and the air there is about 1 K colder. Colder air emits less, so before anything warms, outgoing infrared drops by about 3.7 W/m².", + "10": "No. The emission height would still rise, but the air there would still be at 255 K and would emit the same 240 W/m². There would be no imbalance and no warming.", + "11": "About 255 K. This repeats question 8: 240 W/m² of absorbed sunlight must be emitted, and σT⁴ = 240 gives 255 K, with the surface itself doing the emitting. The measured 288 K is 33 K higher, and the text attributes that difference to greenhouse gases.", + "12": "Mostly from air at about 5 km altitude, the emission height, which is at about 255 K, the emission temperature.", + "13": "Nothing. The raised emission level would be at the same temperature, so it would emit the same 240 W/m², giving no imbalance and no warming. Warming needs both a higher emission height and colder air higher up.", + "14": "Yes. What reaches space is decided high up, where the air is thin and infrared still escapes. More CO₂ makes that thin air absorb more, so the escape moves higher into colder air, and emission falls. Each doubling adds about the same forcing of about 3.7 W/m². The text defers the reason for this per-doubling rule to Pierrehumbert." + }, + "audit": { + "terms": [ + "The escape level has several names: 'emission height' (§3, §4), 'emitting air' (lead paragraph, Summary), 'the escape moves up' (§6), and 'little greenhouse gas above it' (§3). A reader could take these for different things.", + "'radiative forcing' (§4 step 3) is also called 'imbalance' (§4 step 3, §5 'no imbalance') and 'forcing' (§6).", + "'emission temperature' (§1) is mostly written just as '255 K' (§3, §4 step 5, Summary).", + "'heat' (lead paragraph: 'send its heat to space') and 'infrared' (§2 onward) are used for the same outgoing radiation." + ], + "unexplained": [ + "'3.7 W/m² less' (§4 step 3): the text does not show how a 1 K drop produces this number, although σT⁴ would let the reader check it.", + "'Today's 420 ppm gives about 2.2 W/m²' (§6): there is no way to get 2.2 from 420. The reader cannot see why 1.5× the CO₂ gives about 60% of a doubling.", + "'about 5 km up' (§3) and 'on average by about 150 m' (§4 step 1): both are given without any basis, and the text does not say what would make them larger or smaller.", + "'about 330 W/m² ... back down (a measured value)' (§3): this is never connected to the 390 and 240 figures, for example as the net loss of 60 W/m² from the surface, or to any of the questions.", + "'about 3 K' (§4): the text does not explain how feedbacks turn 1 K into 3 K." + ], + "missing_why": [ + "§4 step 4, 'The lapse rate holds, so the whole profile warms together': the text does not say why the lapse rate stays the same under warming.", + "§6, 'Each doubling adds about the same forcing': the logarithmic rule is stated and its reason sent elsewhere, but the 2.2 W/m² figure depends on it.", + "§3, 'a fall of about 6.5 K per km': the mechanism is named (expansion and convection), but not why the value is 6.5 rather than something else.", + "§3, 'Infrared reaches space only from air that has little greenhouse gas above it': 'little' is never quantified, so the 5 km figure cannot be derived." + ], + "excess": [ + "§1, 'with 1361 × 0.70 instead, T = 360 K (87 °C)': a counterfactual detour that the argument does not use.", + "§3, the surface-budget paragraph (390 vs 330 W/m²): a side point that the causal chain and the questions do not need.", + "§2, 'The Sun is hot (about 5800 K), so its light peaks near 0.5 µm': this reader already knows hotter things radiate at shorter wavelengths. Only 'Earth emits near 10 µm' is needed.", + "§3, 'Rising air expands under lower pressure and cools as it expands': this reader already knows a gas cools when it expands." + ] + }, + "gaps": [ + "No derivation of the 3.7 W/m² forcing from the 1 K temperature change.", + "No reason given for why the lapse rate stays fixed as the planet warms.", + "The 5 km emission height and the 150 m rise are asserted without any basis.", + "The logarithmic per-doubling forcing, and so the 2.2 W/m² at 420 ppm, cannot be checked from the text.", + "The text does not explain how water vapour and other feedbacks triple the 1 K to about 3 K.", + "Questions 8/11 and 10/13 are duplicates. The explanation answers both pairs, but the duplication adds no new evidence." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 1, + "13": 2, + "14": 2 + }, + "reasons": { + "12": "Gives the location (about 5 km, the emission height) and the temperature (about 255 K), but explains neither. It never says infrared escapes from that height because too little greenhouse gas remains above it to absorb the radiation again. It also never says the air there must be 255 K because energy balance requires 240 W/m² out. The answer is bare labels without the defining mechanism." + }, + "pass": false + }, + "usage": { + "cost": 0.18852180000000002, + "seconds": 45 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The escape level has several names: 'emission height' (§3, §4), 'emitting air' (lead paragraph, Summary), 'the escape moves up' (§6), and 'little greenhouse gas above it' (§3). A reader could take these for different things." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'radiative forcing' (§4 step 3) is also called 'imbalance' (§4 step 3, §5 'no imbalance') and 'forcing' (§6)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'emission temperature' (§1) is mostly written just as '255 K' (§3, §4 step 5, Summary)." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'heat' (lead paragraph: 'send its heat to space') and 'infrared' (§2 onward) are used for the same outgoing radiation." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'3.7 W/m² less' (§4 step 3): the text does not show how a 1 K drop produces this number, although σT⁴ would let the reader check it." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'Today's 420 ppm gives about 2.2 W/m²' (§6): there is no way to get 2.2 from 420. The reader cannot see why 1.5× the CO₂ gives about 60% of a doubling." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 5 km up' (§3) and 'on average by about 150 m' (§4 step 1): both are given without any basis, and the text does not say what would make them larger or smaller." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'about 330 W/m² ... back down (a measured value)' (§3): this is never connected to the 390 and 240 figures, for example as the net loss of 60 W/m² from the surface, or to any of the questions." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'about 3 K' (§4): the text does not explain how feedbacks turn 1 K into 3 K." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "§4 step 4, 'The lapse rate holds, so the whole profile warms together': the text does not say why the lapse rate stays the same under warming." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "§6, 'Each doubling adds about the same forcing': the logarithmic rule is stated and its reason sent elsewhere, but the 2.2 W/m² figure depends on it." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "§3, 'a fall of about 6.5 K per km': the mechanism is named (expansion and convection), but not why the value is 6.5 rather than something else." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "§3, 'Infrared reaches space only from air that has little greenhouse gas above it': 'little' is never quantified, so the 5 km figure cannot be derived." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "§1, 'with 1361 × 0.70 instead, T = 360 K (87 °C)': a counterfactual detour that the argument does not use." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "§3, the surface-budget paragraph (390 vs 330 W/m²): a side point that the causal chain and the questions do not need." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "§2, 'The Sun is hot (about 5800 K), so its light peaks near 0.5 µm': this reader already knows hotter things radiate at shorter wavelengths. Only 'Earth emits near 10 µm' is needed." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "§3, 'Rising air expands under lower pressure and cools as it expands': this reader already knows a gas cools when it expands." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "No derivation of the 3.7 W/m² forcing from the 1 K temperature change." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "No reason given for why the lapse rate stays fixed as the planet warms." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "The 5 km emission height and the 150 m rise are asserted without any basis." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "The logarithmic per-doubling forcing, and so the 2.2 W/m² at 420 ppm, cannot be checked from the text." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "The text does not explain how water vapour and other feedbacks triple the 1 K to about 3 K." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "Questions 8/11 and 10/13 are duplicates. The explanation answers both pairs, but the duplication adds no new evidence." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153030-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153030-quiz-prose.json new file mode 100644 index 0000000..8898ea2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153030-quiz-prose.json @@ -0,0 +1,230 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "greenhouse", + "at": "2026-10-04T15:30:15", + "model": "default", + "prompt_sha": "a3fd476ca87a", + "reply": { + "answers": { + "1": "The main objects are the Sun's light (about 1361 W/m²), the Earth's surface, the atmosphere and the way its temperature changes with height, greenhouse gases (water vapour, CO₂ and methane), the gases that don't absorb infrared (nitrogen and oxygen), infrared radiation, the emission height (about 5 km up), the emission temperature (255 K), the lapse rate (6.5 K/km), albedo (0.30), the energy balance and radiative forcing.", + "2": "Sunlight passes through the greenhouse gases and is absorbed. Albedo reflects 30% of it, leaving about 240 W/m². The Earth must send the same 240 W/m² to space as infrared. Greenhouse gases absorb infrared and emit it again, so infrared only escapes from air with little gas above it. That air is at the emission height, and it must be at 255 K to emit 240 W/m². Convection fixes how fast the air cools with height (the lapse rate). So the surface is warmer than the emitting air by the lapse rate times the emission height. Air just above the surface sends about 330 W/m² back down to it.", + "3": "Greenhouse gases move the place where infrared escapes up into colder air. Because the emitting air must stay at 255 K to balance the incoming sunlight, and the air warms at 6.5 K per km going down, the surface ends up warmer: 288 K instead of 255 K. Adding more gas lifts the emission height further, so the surface warms more.", + "4": "The key variable is the amount of greenhouse gas, for example CO₂ doubling from 280 to 560 ppm. The emission height rises by about 150 m. The air there is about 1 K colder, so it emits about 3.7 W/m² less (236 instead of 240 W/m²). The Earth gains energy and the whole temperature profile warms until the new emission height is back at 255 K. The surface ends up about 1 K warmer from this mechanism alone, and about 3 K warmer once feedbacks such as water vapour are included. If the lapse rate were 0, nothing would change.", + "5": "Assumption: the lapse rate stays fixed at 6.5 K/km while the Earth warms (the text says in reality it changes a little). Established facts: the measured average surface temperature is 288 K, and satellites see outgoing infrared coming on average from about 5 km up. Another measured fact is that the air sends about 330 W/m² back down to the surface.", + "6": "The Earth absorbs about 240 W/m² of sunlight, after the factor of 4 for geometry and the 30% reflected, and must emit the same amount, which needs an emitter at 255 K. Greenhouse gases absorb infrared and emit it again, so the infrared that reaches space comes from about 5 km up, where little gas remains above. Convection makes the air warm by 6.5 K per km going down, so the surface is about 33 K warmer than that emitting layer, at 288 K. Adding CO₂ lifts the emission height into colder air, which emits less. The Earth then gains energy until that layer is back at 255 K, leaving the surface warmer: about 1 K per doubling directly, about 3 K with feedbacks.", + "7": "The hardest part was step 1 of section 4: why the emission height rises by exactly 150 m. The number is just given, and it seems to be worked backwards from the 3.7 W/m² forcing. It would help to show where 150 m comes from, or to say plainly that it is inferred. The saturation section was also hard: the claim that each doubling gives the same forcing, and how 420 ppm gives 2.2 W/m², are given without reasons. One sentence on why the effect goes by doublings would make it easier.", + "8": "About 255 K (−18 °C). It is set by the energy balance: σT⁴ = 240 W/m². The 240 W/m² comes from 1361 W/m² divided by 4 (sunlight hits a disc of area πR² but the Earth emits from a sphere of area 4πR²), times 0.70 (the part not reflected by albedo).", + "9": "More CO₂ means the infrared has to start from higher up before little gas remains above it, so the emission height rises by about 150 m. Air there is about 1 K colder (254 K), and colder air emits less: about 236 instead of 240 W/m², a drop of about 3.7 W/m².", + "10": "No. The emission height would still rise, but the air there would still be at 255 K and would emit the same 240 W/m². There would be no imbalance and no warming. The warming needs colder air higher up.", + "11": "It comes mostly from about 5 km up, where the air is about 255 K. The temperature has to be 255 K because that is what emits the 240 W/m² needed to balance the absorbed sunlight. The height is where little greenhouse gas remains above, so the infrared escapes instead of being absorbed again. At a lapse rate of 6.5 K/km from a 288 K surface, the air reaches 255 K at about 5 km, and satellites confirm that height. The explanation never derives the 5 km from the amount of gas: it gets it from the lapse rate and the measured surface temperature.", + "12": "(1) Doubling CO₂ raises the emission height by about 150 m. (2) Air there is about 1 K colder. (3) It emits about 236 W/m² against 240 W/m² absorbed, an imbalance of 3.7 W/m² called the radiative forcing. (4) The Earth gains energy, and because convection keeps the lapse rate, the whole profile warms together. (5) Warming stops when the air at the new emission height is back at 255 K and emission matches absorption again. The surface is then about 1 K warmer, or about 3 K with feedbacks such as water vapour. The explanation does not say how long this takes.", + "13": "Yes. What reaches space is decided high up, where the air is thin and infrared still escapes. More CO₂ makes that thin air absorb more, so the escape moves higher into colder air, and less is emitted. Each doubling adds about 3.7 W/m². The explanation does not say why each doubling gives the same forcing; it points to a textbook for that." + }, + "audit": { + "terms": [ + "The layer that emits to space has four names: 'emission height' (§3, §4), 'emitting air' (§1 lead, Summary), 'the escape' / 'where infrared escapes' (§3 title, §6), and 'whatever emits the Earth's energy to space' (§1).", + "The 3.7 W/m² drop is called 'imbalance' (§4 step 3, §5), 'radiative forcing' (§4 step 3) and 'forcing' (§6). The text does say they are the same thing, but in §5 'no imbalance' is not linked to forcing.", + "'the whole profile warms together' (§4 step 4): 'profile' (meaning temperature against height) is never defined. Elsewhere the text says 'the air gets colder with height' or 'lower atmosphere'.", + "'trap heat' (§5, quoting the blanket picture) against 'infrared' and 'energy' everywhere else, without saying that heat here means infrared." + ], + "unexplained": [ + "'The emission height rises, on average by about 150 m' (§4 step 1): no source or derivation is given. The whole numerical chain depends on it, and the reader can't tell whether it is measured, computed, or worked backwards from 3.7 W/m².", + "'Today's 420 ppm gives about 2.2 W/m²' (§6): it doesn't say how 2.2 follows from the doubling rule, or what it means for temperature now (roughly how much warming).", + "'emits about 330 W/m² of infrared back down (a measured value)' (§3): the surface budget is left incomplete. Surface emission of 390 minus 330 coming back leaves 60 W/m², and nothing says where the surface's absorbed sunlight goes, so the paragraph raises a question it doesn't answer.", + "'about 3 K (best estimate of the IPCC)' (§4): there is no range, and no sense of how the feedbacks turn 1 K into 3 K.", + "'6.5 K per km' (§3): it is called measured, but the text doesn't say what sets this value rather than another, or what a larger lapse rate would mean for the surface. That would follow directly from the argument but is never stated." + ], + "missing_why": [ + "'Each doubling adds about the same forcing' (§6): the rule that each doubling adds the same forcing is stated and the reason is sent to another book. The reader can't see why the 280→560 step equals the 560→1120 step.", + "'More gas means the infrared must start higher' (§4 step 1): this is qualitative only. Nothing links the amount of gas to the size of the rise (why 150 m for a doubling).", + "'Convection still sets the lapse rate, so the whole profile warms together' (§4 step 4): it doesn't explain why convection pins the surface to the upper air at a fixed gradient instead of only the upper air warming.", + "'Infrared reaches space only from air that has little greenhouse gas above it' (§3): there is no idea of how much gas counts as 'little', so it is unclear why an average height is a meaningful idea." + ], + "excess": [ + "The Summary (lines 81) repeats §1–§4 with nothing new.", + "'Emission grows fast with temperature: 255 K emits 240 W/m², and 288 K emits 390 W/m²' (§1): the reader already knows hotter things radiate more, and these numbers come up again later.", + "The surface-budget paragraph (§3, 330 W/m²): none of the questions or later steps use it, and it is left incomplete." + ] + }, + "gaps": [ + "Nothing explains where the 150 m rise in emission height comes from. It drives the whole 1 K / 3.7 W/m² calculation.", + "The 5 km emission height is inferred from the 288 K surface and the lapse rate, then used to explain the 288 K surface. Apart from mentioning the satellite observation, the reasoning is close to circular, and the height is never derived from the amount of gas.", + "No mechanism is given for why each doubling adds the same forcing, so the saturation answer rests on a claim the reader has to take on trust.", + "The surface energy budget is incomplete: after the 330 W/m² back-radiation, the remaining terms (absorbed sunlight, convection or evaporation) are missing.", + "Nothing is said about how long the warming takes. Step 5 says when it stops but not over what timescale; ocean lag appears only under 'Not covered'.", + "Feedbacks are given as a number (about 3 K) with only water vapour named, so the reader can't predict how much warming there is beyond the 1 K." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 1, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "1": "Full marks: lists all model entities with correct values.", + "2": "Full marks: the relationships match the model. 'Absorbed' (by the surface) and 'air just above the surface' are loose wording but not wrong.", + "3": "Full marks: emission from a colder height, the emitting air held at 255 K, and the lapse rate giving 288 K.", + "4": "Full marks: CO2 doubling, 150 m rise, 1 K colder, 3.7 W/m², about 1 K without feedbacks and 3 K with, and the zero-lapse-rate case.", + "5": "Partial. The fixed lapse rate is correctly named as an assumption. It omits the single-emission-height assumption, the main simplification in the model. It does not separate model-output estimates (3.7 W/m², 3 K) from observed facts. It presents 'satellites see infrared from about 5 km' as established, but the model derives 5 km from 33 K / 6.5 K/km and does not cite satellite observation.", + "6": "Full marks: a correct, complete chain from energy balance to warming per doubling.", + "7": "Full marks: points to specific gaps (where the 150 m rise comes from, why each doubling adds the same forcing), and both are real gaps given the model's scope.", + "8": "Full marks: 255 K from σT⁴ = 240 W/m², with the factor of 4 and albedo explained.", + "9": "Full marks: higher emission height, colder air there, less emission.", + "10": "Full marks: correct reasoning for the isothermal counterexample.", + "11": "Full marks: 5 km, 255 K, and why that height and that temperature. The 'satellites confirm' remark is not in the model but does not undermine the answer.", + "12": "Full marks: correct steps, and the stopping condition is the air at the new emission height returning to 255 K.", + "13": "Full marks: saturation does not stop the effect, emission moves higher and colder, about 3.7 W/m² per doubling." + }, + "pass": true + }, + "usage": { + "cost": 0.22451879999999996, + "seconds": 62 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The layer that emits to space has four names: 'emission height' (§3, §4), 'emitting air' (§1 lead, Summary), 'the escape' / 'where infrared escapes' (§3 title, §6), and 'whatever emits the Earth's energy to space' (§1)." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The 3.7 W/m² drop is called 'imbalance' (§4 step 3, §5), 'radiative forcing' (§4 step 3) and 'forcing' (§6). The text does say they are the same thing, but in §5 'no imbalance' is not linked to forcing." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'the whole profile warms together' (§4 step 4): 'profile' (meaning temperature against height) is never defined. Elsewhere the text says 'the air gets colder with height' or 'lower atmosphere'." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'trap heat' (§5, quoting the blanket picture) against 'infrared' and 'energy' everywhere else, without saying that heat here means infrared." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'The emission height rises, on average by about 150 m' (§4 step 1): no source or derivation is given. The whole numerical chain depends on it, and the reader can't tell whether it is measured, computed, or worked backwards from 3.7 W/m²." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'Today's 420 ppm gives about 2.2 W/m²' (§6): it doesn't say how 2.2 follows from the doubling rule, or what it means for temperature now (roughly how much warming)." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'emits about 330 W/m² of infrared back down (a measured value)' (§3): the surface budget is left incomplete. Surface emission of 390 minus 330 coming back leaves 60 W/m², and nothing says where the surface's absorbed sunlight goes, so the paragraph raises a question it doesn't answer." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'about 3 K (best estimate of the IPCC)' (§4): there is no range, and no sense of how the feedbacks turn 1 K into 3 K." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'6.5 K per km' (§3): it is called measured, but the text doesn't say what sets this value rather than another, or what a larger lapse rate would mean for the surface. That would follow directly from the argument but is never stated." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'Each doubling adds about the same forcing' (§6): the rule that each doubling adds the same forcing is stated and the reason is sent to another book. The reader can't see why the 280→560 step equals the 560→1120 step." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'More gas means the infrared must start higher' (§4 step 1): this is qualitative only. Nothing links the amount of gas to the size of the rise (why 150 m for a doubling)." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'Convection still sets the lapse rate, so the whole profile warms together' (§4 step 4): it doesn't explain why convection pins the surface to the upper air at a fixed gradient instead of only the upper air warming." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "'Infrared reaches space only from air that has little greenhouse gas above it' (§3): there is no idea of how much gas counts as 'little', so it is unclear why an average height is a meaningful idea." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "The Summary (lines 81) repeats §1–§4 with nothing new." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "'Emission grows fast with temperature: 255 K emits 240 W/m², and 288 K emits 390 W/m²' (§1): the reader already knows hotter things radiate more, and these numbers come up again later." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "The surface-budget paragraph (§3, 330 W/m²): none of the questions or later steps use it, and it is left incomplete." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "Nothing explains where the 150 m rise in emission height comes from. It drives the whole 1 K / 3.7 W/m² calculation." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "The 5 km emission height is inferred from the 288 K surface and the lapse rate, then used to explain the 288 K surface. Apart from mentioning the satellite observation, the reasoning is close to circular, and the height is never derived from the amount of gas." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "No mechanism is given for why each doubling adds the same forcing, so the saturation answer rests on a claim the reader has to take on trust." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "The surface energy budget is incomplete: after the 330 W/m² back-radiation, the remaining terms (absorbed sunlight, convection or evaporation) are missing." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "Nothing is said about how long the warming takes. Step 5 says when it stops but not over what timescale; ocean lag appears only under 'Not covered'." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "Feedbacks are given as a number (about 3 K) with only water vapour named, so the reader can't predict how much warming there is beyond the 1 K." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153148-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153148-quiz-prose.json new file mode 100644 index 0000000..ed6edd4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/runs/2026-10-04T153148-quiz-prose.json @@ -0,0 +1,227 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "greenhouse", + "at": "2026-10-04T15:31:33", + "model": "default", + "prompt_sha": "a3fd476ca87a", + "reply": { + "answers": { + "1": "The Sun's input (1361 W/m²), the Earth's surface, the atmosphere and its layers at different heights, greenhouse gases (water vapour, CO₂, methane) versus non-absorbing N₂ and O₂, infrared radiation, the emission height and the emission temperature (255 K), the lapse rate (6.5 K/km), the albedo (0.30) and radiative forcing.", + "2": "The Earth absorbs 240 W/m² of sunlight (1361/4 × 0.70) and must emit the same amount to space. Greenhouse gases let sunlight through but absorb infrared and emit it again at the local air temperature. Infrared escapes to space only from air with little greenhouse gas above it, at the emission height. That air must be at 255 K to emit 240 W/m². The lapse rate links that height's temperature to the surface: the surface is warmer by 6.5 K for each km of height.", + "3": "Greenhouse gases raise the height from which infrared escapes to space. Because air gets colder with height, the emitting air would radiate less than 240 W/m², so the atmosphere and surface warm until the emitting air is back at 255 K. This leaves the surface warmer by the lapse rate times the emission height.", + "4": "If CO₂ doubles, the emission height rises by about 150 m. Air there is about 1 K colder (6.5 × 0.15), so it emits about 3.7 W/m² less than the Earth absorbs (the radiative forcing). The Earth gains energy until every height has warmed about 1 K, which brings the new emission height back to 255 K. The surface ends about 1 K warmer, or about 3 K (2.5–4 K) once feedbacks such as water vapour are included. With a lapse rate of 0, nothing changes.", + "5": "Assumption: the lapse rate stays fixed at 6.5 K/km while the Earth warms, so every height warms by the same amount (the text says this changes a little in reality). The text also replaces a spread of emission heights with one average height. Observed facts: the average surface temperature is 288 K, the measured lapse rate is about 6.5 K/km, the albedo is 0.30, and the solar input is 1361 W/m².", + "6": "In balance, the Earth must emit the 240 W/m² of sunlight it absorbs, which needs an emitting temperature of 255 K. Greenhouse gases absorb and re-emit infrared, so the radiation escapes only from high air with little gas above it, on average about 5 km up. Air cools by about 6.5 K per km, so the surface sits about 33 K warmer than that layer, at 288 K. Adding CO₂ lifts the escape height into colder air that emits less, creating an imbalance (3.7 W/m² per doubling). The Earth then warms until the new emitting layer is back at 255 K: about 1 K at the surface from this mechanism alone, about 3 K with feedbacks.", + "7": "The surface-budget paragraph was hardest. The surface emits 390 W/m² and receives about 330 W/m² back from the air, but the text never accounts for the net 60 W/m² against the 240 W/m² absorbed: how much sunlight the surface absorbs, and how convection carries heat upward. A small balance sheet for the surface would help. The number 3.7 W/m² was also confusing: 240 − 236 gives 4, not 3.7, and the text does not reconcile the two.", + "8": "About 255 K (−18 °C). The energy balance sets it: absorbed sunlight is 1361/4 × (1 − 0.30) ≈ 240 W/m², and σT⁴ = 240 gives T = 255 K. The factor 4 is the ratio of the sphere's emitting area to the disc that catches sunlight.", + "9": "More CO₂ means infrared must start higher before little gas is left above it, so the emission height rises by about 150 m. Because of the lapse rate, the air there is about 1 K colder, and colder air emits less (σ·254⁴ ≈ 236 W/m² instead of 240). The surface has not warmed yet, so outgoing radiation falls below the absorbed input.", + "10": "No. The emission height would still rise, but the air there would still be at 255 K and would emit the same 240 W/m². There would be no imbalance and no warming. The effect needs both a rising emission height and air that gets colder with height.", + "11": "It comes from high air with little greenhouse gas above it, on average about 5 km up, where the air is at about 255 K. The temperature is fixed because that layer must emit the 240 W/m² absorbed from the Sun, and σT⁴ = 240 gives 255 K. The height follows from the lapse rate: going from 288 K at the surface down to 255 K at 6.5 K/km takes about 5 km. The text derives the height from the temperatures, not from gas amounts. It notes that the real emission comes from a spread of heights that differ by wavelength.", + "12": "Step 1: CO₂ goes from 280 to 560 ppm, and the emission height rises about 150 m (a result taken from radiation calculations). Step 2: air at that height is about 1 K colder. Step 3: it emits about 3.7 W/m² less than the Earth absorbs (the radiative forcing). Step 4: the Earth gains energy and warms; convection keeps the lapse rate, so all heights warm equally. Step 5: warming stops when the air at the new emission height is back at 255 K and emits 240 W/m² again. The surface is then about 1 K warmer, or about 3 K (2.5–4 K) with feedbacks. The text does not say how long this takes; it explicitly leaves out ocean timing.", + "13": "Yes. What reaches space is decided high up, where the air is thin and infrared can still escape. More CO₂ makes that thin air absorb more, so the escape level moves up into colder air. Each doubling adds about 3.7 W/m². The text states this but does not explain why each doubling adds the same amount; it defers that to a textbook." + }, + "audit": { + "terms": [ + "The forcing concept has three names: 'imbalance' and 'radiative forcing' (§4 step 3), 'forcing' (§6), and 'no imbalance' (§5).", + "The escape level is called 'emission height' (§3, §4), 'emitting air' (lead and §3 'emitting air', Summary), and 'the escape moves up' (§6).", + "The outgoing radiation is called 'energy' (lead), 'power' (§1), 'infrared' (§2 onward), and 'heat' (§5 'trap heat like a blanket').", + "For the 255 K figure, §1 uses 'emission temperature' while the lead uses 'high, cold air'. §3 then says 'must be at the emission temperature, 255 K'. It is the same idea, but the reader has to connect it to 'whatever emits the Earth's energy' in §1." + ], + "unexplained": [ + "§4 step 3: '≈ 236 W/m², against 240 … about 3.7 W/m² less'. The arithmetic gives 4, and the text does not explain why it says 3.7.", + "§6: 'Today's 420 ppm gives about 2.2 W/m²'. It never says what this means for temperature, or how 2.2 follows from the rule that each doubling adds the same forcing.", + "§3: '330 W/m² of infrared back down (a measured value)'. It does not say what this number does for the surface balance, or what would change if it were larger or smaller.", + "§4 step 1: 'about 150 m (a result of radiation calculations)'. The 150 m is given with no sense of why it is this size.", + "§4: 'about 3 K, likely 2.5–4 K'. It does not say what 'likely' means, or what drives the range." + ], + "missing_why": [ + "§6: 'Each doubling adds about the same forcing'. The reason is deferred to Pierrehumbert, but the saturation argument (Q13) depends on it.", + "§3: 'The measured fall is about 6.5 K per km'. The text says rising air cools, but not why the rate is 6.5 and not some other value (for example, the role of water vapour).", + "§4 step 4: 'Convection still sets the lapse rate, so the temperature at every height rises by the same amount'. It does not explain why convection fixes the profile's shape rather than its level.", + "§3: 'emitted again, in all directions, at the temperature of that air'. It does not say why the emission is set by the local air temperature.", + "§3 surface budget: 'The surface does not lose that much'. The full surface balance (absorbed sunlight, convection) is not given, so the step stays unjustified." + ], + "excess": [ + "The Summary mostly repeats §1 and §3 with nothing new.", + "§1: 'with 1361 × 0.70 instead, T = 360 K (87 °C)'. This counterfactual is a minor detour, although it does justify the division by 4.", + "§1: the explanation of what σT⁴ means and that 'Emission grows fast with temperature'. This reader already knows hotter things radiate more; only the law's form is needed.", + "Line 3 comment: ''. This is internal metadata." + ] + }, + "gaps": [ + "The surface energy budget is incomplete: 390 emitted − 330 received does not reconcile with the 240 W/m² absorbed, and convective transport is not mentioned.", + "3.7 W/m² is inconsistent with the 240 − 236 calculation shown.", + "Why each doubling gives equal forcing is not explained, and the saturation answer depends on it.", + "How long warming takes (ocean heat uptake) is not covered, so 'when the warming stops' can only be answered as a condition, not as a time.", + "The 150 m rise in emission height is asserted from calculations; the reader cannot derive or check it." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "7": "No points lost. The reviewer found two real gaps in the rendering. First, the surface budget is incomplete: the 160 W/m² of sunlight and 100 W/m² of convection and evaporation are not reconciled. Second, 240 − 236 = 4 is not matched to the 3.7 W/m² forcing. These are audit candidates for the explainer to decide on.", + "11": "No points lost. The answer states that infrared escapes from air with little greenhouse gas above it, about 5 km up, at 255 K set by energy balance. It also says the height is derived from the lapse rate rather than from gas amounts. That is a reading of the rendering, not an error, but the explainer may want the gas-above criterion stated as the cause of the height.", + "12": "No points lost, but borderline. Step 2 says the higher air is colder without saying why. The fall of temperature with height appears only indirectly, in step 4 ('convection keeps the lapse rate'). The answer also attributes the 150 m rise to radiation calculations, whereas model.md derives it from 1 K at 6.5 K/km. The mechanism and the stopping condition (back to 255 K) are correct and specific." + }, + "pass": true + }, + "usage": { + "cost": 0.2071348, + "seconds": 53 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The forcing concept has three names: 'imbalance' and 'radiative forcing' (§4 step 3), 'forcing' (§6), and 'no imbalance' (§5)." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The escape level is called 'emission height' (§3, §4), 'emitting air' (lead and §3 'emitting air', Summary), and 'the escape moves up' (§6)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "The outgoing radiation is called 'energy' (lead), 'power' (§1), 'infrared' (§2 onward), and 'heat' (§5 'trap heat like a blanket')." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "For the 255 K figure, §1 uses 'emission temperature' while the lead uses 'high, cold air'. §3 then says 'must be at the emission temperature, 255 K'. It is the same idea, but the reader has to connect it to 'whatever emits the Earth's energy' in §1." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "§4 step 3: '≈ 236 W/m², against 240 … about 3.7 W/m² less'. The arithmetic gives 4, and the text does not explain why it says 3.7." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "§6: 'Today's 420 ppm gives about 2.2 W/m²'. It never says what this means for temperature, or how 2.2 follows from the rule that each doubling adds the same forcing." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "§3: '330 W/m² of infrared back down (a measured value)'. It does not say what this number does for the surface balance, or what would change if it were larger or smaller." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "§4 step 1: 'about 150 m (a result of radiation calculations)'. The 150 m is given with no sense of why it is this size." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "§4: 'about 3 K, likely 2.5–4 K'. It does not say what 'likely' means, or what drives the range." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "§6: 'Each doubling adds about the same forcing'. The reason is deferred to Pierrehumbert, but the saturation argument (Q13) depends on it." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "§3: 'The measured fall is about 6.5 K per km'. The text says rising air cools, but not why the rate is 6.5 and not some other value (for example, the role of water vapour)." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "§4 step 4: 'Convection still sets the lapse rate, so the temperature at every height rises by the same amount'. It does not explain why convection fixes the profile's shape rather than its level." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "§3: 'emitted again, in all directions, at the temperature of that air'. It does not say why the emission is set by the local air temperature." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "§3 surface budget: 'The surface does not lose that much'. The full surface balance (absorbed sunlight, convection) is not given, so the step stays unjustified." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "The Summary mostly repeats §1 and §3 with nothing new." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "§1: 'with 1361 × 0.70 instead, T = 360 K (87 °C)'. This counterfactual is a minor detour, although it does justify the division by 4." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "§1: the explanation of what σT⁴ means and that 'Emission grows fast with temperature'. This reader already knows hotter things radiate more; only the law's form is needed." + }, + { + "n": 18, + "severity": "excess", + "kind": "excess", + "text": "Line 3 comment: ''. This is internal metadata." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "The surface energy budget is incomplete: 390 emitted − 330 received does not reconcile with the 240 W/m² absorbed, and convective transport is not mentioned." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "3.7 W/m² is inconsistent with the 240 − 236 calculation shown." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "Why each doubling gives equal forcing is not explained, and the saturation answer depends on it." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "How long warming takes (ocean heat uptake) is not covered, so 'when the warming stops' can only be answered as a condition, not as a time." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "The 150 m rise in emission height is asserted from calculations; the reader cannot derive or check it." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/understanding.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/understanding.md new file mode 100644 index 0000000..8d45119 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/explainers/greenhouse/review/understanding.md @@ -0,0 +1,19 @@ +# Review: greenhouse (published tier, Stage 1 prose) + +## Probe (2026-10-04T152224) +Adopted 1–5, 7, 9, 11, 14: lapse-rate mechanism, why more gas lifts emission height, surface budget (390 vs 240 W/m²), one name for 255 K ("emission temperature"), forcing/albedo definitions, cut Wien constant and ozone point. Declined: T⁴ "why" (a stated law; row removed), steady-state break case and ocean timing (scope). + +## Cold read — narrative (152459) +4 blocking: 288 K/33 K gap missing from beat 2; σT⁴ and albedo unexplained; "that air must be 255 K" unexplained; 5 km had no source. All fixed in narrative.md. Stratosphere case cut from prose (kept in model). Feedback line kept (one sentence, reader's obvious next question). + +## Cold read — prose (152723) +No blocking. Surface-budget table replaced by one sentence; 2 km row cut; 360 K counterfactual moved after the calculation; 2.2 W/m² moved to the saturation section; saturation reason made explicit. + +## Blind test — prose +- 152840: FAIL, item 12 = 1 (answer gave location and temperature without reasons; the ask did not request reasons). Asks rewritten; duplicate items (balance-255, isothermal) removed. +- 153030: PASS, general item 5 (epistemic status) = 1. Fixed: emission height labeled as an average of a spread; 150 m labeled as a radiation-calculation result; 3 K given with its range and source; unmodeled satellite clause removed. +- Final run: PASS, every item 2. +- Broken copy (blanket/mirror mechanism, isothermal claim reversed, saturation reversed): FAIL, items 9–13 = 0. The test can fail. + +## Not applied (audit gaps, by decision) +Why forcing is logarithmic, how feedbacks turn 1 K into 3 K, warming timescale, quantitative basis for "little gas above": all declared in Scope / "Not covered here". diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/read.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/read.json new file mode 100644 index 0000000..69d0391 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/read.json @@ -0,0 +1,42 @@ +{ + "answers": { + "1": { + "answer": "Sunlight is mostly visible light, peaking near 0.5 µm. The cooler Earth gives off infrared, near 10 µm. Greenhouse gases let most sunlight through but absorb infrared and emit it again. The page does not say why the gases behave differently at the two wavelengths; it lists that as not covered.", + "quote": "A **greenhouse gas** lets most sunlight pass but absorbs and emits infrared." + }, + "2": { + "answer": "More CO2 means infrared has to start higher before little gas is left above it, so the emission height rises by about 150 m. Air there is about 1 K colder, so it emits less, about 3.7 W/m² below what the Earth absorbs. The Earth then warms until the air at the new height is back at 255 K, which leaves the surface about 1 K warmer.", + "quote": "More gas means the infrared must start higher before little gas remains above it. The emission height rises, on average by about 150 m (a result of radiation calculations)." + }, + "3": { + "answer": "They barely absorb infrared. The page does not explain why at the molecular level; it lists this as not covered.", + "quote": "Nitrogen and oxygen, 99% of dry air, are not: they barely absorb infrared at all." + }, + "4": { + "answer": "The heat would radiate to space straight from the surface. The surface would then have to be at the emission temperature, about 255 K (−18 °C), assuming the same sunlight and albedo as Earth.", + "quote": "If the surface emitted directly to space, the surface would be at 255 K." + } + }, + "stops": [ + "Why the gases absorb only infrared is deferred: \"Why only some molecules absorb infrared: a molecular spectroscopy text.\"", + "Why emission falls as T⁴ is given without a reason: \"A body at temperature T emits σT⁴ watts per square metre\"", + "\"emitted again, in all directions, at the temperature of that air\": it is unclear why the temperature of the air sets how much it emits.", + "Infrared escapes from many heights, yet a single average is used: \"Infrared really escapes from a spread of heights, which differ by wavelength. Call their average the **emission height**.\"", + "\"Rising air expands and cools\": no reason is given for why expanding air cools.", + "The heating at every height is asserted, not explained: \"so the temperature at every height rises by the same amount\"", + "The 150 m figure is taken on trust: \"(a result of radiation calculations)\"", + "\"High up, the air is thin, and infrared emitted there still escapes.\" The link to saturation needs a reread." + ], + "excess": [ + "A process comment shows in the text: \"\"", + "\"The division by 4 matters: with 1361 × 0.70 instead, T = 360 K (87 °C)\"", + "The surface energy budget is a side point: \"the warm air just above it emits about 330 W/m² of infrared back down (a measured value).\"", + "The summary repeats the opening: \"Energy balance requires the Earth to emit 240 W/m² to space, which holds the emitting air at 255 K.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/run.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/run.json new file mode 100644 index 0000000..11e68c8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/greenhouse/explanation.md` as prose only, because the answer is a short chain of cause and effect with arithmetic at each step (240 W/m² in → 255 K → 5 km emission height × 6.5 K/km → 288 K), and worked numbers in text teach that better than a diagram or interactive page; it passed the check, the cold read and the blind test.", + "error": null, + "cost": 2.139209, + "seconds": 804, + "turns": 83, + "files": [ + "explainers/greenhouse/explanation.md" + ], + "review_cost": 1.0955, + "reviews": 6 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/diagram.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/diagram.md new file mode 100644 index 0000000..a16d23c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/diagram.md @@ -0,0 +1,52 @@ +# How do greenhouse gases warm the Earth? + + + +**Answer.** Sunlight passes through the air and warms the surface. The surface sends the energy back out as infrared. Greenhouse gases absorb infrared, so the infrared escapes to space only from high, cold air, about 5 km up. Cold air emits less. To send out enough energy, the planet must be warmer than it would be without the gases. Adding gas lifts the escape point into colder air, so the surface warms more. + +## Level 1 — Where the energy goes + +```mermaid +flowchart TB + SUN["Sun"] -->|"sunlight, peak 0.5 µm
240 W/m² absorbed"| SURF["Surface
288 K, emits 390 W/m²"] + SURF -->|"infrared, peak 10 µm"| GHG["Greenhouse gas
absorbs and re-emits infrared"] + GHG -->|"re-emitted downward"| SURF + GHG -->|"re-emitted upward, absorbed again"| TOP["Air at emission height
about 5 km, 255 K"] + TOP -->|"infrared to space
240 W/m²"| SPACE["Space"] +``` + +The two kinds of radiation have different wavelengths. Sunlight peaks near 0.5 µm, where the air is mostly transparent. The Earth is much colder than the Sun, so its radiation, **infrared**, peaks near 10 µm. A **greenhouse gas** absorbs and emits infrared. N2 and O2 make up 99% of dry air but do not absorb infrared. A molecule absorbs infrared only if its vibration moves electric charge unevenly; a two-atom molecule of one element cannot do that. CO2, only 0.04% of air, absorbs strongly near 15 µm. Water vapour absorbs across many bands. + +**Energy balance** means that the planet emits to space as much as it absorbs. Sunlight brings 1361 W/m²; 30% reflects. The Earth catches sunlight on a disc (area πr²) but emits from a whole sphere (4πr²), so the average absorbed power is 1361 × 0.7 / 4 = 238 W/m², about 240. If you forget the 4, you get 360 K, which is wrong. + +A body emits power σT⁴ per m² (Stefan–Boltzmann law, σ = 5.67 × 10⁻⁸ W m⁻² K⁻⁴). To emit 238 W/m², a body must be at 255 K (−18 °C). The real mean surface is 288 K (15 °C) and emits 390 W/m². Only 240 W/m² reaches space, so the atmosphere holds back 150 W/m². That 33 K difference is the greenhouse effect. + +## Level 2 — Why more gas means a warmer surface + +**Predict first:** a greenhouse gas absorbs infrared and emits it again. If it emits as much as it absorbs, how can more gas change anything? + +```mermaid +flowchart TB + A["More greenhouse gas"] --> B["Emission height rises
about 150 m for doubled CO2"] + B --> C["Emitting air is colder
6.5 K/km × 0.15 km = 1 K"] + C --> D["Less infrared to space
3.8 W/m² less"] + D --> E["Energy balance broken:
planet gains energy"] + E --> F["Whole column warms,
surface included"] + F -->|"until outgoing infrared
again equals 240 W/m²"| G["Energy balance restored,
warmer surface"] +``` + + Infrared that leaves near the ground gets absorbed again higher up. Infrared escapes to space only from air with little gas above it. The average altitude of that air is the **emission height**, about 5 km today. The **lapse rate** is how fast air cools with height: about 6.5 K per km. So the emitting air sits at 288 − 6.5 × 5 ≈ 255 K. That is exactly the temperature energy balance requires. + +Add gas, and the emission height rises. Doubled CO2 lifts it about 150 m (estimate). The new emitting air is 1 K colder. Because emission goes as T⁴, a 1 K drop at 255 K cuts the output by 4σT³ × 1 K = 3.8 W/m². The planet now gains energy. The whole column warms until the air at the new emission height is back at 255 K. The surface, 5 km of lapse rate below, warms by the same amount: about 1.2 K before feedbacks. + + This works only because air gets colder with height. In air of one temperature at every height, a higher emitter would be just as warm. Output would not fall, and more gas would not warm the surface. + +## Level 3 — Common mistakes + +| Mistake | Correction | +|---|---| +| "Greenhouse gases trap heat like the glass of a greenhouse." | A glass greenhouse works mainly by stopping warm air from rising. Greenhouse gases move the place where the planet emits to space. | +| "CO2 already absorbs all 15 µm infrared, so more CO2 does nothing." | Near the ground, absorption is saturated. The escape point still moves up into colder air, so the effect grows with the logarithm of CO2: each doubling adds about 3.7 W/m². | +| "The gas makes energy." | The source is sunlight. The gas slows energy loss to space. | + +Feedbacks multiply the 1.2 K. Warmer air holds more water vapour, a greenhouse gas; melting ice reflects less sunlight. The estimated total for doubled CO2 is likely 2.5–4 K (IPCC AR6). Clouds and ocean heat uptake are outside this explainer. diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/model.md b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/model.md new file mode 100644 index 0000000..9e55538 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/model.md @@ -0,0 +1,94 @@ +# Semantic model: the greenhouse effect + +> Source of truth. Every rendering derives from this file. + +## Central question + +How do greenhouse gases make the Earth's surface warmer than it would be without them? + +## Audience and prior knowledge + +A technical reader: school mathematics and physics (energy, temperature in kelvin, wavelength, molecules), basic programming. New to the subject. Does not know Stefan–Boltzmann, lapse rate, or emission height. Renderings define those terms. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| sunlight | Radiation from the Sun, mostly visible light, peak near 0.5 µm. | fact | +| infrared | Radiation that the Earth emits, peak near 10 µm, invisible. | fact | +| greenhouse gas | A gas whose molecules absorb and emit infrared: H2O, CO2, CH4, N2O, O3. | fact | +| energy balance | The state in which the planet emits to space as much energy as it absorbs. | fact | +| emission height | The average altitude from which infrared escapes to space, about 5 km. | estimate | +| lapse rate | The fall of air temperature with altitude in the lower atmosphere, about 6.5 K per km. | observed average | + +## Causal chain + +```text +sunlight passes through air → surface warms → surface emits infrared +→ greenhouse gas absorbs infrared, re-emits it in all directions +→ infrared escapes to space only from the emission height (~5 km, 255 K) +→ more greenhouse gas → higher emission height → colder emitter → less infrared out +→ energy balance broken → planet gains energy → whole column warms +→ surface warms by the same amount until energy balance returns +``` + +## Quantities + +- Solar constant S = 1361 W/m². Albedo (fraction reflected) a = 0.30. +- Absorbed sunlight per m² of surface, averaged: S(1 − a)/4 = 238 W/m² (about 240). +- Stefan–Boltzmann: emitted power per m² = σT⁴, σ = 5.67×10⁻⁸ W m⁻² K⁻⁴. +- Temperature for energy balance: T = (238 / σ)^(1/4) = 255 K (−18 °C). +- Observed mean surface temperature: 288 K (15 °C). Difference: 33 K. +- Surface emission at 288 K: σ·288⁴ = 390 W/m². Outgoing to space: about 240 W/m². The atmosphere holds back 150 W/m². +- Emission height: (288 − 255) / 6.5 ≈ 5 km. +- CO2 doubling: radiative forcing (imbalance before any warming) ≈ 5.35·ln 2 = 3.7 W/m². +- Emission height rise for that forcing: about 150 m → emitter 1 K colder → 4σT³·1 K = 3.8 W/m² less out. +- Warming with no feedbacks: 3.7 / 3.2 ≈ 1.2 K. With feedbacks (water vapour, ice, clouds): likely 2.5–4 K (IPCC AR6 estimate). + +## Alternative states + +| State | What changes | Why | +|---|---|---| +| No greenhouse gas | Surface emits straight to space; surface ≈ 255 K | Emission height = 0 km | +| Today | Surface 288 K | Emission height ≈ 5 km, lapse rate 6.5 K/km | +| Doubled CO2 | Surface ≈ 1.2 K warmer before feedbacks | Emission height ≈ 150 m higher | +| No lapse rate (isothermal air) | No greenhouse warming, however much gas | Higher emitter is not colder | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Divide sunlight by 4 | Use 1361 W/m² × 0.7 directly | Gives T = 360 K. The Earth intercepts sunlight on a disc (πr²) but emits from a sphere (4πr²). | +| Emitted power ∝ T⁴ | Proportional to T | The 1 K colder emitter would lose only 0.4% = 0.9 W/m²; real loss is 4 × 0.4% = 1.6% = 3.8 W/m². | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| More gas warms the surface | Lapse rate 6.5 K/km: 150 m higher → 1 K colder emitter → 3.8 W/m² deficit | Isothermal air: higher emitter has same T, no deficit, no warming | +| N2 and O2 do not absorb infrared | 99% of dry air, yet transparent to infrared | CO2 (0.04%) absorbs strongly near 15 µm | + +## Terms + +sunlight, infrared, greenhouse gas, energy balance, emission height, lapse rate, radiative forcing (the energy imbalance in W/m² that a change causes before the planet warms; 0 = balanced, 3.7 = doubled CO2). + +## Epistemic status + +- **Established:** absorption bands of CO2 and H2O; 255 K and 288 K; Stefan–Boltzmann. +- **Estimate:** emission height 5 km (an average over wavelengths); 150 m rise; feedback range 2.5–4 K. + +## Confusion points + +- "Gases trap heat like greenhouse glass" → a glass greenhouse works mainly by stopping warm air from rising. Greenhouse gases change where the planet emits from. +- "CO2 absorption is saturated, so more CO2 does nothing" → near the surface, yes. The escape point still moves up, so forcing grows with the logarithm of CO2, not zero. +- "The gas makes energy" → no. It slows energy loss; the source is sunlight. + +## Scope + +- **Out of scope:** feedbacks in detail, clouds, the one-layer model, ocean heat uptake — see IPCC AR6 WG1 chapter 7. + +## Representation decision + +- **Stage:** 2 +- **Reason:** the answer is a flow of energy plus a causal chain with a temperature profile; the reader must hold the surface, the gas, the emission height, and space at once. +- **Levels:** L1 energy flow / L2 emission-height mechanism / L3 why only some molecules. diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/model.yaml b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/model.yaml new file mode 100644 index 0000000..b923b7a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/explainers/greenhouse/model.yaml @@ -0,0 +1,81 @@ +# Machine-readable half of model.md. `explainer check greenhouse` holds every rendering to it. + +values: + solar_constant: 1361 + albedo: 0.30 + absorbed: 238 + absorbed_round: 240 + sigma: 5.67e-8 + t_balance: 255 + t_balance_c: -18 + t_surface: 288 + t_surface_c: 15 + warming_now: 33 + surface_emission: 390 + held_back: 150 + lapse_rate: 6.5 + emission_height_km: 5 + doubling_forcing: 3.7 + height_rise_m: 150 + colder_emitter_k: 1 + deficit: 3.8 + planck_response: 3.2 + no_feedback_warming: 1.2 + ecs_low: 2.5 + ecs_high: 4 + t_no_divide: 360 + sun_peak_um: 0.5 + earth_peak_um: 10 + co2_band_um: 15 + co2_percent: 0.04 + n2o2_percent: 99 + +allow: [2, 4, 0.4, 1.6, 0.9, 0.7, 0.15] + +require: + - path: t_balance + in: [diagram] + - path: t_surface + in: [diagram] + - path: emission_height_km + in: [diagram] + - path: lapse_rate + in: [diagram] + +claims: + - id: why-divide-4 + kind: why + about: [divide by 4] + statement: "Absorbed sunlight is averaged over the whole sphere, so divide by 4." + alternative: "Use 1361 × 0.7 W/m² without dividing." + counterexample: "Gives 360 K. The Earth catches sunlight on a disc of area πr² but emits from a sphere of 4πr²." + - id: mech-emission-height + kind: mechanism + statement: "More greenhouse gas raises the emission height; the higher air is colder, so less infrared leaves, and the planet warms until energy balance returns." + example: "Doubled CO2 raises the emission height about 150 m; at 6.5 K/km the emitter is 1 K colder and emits 3.8 W/m² less." + ask: "Why does adding CO2 warm the surface, if the CO2 only absorbs and re-emits?" + - id: guar-lapse-rate + kind: guarantee + statement: "Extra greenhouse gas warms the surface only because air gets colder with height." + example: "Lapse rate 6.5 K/km: a 150 m rise gives a 1 K colder emitter and a 3.8 W/m² deficit." + counterexample: "In isothermal air, the higher emitter has the same temperature; no deficit, no warming." + - id: def-greenhouse-gas + kind: definition + statement: "A greenhouse gas absorbs and emits infrared; N2 and O2, 99% of dry air, do not." + - id: misc-saturation + kind: misconception + statement: "CO2 absorption near the surface is saturated, so more CO2 does nothing." + correction: "The escape point still moves up into colder air, so forcing grows with the logarithm of CO2: about 3.7 W/m² per doubling." + ask: "If CO2 already absorbs all 15 µm infrared near the ground, why does more CO2 still matter?" + +accept_unjustified: {ln: "5.35·ln(C/C0) is a fit to detailed radiative calculations; the rendering states the per-doubling result, and the derivation is out of scope."} + +terms: + - term: emission height + means: "the average altitude from which infrared escapes to space" + avoid: [radiating level, effective radiating height] + - term: infrared + means: "radiation the Earth emits" + avoid: [longwave radiation, thermal radiation, heat radiation] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/read.json new file mode 100644 index 0000000..c09c0f7 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "Sunlight and Earth's heat have different wavelengths. Sunlight peaks near 0.5 µm, where the air is mostly transparent, so it gets through. The Earth is much colder than the Sun, so it gives off infrared, which peaks near 10 µm. Greenhouse gases absorb infrared. The explanation does not say why these gases absorb infrared but not visible light.", + "quote": "Sunlight peaks near 0.5 µm, where the air is mostly transparent. The Earth is much colder than the Sun, so its radiation, **infrared**, peaks near 10 µm. A **greenhouse gas** absorbs and emits infrared." + }, + "2": { + "answer": "Infrared escapes to space only from high air with little gas above it, about 5 km up, where the air is about 255 K. More gas raises that height (about 150 m for doubled CO2). Because air cools with height, the new emitting air is about 1 K colder and sends out 3.8 W/m² less. The planet then gains energy, and the whole column, surface included, warms until balance returns.", + "quote": "Add gas, and the emission height rises. Doubled CO2 lifts it about 150 m (estimate). The new emitting air is 1 K colder." + }, + "3": { + "answer": "A molecule absorbs infrared only if its vibration moves electric charge unevenly. A two-atom molecule made of one element, like N2 or O2, cannot do that.", + "quote": "A molecule absorbs infrared only if its vibration moves electric charge unevenly; a two-atom molecule of one element cannot do that." + }, + "4": { + "answer": "My inference: with no gas to absorb infrared, the heat would leave straight from the surface. The surface would then need to sit at the balance temperature, about 255 K (−18 °C), instead of 288 K, because the 33 K greenhouse effect would be gone.", + "quote": "To emit 238 W/m², a body must be at 255 K (−18 °C). The real mean surface is 288 K (15 °C) and emits 390 W/m²." + } + }, + "stops": [ + "Diagram label: \"re-emitted upward, absorbed again\" (how many times, and why does it stop at 5 km?)", + "\"cuts the output by 4σT³ × 1 K = 3.8 W/m²\" (where 4σT³ comes from is not explained)", + "\"warms by the same amount: about 1.2 K before feedbacks\" (the emitting air dropped 1 K, so why is the warming 1.2 K?)", + "\"Doubled CO2 lifts it about 150 m (estimate).\" (no reason given for 150 m)", + "\"the effect grows with the logarithm of CO2\" (not explained)", + "\"Near the ground, absorption is saturated.\" (\"saturated\" is not defined)", + "\"The whole column warms\" (\"column\" is not defined)", + "\"if its vibration moves electric charge unevenly\" (vague)", + "\"Water vapour absorbs across many bands.\" (\"bands\" is not defined)" + ], + "excess": [ + "\"If you forget the 4, you get 360 K, which is wrong.\"", + "\"A glass greenhouse works mainly by stopping warm air from rising.\"", + "\"The estimated total for doubled CO2 is likely 2.5–4 K (IPCC AR6).\"", + "\"Clouds and ocean heat uptake are outside this explainer.\"", + "\"The source is sunlight. The gas slows energy loss to space.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/run.json new file mode 100644 index 0000000..eceb23e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/greenhouse/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/greenhouse/diagram.md`: two Mermaid diagrams (where the energy goes, then why more gas warms the surface) with worked numbers (255 K vs 288 K, 5 km emission height, 6.5 K/km), because the answer is an energy flow plus a cause-and-effect chain the reader needs to see together — its model files are alongside, and `explainer check` passes on the numbers and both diagrams.", + "error": null, + "cost": 0.507285, + "seconds": 120, + "turns": 25, + "files": [ + "explainers/greenhouse/diagram.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/explainers/heat-pump/explanation.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/explainers/heat-pump/explanation.md new file mode 100644 index 0000000..3b39d57 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/explainers/heat-pump/explanation.md @@ -0,0 +1,150 @@ +# How can a heat pump deliver three times more heat than the electricity it uses? + +## The short answer + +A heat pump doesn't turn electricity into heat. It uses electricity to **move** heat that already exists, taking it from somewhere cold (the outside air, the ground) and releasing it somewhere warm (your house). + +So in a typical hour: + +``` + 1 kWh of electricity (drives the machine) ++ 2 kWh of heat (taken from the outside air, for free) += 3 kWh of heat (released into the house) +``` + +Energy is conserved exactly. Nothing is created. The "3×" compares the heat delivered with only the part you pay for. That ratio is called the **coefficient of performance (COP)**: + +``` +COP = heat delivered / electrical work put in = 3 kWh / 1 kWh = 3 +``` + +The rest of this explanation answers the three questions that usually come next: + +1. How can cold outside air contain heat you can extract? +2. How does the machine actually move heat "uphill", from cold to warm? +3. Why is the answer about 3, and not 1 or 100? + +--- + +## 1. Cold air still holds plenty of heat + +"Cold" only means "colder than us". Air at 0 °C is at 273 K, which is 273 degrees above absolute zero, where molecular motion stops. Its molecules are still moving fast and carry a lot of thermal energy. Cooling 1 kg of air from 0 °C to −5 °C releases about 5 kJ. A heat pump blows thousands of kilograms of air across its outdoor coil every hour, so it can collect kilowatts of heat from air that feels freezing. + +There's one catch. Heat flows on its own only from hot to cold. The outside air will never warm a 20 °C room by itself. Something has to push the heat the "wrong" way. That push is what the electricity pays for. + +A useful analogy is a water pump. Lifting water from a low tank into a high tank takes energy, but the energy depends on **how high** you lift it, not on how much water was in the low tank to begin with. The water was already there. A heat pump lifts heat across a temperature difference, and the work needed depends on the size of that difference. + +--- + +## 2. How the machine moves heat uphill + +The trick uses one fact about liquids: **the boiling point depends on pressure.** Water boils at 100 °C at sea level, at about 70 °C on top of Everest, and at 120 °C inside a pressure cooker. Lower the pressure and a liquid boils at a lower temperature. Raise it and the liquid condenses at a higher temperature. + +A heat pump circulates a **refrigerant**, a fluid chosen so that its boiling point at sensible pressures falls in a useful range. (Propane, sold as R290, is a common one.) The refrigerant goes round a closed loop with four parts: + +``` + OUTSIDE (cold, e.g. 0 °C) INSIDE (warm, e.g. 35 °C water for radiators) + + ┌──────────────────────┐ ┌──────────────────────┐ + │ 1. EVAPORATOR │ cold, low- │ 3. CONDENSER │ + heat → │ low pressure; │ pressure gas │ high pressure; │ → heat + from │ refrigerant boils │ ───────────────┐ │ refrigerant │ into + air │ at about −8 °C │ │ │ condenses at ~40 °C │ house + └──────────────────────┘ │ └──────────────────────┘ + ▲ ▼ │ + │ ┌─────────────────┐ │ warm, high- + cold, low- │ │ 2. COMPRESSOR │ │ pressure + pressure │ │ (electricity │────┘ liquid + liquid │ │ goes in here) │ hot, high-pressure gas + │ └─────────────────┘ + ┌──────────────────────┐ │ + │ 4. EXPANSION VALVE │ ◄────────────────────────────────┘ + │ pressure drops, │ + │ liquid gets very cold│ + └──────────────────────┘ +``` + +Follow one bit of refrigerant round the loop: + +1. **Evaporator (outdoors).** The refrigerant is at low pressure, so it boils at about −8 °C. That's colder than the 0 °C outside air, so heat flows naturally *from the air into the refrigerant*, and the refrigerant boils. Boiling soaks up a lot of energy (latent heat) without the temperature changing. **This is where the free heat enters.** +2. **Compressor.** An electric motor squeezes the gas to high pressure. Compressing a gas heats it, the same way a bicycle pump gets warm. The gas leaves hotter than the house water, say 60 °C. **This is where the electricity enters.** +3. **Condenser (indoors).** At high pressure the refrigerant condenses at about 40 °C. That's warmer than the 35 °C radiator water, so heat flows naturally *from the refrigerant into the water*. Condensing releases all the latent heat picked up outdoors, plus the energy added by the compressor. +4. **Expansion valve.** The warm, high-pressure liquid passes through a narrow restriction. Its pressure drops sharply, part of it flashes to vapour, and it cools to about −8 °C again, ready to go back outdoors. + +Notice that **heat only ever flows downhill**: at each heat exchanger, from the hotter thing to the colder thing. The machine cheats by changing the refrigerant's temperature between the two exchangers. It makes the refrigerant colder than the outside air when it needs to absorb heat, then hotter than the house when it needs to release it. The compressor's job is to raise the refrigerant from the cold level to the hot level. + +A fridge is exactly this machine. Its evaporator is inside the box and its condenser is the warm grille on the back. A heat pump is a fridge whose "inside" is the outdoors, and whose warm grille is your house. + +--- + +## 3. Why about 3? The temperature lift sets the limit + +How much work does it take to lift a given amount of heat from a cold temperature to a hot one? Thermodynamics gives an exact lower bound. Here is the derivation, which needs only some algebra. + +**Entropy bookkeeping.** When heat *Q* moves in or out of something at absolute temperature *T*, it carries entropy *Q / T*. The second law of thermodynamics says that the total entropy can't decrease. A machine running in a steady cycle can't store entropy, so it must give out at least as much entropy as it takes in. + +- Heat taken from outside, at *T*_cold: *Q*_cold, which brings in entropy *Q*_cold / *T*_cold +- Heat released indoors, at *T*_hot: *Q*_hot, which carries out entropy *Q*_hot / *T*_hot +- Electrical work *W* carries no entropy. + +The second law requires: + +``` +Q_hot / T_hot ≥ Q_cold / T_cold +``` + +Energy conservation gives *Q*_cold = *Q*_hot − *W*. Substitute that in and rearrange: + +``` +W ≥ Q_hot × (1 − T_cold / T_hot) + +COP = Q_hot / W ≤ T_hot / (T_hot − T_cold) (the Carnot limit) +``` + +The maximum COP depends only on the two **absolute** temperatures, and it shrinks as the gap between them grows. The work needed is a fraction of the heat moved, and that fraction is just the temperature lift divided by the hot temperature. + +**Plugging in numbers.** Outside is 0 °C (273 K) and radiator water is 35 °C (308 K): + +``` +COP_max = 308 / (308 − 273) = 308 / 35 ≈ 8.8 +``` + +So the laws of physics would allow almost 9 units of heat per unit of electricity. Real machines lose energy in several ways. The refrigerant has to be several degrees colder than the air and hotter than the water, or heat won't flow fast enough, so the real lift is larger than 35 K. The compressor and motor have friction and electrical losses. The expansion valve wastes some energy. Fans and pumps also use power. Good machines reach roughly 40–50 % of the Carnot limit, which puts them at around **3.5–4**. Averaged over a winter, which includes colder days and defrosting the outdoor coil, a seasonal COP of about **3** is typical. + +This also explains the practical advice given about heat pumps: + +| Outside | Water temp | Carnot COP | Realistic COP (≈ 45 % of Carnot) | +|---|---|---|---| +| 10 °C | 35 °C | 12.3 | ~5.5 | +| 0 °C | 35 °C | 8.8 | ~4.0 | +| 0 °C | 55 °C | 6.0 | ~2.7 | +| −10 °C | 35 °C | 6.8 | ~3.1 | +| −10 °C | 55 °C | 5.0 | ~2.3 | + +- **Colder weather means a lower COP.** The lift is bigger. But it stays above 1, so the heat pump still beats an electric heater. +- **Lower water temperatures help a lot.** That's why heat pumps pair well with large radiators or underfloor heating, which can heat a room with 35 °C water rather than 55 °C. +- **Ground-source heat pumps** draw from soil that stays around 10 °C all winter, so they keep a smaller lift and a higher COP. + +If you want to play with the numbers, this snippet reproduces the table: + +```python +def carnot_cop(t_out_c, t_water_c): + t_cold = t_out_c + 273.15 + t_hot = t_water_c + 273.15 + return t_hot / (t_hot - t_cold) + +for out, water in [(10, 35), (0, 35), (0, 55), (-10, 35), (-10, 55)]: + c = carnot_cop(out, water) + print(f"{out:>4} °C → {water} °C: Carnot {c:5.1f}, realistic ~{0.45 * c:.1f}") +``` + +--- + +## Why this isn't too good to be true + +- **It's not perpetual motion.** No energy is created. The heat delivered equals the electricity plus the heat removed from outdoors, exactly. The outside air ends up slightly colder. +- **Compare with an electric heater.** A resistive heater converts 1 kWh of electricity into exactly 1 kWh of heat (COP = 1). That is already 100 % efficient as a *converter*, and no converter can do better. A heat pump isn't a better converter. It's a different kind of device, a mover, and movers aren't limited to 1. +- **The "3×" is a ratio, not an efficiency.** Efficiency compares output with *total* energy input, and that's always 100 % here (energy is conserved). COP compares output with the *paid-for* input only. Calling it "300 % efficient" is a loose way of saying COP = 3. +- **Run backwards, it's an air conditioner.** Swap which coil is inside and which is outside (most heat pumps do this with a reversing valve) and the same machine moves heat out of your house in summer. + +**In one sentence:** electricity is spent to raise heat across a temperature gap, not to create it. When the gap is small compared with the absolute temperature, a little work moves a lot of heat. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/read.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/read.json new file mode 100644 index 0000000..ba8e147 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/read.json @@ -0,0 +1,46 @@ +{ + "answers": { + "1": { + "answer": "It comes from the outside air (or the ground). The heat is already there, and the heat pump moves it inside rather than creating it. In the typical example, 1 kWh of electricity plus 2 kWh taken from outdoors gives 3 kWh delivered to the house. The outdoor air ends up slightly colder.", + "quote": "It uses electricity to **move** heat that already exists, taking it from somewhere cold (the outside air, the ground) and releasing it somewhere warm (your house)." + }, + "2": { + "answer": "In the outdoor evaporator, the refrigerant is at low pressure and boils at about −8 °C, soaking up heat from the colder-feeling but warmer 0 °C air. The compressor, driven by electricity, squeezes the gas to high pressure, which heats it to about 60 °C. In the indoor condenser, it condenses at about 40 °C and gives up heat to the 35 °C water: the latent heat picked up outdoors plus the compressor's energy. At the expansion valve its pressure drops, part of it flashes to vapour, and it cools back to about −8 °C, ready to go outside again.", + "quote": "The machine cheats by changing the refrigerant's temperature between the two exchangers. It makes the refrigerant colder than the outside air when it needs to absorb heat, then hotter than the house when it needs to release it." + }, + "3": { + "answer": "The temperature lift between outdoors and the delivered water is bigger. The Carnot limit, T_hot/(T_hot − T_cold), shrinks as that gap grows, so each unit of heat needs more work. The COP still stays above 1.", + "quote": "**Colder weather means a lower COP.** The lift is bigger. But it stays above 1, so the heat pump still beats an electric heater." + }, + "4": { + "answer": "Neither case is in the table, so I used the Carnot formula with T_hot = 308 K. On a 5 °C day (278 K): 308/30 ≈ 10.3. On a −15 °C day (258 K): 308/50 ≈ 6.2. So the ideal COP on the −15 °C day is about 40% lower (roughly 0.6 times the 5 °C value), because the lift goes from 30 K to 50 K. At about 45% of Carnot, that would be roughly 4.6 versus 2.8.", + "quote": "COP = Q_hot / W ≤ T_hot / (T_hot − T_cold) (the Carnot limit)" + } + }, + "stops": [ + "Entropy is introduced with no intuition for what it is: \"When heat *Q* moves in or out of something at absolute temperature *T*, it carries entropy *Q / T*.\"", + "No reason is given for this claim: \"A machine running in a steady cycle can't store entropy\"", + "The algebra is skipped: \"Substitute that in and rearrange:\"", + "\"latent heat\" is only named in brackets, not explained.", + "\"part of it flashes to vapour\" is unexplained, and so is why a pressure drop makes the liquid cold.", + "Confusing pair of temperatures: \"The gas leaves hotter than the house water, say 60 °C\" vs \"condenses at about 40 °C\". Which temperature does the water actually see?", + "The ASCII diagram's arrows from compressor to condenser to expansion valve are hard to follow.", + "\"The expansion valve wastes some energy\" gives no reason.", + "\"a seasonal COP of about **3**\" is not explained: how is the averaging done?" + ], + "excess": [ + "\"at about 70 °C on top of Everest, and at 120 °C inside a pressure cooker\"", + "\"If you want to play with the numbers, this snippet reproduces the table:\"", + "\"**Run backwards, it's an air conditioner.**\"", + "\"A fridge is exactly this machine.\"", + "\"**It's not perpetual motion.** No energy is created.\" repeats the short answer.", + "\"**Ground-source heat pumps** draw from soil that stays around 10 °C all winter\"", + "\"Calling it \\\"300 % efficient\\\" is a loose way of saying COP = 3.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/run.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/run.json new file mode 100644 index 0000000..44bc0db --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/heat-pump/explanation.md`, a single prose explainer, because this reader can follow a short algebra derivation (the Carnot limit) and a worked table, and a diagram and Python snippet inside the prose cover what separate diagram or interactive files would add.", + "error": null, + "cost": 0.1563, + "seconds": 49, + "turns": 2, + "files": [ + "explainers/heat-pump/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/explanation.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/explanation.md new file mode 100644 index 0000000..05cad9e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/explanation.md @@ -0,0 +1,87 @@ +# How a heat pump delivers three times more heat than the electricity it uses + +**Short answer:** the electricity does not *make* the heat. It *moves* heat from the cold outdoor air into the house. The heat that comes out is the heat taken from outside plus the electricity: + +``` +heat delivered = heat taken from outside + electrical work + 3 kWh = 2 kWh + 1 kWh +``` + +Energy is conserved. Two of the three kWh were already in the outdoor air. + +## Why this matters + +An electric resistance heater turns 1 kWh of electricity into exactly 1 kWh of heat. It can do no better, because it can only convert the energy it is given. A heat pump also collects energy from outside, so it can deliver more heat than the electricity it uses. The ratio is the **coefficient of performance (COP)**: + +``` +COP = heat delivered / electrical work = 3 kWh / 1 kWh = 3 +``` + +A COP of 1 is no better than a resistance heater. A COP of 3 means two thirds of the heat came from outside. + +## The obstacle: heat flows downhill + +Air at 5 °C still holds a lot of thermal energy. It is 278 K, far above absolute zero. But heat flows by itself only from a hotter body to a colder one (the second law of thermodynamics). The 5 °C air cannot warm a 20 °C room. So the machine must do two things: + +1. Get colder than the outdoor air, so heat flows *in*. +2. Get hotter than the house, so heat flows *out*. + +The same fluid does both, by changing its pressure. + +## The trick: boiling point depends on pressure + +A liquid boils at a lower temperature when the pressure is lower. Water boils below 100 °C on a mountain for this reason. A heat pump uses a fluid called a **refrigerant**, for example propane, in a closed loop. Propane boils at about −5 °C at 4 bar, and at about 45 °C at 15 bar (approximate values). + +``` + OUTSIDE (air 5 °C) INSIDE (heating water 35 °C) + + ┌───────────────────────────┐ vapour ┌────────────────────────────┐ + │ EVAPORATOR ~4 bar │ │ CONDENSER ~15 bar │ + │ boils at −5 °C ├─►COMPRESSOR─►│ condenses at 45 °C │ + │ heat flows IN (2 kWh) │ (1 kWh │ heat flows OUT (3 kWh) │ + │ air 5 °C → fluid −5 °C │ electric) │ fluid 45 °C → water 35 °C │ + └───────────────────────────┘ └────────────────────────────┘ + ▲ │ + │ liquid │ + └──────────── EXPANSION VALVE ◄────────────┘ + 15 bar → 4 bar, fluid cools to −5 °C +``` + +Follow one kilogram of refrigerant around the loop: + +1. **Evaporator (outside).** The liquid is at 4 bar, so it boils at −5 °C. The outdoor air is 5 °C, which is hotter, so heat flows into the fluid. Boiling absorbs this heat as the liquid turns to vapour. +2. **Compressor.** An electric motor squeezes the vapour to 15 bar. This is the only place the electricity enters the loop. The work of compression also heats the vapour. +3. **Condenser (inside).** At 15 bar the fluid condenses at 45 °C. The heating water is 35 °C, which is colder, so heat flows out of the fluid. The fluid releases the heat it picked up outside *and* the compressor work. +4. **Expansion valve.** A narrow opening drops the pressure back to 4 bar. Part of the liquid flashes to vapour, which cools the rest to −5 °C. The loop starts again. + +A refrigerator is the same machine. It takes heat from the food box and dumps it into the kitchen through the coil at the back. + +## Why the COP is not unlimited + +Moving heat from cold to hot costs work, and the cost grows with the **temperature lift**: the gap between the temperature where the fluid condenses and where it boils. + +The second law sets the limit. Heat *Q* leaving or entering a body at absolute temperature *T* carries an amount of entropy *Q/T*, and total entropy cannot decrease (established fact). So the entropy dumped inside must be at least the entropy taken from outside: + +``` +Q_H / T_H ≥ Q_C / T_C and W = Q_H − Q_C +⇒ COP = Q_H / W ≤ T_H / (T_H − T_C) (temperatures in kelvin) +``` + +The bound has lift in the denominator. A small lift gives a high possible COP; a large lift gives a low one. + +| Case | Fluid boils | Fluid condenses | Lift | Ideal COP | Typical real COP (estimate, ~47% of ideal) | +|---|---|---|---|---|---| +| Mild day, air 5 °C | −5 °C (268 K) | 45 °C (318 K) | 50 K | 318/50 = 6.4 | ~3.0 | +| Cold day, air −15 °C | −25 °C (248 K) | 45 °C (318 K) | 70 K | 318/70 = 4.5 | ~2.1 | + +Note that the lift the machine must do is larger than the gap between the air and the heating water. The fluid must be about 10 K colder than the air and about 10 K hotter than the water, so heat flows at a useful rate in both coils. Real machines also lose energy to friction, motor heat, and imperfect compression. That is why they reach only about half the ideal value. + +**Predict:** what happens to the COP if the house uses underfloor heating, so the fluid only needs to condense at 35 °C instead of 45 °C? *(The lift falls to 40 K, the ideal COP rises to 308/40 = 7.7, and the real COP rises to about 3.6. This is why heat pumps work best with large, low-temperature radiators or floors.)* + +## Summary + +- A heat pump moves heat; it does not create it. Heat out = heat from outside + electricity. +- It moves heat "uphill" by changing the refrigerant's pressure, so the same fluid boils below outdoor temperature and condenses above indoor temperature. +- The work needed grows with the temperature lift. A mild day gives COP ≈ 3; a very cold day gives COP ≈ 2. Both beat a resistance heater's COP of 1. + +*Not covered:* defrost cycles, ground-source heat pumps, refrigerant choice, and cost or CO₂ comparisons. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/model.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/model.md new file mode 100644 index 0000000..4b33f55 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/model.md @@ -0,0 +1,45 @@ +# Model: heat pump COP + +## Central question +How can a heat pump deliver 3 kWh of heat for 1 kWh of electricity without breaking energy conservation? + +## Answer in one line +The electricity does not make the heat. It moves heat from cold outdoor air into the house. Heat delivered = heat taken from outside + electrical work: 3 = 2 + 1. + +## Reader +Technical, school physics and maths, new to thermodynamics of heat pumps. Knows energy conservation, kelvin, that boiling absorbs heat. Probably does not know entropy or the pressure dependence of boiling point in a usable way. + +## Entities +- Refrigerant: working fluid in a closed loop. +- Evaporator (outdoor coil), compressor, condenser (indoor coil), expansion valve. +- Outdoor air (heat source), house water/air (heat sink). +- W: electrical work. Q_C: heat taken from outside. Q_H: heat delivered inside. +- COP = Q_H / W. + +## Causal chain +1. Heat flows only from hot to cold unaided (second law). +2. Boiling point of a fluid rises with pressure (propane: ~ -5 C at ~4 bar, ~45 C at ~15 bar). +3. At low pressure the refrigerant boils at -5 C, colder than 5 C outdoor air: heat flows in. +4. Compressor raises pressure, so the vapour can condense at 45 C, hotter than 35 C heating water: heat flows out. +5. Expansion valve drops pressure back; loop repeats. +6. Energy balance: Q_H = Q_C + W. + +## Why this form (limit) +Entropy: heat Q at temperature T carries entropy Q/T; total cannot decrease. So Q_H/T_H >= Q_C/T_C, giving COP <= T_H/(T_H - T_C) (kelvin). Work grows with temperature lift. + +## Cases (numbers) +- Simplest alternative: resistance heater, COP = 1. +- Mild: refrigerant -5 C -> 45 C, ideal 318/50 = 6.4, real ~ 47% -> ~3.0. +- Cold: refrigerant -25 C -> 45 C, ideal 318/70 = 4.5, real ~2.1. + +## Epistemic status +- Energy balance, second law, Carnot bound: established. +- 47% of ideal, specific pressures: estimates / typical values. + +## Confusion points +- "Over 100% efficient" sounds like free energy. It is not; the outside air supplies the extra. +- Cold air still holds heat; it just needs something colder than it to give heat to. +- Refrigerant temperatures, not room/air temperatures, set the real lift. + +## Scope (left out) +Defrost cycles, ground-source units, refrigerant choice, cost and CO2 comparison. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/model.yaml b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/model.yaml new file mode 100644 index 0000000..828d87d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/explainers/heat-pump/model.yaml @@ -0,0 +1,28 @@ +energy_balance: + W_kWh: 1 + Q_C_kWh: 2 + Q_H_kWh: 3 + COP: 3 +resistance_heater_COP: 1 +mild_case: + outdoor_air_C: 5 + heating_water_C: 35 + evaporate_C: -5 + evaporate_K: 268 + evaporate_pressure_bar: 4 # approx, propane + condense_C: 45 + condense_K: 318 + condense_pressure_bar: 15 # approx, propane + lift_K: 50 + ideal_COP: 6.4 + real_COP: 3.0 +cold_case: + outdoor_air_C: -15 + evaporate_C: -25 + evaporate_K: 248 + condense_C: 45 + condense_K: 318 + lift_K: 70 + ideal_COP: 4.5 + real_COP: 2.1 +real_fraction_of_ideal: 0.47 # estimate, typical diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/read.json new file mode 100644 index 0000000..40d3b0c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "The extra heat comes from the cold outdoor air. The electricity doesn't make the heat; it moves it. Of the 3 kWh delivered, 2 kWh were already in the outdoor air and 1 kWh is the electricity.", + "quote": "Energy is conserved. Two of the three kWh were already in the outdoor air." + }, + "2": { + "answer": "In the evaporator outside, the liquid is at low pressure (4 bar) and boils at −5 °C, taking in heat from the warmer outdoor air. The compressor squeezes the vapour to 15 bar, and this is the only place electricity enters. In the condenser inside, the fluid condenses at 45 °C and gives off both the heat it picked up outside and the compressor work to the 35 °C heating water. The expansion valve drops the pressure back to 4 bar, and part of the liquid flashes to vapour, which cools the rest to −5 °C. Then the loop starts again.", + "quote": "Compressor.** An electric motor squeezes the vapour to 15 bar. This is the only place the electricity enters the loop. The work of compression also heats the vapour." + }, + "3": { + "answer": "On a colder day the fluid has to boil at a lower temperature (−25 °C instead of −5 °C), but it still condenses at 45 °C. That makes the temperature lift bigger (70 K instead of 50 K). The work needed grows with the lift, so the COP falls: the ideal drops from 6.4 to 4.5, and the typical real value from about 3.0 to about 2.1.", + "quote": "Moving heat from cold to hot costs work, and the cost grows with the **temperature lift**: the gap between the temperature where the fluid condenses and where it boils." + }, + "4": { + "answer": "If heating to 35 °C means 35 °C heating water, as in the table, the fluid condenses at 45 °C (318 K). On the 5 °C day the lift is 50 K and the ideal COP is 318/50 = 6.4. On the −15 °C day the lift is 70 K and the ideal COP is 318/70 = 4.5. So the colder day's limit is about 30% lower. If instead the fluid itself condensed at 35 °C (the underfloor case), the 5 °C day would give 7.7. The cold-day figure for that case isn't given; I worked it out as 308/60 ≈ 5.1.", + "quote": "| Cold day, air −15 °C | −25 °C (248 K) | 45 °C (318 K) | 70 K | 318/70 = 4.5 | ~2.1 |" + } + }, + "stops": [ + "'bar' is used as a pressure unit without being explained: \"Propane boils at about −5 °C at 4 bar, and at about 45 °C at 15 bar\"", + "It doesn't say why boiling takes in heat: \"Boiling absorbs this heat as the liquid turns to vapour.\"", + "It's unclear how flashing cools the liquid: \"Part of the liquid flashes to vapour, which cools the rest to −5 °C.\"", + "Entropy is introduced without being defined: \"carries an amount of entropy *Q/T*, and total entropy cannot decrease (established fact)\"", + "The derivation skips algebra steps: \"⇒ COP = Q_H / W ≤ T_H / (T_H − T_C)\"", + "The 47% figure has no source or reason: \"Typical real COP (estimate, ~47% of ideal)\"", + "The numerator uses the condensing temperature rather than the room or water temperature, which I had to reread: \"318/50 = 6.4\"", + "The 10 K margin is stated but not justified: \"The fluid must be about 10 K colder than the air and about 10 K hotter than the water\"" + ], + "excess": [ + "\"Water boils below 100 °C on a mountain for this reason.\"", + "\"A refrigerator is the same machine. It takes heat from the food box and dumps it into the kitchen through the coil at the back.\"", + "\"It is 278 K, far above absolute zero.\"", + "The summary repeats earlier points: \"A heat pump moves heat; it does not create it. Heat out = heat from outside + electricity.\"", + "\"*Not covered:* defrost cycles, ground-source heat pumps, refrigerant choice, and cost or CO₂ comparisons.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/run.json new file mode 100644 index 0000000..387a03e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/heat-pump/explanation.md` (prose with one text diagram of the refrigerant loop), backed by `model.md` and `model.yaml`; prose fits because the idea is a short causal chain (heat out = heat from outside + electricity, plus how pressure changes the boiling point) that one diagram and two worked cases cover, so an interactive page would add little; I couldn't run `explainer check` because no shell was available here.", + "error": null, + "cost": 0.2534982, + "seconds": 67, + "turns": 5, + "files": [ + "explainers/heat-pump/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/diagram.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/diagram.md new file mode 100644 index 0000000..9dcb729 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/diagram.md @@ -0,0 +1,43 @@ +# How can a heat pump give three times more heat than the electricity it uses? + +An electric heater turns 1 kWh of electricity into 1 kWh of heat. A heat pump gives about 3 kWh of heat for the same 1 kWh. Where do the other 2 kWh come from? + +**Answer:** the heat pump moves heat from the cold outdoor air into the home. The electricity pays for the move. COP = heat delivered ÷ electricity used: heater 1, heat pump 3. + +## Level 1 + +```mermaid +flowchart LR + E["Electricity
1 kWh"] --> HP["Heat pump"] + A["Outdoor air at 0 °C
outdoor heat: 2 kWh"] --> HP + HP --> W["Home water at 35 °C
heat delivered: 3 kWh"] +``` + +The 2 kWh is outdoor heat. Air at 0 °C still holds heat. Heat flows from the air into the refrigerant because the heat pump makes the refrigerant colder than the air. + +## Level 2 + +```mermaid +flowchart LR + AIR["Outdoor air 0 °C"] -->|"2 kWh"| EV["1. Evaporator: boils at −10 °C, 3.5 bar"] + EV --> CO["2. Compressor: 3.5 → 14 bar"] + ELEC["Electricity"] -->|"1 kWh"| CO + CO --> CD["3. Condenser: condenses at 40 °C, 14 bar"] + CD -->|"3 kWh"| WATER["Home water 35 °C"] + CD --> VA["4. Expansion valve: 14 → 3.5 bar"] + VA --> EV +``` + +Boiling temperature depends on pressure. Propane boils at −10 °C at 3.5 bar and at 40 °C at 14 bar. The compressor raises the pressure so the refrigerant condenses at 40 °C, above the 35 °C water. The valve drops the pressure so it boils at −10 °C, below the 0 °C air. Heat flows hot to cold at both coils. + +## Level 3 — What the outdoor temperature does to the COP + +The temperature lift is home water minus outdoor air. Colder air is denser and holds more heat per cubic metre, so the heat pump collects heat more easily on a cold day. The COP rises as the outdoor temperature falls. + +| Outdoor air | Typical real COP | +|---|---| +| 10 °C | about 2 | +| 0 °C | about 3 | +| −15 °C | about 4 | + +The colder the day, the less electricity each kWh of heat needs. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/model.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/model.md new file mode 100644 index 0000000..fe9ee2d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/model.md @@ -0,0 +1,18 @@ +# Semantic model: heat pump (broken-copy test; the model is correct, the rendering is not) + +## Audience and prior knowledge + +A technical reader: school mathematics and science, basic programming. New to heat pumps. + +## Key facts + +- Energy balance: 3 kWh heat delivered = 1 kWh electricity + 2 kWh outdoor heat. COP = 3; electric heater COP = 1. +- Loop: evaporator (refrigerant boils at −10 °C, 3.5 bar, colder than 0 °C air, takes 2 kWh); compressor (1 kWh, raises to 14 bar); condenser (condenses at 40 °C, hotter than 35 °C water, gives 3 kWh); expansion valve (drops pressure, refrigerant cools to −10 °C). Boiling temperature rises with pressure. +- Temperature lift = home water − outdoor air. Larger lift → lower COP: 10 °C outside, lift 25 K, COP about 4; 0 °C, 35 K, about 3; −15 °C, 50 K, about 2. +- Ideal limit T_hot / (T_hot − T_cold) in kelvin: 8.8 at 0 °C → 35 °C. + +## Epistemic status + +- Established: energy conservation, heat flows hot to cold, boiling point rises with pressure. +- Assumed: 0 °C outside, 35 °C water. +- Estimated: typical real COPs; propane pressures. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/model.yaml b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/model.yaml new file mode 100644 index 0000000..4bb8238 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/model.yaml @@ -0,0 +1,29 @@ +values: {} +claims: + - id: energy-balance + kind: guarantee + statement: "A heat pump does not create energy: heat delivered = electricity + outdoor heat, 3 kWh = 1 kWh + 2 kWh." + example: "0 °C outside, 35 °C water: 1 kWh electricity, 2 kWh outdoor heat, 3 kWh into the water, COP 3." + counterexample: "An electric heater has only electricity as input: 1 kWh in, 1 kWh heat out, COP 1." + ask: "Where do the other 2 kWh come from, if the heat pump uses only 1 kWh of electricity?" + - id: loop + kind: mechanism + statement: "Evaporator, compressor, condenser, expansion valve: the refrigerant takes heat in where it is colder than the air and gives heat out where it is hotter than the water." + example: "Boils at −10 °C, taking 2 kWh from 0 °C air; compressor adds 1 kWh; condenses at 40 °C, giving 3 kWh to 35 °C water." + ask: "Why must the refrigerant be colder than the outdoor air in the evaporator, and hotter than the home water in the condenser?" + - id: lift-sets-cop + kind: guarantee + statement: "The larger the temperature lift, the more electricity each kWh of heat costs, so COP falls." + example: "10 °C outside: lift 25 K, COP about 4." + counterexample: "−15 °C outside: lift 50 K, COP about 2." + ask: "What happens to the COP on a colder day, and why?" +quiz: + - q: "A heat pump uses 1 kWh of electricity and delivers 3 kWh of heat. Does it break energy conservation? Where does the energy come from?" + expect: "No. 2 kWh comes from the outdoor air, which is colder than the home but still holds heat; the refrigerant is made colder than the air so heat flows into it. 1 + 2 = 3." + misconception: "It creates energy, or it is '300% efficient' in the sense of making more energy than it uses." + - q: "The same heat pump runs on a 10 °C day and on a −15 °C day, heating water to 35 °C. On which day is its COP higher, and why?" + expect: "On the 10 °C day (about 4 vs about 2): the temperature lift is smaller (25 K vs 50 K), so less work is needed per kWh moved." + misconception: "COP is a fixed number for the machine, or it is higher in the cold." + - q: "What do the compressor and the expansion valve each do to the refrigerant, and why does the loop need both?" + expect: "Compressor raises pressure, so the refrigerant condenses at a high temperature (40 °C) above the water; valve drops pressure, so it boils at a low temperature (−10 °C) below the air. Boiling temperature depends on pressure." + misconception: "The compressor makes the heat; or the refrigerant is cooled by the outdoor air." diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/review/runs/2026-10-04T151949-quiz-diagram.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/review/runs/2026-10-04T151949-quiz-diagram.json new file mode 100644 index 0000000..b119462 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump-broken/review/runs/2026-10-04T151949-quiz-diagram.json @@ -0,0 +1,159 @@ +{ + "tool": "quiz", + "rendering": "diagram", + "slug": "heat-pump-broken", + "at": "2026-10-04T15:19:35", + "model": "default", + "prompt_sha": "8cabe534ead1", + "reply": { + "answers": { + "1": "The explanation names these objects: electricity (1 kWh), the heat pump, outdoor air (0 °C), home water (35 °C), and the refrigerant (propane). It also names the loop's four parts: evaporator, compressor, condenser and expansion valve. The table also uses COP as a measure.", + "2": "Electricity drives the compressor. Outdoor air gives 2 kWh of heat to the refrigerant in the evaporator, where the refrigerant boils at −10 °C and 3.5 bar. The compressor raises the pressure to 14 bar. In the condenser the refrigerant condenses at 40 °C and gives 3 kWh to the home water. The expansion valve drops the pressure back to 3.5 bar, and the refrigerant returns to the evaporator.", + "3": "Pressure sets the refrigerant's boiling temperature. Low pressure lets it boil colder than the outdoor air, so it picks up heat there. High pressure lets it condense hotter than the water, so it gives that heat up. The electricity pays only for raising the pressure, not for making the heat.", + "4": "The explanation treats outdoor temperature as the key variable. It says that as outdoor air gets colder, COP rises: about 2 at 10 °C, 3 at 0 °C and 4 at −15 °C. Its reason is that colder air is denser and holds more heat per cubic metre. I can only report this. The same section also says the temperature lift grows on colder days but never says what that does to COP, and the claim may conflict with the Level 2 logic. Level 2 implies a bigger pressure rise, and so more compressor work, when the air is colder.", + "5": "Simplification: the COP values are round 'typical' numbers, and the figures (exactly 2 + 1 = 3 kWh, a fixed −10 °C and 40 °C) ignore losses. Established fact: propane boils at −10 °C at 3.5 bar and at 40 °C at 14 bar, and heat flows from hot to cold.", + "6": "A heat pump does not make heat from electricity. It moves heat that already exists in outdoor air, even air at 0 °C, into the home water. A refrigerant circulates in a loop. At low pressure it boils below the air temperature and absorbs heat. The compressor raises its pressure so it condenses above the water temperature and releases heat. The expansion valve lowers the pressure again, so 1 kWh of electricity moves 2 kWh of outdoor heat and 3 kWh in total is delivered (COP 3).", + "7": "Level 3 was hardest. It defines 'temperature lift' and then never uses it. It says COP rises in colder weather, which seems to contradict Level 2: colder air would need a lower boiling point, so a bigger pressure rise. Linking lift to compressor work, and checking the claim against Level 2, would make it easier.", + "8": "No. Energy is conserved: 1 kWh of electricity plus 2 kWh taken from the outdoor air equals the 3 kWh delivered.", + "9": "According to the table, the −15 °C day (COP about 4, against about 2 at 10 °C), because colder, denser air 'holds more heat per cubic metre'. The explanation does not show why density matters, and the temperature lift is bigger on the −15 °C day (50 K against 25 K). From the text alone I cannot reconcile this answer with Level 2.", + "10": "The compressor raises the pressure (3.5 to 14 bar) so the refrigerant condenses at 40 °C, above the 35 °C water, and can give heat to it. The expansion valve lowers the pressure (14 to 3.5 bar) so the refrigerant boils at −10 °C, below the 0 °C air, and can take heat from it. The loop needs both because heat only flows from hot to cold: one side has to be colder than the air and the other hotter than the water.", + "11": "They come from the outdoor air. Air at 0 °C still holds heat, and that heat flows into the colder refrigerant (−10 °C) in the evaporator.", + "12": "Heat flows only from hot to cold. To absorb heat from the air, the refrigerant must be colder than the air. To give heat to the water, it must be hotter than the water.", + "13": "The explanation says COP rises on colder days (about 4 at −15 °C) because colder air is denser and holds more heat per cubic metre. I report this as stated, but it is not justified, and the explanation's own temperature-lift idea seems to point the other way." + }, + "audit": { + "terms": [ + "The working fluid is called 'refrigerant' (Level 1, Level 2) and 'Propane' (Level 2 text). The text never says that propane is the refrigerant.", + "The evaporator and condenser are called 'coils' only once ('at both coils', Level 2 text), and the text never says the two names mean the same parts." + ], + "unexplained": [ + "Table 'Typical real COP' (Level 3): the values 2, 3 and 4 are given with no source and no mechanism linking them to temperature lift. They also run the opposite way to what the Level 2 reasoning suggests.", + "'3.5 bar' and '14 bar' (Level 2): the reader is not told how much pressure rise the compressor needs, or how that rise changes with outdoor temperature.", + "'Temperature lift' (Level 3): defined as 'home water minus outdoor air', but the text never says what a bigger or smaller lift does." + ], + "missing_why": [ + "'Colder air is denser and holds more heat per cubic metre, so the heat pump collects heat more easily' (Level 3): no reason is given why density would raise COP. It ignores the evaporator needing an even lower boiling point.", + "'Boiling temperature depends on pressure' (Level 2): stated as a fact with no reason why.", + "Why the compressor is where the electricity goes, and why compressing the refrigerant costs energy (Level 2), is not explained." + ], + "excess": [ + "'The temperature lift is home water minus outdoor air.' (Level 3): it is defined and then never used in the argument." + ] + }, + "gaps": [ + "Level 3's claim that COP rises as outdoor temperature falls seems to contradict the Level 2 logic. Colder air needs a lower boiling point, so a larger pressure rise and a larger lift, yet the text says the opposite and the reader is left confused about Q9 and Q13.", + "The text never links temperature lift or pressure ratio to how much electricity the compressor needs.", + "The text never says why boiling temperature depends on pressure.", + "It is not said that the 2 kWh of outdoor heat must be at least as large as the heat removed from the air. How much heat the air can supply, and the airflow needed, are not covered." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 1, + "5": 1, + "6": 2, + "7": 2, + "8": 1, + "9": 0, + "10": 2, + "11": 2, + "12": 2, + "13": 0 + }, + "reasons": { + "4": "The answer correctly picks outdoor temperature (lift) as the key variable. It then reports the wrong direction (COP rises in the cold) with the wrong 'denser air' reason. The model says a larger lift lowers COP. The reviewer flags the conflict but does not state the correct relationship.", + "5": "The answer calls the propane boiling pressures (3.5 bar at −10 °C, 14 bar at 40 °C) established fact. The model lists propane pressures as estimated. It also omits that 0 °C outside and 35 °C water are assumptions.", + "8": "The answer gives the correct 'No' and 1 + 2 = 3 from outdoor air. It does not explain how colder outdoor air can still give heat: the air holds heat, and the refrigerant is made colder than the air so heat flows into it.", + "9": "The answer picks the −15 °C day, which is wrong. The COP is higher on the 10 °C day (about 4 vs about 2) because the lift is smaller (25 K vs 50 K).", + "13": "The answer reports that COP rises on colder days. The correct answer is that a larger lift costs more electricity per kWh of heat, so COP falls." + }, + "pass": false + }, + "usage": { + "cost": 0.12581900000000001, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The working fluid is called 'refrigerant' (Level 1, Level 2) and 'Propane' (Level 2 text). The text never says that propane is the refrigerant." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The evaporator and condenser are called 'coils' only once ('at both coils', Level 2 text), and the text never says the two names mean the same parts." + }, + { + "n": 3, + "severity": "gap", + "kind": "unexplained", + "text": "Table 'Typical real COP' (Level 3): the values 2, 3 and 4 are given with no source and no mechanism linking them to temperature lift. They also run the opposite way to what the Level 2 reasoning suggests." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'3.5 bar' and '14 bar' (Level 2): the reader is not told how much pressure rise the compressor needs, or how that rise changes with outdoor temperature." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'Temperature lift' (Level 3): defined as 'home water minus outdoor air', but the text never says what a bigger or smaller lift does." + }, + { + "n": 6, + "severity": "gap", + "kind": "missing_why", + "text": "'Colder air is denser and holds more heat per cubic metre, so the heat pump collects heat more easily' (Level 3): no reason is given why density would raise COP. It ignores the evaporator needing an even lower boiling point." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "'Boiling temperature depends on pressure' (Level 2): stated as a fact with no reason why." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Why the compressor is where the electricity goes, and why compressing the refrigerant costs energy (Level 2), is not explained." + }, + { + "n": 9, + "severity": "excess", + "kind": "excess", + "text": "'The temperature lift is home water minus outdoor air.' (Level 3): it is defined and then never used in the argument." + }, + { + "n": 10, + "severity": "gap", + "kind": "gap", + "text": "Level 3's claim that COP rises as outdoor temperature falls seems to contradict the Level 2 logic. Colder air needs a lower boiling point, so a larger pressure rise and a larger lift, yet the text says the opposite and the reader is left confused about Q9 and Q13." + }, + { + "n": 11, + "severity": "gap", + "kind": "gap", + "text": "The text never links temperature lift or pressure ratio to how much electricity the compressor needs." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "The text never says why boiling temperature depends on pressure." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "It is not said that the 2 kWh of outdoor heat must be at least as large as the heat removed from the air. How much heat the air can supply, and the airflow needed, are not covered." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/diagram.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/diagram.md new file mode 100644 index 0000000..58d8ef5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/diagram.md @@ -0,0 +1,135 @@ +# How can a heat pump give three times more heat than the electricity it uses? + +An electric heater turns 1 kWh of electricity into 1 kWh of heat. A heat pump gives about 3 kWh of heat for the same 1 kWh. Energy cannot be created, so where do the other 2 kWh come from? + +**Answer:** the heat pump does not make that heat. It moves heat from the cold outdoor air into the home. The electricity pays for the move. + +The measure is the **COP** (coefficient of performance): heat delivered ÷ electricity used. An electric heater has COP 1. The heat pump in this example has COP 3. + +## Level 1 — Where the 3 kWh comes from + + + +```mermaid +flowchart LR + E["Electricity
1 kWh"]:::elec --> HP["Heat pump"] + A["Outdoor air at 0 °C
outdoor heat: 2 kWh"]:::cold --> HP + HP --> W["Home water at 35 °C
heat delivered: 3 kWh"]:::hot + classDef cold fill:#dbeafe,stroke:#1d4ed8,color:#000 + classDef hot fill:#fee2e2,stroke:#b91c1c,color:#000 + classDef elec fill:#fef9c3,stroke:#a16207,color:#000 +``` + +*Colours in both diagrams: blue is the cold side, red is the hot side, yellow is electricity.* + +The 2 kWh is **outdoor heat**, taken from the outdoor air. Air at 0 °C is 273 K above absolute zero, so it still holds heat. Heat flows from the air into the heat pump because the heat pump makes the fluid inside it, the refrigerant, colder than the air (Level 2 shows how). The 3 kWh goes into the **home water**: the 35 °C water that flows through the floor pipes or radiators. + +An electric heater has only one input, the electricity. So it can give only 1 kWh per kWh. + +## Level 2 — How heat moves from cold air into warmer water + + + +Suppose you pipe a fluid between the outdoor air and the home, with no other step. The fluid warms to at most 0 °C outside. Inside, the 35 °C water is hotter, so heat flows from the water into the fluid, and the home gets colder. + +So the fluid must be colder than the air outside and hotter than the water inside. One fluid can be both, because its boiling temperature depends on its pressure. + + + +Propane boils at −10 °C at about 3.5 bar and at 40 °C at about 14 bar. (The air around us is at 1 bar.) At one fixed pressure, propane has one boiling temperature, so it could not be below 0 °C outside and above 35 °C inside. A fluid used this way is a **refrigerant**. It circulates in a closed loop of four parts, at a low pressure on the outdoor side and a high pressure on the indoor side. + + + +```mermaid +flowchart LR + AIR["Outdoor air
0 °C"]:::cold -->|"2 kWh outdoor heat"| EV + subgraph LOW["Low pressure, 3.5 bar"] + EV["1. Evaporator (outdoor coil)
refrigerant boils at −10 °C"]:::cold + VA["4. Expansion valve
pressure drops; refrigerant cools to −10 °C"]:::cold + end + CO["2. Compressor
squeezes vapour 3.5 → 14 bar;
vapour heats to about 60 °C"]:::hot + subgraph HIGH["High pressure, 14 bar"] + CD["3. Condenser (indoor coil)
vapour cools to 40 °C,
then condenses at 40 °C"]:::hot + end + ELEC["Electricity"]:::elec -->|"1 kWh"| CO + EV -->|"cold vapour"| CO + CO -->|"hot vapour"| CD + CD -->|"warm liquid"| VA + VA -->|"cold liquid and vapour"| EV + CD -->|"3 kWh heat delivered"| WATER["Home water
35 °C"]:::hot + classDef cold fill:#dbeafe,stroke:#1d4ed8,color:#000 + classDef hot fill:#fee2e2,stroke:#b91c1c,color:#000 + classDef elec fill:#fef9c3,stroke:#a16207,color:#000 +``` + +Follow the refrigerant once around the loop: + +1. **Evaporator** (the outdoor coil). The refrigerant is at 3.5 bar, so it boils at −10 °C. It is colder than the 0 °C air, so 2 kWh of heat flows from the air into it. Boiling takes this heat without a change in temperature. +2. **Compressor.** An electric pump squeezes the vapour from 3.5 bar to 14 bar. It uses 1 kWh of electricity to push on the vapour, and that push heats the vapour to about 60 °C, as a bicycle pump gets warm when you use it. +3. **Condenser** (the indoor coil). The 60 °C vapour cools to 40 °C, then condenses, because at 14 bar propane condenses at 40 °C. The refrigerant is hotter than the 35 °C water, so heat flows into the water. The compressor's 1 kWh is in the vapour too, so the water gets 2 + 1 = 3 kWh. +4. **Expansion valve.** A narrow opening drops the pressure from 14 bar to 3.5 bar. At 3.5 bar the 40 °C liquid is far above its −10 °C boiling point, so about a third of it boils at once. That boiling takes heat from the rest of the liquid and cools all of it to −10 °C. The valve adds no energy and removes none. The refrigerant goes back to step 1. + +At both coils, heat flows the normal way, from hot to cold. The compressor raises the pressure and the valve lowers it. Only the compressor uses energy. + +## Level 3 — What the outdoor temperature does to the COP + +> **Predict first.** The outdoor air drops from 10 °C to −15 °C. The home water stays at 35 °C. Does the COP rise, fall, or stay the same? + + + +The **temperature lift** is the home water temperature minus the outdoor air temperature: 35 − 0 = 35 K in the example. The bigger the lift, the more electricity each kWh of heat needs. + +| Outdoor air | Temperature lift | Ideal limit | Typical real COP (estimate) | +|---|---|---|---| +| 10 °C | 25 K | 12.3 | about 4 | +| 0 °C | 35 K | 8.8 | about 3 | +| −15 °C | 50 K | 6.2 | about 2 | +| −25 °C | 60 K | 5.1 | about 1.5 | + +The COP falls. At −25 °C the heat pump still beats a heater, but only just; many units add a backup electric heater (COP 1) for the coldest days. Hotter water also raises the lift. Old radiators need 55 °C water: at 0 °C outside, the lift is 55 K and the COP is about 2.2. + +### The ideal limit + + + +Physics sets an **ideal limit** on the COP (physicists call it the Carnot limit). For 35 °C water and 0 °C air it is 308.15 / 35 = 8.8. In symbols: + +ideal limit = T_hot / (T_hot − T_cold) + +Here T_hot is the home water temperature (308.15 K) and T_cold is the outdoor air temperature (273.15 K). A bigger lift makes the bottom of the fraction bigger, so the limit falls. + + + +Use kelvin, not Celsius. In Celsius the formula gives 35 / (35 − 0) = 1, which would say that a heat pump can never beat a heater. + +### Where the limit comes from + +The limit comes from **entropy**. Heat Q that passes at kelvin temperature T carries entropy Q/T, in kWh/K. So 1 kWh carries more entropy at a low temperature than at a high one. Electricity carries no entropy. Nothing can destroy entropy. This is the same law that stops heat flowing from cold to hot by itself. For example, 1 kWh flowing by itself from 308.15 K water to 273.15 K air carries away 1/308.15 and brings 1/273.15, a net gain of 0.00042 kWh/K. The reverse flow would destroy that entropy, so it never happens by itself. + +The refrigerant ends each cycle where it started, so it cannot store entropy. The electricity brings none. So the water must get at least as much entropy as the air loses. Write Q_hot for the heat delivered, Q_cold for the outdoor heat, and W for the electricity: + +Q_hot / T_hot ≥ Q_cold / T_cold + +- **Allowed:** COP 3. The air loses 2 / 273.15 = 0.00732 kWh/K. The water gets 3 / 308.15 = 0.00974 kWh/K. 0.00974 ≥ 0.00732. +- **Forbidden:** a claimed COP 12, with 1 kWh of electricity and 11 kWh from the air. The air loses 11 / 273.15 = 0.0403 kWh/K. The water gets 12 / 308.15 = 0.0389 kWh/K. Entropy would be destroyed, so no machine can do this. + +Energy conservation gives Q_cold = Q_hot − W. Put this into the inequality: + +1. Q_hot / T_hot ≥ (Q_hot − W) / T_cold +2. Multiply by T_cold: Q_hot × T_cold / T_hot ≥ Q_hot − W +3. Rearrange: W ≥ Q_hot × (T_hot − T_cold) / T_hot +4. Divide: COP = Q_hot / W ≤ T_hot / (T_hot − T_cold) + +So the minimum electricity for 3 kWh is 3 × 35 / 308.15 = 0.34 kWh. A real heat pump uses 1 kWh. + +### Why a real heat pump gets 3, not 8.8 + +Heat flows through a coil only across a temperature gap, and a bigger gap moves heat faster through a smaller coil. So the refrigerant boils 10 K below the air (−10 °C) and condenses 5 K above the water (40 °C). Its own lift, the **refrigerant lift**, is 40 − (−10) = 50 K, so the ideal limit for the refrigerant is 313.15 / 50 = 6.3. Losses in the compressor, fans, and pumps take the COP from 6.3 down to about 3. + +## Summary + +A heat pump does not make 3 kWh of heat from 1 kWh of electricity. It moves 2 kWh of outdoor heat into the home, and the 1 kWh of electricity pays for the move and ends up as heat too. Pressure sets the boiling temperature, so the refrigerant is colder than the air outside and hotter than the water inside. Heat flows the normal way at both coils. The colder the day, the more electricity each kWh of heat needs. + +A fridge uses the same loop, with the cold coil inside the fridge. + +**Not covered here:** ground-source heat pumps (warmer, steadier cold side, so a higher COP); defrosting the outdoor coil below about 5 °C; money and CO₂ compared with a gas boiler; seasonal COP (the average over a winter); refrigerant choice and safety; why the loop uses a boiling fluid and not a warmed gas. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/model.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/model.md new file mode 100644 index 0000000..f6e9b7a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/model.md @@ -0,0 +1,160 @@ +# Semantic model: how a heat pump delivers more heat than the electricity it uses + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How can a heat pump put 3 kWh of heat into a home for each 1 kWh of electricity, when energy cannot be created? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (energy conservation, temperature, kelvin, pressure, boiling and condensing, heat flows from hot to cold), basic programming. New to thermodynamic cycles. Does not know: refrigerant, evaporator, compressor, condenser, expansion valve, COP, entropy, that boiling temperature depends on pressure well enough to use it, the ideal limit (Carnot). + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Electric heater | A resistor that converts electricity into heat, 1 kWh in, 1 kWh out. | fact | +| Heat pump | A machine that uses electricity to move heat from a cold place (outdoor air) to a warmer place (home water). | fact | +| Outdoor air | The cold source; at 0 °C (273 K) it still holds thermal energy that can be taken out. | fact | +| Home water | The hot side: water at 35 °C that flows through underfloor pipes or radiators. | assumption (typical) | +| Refrigerant | A fluid in a closed loop that boils and condenses at useful temperatures (for example propane). | fact | +| Evaporator | Outdoor heat exchanger where the cold, low-pressure refrigerant boils and takes heat from the air. | fact | +| Compressor | Electric pump that squeezes the refrigerant vapour to high pressure; the work done on the gas makes it hot (as a bicycle pump gets warm). | fact | +| Condenser | Indoor heat exchanger where the hot, high-pressure refrigerant condenses and gives heat to the home water. | fact | +| Expansion valve | Narrow opening that drops the refrigerant pressure; part of the liquid boils at once, and that boiling cools the rest. | fact | +| COP | Coefficient of performance: heat delivered ÷ electricity used. | definition | +| Temperature lift | Home water temperature minus outdoor air temperature (35 K in the example). | definition | +| Refrigerant lift | Condensing temperature minus boiling temperature of the refrigerant (50 K in the example); larger than the temperature lift because heat exchangers need a temperature gap. | definition | +| Entropy | Bookkeeping quantity: heat Q passing at absolute temperature T carries Q/T of entropy; entropy can be created but never destroyed. | established fact | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Outdoor air | gives heat to | Evaporator | because the refrigerant there (−10 °C) is colder than the air (0 °C) | +| Evaporator | sends low-pressure vapour to | Compressor | | +| Compressor | sends hot high-pressure vapour to | Condenser | electricity enters here | +| Condenser | gives heat to | Home water | because the refrigerant there (40 °C) is hotter than the water (35 °C) | +| Condenser | sends high-pressure liquid to | Expansion valve | | +| Expansion valve | sends a cold low-pressure mix of liquid and vapour to | Evaporator | | +| Temperature lift | sets the minimum electricity per kWh of heat | COP | larger lift → lower COP | + +## Dependencies (learning order) + +1. Energy conservation: heat delivered = electricity + outdoor heat. +2. Heat flows only from hot to cold by itself. +3. Boiling temperature rises with pressure. +4. The refrigerant loop uses (3) to make the refrigerant colder than outdoor air and hotter than home water, so (2) works for it at both ends. +5. Moving heat from cold to hot costs work; the work grows with the temperature lift. + +## Causal chain + +```text +Expansion valve drops pressure → refrigerant boils at −10 °C → colder than 0 °C air → heat flows in (2 kWh) +Compressor adds 1 kWh of work → pressure and temperature rise → condenses at 40 °C → hotter than 35 °C water → heat flows out (3 kWh) +``` + +## Temporal sequence + +1. Evaporator: cold refrigerant at about 3.5 bar boils at −10 °C and takes 2 kWh from 0 °C outdoor air. +2. Compressor: uses 1 kWh of electricity to squeeze the vapour from about 3.5 bar to about 14 bar (the air around us is 1 bar). The work ends up as heat in the vapour, which leaves at about 60 °C. +3. Condenser: the vapour cools to 40 °C, condenses at 40 °C, and gives 3 kWh to 35 °C home water. +4. Expansion valve: pressure drops from about 14 bar to about 3.5 bar. At 3.5 bar the 40 °C liquid is far above its −10 °C boiling point, so about a third of it boils at once. That boiling takes heat from the rest of the liquid and cools everything to −10 °C. The valve adds and removes no energy. The loop repeats. + +## Quantities + +- Symbols, mapped once: Q_hot = heat delivered (3 kWh), Q_cold = outdoor heat (2 kWh), W = electricity (1 kWh). +- Energy balance (0 °C outside, 35 °C water): heat delivered 3 kWh = electricity 1 kWh + outdoor heat 2 kWh. +- COP = heat delivered / electricity = 3 / 1 = 3. Electric heater COP = 1. +- Ideal limit (Carnot): ideal COP = T_hot / (T_hot − T_cold), temperatures in kelvin. At 35 °C (308.15 K) and 0 °C (273.15 K): 308.15 / 35 = 8.8. +- Propane boiling pressures (approximate): −10 °C at about 3.5 bar; 40 °C at about 14 bar. +- Effect of outdoor temperature (water 35 °C): + +| Outdoor | Temperature lift | Ideal COP | Typical real COP | +|---|---|---|---| +| 10 °C | 25 K | 12.3 | about 4 | +| 0 °C | 35 K | 8.8 | about 3 | +| −15 °C | 50 K | 6.2 | about 2 | + +- Water at 55 °C (old radiators), 0 °C outside: lift 55 K, ideal 328.15 / 55 = 6.0, typical real about 2.2. +- Why real is about a third of ideal: heat flows through a heat exchanger only across a temperature gap, and a bigger gap moves heat faster through a smaller coil. So the refrigerant boils at −10 °C (10 K below the air) and condenses at 40 °C (5 K above the water). Its refrigerant lift is 50 K, not 35 K: ideal 313.15 / 50 = 6.3. The compressor, fans, and pumps lose about half of the rest. +- Very cold weather: at −25 °C outside, lift 60 K, ideal 308.15 / 60 = 5.1, typical real about 1.5 (estimate). Real COP stays above 1 in most winters; many units add a backup electric heater (COP 1) for the coldest days. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Colder outside | COP falls | larger lift needs more work per kWh moved | +| Hotter water (radiators) | COP falls | larger lift | +| No compressor (just a pump circulating fluid) | no heating | fluid cannot be hotter than the air it came from | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Compress the refrigerant (spend work) | Just circulate a fluid between outside and inside | Fluid warmed by 0 °C air reaches at most 0 °C; in a 20 °C home heat flows from home to fluid, so the home loses heat. | +| Change pressure to change boiling temperature | Use a fixed-pressure fluid | At one pressure the fluid boils and condenses at one temperature; it cannot be both below 0 °C outside and above 35 °C inside. | +| Ideal limit uses kelvin | Use Celsius in T_hot / (T_hot − T_cold) | 35 / (35 − 0) = 1, which says a heat pump cannot beat a heater. Kelvin gives 308.15 / 35 = 8.8. | +| COP above 1 | "Energy out cannot exceed energy in" | It does not: 3 kWh out = 1 kWh electricity + 2 kWh outdoor heat. The heater has COP 1 because electricity is its only input. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| COP above 1 because heat is moved, not made | 0 °C outside, 35 °C water: 1 kWh in, 2 kWh from air, 3 kWh out | Electric heater makes heat from electricity only: 1 kWh in, 1 kWh out, COP 1 | +| Larger lift → lower COP | 10 °C outside: lift 25 K, COP about 4 | −15 °C outside: lift 50 K, COP about 2 | +| Heat flows in at the evaporator because refrigerant is colder than the air | refrigerant −10 °C, air 0 °C | if the refrigerant were at 5 °C, heat would flow from refrigerant to air | +| No machine beats the ideal limit | COP 3: entropy taken 2 / 273.15 = 0.00732 kWh/K ≤ given 3 / 308.15 = 0.00974 kWh/K | claimed COP 12: taken 11 / 273.15 = 0.0403 > given 12 / 308.15 = 0.0389, so entropy would be destroyed | + +## Terms + +- **Heat pump**: moves heat from a cold place to a warmer place, using electricity. +- **Refrigerant**: the working fluid in the closed loop. +- **Evaporator / compressor / condenser / expansion valve**: the four parts of the loop, as above. +- **COP**: heat delivered ÷ electricity used. COP 1: no better than a heater. COP 3: three times the heat. COP 8.8: the ideal limit at 0 °C → 35 °C. +- **Outdoor heat**: the heat taken from the air (2 kWh). Not "free energy". +- **Temperature lift**: home water minus outdoor air. 25 K: easy, COP about 4. 50 K: hard, COP about 2. +- **Refrigerant lift**: refrigerant condensing minus boiling temperature; 50 K in the example. Used only to explain why real COP is below ideal. +- **Ideal limit** (ideal COP): the highest COP physics allows for a given pair of temperatures. Not "efficiency"; "Carnot" only at first mention. +- **Entropy**: Q/T for heat Q at kelvin temperature T; never destroyed. Used only in L3. + +## Epistemic status + +- **Observed / established:** energy conservation; heat flows hot to cold; boiling point rises with pressure; entropy is never destroyed; the Carnot limit. +- **Mathematical consequence:** COP = 1 + outdoor heat / electricity; ideal-limit values. +- **Assumed:** 0 °C outside, 35 °C water, refrigerant at −10 °C and 40 °C. +- **Estimated:** typical real COPs (about 4, 3, 2, 2.2, 1.5); propane pressures (about 3.5 and 14 bar); vapour at about 60 °C after the compressor; about a third of the liquid boils in the valve. + +## Confusion points + +- "COP 3 breaks energy conservation" → 3 kWh out = 1 kWh electricity + 2 kWh outdoor heat. +- "0 °C air has no heat to give" → 0 °C is 273 K; air holds thermal energy far above absolute zero; it gives heat to anything colder, such as −10 °C refrigerant. +- "The COP is a fixed property of the machine" → it depends on the temperature lift; the same unit gives about 4 at 10 °C and about 2 at −15 °C. +- "The compressor's electricity is lost" → the compressor's work also ends up as heat in the home; it is part of the 3 kWh. + +## Why the ideal limit has this form (L3) + +Heat Q passing at absolute temperature T carries entropy Q/T. Entropy can be created but never destroyed; heat flowing by itself from hot to cold creates it (1 kWh from 308.15 K to 273.15 K: −1/308.15 + 1/273.15 = +0.00042 kWh/K). The loop returns to its start each cycle, so it cannot store entropy, and electricity (work) carries no entropy. So the entropy given to the home water must be at least the entropy taken from the air: Q_hot / T_hot ≥ Q_cold / T_cold. + +- Holds: COP 3. Taken: 2 / 273.15 = 0.00732 kWh/K. Given: 3 / 308.15 = 0.00974 kWh/K. 0.00974 ≥ 0.00732. +- Breaks: a claimed COP 12 (1 kWh electricity, 11 kWh from air, 12 kWh out). Taken: 11 / 273.15 = 0.0403. Given: 12 / 308.15 = 0.0389. Entropy would be destroyed, so no machine can do it. +- Algebra: put Q_cold = Q_hot − W into the inequality: Q_hot / T_hot ≥ (Q_hot − W) / T_cold, so W ≥ Q_hot × (T_hot − T_cold) / T_hot, so COP = Q_hot / W ≤ T_hot / (T_hot − T_cold) = 308.15 / 35 = 8.8. +- Meaning: the minimum electricity for 3 kWh is 3 × 35 / 308.15 = 0.34 kWh. The real machine uses 1 kWh. + +## Scope + +- **Out of scope:** ground-source and water-source heat pumps — same cycle, warmer and steadier cold side, so higher COP. +- **Out of scope:** defrost cycles below about 5 °C, where frost on the evaporator is melted. +- **Out of scope:** cost and CO₂ comparisons with gas boilers — depend on electricity and gas prices and the grid mix. +- **Out of scope:** seasonal COP (SCOP) — the average COP over a heating season. +- **Out of scope:** refrigerant choice and safety — propane, R32, CO₂. +- **Out of scope:** why a boiling fluid and not a warmed gas — boiling takes much heat at one fixed temperature, so the coils can be small. +- **Connection:** a fridge and an air conditioner use the same loop, with the cold side inside. + +## Representation decision + +- **Stage:** 2 +- **Reason:** the mechanism is a closed loop of four parts with two heat exchanges; the reader must hold the loop, the temperature at each point, and the energy balance at once. A diagram shows the loop; a table shows how the lift changes the COP. +- **Levels:** L1 energy balance (where the 3 kWh comes from) / L2 the refrigerant loop / L3 why moving heat from cold to hot costs work, and how much / Scope. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/model.yaml b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/model.yaml new file mode 100644 index 0000000..eccb43d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/model.yaml @@ -0,0 +1,136 @@ +# Machine-readable half of model.md. `explainer check heat-pump` holds every rendering to it. + +values: + energy: # kWh, at 0 °C outside and 35 °C home water + electricity: 1 + outdoor_heat: 2 + heat_delivered: 3 + cop: + heat_pump: 3 + heater: 1 + temps: # °C + outdoor: 0 + water: 35 + room: 20 + boil: -10 + condense: 40 + discharge: 60 + warm_refrigerant_case: 5 + kelvin: + outdoor: 273.15 + water: 308.15 + condense: 313.15 + radiator_water: 328.15 + pressure_bar: + low: 3.5 + high: 14 + air: 1 + lift_k: + example: 35 + refrigerant: 50 + evaporator_gap: 10 + condenser_gap: 5 + ideal: + example: 8.8 + refrigerant: 6.3 + sweep: # outdoor °C, lift K, ideal COP, typical real COP; water at 35 °C + mild: {outdoor: 10, lift: 25, ideal: 12.3, real: 4} + freezing: {outdoor: 0, lift: 35, ideal: 8.8, real: 3} + cold: {outdoor: -15, lift: 50, ideal: 6.2, real: 2} + very_cold: {outdoor: -25, lift: 60, ideal: 5.1, real: 1.5} + radiators: {water: 55, lift: 55, ideal: 6.0, real: 2.2} + entropy: # kWh/K + taken: 0.00732 + given: 0.00974 + bad_cop: 12 + bad_outdoor_heat: 11 + bad_taken: 0.0403 + bad_given: 0.0389 + hot_to_cold_net: 0.00042 + kelvin_rounded: 273 + min_electricity: 0.34 + celsius_wrong: 1 + +allow: [2, 3, 4] + +require: + - path: energy/outdoor_heat + in: [diagram] + - path: energy/heat_delivered + in: [diagram] + - path: temps/boil + in: [diagram] + - path: temps/condense + in: [diagram] + - path: ideal/example + in: [diagram] + +claims: + - id: energy-balance + kind: guarantee + statement: "A heat pump does not create energy: heat delivered = electricity + outdoor heat, 3 kWh = 1 kWh + 2 kWh." + example: "0 °C outside, 35 °C water: 1 kWh electricity, 2 kWh outdoor heat, 3 kWh into the water, COP 3." + counterexample: "An electric heater has only electricity as input: 1 kWh in, 1 kWh heat out, COP 1." + ask: "Where do the other 2 kWh come from, if the heat pump uses only 1 kWh of electricity?" + + - id: why-compress + kind: why + statement: "The heat pump must spend work, because heat flows by itself only from hot to cold." + alternative: "Circulate a fluid between the outdoor air and the home, with no compressor." + counterexample: "Fluid warmed by 0 °C air is at most 0 °C; in a 20 °C home heat flows from the home into the fluid." + + - id: pressure-sets-boiling + kind: mechanism + statement: "Boiling temperature rises with pressure, so the loop runs at two pressures: low where it must be cold, high where it must be hot." + example: "Propane boils at −10 °C at about 3.5 bar and at 40 °C at about 14 bar." + alternative: "One fixed pressure gives one boiling temperature, which cannot be below 0 °C outdoors and above 35 °C indoors." + + - id: loop + kind: mechanism + statement: "Evaporator, compressor, condenser, expansion valve: the refrigerant takes heat in where it is colder than the air and gives heat out where it is hotter than the water." + example: "Boils at −10 °C, taking 2 kWh from 0 °C air; compressor adds 1 kWh; condenses at 40 °C, giving 3 kWh to 35 °C water; valve drops the pressure, a third of the liquid boils, the rest cools to −10 °C." + ask: "Why must the refrigerant be colder than the outdoor air in the evaporator, and hotter than the home water in the condenser?" + + - id: lift-sets-cop + kind: guarantee + statement: "The larger the temperature lift, the more electricity each kWh of heat costs, so COP falls." + example: "10 °C outside: lift 25 K, COP about 4." + counterexample: "−15 °C outside: lift 50 K, COP about 2." + ask: "What happens to the COP on a colder day, and why?" + + - id: ideal-limit + kind: limit + statement: "No heat pump can beat ideal COP = T_hot / (T_hot − T_cold), in kelvin, because entropy cannot be destroyed." + example: "COP 3 at 273.15 K → 308.15 K: entropy taken 0.00732 kWh/K, given 0.00974 kWh/K; allowed. Ideal 8.8." + counterexample: "Claimed COP 12: taken 0.0403 kWh/K, given 0.0389 kWh/K; entropy would be destroyed." + + - id: why-kelvin + kind: why + about: [ideal COP] + statement: "The ideal limit uses kelvin, because entropy is Q/T with T measured from absolute zero." + alternative: "Use Celsius." + counterexample: "35 / (35 − 0) = 1, which says a heat pump cannot beat a heater; kelvin gives 8.8." + +terms: + - term: outdoor heat + means: "the heat taken from the outdoor air, 2 kWh in the example" + avoid: [free energy, free heat] + - term: ideal limit + means: "the highest COP allowed for two temperatures; COP is not an efficiency" + avoid: [efficiency of 300, 300% efficient, 300 % efficient] + - term: temperature lift + means: "home water minus outdoor air; the refrigerant lift is a separate term" + avoid: [temperature difference of the refrigerant] + +budget: {diagram: 1100, reason: "three levels: energy balance, the loop with two pressures, and the ideal limit with a case that holds and one that breaks"} + +quiz: + - q: "A heat pump uses 1 kWh of electricity and delivers 3 kWh of heat. Does it break energy conservation? Where does the energy come from?" + expect: "No. 2 kWh comes from the outdoor air, which is colder than the home but still holds heat; the refrigerant is made colder than the air so heat flows into it. 1 + 2 = 3." + misconception: "It creates energy, or it is '300% efficient' in the sense of making more energy than it uses." + - q: "The same heat pump runs on a 10 °C day and on a −15 °C day, heating water to 35 °C. On which day is its COP higher, and why?" + expect: "On the 10 °C day (about 4 vs about 2): the temperature lift is smaller (25 K vs 50 K), so less work is needed per kWh moved." + misconception: "COP is a fixed number for the machine, or it is higher in the cold." + - q: "What do the compressor and the expansion valve each do to the refrigerant, and why does the loop need both?" + expect: "Compressor raises pressure, so the refrigerant condenses at a high temperature (40 °C) above the water; valve drops pressure, so it boils at a low temperature (−10 °C) below the air. Boiling temperature depends on pressure." + misconception: "The compressor makes the heat; or the refrigerant is cooled by the outdoor air." diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/narrative.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/narrative.md new file mode 100644 index 0000000..7b87a3e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/narrative.md @@ -0,0 +1,60 @@ +# Narrative: heat-pump + +> Pass 3. `model.md` says what is true. This file says the path a first-time reader takes to it. + +## 1. Reader before and after + +- **Before:** a technical reader who knows energy conservation, kelvin, pressure, boiling and condensing, and that heat flows from hot to cold by itself. New to heat pumps, refrigerant loops, COP, and entropy. +- **After:** "A heat pump does not make heat, it moves it: 1 kWh of electricity carries 2 kWh of heat from the cold outdoor air into the home, so 3 kWh arrives. It does this by changing the refrigerant's pressure, which changes its boiling temperature, and the colder it is outside the more electricity each kWh costs." + +## 2. Question and motive + +- **Question:** How can a heat pump give 3 kWh of heat for 1 kWh of electricity, when energy cannot be created? Result in words: it does not create the 2 extra kWh; it moves them from the outdoor air into the home, and the electricity pays for the move. +- **Why care:** an electric heater gives 1 kWh of heat for 1 kWh. A ratio of 3 sounds like it breaks conservation of energy, so the answer must be about where the energy comes from. +- **Why this approach:** first balance the energy (it shows where the extra comes from), then ask how heat can flow from cold air to warm water (that question leads to the loop), then ask how much the move must cost (that leads to the ideal limit). + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| COP | heat delivered ÷ electricity used | heater 1 / 1 = 1; heat pump 3 / 1 = 3 | 1 | +| outdoor heat | heat taken from the outdoor air | 2 kWh from 0 °C air | 2 | +| refrigerant | fluid in the closed loop | propane | 4 | +| evaporator, compressor, condenser, expansion valve | the four parts of the loop | each with its temperature and pressure in the diagram | 5 | +| bar | pressure; the air around us is 1 bar | 3.5 bar and 14 bar | 4 | +| temperature lift | home water minus outdoor air | 35 − 0 = 35 K | 7 | +| ideal limit (Carnot) | highest COP physics allows for two temperatures | 308.15 / 35 = 8.8 | 8 | +| entropy | heat Q at kelvin temperature T carries Q/T; never destroyed | 2 / 273.15 = 0.00732 kWh/K | 9 | +| Q_hot, Q_cold, W | heat delivered, outdoor heat, electricity | 3, 2, 1 kWh | 9 | +| refrigerant lift | condensing minus boiling temperature | 40 − (−10) = 50 K | 10 | +| diagram colours | blue = cold side, red = hot side, yellow = electricity | the loop diagram | 5 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How can 1 kWh give 3 kWh? | — | heater vs heat pump numbers | "An electric heater turns 1 kWh of electricity into 1 kWh of heat. A heat pump gives about 3 kWh of heat for the same 1 kWh. Energy cannot be created, so where do the other 2 kWh come from? COP (heat delivered ÷ electricity used) is 1 for the heater and 3 for the heat pump." Answer up front: it moves heat, it does not make it. | the puzzle, the answer in one line, the word COP | +| 2 | notice | Is energy created? | so | L1 energy diagram: 1 + 2 = 3; caption explains colours (blue cold side, red hot side, yellow electricity) | "No. 2 kWh is outdoor heat, taken from the outdoor air. 0 °C air is 273 K above absolute zero and still holds heat. The 3 kWh goes into the home water: the 35 °C water that flows through the floor pipes or radiators." | the heat is moved, not made; the home water | +| 3 | objection | Heat flows only from hot to cold. How can it go from 0 °C air into 35 °C water? | but | — | "To carry heat from outside to inside, pipe a fluid between the two. With no other step, the fluid warms to at most 0 °C outside. Inside, the 35 °C water is hotter, so heat flows from the water into the fluid: the home gets colder. The fluid must be colder than the air outside and hotter than the water inside." | the problem the machine must solve | +| 4 | approach | How can one fluid be both? | therefore | — | "Change its pressure. Propane, the fluid used here, boils at −10 °C at 3.5 bar and at 40 °C at 14 bar (the air around us is 1 bar). We call this fluid the refrigerant." | the loop runs at two pressures | +| 5 | mechanism | How does the loop use this? | so | L2 loop diagram, same colours and the same kWh as L1 | the four parts in order: "Evaporator (outdoor coil): refrigerant at 3.5 bar boils at −10 °C, colder than the 0 °C air, so 2 kWh flows in. Compressor: an electric pump squeezes the vapour to 14 bar; the work heats it to about 60 °C, as a bicycle pump gets warm. Condenser (indoor coil): the vapour condenses at 40 °C, hotter than the 35 °C water, so 3 kWh flows out. Expansion valve: a narrow opening drops the pressure to 3.5 bar; about a third of the liquid boils at once, and that boiling cools the rest to −10 °C." | the full loop; the compressor raises the pressure, the valve lowers it | +| 6 | notice | Where does the electricity go? | and | same diagram, compressor | "The compressor's 1 kWh heats the vapour, so it reaches the water too: 1 + 2 = 3 kWh at the condenser." | 3 = 1 + 2 at the condenser | +| 7 | test (predict) | What happens on a colder day? | so | predict prompt: "Outside drops from 10 °C to −15 °C. Does the COP rise, fall, or stay the same?" Then the table. | "The temperature lift is home water minus outdoor air: 35 K at 0 °C. COP is about 4 at 10 °C (lift 25 K), about 3 at 0 °C, about 2 at −15 °C (lift 50 K). Old radiators need 55 °C water: lift 55 K at 0 °C, COP about 2.2." | larger lift → lower COP; why comes next | +| 8 | approach | Why does a larger lift cost more? Is there a limit? | so | formula with the instance first | "Physics sets an ideal limit (the Carnot limit): 308.15 / 35 = 8.8 for 35 °C water and 0 °C air. In symbols, ideal COP = T_hot / (T_hot − T_cold), where T_hot is the home water (308.15 K) and T_cold the outdoor air (273.15 K). Use kelvin: in Celsius 35 / (35 − 0) = 1, which would say a heat pump can never beat a heater. A larger lift makes the bottom bigger, so the limit falls." | the limit, its numbers, why kelvin | +| 9 | mechanism | Why that formula? | because | L3: rule, two cases, steps | "Heat Q that passes at kelvin temperature T carries entropy Q/T. Heat flowing by itself from hot to cold makes entropy: 1 kWh from 308.15 K to 273.15 K loses 1/308.15 and gains 1/273.15, net +0.00042 kWh/K. Nothing can destroy entropy; that is the same law that stops heat flowing from cold to hot by itself. The loop ends each cycle where it started, so it cannot store entropy. So the water must get at least as much entropy as the air loses. Name Q_hot = heat delivered, Q_cold = outdoor heat, W = electricity. COP 3: air loses 2/273.15 = 0.00732, water gets 3/308.15 = 0.00974. Allowed. COP 12 (11 kWh from air): air loses 0.0403, water gets 0.0389. Forbidden. Put Q_cold = Q_hot − W into Q_hot/T_hot ≥ Q_cold/T_cold and solve for Q_hot/W, step by step: you get the limit." | where the limit comes from | +| 10 | notice | Why is a real one 3, not 8.8? | but | — | "Heat flows through a coil only across a temperature gap. So the refrigerant boils 10 K below the air and condenses 5 K above the water. Its own lift, the refrigerant lift, is 40 − (−10) = 50 K, and 313.15 / 50 = 6.3. The compressor, fans, and pumps lose about half of the rest." | the gap's two causes | +| 11 | close | — | so | — | "A heat pump does not make 3 kWh from 1 kWh. It moves 2 kWh of outdoor heat into the home, and the 1 kWh of electricity pays for the move and ends up as heat too. Pressure sets the boiling temperature, so the refrigerant is colder than the air outside and hotter than the water inside, and heat flows the normal way at both ends. The colder the day, the more each kWh costs." Connection: a fridge is the same loop, with the cold coil inside the fridge. Not covered here: a short list. | the after-sentence | + +## 5. Concrete to symbol + +COP: 3 kWh / 1 kWh = 3, then the word. Ideal limit: 308.15 / 35 = 8.8 first, then T_hot / (T_hot − T_cold), with "here T_hot is 308.15 K, the 35 °C water". Entropy: 2 kWh at 273.15 K → 0.00732 kWh/K, then Q/T. + +## 6. Links between representations + +The kWh numbers in the L1 diagram are the same numbers on the evaporator, compressor, and condenser in the L2 diagram. Blue nodes are the cold side, red the hot side, yellow the electricity, in both diagrams; the text says so under the first. + +## 7. Close the loop + +- **Answer:** the heat pump does not make 3 kWh from 1 kWh. It moves 2 kWh of outdoor heat into the home, and the 1 kWh of electricity both pays for the move and ends up as heat. +- **General argument:** changing the pressure changes the boiling temperature, so the refrigerant is colder than the air outside and hotter than the water inside; heat flows the normal way at both ends. The cost per kWh grows with the temperature lift. +- **Payoff:** the reader can predict the COP falls in cold weather and with hot radiators. Connection: a fridge is the same loop. Scope list. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/decisions.yaml new file mode 100644 index 0000000..241bc31 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/decisions.yaml @@ -0,0 +1,485 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T150833-probe#1: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#10: + decision: declined + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#11: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#12: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#13: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#14: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#15: + decision: declined + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#2: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#3: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#4: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#5: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#6: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#7: + decision: declined + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#8: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T150833-probe#9: + decision: adopted + note: 7 to Scope; 10 adds length without changing the argument; 15 is required template + metadata +2026-10-04T151129-coldread-narrative#1: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#10: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#11: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#12: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#13: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#14: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#15: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#16: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#17: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#18: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#19: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#2: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#20: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#21: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#22: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#23: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#24: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#25: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#26: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#27: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#28: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#29: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#3: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#30: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#31: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#32: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#33: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#34: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#35: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#36: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#37: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#38: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#39: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#4: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#40: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#41: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#42: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#43: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#44: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#45: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#46: + decision: declined + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#47: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#5: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#6: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#7: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#8: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151129-coldread-narrative#9: + decision: adopted + note: Said column written out in full; L1/L3/scope are level labels the rendering + names in headings +2026-10-04T151329-coldread-diagram#1: + decision: adopted + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151329-coldread-diagram#2: + decision: declined + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151329-coldread-diagram#3: + decision: adopted + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151329-coldread-diagram#4: + decision: adopted + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151329-coldread-diagram#5: + decision: adopted + note: '2: 1 bar gives the pressures a meaning' +2026-10-04T151455-quiz-diagram#1: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#10: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#11: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#12: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#13: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#14: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#15: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#16: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#17: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#18: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#19: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#2: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#20: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#21: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#22: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#23: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#24: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#25: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#3: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#4: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#5: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#6: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#7: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#8: + decision: adopted + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151455-quiz-diagram#9: + decision: declined + note: '3: names given together once; 5: separated by definition; 7,9,10,12,13,15,21,23,25: + depth beyond this reader''s question, estimates labelled; 16,18: carry confusion + point and tie entropy to known rule; 19,20: close and connection required by narrative' +2026-10-04T151658-quiz-diagram#1: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#10: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#11: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#12: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#13: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#14: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#15: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#16: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#17: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#18: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#19: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#2: + decision: adopted + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#20: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#21: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#22: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#23: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#24: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#3: + decision: adopted + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#4: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#5: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#6: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#7: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#8: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' +2026-10-04T151658-quiz-diagram#9: + decision: declined + note: 'rest: depth beyond the question (entropy foundations, compression physics, + coil sizing), estimates already labelled, close and scope required' diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T150833-probe.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T150833-probe.json new file mode 100644 index 0000000..ddc4656 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T150833-probe.json @@ -0,0 +1,218 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "heat-pump", + "at": "2026-10-04T15:08:33", + "model": "default", + "prompt_sha": "7350959dd026", + "reply": { + "gaps": [ + { + "rule": 3, + "severity": "main", + "about": "'Entropy' is used in L3 (Q/T, 'cannot store entropy', 'cannot be destroyed') but is not in the Terms section, and the Audience section does not list it as prior knowledge.", + "why_it_matters": "L3 is the level that explains why moving heat uphill costs work. Its whole argument depends on a word this reader has never been given a meaning for, so they can't check any step of it.", + "suggested_addition": "Add to Terms: 'Entropy (here only): heat Q passing at absolute temperature T carries Q/T of entropy. Taking 2 kWh from 273.15 K air removes 2/273.15 = 0.00732 kWh/K. Giving 3 kWh to 308.15 K water adds 3/308.15 = 0.00974 kWh/K. Heat flowing by itself from hot to cold always increases the total.' Give one example of that last fact: 1 kWh moving from 308 K to 273 K gives −0.00325 + 0.00366 = +0.00042 kWh/K." + }, + { + "rule": 2, + "severity": "main", + "about": "L3 says entropy given to the hot side ≥ entropy taken from the cold side, with no numbers showing it holding or failing. The algebra from that inequality to W ≥ Q_hot(T_hot−T_cold)/T_hot is also left out, and so is the step from that bound to COP_max = T_hot/(T_hot−T_cold).", + "why_it_matters": "The ideal limit (8.8) is the only bound in the explanation. Without a case that violates it, the reader can't see what the inequality forbids, and the formula in Quantities seems to come from nowhere.", + "suggested_addition": "Holds: COP 3 at 0 °C → 35 °C gives 2/273.15 = 0.00732 taken and 3/308.15 = 0.00974 given; 0.00974 ≥ 0.00732. Breaks: a claimed COP 12 (11 kWh from air, 1 kWh electricity, 12 kWh out) gives 11/273.15 = 0.0403 taken but only 12/308.15 = 0.0389 given, so entropy would be destroyed, which is forbidden. Show the algebra: substitute Q_cold = Q_hot − W, so Q_hot/T_hot ≥ (Q_hot − W)/T_cold, so W ≥ Q_hot(1 − T_cold/T_hot). Then Q_hot/W ≤ T_hot/(T_hot−T_cold) = 308.15/35 = 8.8." + }, + { + "rule": 4, + "severity": "main", + "about": "The expansion-valve step says the pressure drop 'makes it cold' but never says how. The model also calls the output 'cold low-pressure liquid', when it is really a mixture of liquid and vapour.", + "why_it_matters": "The reader is told that boiling temperature depends on pressure but is never shown it being used here. 40 °C liquid at 3.5 bar is far above its −10 °C boiling point, so part of it boils at once and that boiling cools the rest. Without this, getting from 40 °C to −10 °C with no energy removed looks like magic.", + "suggested_addition": "'At 3.5 bar propane boils at −10 °C, so the 40 °C liquid boils instantly. Roughly a third of it turns to vapour (estimate; recompute from propane tables), and the heat that boiling uses comes from the liquid itself, cooling everything to −10 °C. The valve adds or removes 0 kWh; it only drops the pressure.'" + }, + { + "rule": 1, + "severity": "main", + "about": "Why squeezing vapour makes it hot. The model states this but never explains it, and offers no comparison.", + "why_it_matters": "The compressor is the one place where electricity enters. Without a reason, 'pressure rises → temperature rises' is just a rule to memorise, and the reader can't tell why the work 'ends up as heat in the vapour'.", + "suggested_addition": "'Pushing the piston or scroll does work on the gas molecules and speeds them up, which is a higher temperature (as when a bicycle pump gets warm). Vapour enters at about −10 °C and 3.5 bar and leaves at about 14 bar and roughly 60 °C (estimate). It first cools to 40 °C in the condenser, then condenses.' Also say that this means the condenser receives vapour above 40 °C." + }, + { + "rule": 3, + "severity": "main", + "about": "'Temperature lift' names two different things: water minus air (35 K, used in the table and the ideal COP) and refrigerant minus refrigerant (50 K, used in the 'why real is a third' line). T_hot is likewise 308.15 K in one place and 313.15 K in another.", + "why_it_matters": "The reader will see that the lift is 35 K and also 50 K for the same situation. The two ideal COPs (8.8 and 6.3) will look like a contradiction.", + "suggested_addition": "Name them separately. 'Lift' = water − air = 35 K, ideal 8.8. 'Refrigerant lift' = condensing − boiling = 40 − (−10) = 50 K, ideal 313.15/50 = 6.3. State that the extra 15 K (10 K at the evaporator + 5 K at the condenser) is what heat exchangers need to make heat flow." + }, + { + "rule": 1, + "severity": "edge", + "about": "Why run the refrigerant at −10 °C and 40 °C instead of right at 0 °C and 35 °C. The cost is stated (refrigerant lift 50 K instead of 35 K) but not the reason.", + "why_it_matters": "It explains part of why real is below ideal, and it explains a trade-off: a bigger heat exchanger means a smaller temperature gap, which means a higher COP.", + "suggested_addition": "'Heat flow through an exchanger is roughly proportional to the temperature difference. At 0 K difference nothing flows. Halving the evaporator gap from 10 K to 5 K needs about twice the coil area but raises the ideal from 313.15/50 = 6.3 to 313.15/45 = 7.0.'" + }, + { + "rule": 1, + "severity": "edge", + "about": "Why use a fluid that boils and condenses rather than just heating and cooling a gas.", + "why_it_matters": "Boiling takes a lot of heat at one fixed temperature. That is why the temperatures at each exchanger are fixed at single values. A gas-only cycle would change temperature along the coil.", + "suggested_addition": "'Propane takes about 390 kJ/kg to boil at −10 °C (estimate), so 2 kWh = 7200 kJ needs about 18 kg to circulate, all at −10 °C. Warming propane gas by 10 K takes only about 17 kJ/kg (cp ≈ 1.7 kJ/kg·K), so it would need about 20 times the flow.'" + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts have more than one name: 'ideal limit' / 'Carnot' / 'COP_max' / 'Ideal COP'; 'outdoor heat' / 'heat taken from outside' / 'heat from the air' / Q_cold; 'electricity' / 'work' / W; 'heat delivered' / Q_hot / 'heat out'; 'typical real COP' / 'COP'.", + "why_it_matters": "In L3 the reader has to work out that W is the electricity and Q_cold is the outdoor heat, and that 'Carnot' is the same thing as 'ideal limit'.", + "suggested_addition": "Choose one name for each and map the symbols once: 'Q_hot = heat delivered (3 kWh), Q_cold = outdoor heat (2 kWh), W = electricity (1 kWh), COP_max = ideal limit (Carnot).'" + }, + { + "rule": 5, + "severity": "edge", + "about": "Pressures in bar (3.5 and 14) are given with no reference point.", + "why_it_matters": "The reader can't tell whether 14 bar is mild or extreme, which matters for why the parts are sealed and strong.", + "suggested_addition": "'Air around you is about 1 bar; a car tyre is about 3.2 bar absolute; the condenser side at 14 bar is about 4 times tyre pressure.'" + }, + { + "rule": 5, + "severity": "edge", + "about": "The ratio of real to ideal (3/8.8 ≈ 0.34) is mentioned but never shown as a measure with low and high values.", + "why_it_matters": "It tells the reader how close a machine gets to physics' limit, and it separates poor engineering from a hard lift.", + "suggested_addition": "'Fraction of ideal: 3/8.8 = 0.34 at 0 °C. At 10 °C: 4/12.3 = 0.33. At −15 °C: 2/6.2 = 0.32. Roughly constant, so the lift, not the machine, causes most of the drop.'" + }, + { + "rule": 4, + "severity": "edge", + "about": "Real COP at outdoor temperatures where it gets close to 1 is missing. The table stops at −15 °C.", + "why_it_matters": "A learner will ask whether it ever becomes worse than a heater. This is the natural place where the claim 'COP above 1' would break.", + "suggested_addition": "Add a row: −25 °C outside, lift 60 K, ideal 308.15/60 = 5.1, typical real about 1.5 (estimate; recompute). State that the real COP stays above 1 well beyond typical winter temperatures, but backup resistive heaters (COP 1) sometimes take over." + } + ], + "excess": [ + "Why-this-form row about Celsius: the second example, '−15 °C outside and 0 °C water gives 0/15 = 0, kelvin 18.2', uses an unrealistic 0 °C water temperature and adds nothing beyond the first example (35/35 = 1).", + "Refrigerant product names (R290, R32, R744) in Entities and Scope; 'propane' alone is enough.", + "Alternative state 'Reverse the loop (fridge, air conditioner)'; it isn't needed to answer the central question.", + "Representation decision section (stage, levels); this is planning metadata, not content for the reader." + ], + "next_questions": [ + { + "q": "Why does squeezing the vapour make it hot, and why does going through the valve make the liquid cold without removing energy?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "What is entropy, and why does heat carry Q/T of it?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How cold can it get outside before the heat pump is no better than an electric heater (COP 1)?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Is it actually cheaper or lower in CO₂ than a gas boiler?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Why is the real COP (3) so far below the ideal limit (8.8), and could better engineering close the gap?", + "answered_by_model": true, + "declared_out_of_scope": false + } + ] + }, + "usage": { + "cost": 0.1754818, + "seconds": 61 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 3", + "text": "'Entropy' is used in L3 (Q/T, 'cannot store entropy', 'cannot be destroyed') but is not in the Terms section, and the Audience section does not list it as prior knowledge. → Add to Terms: 'Entropy (here only): heat Q passing at absolute temperature T carries Q/T of entropy. Taking 2 kWh from 273.15 K air removes 2/273.15 = 0.00732 kWh/K. Giving 3 kWh to 308.15 K water adds 3/308.15 = 0.00974 kWh/K. Heat flowing by itself from hot to cold always increases the total.' Give one example of that last fact: 1 kWh moving from 308 K to 273 K gives −0.00325 + 0.00366 = +0.00042 kWh/K." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "L3 says entropy given to the hot side ≥ entropy taken from the cold side, with no numbers showing it holding or failing. The algebra from that inequality to W ≥ Q_hot(T_hot−T_cold)/T_hot is also left out, and so is the step from that bound to COP_max = T_hot/(T_hot−T_cold). → Holds: COP 3 at 0 °C → 35 °C gives 2/273.15 = 0.00732 taken and 3/308.15 = 0.00974 given; 0.00974 ≥ 0.00732. Breaks: a claimed COP 12 (11 kWh from air, 1 kWh electricity, 12 kWh out) gives 11/273.15 = 0.0403 taken but only 12/308.15 = 0.0389 given, so entropy would be destroyed, which is forbidden. Show the algebra: substitute Q_cold = Q_hot − W, so Q_hot/T_hot ≥ (Q_hot − W)/T_cold, so W ≥ Q_hot(1 − T_cold/T_hot). Then Q_hot/W ≤ T_hot/(T_hot−T_cold) = 308.15/35 = 8.8." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 4", + "text": "The expansion-valve step says the pressure drop 'makes it cold' but never says how. The model also calls the output 'cold low-pressure liquid', when it is really a mixture of liquid and vapour. → 'At 3.5 bar propane boils at −10 °C, so the 40 °C liquid boils instantly. Roughly a third of it turns to vapour (estimate; recompute from propane tables), and the heat that boiling uses comes from the liquid itself, cooling everything to −10 °C. The valve adds or removes 0 kWh; it only drops the pressure.'" + }, + { + "n": 4, + "severity": "main", + "kind": "rule 1", + "text": "Why squeezing vapour makes it hot. The model states this but never explains it, and offers no comparison. → 'Pushing the piston or scroll does work on the gas molecules and speeds them up, which is a higher temperature (as when a bicycle pump gets warm). Vapour enters at about −10 °C and 3.5 bar and leaves at about 14 bar and roughly 60 °C (estimate). It first cools to 40 °C in the condenser, then condenses.' Also say that this means the condenser receives vapour above 40 °C." + }, + { + "n": 5, + "severity": "main", + "kind": "rule 3", + "text": "'Temperature lift' names two different things: water minus air (35 K, used in the table and the ideal COP) and refrigerant minus refrigerant (50 K, used in the 'why real is a third' line). T_hot is likewise 308.15 K in one place and 313.15 K in another. → Name them separately. 'Lift' = water − air = 35 K, ideal 8.8. 'Refrigerant lift' = condensing − boiling = 40 − (−10) = 50 K, ideal 313.15/50 = 6.3. State that the extra 15 K (10 K at the evaporator + 5 K at the condenser) is what heat exchangers need to make heat flow." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 1", + "text": "Why run the refrigerant at −10 °C and 40 °C instead of right at 0 °C and 35 °C. The cost is stated (refrigerant lift 50 K instead of 35 K) but not the reason. → 'Heat flow through an exchanger is roughly proportional to the temperature difference. At 0 K difference nothing flows. Halving the evaporator gap from 10 K to 5 K needs about twice the coil area but raises the ideal from 313.15/50 = 6.3 to 313.15/45 = 7.0.'" + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 1", + "text": "Why use a fluid that boils and condenses rather than just heating and cooling a gas. → 'Propane takes about 390 kJ/kg to boil at −10 °C (estimate), so 2 kWh = 7200 kJ needs about 18 kg to circulate, all at −10 °C. Warming propane gas by 10 K takes only about 17 kJ/kg (cp ≈ 1.7 kJ/kg·K), so it would need about 20 times the flow.'" + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts have more than one name: 'ideal limit' / 'Carnot' / 'COP_max' / 'Ideal COP'; 'outdoor heat' / 'heat taken from outside' / 'heat from the air' / Q_cold; 'electricity' / 'work' / W; 'heat delivered' / Q_hot / 'heat out'; 'typical real COP' / 'COP'. → Choose one name for each and map the symbols once: 'Q_hot = heat delivered (3 kWh), Q_cold = outdoor heat (2 kWh), W = electricity (1 kWh), COP_max = ideal limit (Carnot).'" + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 5", + "text": "Pressures in bar (3.5 and 14) are given with no reference point. → 'Air around you is about 1 bar; a car tyre is about 3.2 bar absolute; the condenser side at 14 bar is about 4 times tyre pressure.'" + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 5", + "text": "The ratio of real to ideal (3/8.8 ≈ 0.34) is mentioned but never shown as a measure with low and high values. → 'Fraction of ideal: 3/8.8 = 0.34 at 0 °C. At 10 °C: 4/12.3 = 0.33. At −15 °C: 2/6.2 = 0.32. Roughly constant, so the lift, not the machine, causes most of the drop.'" + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 4", + "text": "Real COP at outdoor temperatures where it gets close to 1 is missing. The table stops at −15 °C. → Add a row: −25 °C outside, lift 60 K, ideal 308.15/60 = 5.1, typical real about 1.5 (estimate; recompute). State that the real COP stays above 1 well beyond typical winter temperatures, but backup resistive heaters (COP 1) sometimes take over." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "Why-this-form row about Celsius: the second example, '−15 °C outside and 0 °C water gives 0/15 = 0, kelvin 18.2', uses an unrealistic 0 °C water temperature and adds nothing beyond the first example (35/35 = 1)." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Refrigerant product names (R290, R32, R744) in Entities and Scope; 'propane' alone is enough." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Alternative state 'Reverse the loop (fridge, air conditioner)'; it isn't needed to answer the central question." + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Representation decision section (stage, levels); this is planning metadata, not content for the reader." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151129-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151129-coldread-narrative.json new file mode 100644 index 0000000..16bb54c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151129-coldread-narrative.json @@ -0,0 +1,493 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "heat-pump", + "at": "2026-10-04T15:11:29", + "model": "default", + "prompt_sha": "c1be6734e6da", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: heater vs heat pump numbers. Said: define COP; heater 1, heat pump 3", + "unresolved": [], + "unsaid": [], + "leap": [ + "The words never say why a COP of 3 is puzzling. Nothing links 'heat pump 3' to conservation of energy, so the puzzle exists only in the 'Reader's question' column, which the reader does not get." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (notice)", + "text": "Shown: L1 energy diagram: 1 + 2 = 3. Said: the 2 kWh is outdoor heat; 0 °C air is 273 K and still holds heat", + "unresolved": [ + "L1" + ], + "unsaid": [ + "The L1 diagram's colour coding. The plan says the text explains blue/red/yellow under the first diagram, but this beat's 'Said' does not.", + "The question 'is energy created?' is never asked aloud. The answer arrives before the question." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "Shown: —. Said: a plain fluid loop with no compressor reaches at most 0 °C and would cool a 20 °C home", + "unresolved": [ + "compressor", + "fluid loop" + ], + "unsaid": [], + "leap": [ + "The text brings in a 'fluid loop' with no reason given for why heat would be moved by a loop of fluid at all.", + "The objection itself, that heat flows hot to cold, so how can it go from 0 °C air indoors, is not in 'Said'. Only the failed design is.", + "There is a jump from '20 °C home' here to '35 °C water' later. The heating water is never introduced." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (approach)", + "text": "Shown: —. Said: boiling temperature rises with pressure; propane boils at −10 °C at 3.5 bar, at 40 °C at 14 bar", + "unresolved": [ + "propane (its role as the fluid in the loop is not said; the word 'refrigerant' never appears in 'Said')" + ], + "unsaid": [], + "leap": [ + "It is not said how these two boiling points help. The tie to 'colder than 0 °C air, hotter than the water' waits until beat 5." + ], + "excess": [ + "'boiling temperature rises with pressure': the reader already knows boiling and pressure. This is useful as a reminder but borderline." + ], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: L2 loop diagram. Said: four steps, each with temperatures and kWh; why −10 °C is colder than the air and 40 °C hotter than the water", + "unresolved": [ + "the names of the four parts (evaporator, compressor, condenser, expansion valve) if they are labels in the diagram; 'Said' does not name or define them", + "refrigerant", + "the water" + ], + "unsaid": [ + "The blue/red/yellow colour coding of the diagram nodes is never explained in 'Said'.", + "The diagram's kWh numbers are the same as in L1 (the plan's link), but this is not said." + ], + "leap": [ + "How the pressure is raised and lowered (by the compressor and the expansion valve) is not stated." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (notice)", + "text": "Shown: same diagram, compressor. Said: the compressor's 1 kWh ends up in the water too", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (test/predict)", + "text": "Shown: predict prompt, then table. Said: define temperature lift; COP about 4 / 3 / 2 at 10 / 0 / −15 °C; radiators at 55 °C", + "unresolved": [], + "unsaid": [ + "The contents of the predict prompt are not stated." + ], + "leap": [ + "'radiators at 55 °C' is dropped in with no stated COP and no reason why hotter radiators matter.", + "Why a larger lift lowers COP is not given here. The reason comes in beats 8–9." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (approach)", + "text": "Shown: formula. Said: ideal limit T_hot / (T_hot − T_cold) in kelvin; 8.8; Celsius gives 1", + "unresolved": [ + "T_hot", + "T_cold" + ], + "unsaid": [ + "The concrete 308.15 / 35 is not in 'Said'. Only '8.8' is, so the reader cannot check where 8.8 comes from." + ], + "leap": [ + "The formula is asserted with no reason. The reason is deferred to beat 9.", + "'Celsius gives 1' is too terse to follow: the reader is not shown 35/35, and the beat does not say why Celsius is wrong." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (mechanism)", + "text": "Shown: L3 inequality, two cases. Said: entropy Q/T is never destroyed; COP 3 passes, COP 12 fails; algebra", + "unresolved": [ + "entropy", + "Q", + "Q/T", + "L3", + "Q_hot, Q_cold, W (presumably in the inequality, never defined in 'Said')", + "COP 12 (where 12 comes from)" + ], + "unsaid": [ + "The L3 inequality itself is not spoken. 'algebra' is all the reader gets." + ], + "leap": [ + "'never destroyed' is stated as a rule. It is not tied to what the reader already knows, that heat flows hot to cold by itself.", + "How the inequality turns into T_hot / (T_hot − T_cold) is compressed into 'algebra'." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 10 (notice)", + "text": "Shown: —. Said: refrigerant lift 50 K (ideal 6.3) plus compressor losses", + "unresolved": [ + "refrigerant lift", + "refrigerant" + ], + "unsaid": [], + "leap": [ + "'ideal 6.3' is not derived in 'Said' (313.15/50). The reader has to work out that it uses 40 °C and −10 °C instead of 35 °C and 0 °C." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 11 (close)", + "text": "Shown: —. Said: answer in the opening words; connection to the fridge; scope", + "unresolved": [ + "scope" + ], + "unsaid": [], + "leap": [ + "The fridge connection is named but not explained." + ], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 9: 'entropy Q/T is never destroyed' is given as an unexplained rule to a reader new to entropy, and the step to the formula is only 'algebra'. The ideal limit therefore rests on nothing the reader can follow.", + "Beat 9: the symbols Q_hot, Q_cold, W and the 'L3 inequality' are never defined or spoken.", + "Beat 11: 'answer in the opening words' is a placeholder. The answer and the general argument are never actually said.", + "Beat 5: the four parts and the word 'refrigerant' are never named or defined in 'Said', yet beats 6 and 10 depend on 'compressor' and 'refrigerant lift'.", + "Beat 1/2: the puzzle (3 out for 1 in seems to break conservation of energy) is never said, so the reader is not told what question is being answered." + ], + "cuts": [ + "The terse 'Celsius gives 1' in beat 8: either spell it out (35/35) or cut it. As it stands it costs a reread and gains nothing.", + "'radiators at 55 °C' in beat 7: it is not used later in the beats, so either give its COP or drop it.", + "Beat 4's restatement that boiling temperature rises with pressure: the reader knows this, so lead with the propane numbers." + ], + "question": { + "text": "How can a heat pump deliver 3 kWh of heat for 1 kWh of electricity without creating energy?", + "known_at": "Only implicitly, at beat 1 (heater 1 vs heat pump 3) and beat 2. The question is never put in words in 'Said'.", + "result_stated": "Partly, at beat 2: 'the 2 kWh is outdoor heat'. The full answer is promised at beat 11 ('answer in the opening words') but never written." + }, + "motive": "Reason to care: missing in 'Said'. The heater-vs-heat-pump comparison in beat 1 hints at it, but conservation of energy is never mentioned. Reason for the approach: missing. The sequence (energy balance, then the uphill problem, then the cost limit) is never announced.", + "close": "Not achieved in the beats. Beat 11 'Said' reads 'answer in the opening words; connection to the fridge; scope', which is a placeholder, and the general argument about pressure and boiling temperature is not restated.", + "recap": "A heat pump doesn't make heat, it moves it: squeezing and releasing a refrigerant makes it boil colder than the outdoor air and condense hotter than the indoor water, so 1 kWh of electricity carries 2 kWh in, and the bigger the temperature gap the more each kWh costs.", + "strongest": "Beats 3–4: the failed no-compressor loop ('reaches at most 0 °C and would cool a 20 °C home') followed by propane's two boiling points at 3.5 and 14 bar. Together they make the trick feel necessary and concrete.", + "weakest": "Beat 9: entropy, Q/T, an unspoken inequality, undefined symbols and 'algebra', all in one beat for a reader new to entropy.", + "suggestions": [ + "In beat 1, say explicitly that energy cannot be created, so a COP of 3 looks impossible.", + "In beat 2 or 5, say the colour coding aloud (blue = cold side, red = hot side, yellow = electricity).", + "In beat 5, name the four parts and the word 'refrigerant' as each appears, and say the compressor raises the pressure and the valve lowers it.", + "Introduce the heating water (at 35 °C) before beat 3 relies on it.", + "In beat 9, give 2 / 273.15 = 0.00732 kWh/K before the symbol Q/T, define Q_hot, Q_cold and W, and link 'never destroyed' to heat flowing hot to cold by itself.", + "In beat 8, show 308.15 / 35 = 8.8 before the formula.", + "In beat 11, write out the answer sentence and the general argument." + ] + }, + "usage": { + "cost": 0.1379862, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 9: 'entropy Q/T is never destroyed' is given as an unexplained rule to a reader new to entropy, and the step to the formula is only 'algebra'. The ideal limit therefore rests on nothing the reader can follow." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 9: the symbols Q_hot, Q_cold, W and the 'L3 inequality' are never defined or spoken." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 11: 'answer in the opening words' is a placeholder. The answer and the general argument are never actually said." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: the four parts and the word 'refrigerant' are never named or defined in 'Said', yet beats 6 and 10 depend on 'compressor' and 'refrigerant lift'." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 1/2: the puzzle (3 out for 1 in seems to break conservation of energy) is never said, so the reader is not told what question is being answered." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "The terse 'Celsius gives 1' in beat 8: either spell it out (35/35) or cut it. As it stands it costs a reread and gains nothing." + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "'radiators at 55 °C' in beat 7: it is not used later in the beats, so either give its COP or drop it." + }, + { + "n": 8, + "severity": "excess", + "kind": "cut", + "text": "Beat 4's restatement that boiling temperature rises with pressure: the reader knows this, so lead with the propane numbers." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): The words never say why a COP of 3 is puzzling. Nothing links 'heat pump 3' to conservation of energy, so the puzzle exists only in the 'Reader's question' column, which the reader does not get." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (notice): L1" + }, + { + "n": 11, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (notice): The L1 diagram's colour coding. The plan says the text explains blue/red/yellow under the first diagram, but this beat's 'Said' does not." + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (notice): The question 'is energy created?' is never asked aloud. The answer arrives before the question." + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): compressor" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): fluid loop" + }, + { + "n": 15, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): The text brings in a 'fluid loop' with no reason given for why heat would be moved by a loop of fluid at all." + }, + { + "n": 16, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): The objection itself, that heat flows hot to cold, so how can it go from 0 °C air indoors, is not in 'Said'. Only the failed design is." + }, + { + "n": 17, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): There is a jump from '20 °C home' here to '35 °C water' later. The heating water is never introduced." + }, + { + "n": 18, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (approach): propane (its role as the fluid in the loop is not said; the word 'refrigerant' never appears in 'Said')" + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (approach): It is not said how these two boiling points help. The tie to 'colder than 0 °C air, hotter than the water' waits until beat 5." + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): the names of the four parts (evaporator, compressor, condenser, expansion valve) if they are labels in the diagram; 'Said' does not name or define them" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): refrigerant" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): the water" + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The blue/red/yellow colour coding of the diagram nodes is never explained in 'Said'." + }, + { + "n": 24, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The diagram's kWh numbers are the same as in L1 (the plan's link), but this is not said." + }, + { + "n": 25, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): How the pressure is raised and lowered (by the compressor and the expansion valve) is not stated." + }, + { + "n": 26, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (test/predict): The contents of the predict prompt are not stated." + }, + { + "n": 27, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test/predict): 'radiators at 55 °C' is dropped in with no stated COP and no reason why hotter radiators matter." + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test/predict): Why a larger lift lowers COP is not given here. The reason comes in beats 8–9." + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (approach): T_hot" + }, + { + "n": 30, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (approach): T_cold" + }, + { + "n": 31, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (approach): The concrete 308.15 / 35 is not in 'Said'. Only '8.8' is, so the reader cannot check where 8.8 comes from." + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (approach): The formula is asserted with no reason. The reason is deferred to beat 9." + }, + { + "n": 33, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (approach): 'Celsius gives 1' is too terse to follow: the reader is not shown 35/35, and the beat does not say why Celsius is wrong." + }, + { + "n": 34, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): entropy" + }, + { + "n": 35, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): Q" + }, + { + "n": 36, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): Q/T" + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): L3" + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): Q_hot, Q_cold, W (presumably in the inequality, never defined in 'Said')" + }, + { + "n": 39, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (mechanism): COP 12 (where 12 comes from)" + }, + { + "n": 40, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 9 (mechanism): The L3 inequality itself is not spoken. 'algebra' is all the reader gets." + }, + { + "n": 41, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (mechanism): 'never destroyed' is stated as a rule. It is not tied to what the reader already knows, that heat flows hot to cold by itself." + }, + { + "n": 42, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (mechanism): How the inequality turns into T_hot / (T_hot − T_cold) is compressed into 'algebra'." + }, + { + "n": 43, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 10 (notice): refrigerant lift" + }, + { + "n": 44, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 10 (notice): refrigerant" + }, + { + "n": 45, + "severity": "edge", + "kind": "leap", + "text": "Beat 10 (notice): 'ideal 6.3' is not derived in 'Said' (313.15/50). The reader has to work out that it uses 40 °C and −10 °C instead of 35 °C and 0 °C." + }, + { + "n": 46, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 11 (close): scope" + }, + { + "n": 47, + "severity": "edge", + "kind": "leap", + "text": "Beat 11 (close): The fridge connection is named but not explained." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151329-coldread-diagram.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151329-coldread-diagram.json new file mode 100644 index 0000000..e2e1650 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151329-coldread-diagram.json @@ -0,0 +1,95 @@ +{ + "tool": "coldread", + "rendering": "diagram", + "slug": "heat-pump", + "at": "2026-10-04T15:13:29", + "model": "default", + "prompt_sha": "b7a65e2b208a", + "reply": { + "diagrams": [ + { + "at": "Diagram 1 (Level 1 — Where the 3 kWh comes from)", + "text": "Three boxes feed into or out of a 'Heat pump' box. 'Electricity 1 kWh' (yellow) and 'Outdoor air at 0 °C / outdoor heat: 2 kWh' (blue) go in. 'Home water at 35 °C / heat delivered: 3 kWh' (red) comes out. A caption below explains the colours. It answers the opening puzzle: where do the extra 2 kWh come from?", + "unresolved": [], + "leap": [], + "excess": [ + "\"Energy is conserved: 1 kWh of electricity + 2 kWh of outdoor heat = 3 kWh of heat delivered.\" The diagram has just shown this sum, and the reader knows energy conservation.", + "\"Air at 0 °C is 273 K above absolute zero, so it still holds heat.\" This reader knows kelvin. It is brief and heads off a 'how can cold air give heat?' doubt, so it costs little." + ], + "purpose_unclear": false + }, + { + "at": "Diagram 2 (Level 2 — the refrigerant loop)", + "text": "A four-part loop split into a 'Low pressure, 3.5 bar' group (1. Evaporator, outdoor coil, boils at −10 °C; 4. Expansion valve, pressure drops, cools to −10 °C) and a 'High pressure, 14 bar' group (2. Compressor, vapour squeezed, heats to about 60 °C; 3. Condenser, indoor coil, condenses at 40 °C). Arrows: cold vapour → hot vapour → warm liquid → cold liquid and vapour. Outdoor air sends 2 kWh in, electricity sends 1 kWh to the compressor, and the condenser sends 3 kWh to the home water.", + "unresolved": [], + "leap": [ + "The compressor label says the vapour heats \"to about 60 °C\", but the next box says it \"condenses at 40 °C\". Neither the diagram nor step 3 says that the hot vapour first cools from 60 °C to 40 °C, giving up heat to the water, before it condenses. A careful reader stops to ask why the temperature drops by 20 K.", + "The compressor is drawn inside the \"High pressure, 14 bar\" group, yet it takes in 3.5 bar vapour. It is really the boundary between the two pressures, so the grouping slightly misleads." + ], + "excess": [ + "\"Heat flows by itself only from hot to cold.\" This reader already knows it, though it sets up the argument.", + "\"(The air around us is at 1 bar.)\" This reader knows pressure. It is a small aside." + ], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Level 1 prose: \"Energy is conserved: 1 kWh of electricity + 2 kWh of outdoor heat = 3 kWh of heat delivered.\" It repeats the diagram.", + "Level 2: \"(The air around us is at 1 bar.)\"", + "Level 2 opening: \"Heat flows by itself only from hot to cold.\" The reader knows this; keep only the thought experiment that follows it." + ], + "question": { + "text": "How can a heat pump deliver about 3 kWh of heat for 1 kWh of electricity without breaking energy conservation? A follow-on question: what limits that ratio?", + "known_at": "The title and the first paragraph: \"Energy cannot be created, so where do the other 2 kWh come from?\"", + "result_stated": "Yes, in words, at once: \"**Answer:** the heat pump does not make that heat. It moves heat from the cold outdoor air into the home. The electricity pays for the move.\"" + }, + "motive": "The reason to care is the apparent paradox: \"An electric heater turns 1 kWh of electricity into 1 kWh of heat. A heat pump gives about 3 kWh of heat for the same 1 kWh. Energy cannot be created...\" The reason for the approach is given in Level 2: piping a fluid with no other step would make \"the home gets colder\", so \"the fluid must be colder than the air outside and hotter than the water inside. One fluid can be both, because its boiling temperature depends on its pressure.\" Level 3 is motivated by the \"Predict first\" question about colder days.", + "close": "Yes. The Summary says: \"A heat pump does not make 3 kWh of heat from 1 kWh of electricity. It moves 2 kWh of outdoor heat into the home, and the 1 kWh of electricity pays for the move and ends up as heat too. Pressure sets the boiling temperature... Heat flows the normal way at both coils. The colder the day, the more each kWh costs.\" The entropy limit is not restated in the summary.", + "recap": "A heat pump doesn't make heat: it uses electricity to change the refrigerant's pressure, so the refrigerant boils colder than the outdoor air and condenses hotter than the home water, carrying 2 kWh in from outside plus the 1 kWh of electricity.", + "strongest": "Diagram 2, read together with the numbered walk-through and the earlier thought experiment (\"the home gets colder\"). Each coil's temperature is set against its neighbour (−10 °C vs 0 °C air, 40 °C vs 35 °C water), so \"At both coils, heat flows the normal way\" lands concretely.", + "weakest": "Diagram 2's compressor box, \"heats to about 60 °C\", followed by \"condenses at 40 °C\" with no word about the cooling in between. A smaller point: the closing line \"these two pressure changes do all the work\" sits oddly next to \"The valve adds no energy and removes none.\"", + "suggestions": [ + "In step 3, add a clause such as: 'the 60 °C vapour first cools to 40 °C, giving heat to the water, then condenses'.", + "Draw the compressor (and the valve) on the boundary between the two pressure groups, or label it '3.5 → 14 bar'.", + "Change \"do all the work\" to something like \"make this possible\", so it does not clash with the valve doing no work.", + "Consider one sentence in the Summary that restates the entropy limit in words: the bigger the lift, the more entropy must be dumped, so the more work is required." + ] + }, + "usage": { + "cost": 0.11641279999999998, + "seconds": 32 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Level 1 prose: \"Energy is conserved: 1 kWh of electricity + 2 kWh of outdoor heat = 3 kWh of heat delivered.\" It repeats the diagram." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Level 2: \"(The air around us is at 1 bar.)\"" + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Level 2 opening: \"Heat flows by itself only from hot to cold.\" The reader knows this; keep only the thought experiment that follows it." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Diagram 2 (Level 2 — the refrigerant loop): The compressor label says the vapour heats \"to about 60 °C\", but the next box says it \"condenses at 40 °C\". Neither the diagram nor step 3 says that the hot vapour first cools from 60 °C to 40 °C, giving up heat to the water, before it condenses. A careful reader stops to ask why the temperature drops by 20 K." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Diagram 2 (Level 2 — the refrigerant loop): The compressor is drawn inside the \"High pressure, 14 bar\" group, yet it takes in 3.5 bar vapour. It is really the boundary between the two pressures, so the grouping slightly misleads." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151455-quiz-diagram.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151455-quiz-diagram.json new file mode 100644 index 0000000..1599bcf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151455-quiz-diagram.json @@ -0,0 +1,243 @@ +{ + "tool": "quiz", + "rendering": "diagram", + "slug": "heat-pump", + "at": "2026-10-04T15:14:31", + "model": "default", + "prompt_sha": "c613363175a6", + "reply": { + "answers": { + "1": "The heat pump; the electricity it uses; the outdoor air (the cold source, giving 'outdoor heat'); the home water (35 °C, the hot side); the refrigerant (propane in the example), which moves in a closed loop through four parts: the evaporator (outdoor coil), the compressor, the condenser (indoor coil) and the expansion valve. Measures: COP, temperature lift, the ideal (Carnot) limit, and entropy.", + "2": "Electricity drives the compressor. The refrigerant takes heat from the outdoor air in the evaporator by boiling at low pressure (3.5 bar, −10 °C). The compressor raises it to high pressure (14 bar), which heats the vapour to about 60 °C. In the condenser it cools and condenses at 40 °C, giving heat to the 35 °C water. The expansion valve drops the pressure again, so part of the liquid boils and the rest cools to −10 °C. COP is heat delivered ÷ electricity. It is limited by the lift between the outdoor air and the water, through the entropy rule.", + "3": "The compressor and valve set two pressures, so the refrigerant boils below the outdoor temperature and condenses above the water temperature. Heat therefore flows naturally into the refrigerant outside and out of it inside. The electricity pays for this move, so the delivered heat is the outdoor heat plus the electricity.", + "4": "The temperature lift (water temperature minus outdoor temperature) matters most. If it gets larger (a colder day or hotter water), COP falls: 10 °C gives about 4, 0 °C about 3, −15 °C about 2 and −25 °C about 1.5. The ideal limit T_hot/(T_hot−T_cold) falls as the bottom of the fraction grows. Entropy explains why: heat taken in at a lower T carries more entropy per kWh, so more of the delivered heat must come from electricity.", + "5": "Assumptions and simplifications: the refrigerant boils 10 K below the air and condenses 5 K above the water; the real COP figures are labelled estimates; the ideal limit assumes no entropy is created. Facts: propane boils at −10 °C at about 3.5 bar and at 40 °C at about 14 bar; energy is conserved (Q_cold = Q_hot − W).", + "6": "A heat pump does not make heat; it moves heat from cold outdoor air into warmer home water, and the electricity pays for the move. A refrigerant's boiling point depends on pressure, so a closed loop holds it at low pressure outside (boiling below air temperature) and high pressure inside (condensing above water temperature). The compressor raises the pressure using electricity, and the valve drops it, so heat flows the normal hot-to-cold way at both coils. The delivered heat is the outdoor heat plus the compressor's energy, so COP can be about 3. Entropy limits COP to T_hot/(T_hot−T_cold), so a larger lift (colder day, hotter water) means a lower COP, and real machines get much less than this limit because they need temperature gaps at the coils and have losses.", + "7": "The entropy section was hardest. Entropy appears only as 'Q/T, which cannot be destroyed', with no sense of what it is or what the kWh/K unit means. The inequality also leaves out the electricity's entropy without saying why. One sentence saying electricity (work) carries no entropy, and a short intuition for what entropy measures, would have helped.", + "8": "No. The 3 kWh is 2 kWh of heat taken from the outdoor air plus the 1 kWh of electricity, which ends up in the vapour and then in the water. Energy in equals energy out.", + "9": "On the 10 °C day. The lift is 25 K instead of 50 K. The ideal limit is 12.3 instead of 6.2, and the typical COP is about 4 instead of about 2. A smaller lift means less electricity for each kWh of heat.", + "10": "The compressor uses electricity to squeeze the vapour from 3.5 to 14 bar, which heats it to about 60 °C, so it can condense at 40 °C above the water temperature. The expansion valve drops the pressure from 14 back to 3.5 bar through a narrow opening. About a third of the liquid boils at once, which cools all of it to −10 °C, below the air temperature; the valve adds and removes no energy. Both are needed because one fixed pressure gives one boiling point, and the fluid must be colder than the air outside and hotter than the water inside, so the loop needs a high-pressure side and a low-pressure side.", + "11": "From the outdoor air. Air at 0 °C (273 K) still holds heat. The refrigerant boiling at −10 °C in the outdoor coil is colder than the air, so 2 kWh flows into it.", + "12": "Heat flows by itself only from hot to cold. To take heat from the air, the refrigerant must be colder than the air: −10 °C against 0 °C. If it were warmer, no heat would flow in.", + "13": "COP falls, for example from about 3 at 0 °C to about 2 at −15 °C. The lift grows, so the ideal limit T_hot/(T_hot−T_cold) shrinks. In entropy terms, each kWh taken from colder air carries more entropy, so more electricity is needed for each kWh delivered." + }, + "audit": { + "terms": [ + "The cost of heat is called 'cost' in the Level 3 title ('Why the cost rises on a colder day'), 'electricity each kWh of heat costs' (line 80) and 'the more each kWh costs' (Summary), but 'Not covered here: cost and CO₂' uses cost to mean money. The two meanings conflict.", + "The limit has three names: 'ideal limit' (table column, line 95), 'the Carnot limit' (line 95) and 'ideal COP' (formula, line 97).", + "Each coil has two names: 'Evaporator' / 'outdoor coil' and 'Condenser' / 'indoor coil' (diagram and steps 1 and 3). Both names are given together, but the body text and the Level 3 section use 'coil' alone.", + "The electricity is called 'electricity' (Levels 1–2), 'W' and 'electricity' (line 109), and 'work' ('The work on the gas', line 68; 'do all the work', line 72). The text never says that 'work' and the electricity are the same thing.", + "Two lifts: 'temperature lift' or 'lift' (water minus air, line 80) and 'refrigerant lift' (line 127). They are told apart only at the end, and the table's ideal limit uses the first." + ], + "unexplained": [ + "Entropy values in 'kWh/K' (0.00042, 0.00732, 0.00974, 0.0403, 0.0389, line 107 onward). Entropy is new to this reader, and the text never says what a larger or smaller value means beyond 'cannot be destroyed'.", + "The 'Typical real COP (estimate)' column (line 82): no basis is given for the estimates, or for 'COP is about 2.2' with 55 °C radiators.", + "'The compressor, fans, and pumps lose about half of the rest' (line 127): it is unclear what 'the rest' refers to, and why the loss is about half.", + "'about a third of it boils at once' (line 70): the fraction is stated without saying where it comes from.", + "'boils 10 K below the air … condenses 5 K above the water' (line 127): the trade-off is named (bigger gap, smaller coil), but there is no sense of what changes if the gap is smaller or larger." + ], + "missing_why": [ + "Q_hot/T_hot ≥ Q_cold/T_cold (line 111) ignores the 1 kWh of electricity on the entropy side. The text never says that electricity (work) carries no entropy, which the derivation depends on.", + "'Heat Q that passes at kelvin temperature T carries entropy Q/T' (line 107) is stated without any reason why entropy has this form.", + "'The work on the gas heats it to about 60 °C' (line 68): there is no reason given for why compressing a gas heats it, or why to 60 °C specifically. The bicycle pump is only an analogy.", + "'these two pressure changes do all the work' (line 72) conflicts with 'The valve adds no energy and removes none' (line 70), and the text does not resolve this.", + "Why 'The refrigerant ends each cycle where it started, so it cannot store entropy' means the entropy balance must hold for each cycle is stated briefly, without that argument being spelled out." + ], + "excess": [ + "'Air at 0 °C is 273 K above absolute zero, so it still holds heat' (line 25): this reader already knows kelvin.", + "'(The air around us is at 1 bar.)' (line 39): this reader already knows pressure.", + "The worked example of 1 kWh flowing from water to air and the entropy gain of 0.00042 (line 107) repeats the hot-to-cold rule the reader already knows.", + "The Summary (line 131) repeats Levels 1–3 and adds nothing new.", + "'A fridge uses the same loop' (line 133) is a detour that neither the questions nor the argument uses." + ] + }, + "gaps": [ + "Entropy is never explained as a concept; it is defined only as Q/T plus a rule that it cannot be destroyed.", + "The text does not say that electrical work brings no entropy, which the COP limit derivation needs.", + "There is no basis for the real-COP estimates, or for the 'lose about half' of the gap between 6.3 and 3.", + "'Cost' means electricity per kWh in the body but money in 'Not covered', which is confusing.", + "The text does not explain why compressing the vapour raises its temperature, beyond the bicycle-pump analogy." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 1, + "6": 2, + "7": 2, + "8": 1, + "9": 2, + "10": 2, + "11": 2, + "12": 1, + "13": 2 + }, + "reasons": { + "5": "The answer calls the propane pressures (about 3.5 and 14 bar) facts, but model.md lists them under Estimated. It also leaves out the main stated assumptions, 0 °C outside and 35 °C water. The real-COP estimates and the 10 K and 5 K coil gaps are labelled correctly.", + "8": "The answer gives the 'No' and the balance 2 + 1 = 3 correctly. It does not say that the outdoor air still holds heat even though it is cold. It also does not say that the refrigerant is made colder than the air so heat flows into it. Both are part of the expected answer.", + "11": "No loss. The answer names the outdoor air and the 2 kWh specifically and explains why heat flows into the refrigerant. The balance 3 = 1 + 2 is not written out but follows directly from the question's premise.", + "12": "The outdoor half is correct and specific: heat flows hot to cold, so the refrigerant must be colder than the air (−10 °C against 0 °C). The expected answer also covers the loop and the indoor half, where the refrigerant must be hotter than the water to give heat out. The answer leaves this out. 'No heat would flow in' is weaker than the model's point that heat would flow out to the air. Note: the expected text for this item is broader than its question asks, so the rubric may need aligning.", + "pass_note": "Fails because quiz items 8 and 12 score 1 instead of 2. No general item scores 0." + }, + "pass": false + }, + "usage": { + "cost": 0.2533404, + "seconds": 66 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The cost of heat is called 'cost' in the Level 3 title ('Why the cost rises on a colder day'), 'electricity each kWh of heat costs' (line 80) and 'the more each kWh costs' (Summary), but 'Not covered here: cost and CO₂' uses cost to mean money. The two meanings conflict." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "The limit has three names: 'ideal limit' (table column, line 95), 'the Carnot limit' (line 95) and 'ideal COP' (formula, line 97)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "Each coil has two names: 'Evaporator' / 'outdoor coil' and 'Condenser' / 'indoor coil' (diagram and steps 1 and 3). Both names are given together, but the body text and the Level 3 section use 'coil' alone." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "The electricity is called 'electricity' (Levels 1–2), 'W' and 'electricity' (line 109), and 'work' ('The work on the gas', line 68; 'do all the work', line 72). The text never says that 'work' and the electricity are the same thing." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "Two lifts: 'temperature lift' or 'lift' (water minus air, line 80) and 'refrigerant lift' (line 127). They are told apart only at the end, and the table's ideal limit uses the first." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Entropy values in 'kWh/K' (0.00042, 0.00732, 0.00974, 0.0403, 0.0389, line 107 onward). Entropy is new to this reader, and the text never says what a larger or smaller value means beyond 'cannot be destroyed'." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "The 'Typical real COP (estimate)' column (line 82): no basis is given for the estimates, or for 'COP is about 2.2' with 55 °C radiators." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'The compressor, fans, and pumps lose about half of the rest' (line 127): it is unclear what 'the rest' refers to, and why the loss is about half." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'about a third of it boils at once' (line 70): the fraction is stated without saying where it comes from." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "'boils 10 K below the air … condenses 5 K above the water' (line 127): the trade-off is named (bigger gap, smaller coil), but there is no sense of what changes if the gap is smaller or larger." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Q_hot/T_hot ≥ Q_cold/T_cold (line 111) ignores the 1 kWh of electricity on the entropy side. The text never says that electricity (work) carries no entropy, which the derivation depends on." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'Heat Q that passes at kelvin temperature T carries entropy Q/T' (line 107) is stated without any reason why entropy has this form." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "'The work on the gas heats it to about 60 °C' (line 68): there is no reason given for why compressing a gas heats it, or why to 60 °C specifically. The bicycle pump is only an analogy." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "'these two pressure changes do all the work' (line 72) conflicts with 'The valve adds no energy and removes none' (line 70), and the text does not resolve this." + }, + { + "n": 15, + "severity": "gap", + "kind": "missing_why", + "text": "Why 'The refrigerant ends each cycle where it started, so it cannot store entropy' means the entropy balance must hold for each cycle is stated briefly, without that argument being spelled out." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "'Air at 0 °C is 273 K above absolute zero, so it still holds heat' (line 25): this reader already knows kelvin." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "'(The air around us is at 1 bar.)' (line 39): this reader already knows pressure." + }, + { + "n": 18, + "severity": "excess", + "kind": "excess", + "text": "The worked example of 1 kWh flowing from water to air and the entropy gain of 0.00042 (line 107) repeats the hot-to-cold rule the reader already knows." + }, + { + "n": 19, + "severity": "excess", + "kind": "excess", + "text": "The Summary (line 131) repeats Levels 1–3 and adds nothing new." + }, + { + "n": 20, + "severity": "excess", + "kind": "excess", + "text": "'A fridge uses the same loop' (line 133) is a detour that neither the questions nor the argument uses." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "Entropy is never explained as a concept; it is defined only as Q/T plus a rule that it cannot be destroyed." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "The text does not say that electrical work brings no entropy, which the COP limit derivation needs." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "There is no basis for the real-COP estimates, or for the 'lose about half' of the gap between 6.3 and 3." + }, + { + "n": 24, + "severity": "gap", + "kind": "gap", + "text": "'Cost' means electricity per kWh in the body but money in 'Not covered', which is confusing." + }, + { + "n": 25, + "severity": "gap", + "kind": "gap", + "text": "The text does not explain why compressing the vapour raises its temperature, beyond the bicycle-pump analogy." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151658-quiz-diagram.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151658-quiz-diagram.json new file mode 100644 index 0000000..fff9b3c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/runs/2026-10-04T151658-quiz-diagram.json @@ -0,0 +1,232 @@ +{ + "tool": "quiz", + "rendering": "diagram", + "slug": "heat-pump", + "at": "2026-10-04T15:16:46", + "model": "default", + "prompt_sha": "a1f778fe9ff6", + "reply": { + "answers": { + "1": "The main objects are electricity (1 kWh), outdoor air (the cold source, 0 °C in the example), home water (the hot sink, 35 °C) and the heat pump itself. The heat pump contains a refrigerant (propane) that circulates through a closed loop of four parts: the evaporator (outdoor coil), the compressor, the condenser (indoor coil) and the expansion valve. The measures used are COP, temperature lift, the ideal (Carnot) limit and entropy.", + "2": "Electricity drives the compressor. Outdoor air gives heat to the refrigerant in the evaporator, because the refrigerant boils there at -10 °C, below the air temperature. The compressor raises the refrigerant from 3.5 to 14 bar, which heats the vapour to about 60 °C. In the condenser the refrigerant condenses at 40 °C, above the water temperature, so heat flows into the home water. The expansion valve drops the pressure back to 3.5 bar, and part of the liquid boils, cooling the rest to -10 °C. Pressure sets the boiling temperature, so the low-pressure side is cold and the high-pressure side is hot. COP is heat delivered divided by electricity used. COP falls as the temperature lift (water temperature minus air temperature) grows, and entropy sets an upper limit on it.", + "3": "Electricity runs a compressor that raises the refrigerant's pressure, and with it the boiling/condensing temperature. That lets the refrigerant soak up heat from cold outdoor air at low pressure and release it into warmer home water at high pressure, with heat flowing hot-to-cold at each coil. So the water gets the moved outdoor heat plus the electricity: 2 + 1 = 3 kWh.", + "4": "The key variable is the temperature lift. If the outdoor air gets colder, or the water must be hotter, the lift grows and COP falls. In the table, COP is about 4 at 10 °C, about 3 at 0 °C, about 2 at -15 °C and about 1.5 at -25 °C. The reason is entropy. Heat taken at a lower temperature carries more entropy per kWh, and the water must receive at least that much entropy, so more of the delivered heat must come from electricity. The ideal limit T_hot/(T_hot - T_cold) falls as the bottom of the fraction grows.", + "5": "Assumptions/simplifications: the 'typical real COP' values are labelled estimates; the refrigerant is assumed to boil exactly 10 K below the air and condense 5 K above the water; the expansion valve is said to add and remove no energy. Established facts: propane boils at -10 °C at about 3.5 bar and at 40 °C at about 14 bar; energy is conserved; entropy cannot be destroyed.", + "6": "A heat pump does not create heat: it moves heat from cold outdoor air into warmer home water, and the electricity it uses ends up as heat too, so 1 kWh in can give 3 kWh out. It does this with a refrigerant whose boiling temperature depends on pressure. At low pressure the refrigerant boils below the outdoor air temperature and absorbs heat. At high pressure, after the compressor, it condenses above the water temperature and releases heat. The expansion valve drops the pressure again to close the loop. Entropy cannot be destroyed, which sets an upper limit on COP of T_hot/(T_hot - T_cold) in kelvin, so a bigger temperature lift (a colder day or hotter water) means a lower COP. Real units reach about a third of that limit because of the coil temperature gaps and machine losses.", + "7": "The entropy section was hardest. It says heat Q at temperature T 'carries entropy Q/T' and that entropy cannot be destroyed, but it never says what entropy is or why it has the form Q/T. A sentence or two of intuition would have made the derivation feel earned, not just checkable. A second point: the expansion valve 'boils a third of the liquid' with no energy added, and it isn't obvious how that conserves energy until you see that the heat comes from the liquid itself.", + "8": "No, it does not break energy conservation. Of the 3 kWh, 1 kWh is the electricity, which ends up as heat in the vapour, and 2 kWh is heat taken from the outdoor air. Air at 0 °C is 273 K above absolute zero, so it still holds heat. It flows into the refrigerant because the refrigerant boils colder (-10 °C) in the evaporator.", + "9": "COP is higher on the 10 °C day. The lift there is 25 K (ideal limit 12.3, typical real COP about 4), against 50 K at -15 °C (ideal limit 6.2, typical about 2). A bigger lift needs more electricity per kWh of heat, because heat taken at a lower temperature carries more entropy, and the water must receive at least that much.", + "10": "The compressor uses electricity to squeeze the vapour from 3.5 to 14 bar, which heats it to about 60 °C and raises its condensing temperature to 40 °C, so it can give heat to the 35 °C water. The expansion valve is a narrow opening that drops the pressure from 14 back to 3.5 bar. About a third of the liquid boils at once, cooling the whole flow to -10 °C, so it can take heat from the 0 °C air. The loop needs both because the refrigerant must be at high pressure (hot) at the indoor coil and low pressure (cold) at the outdoor coil. One raises the pressure and the other lowers it. Only the compressor uses energy.", + "11": "They are heat taken from the outdoor air in the evaporator. The refrigerant boils at -10 °C there, colder than the 0 °C air, so heat flows into it by itself. That heat is carried around the loop and released in the condenser, along with the compressor's 1 kWh.", + "12": "Heat flows by itself only from hot to cold. If the refrigerant were not colder than the outdoor air, no heat would flow into it. If it were not hotter than the water, heat would flow from the water into the refrigerant and the home would get colder. A temperature gap also moves heat faster through a smaller coil, which is why the explanation uses 10 K and 5 K gaps.", + "13": "COP falls, for example from about 3 at 0 °C to about 2 at -15 °C and about 1.5 at -25 °C. The temperature lift grows, so the entropy limit T_hot/(T_hot - T_cold) falls and each kWh of heat needs more electricity. Many units add a backup electric heater (COP 1) for the coldest days." + }, + "audit": { + "terms": [ + "The same limit has two names: 'ideal limit' (Level 3 table, 'The ideal limit') and 'Carnot limit' ('physicists call it the Carnot limit').", + "Two words for the refrigerant: 'fluid' (Level 1: 'makes its fluid colder'; Level 2: 'pipe a fluid') and 'refrigerant' (named only partway through Level 2).", + "Two words for the refrigerant vapour: 'vapour' (diagram, steps 2-3) and 'gas' (step 2: 'push on the gas', 'heats the gas'). A reader may wonder if the change of word means a change of state.", + "Two words for the electricity input: 'electricity' and 'W' in the formulas. Its relation to 'push' or work in step 2 is not made explicit.", + "Two names for the heat taken from the air: 'outdoor heat' (Levels 1 and 2) and 'Q_cold' (Level 3). The mapping is given, but 'heat delivered' becomes 'Q_hot', and the air side of the entropy argument talks about 'the air loses'." + ], + "unexplained": [ + "'Losses in the compressor, fans, and pumps take the COP from 6.3 down to about 3': the size of these losses is not explained, and they halve the COP.", + "Table column 'Typical real COP (estimate)': no source or method, so the reader can't tell how firm 'about 4/3/2/1.5' are.", + "'Old radiators need 55 °C water ... the COP is about 2.2': no ideal-limit figure is given for this case, and it's not shown how 2.2 was obtained.", + "'about a third of it boils at once' at the expansion valve: there is no basis for 'a third', and nothing says what a larger or smaller fraction would mean.", + "The coil gaps 'boils 10 K below the air ... condenses 5 K above the water': the reader isn't told why these values or what changes with bigger or smaller gaps beyond 'faster through a smaller coil'. The text also doesn't say whether the -10 °C boiling point moves on a -15 °C day." + ], + "missing_why": [ + "'Heat Q that passes at kelvin temperature T carries entropy Q/T': stated without any reason why entropy has this form or what entropy is. The whole limit rests on it.", + "'its boiling temperature depends on its pressure': it isn't explained why higher pressure raises the boiling point. Only propane data is given.", + "'that push heats the gas': only the bicycle-pump analogy is offered, with no reason why compressing a gas raises its temperature.", + "'Nothing can destroy entropy': stated as a law, with the link to hot-to-cold flow asserted rather than motivated.", + "'The refrigerant ends each cycle where it started, so it cannot store entropy': it isn't explained why the entropy bookkeeping uses the water and air temperatures rather than the refrigerant's own. This only comes up later as 'refrigerant lift'." + ], + "excess": [ + "'Air at 0 °C is 273 K above absolute zero, so it still holds heat': this reader already knows kelvin.", + "'(The air around us is at 1 bar.)': this reader already knows pressure.", + "'An electric heater has only one input, the electricity. So it can give only 1 kWh per kWh.': repeats the opening and the COP-1 statement.", + "'A fridge uses the same loop...' and the 'Not covered here' list: detours that neither the questions nor the argument use." + ] + }, + "gaps": [ + "Entropy is used as the basis of the limit but never explained beyond 'Q/T' and 'cannot be destroyed'.", + "The text doesn't say how the refrigerant's boiling temperature or pressure adjusts when the outdoor temperature changes. The loop is described only for 0 °C air (-10 °C boiling), so it's unclear how it works on a -15 °C day.", + "There is no explanation of why real COP is only about a third of the ideal limit, beyond naming the losses.", + "Energy accounting at the expansion valve, where liquid boils with no energy added, is only lightly explained.", + "The real COP figures are estimates with no stated basis." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 1, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "5": "Misclassifies the propane boiling pressures (about 3.5 and 14 bar) as established facts. model.md labels them estimates. It also lists 'the valve adds and removes no energy' as an assumption, when it is a stated fact of the cycle. It omits other estimates the model labels, such as vapour at about 60 °C and about a third of the liquid boiling. It also omits the core assumed conditions of 0 °C outside and 35 °C water." + }, + "pass": true + }, + "usage": { + "cost": 0.23809480000000002, + "seconds": 60 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same limit has two names: 'ideal limit' (Level 3 table, 'The ideal limit') and 'Carnot limit' ('physicists call it the Carnot limit')." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Two words for the refrigerant: 'fluid' (Level 1: 'makes its fluid colder'; Level 2: 'pipe a fluid') and 'refrigerant' (named only partway through Level 2)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "Two words for the refrigerant vapour: 'vapour' (diagram, steps 2-3) and 'gas' (step 2: 'push on the gas', 'heats the gas'). A reader may wonder if the change of word means a change of state." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "Two words for the electricity input: 'electricity' and 'W' in the formulas. Its relation to 'push' or work in step 2 is not made explicit." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "Two names for the heat taken from the air: 'outdoor heat' (Levels 1 and 2) and 'Q_cold' (Level 3). The mapping is given, but 'heat delivered' becomes 'Q_hot', and the air side of the entropy argument talks about 'the air loses'." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'Losses in the compressor, fans, and pumps take the COP from 6.3 down to about 3': the size of these losses is not explained, and they halve the COP." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "Table column 'Typical real COP (estimate)': no source or method, so the reader can't tell how firm 'about 4/3/2/1.5' are." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'Old radiators need 55 °C water ... the COP is about 2.2': no ideal-limit figure is given for this case, and it's not shown how 2.2 was obtained." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'about a third of it boils at once' at the expansion valve: there is no basis for 'a third', and nothing says what a larger or smaller fraction would mean." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "The coil gaps 'boils 10 K below the air ... condenses 5 K above the water': the reader isn't told why these values or what changes with bigger or smaller gaps beyond 'faster through a smaller coil'. The text also doesn't say whether the -10 °C boiling point moves on a -15 °C day." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'Heat Q that passes at kelvin temperature T carries entropy Q/T': stated without any reason why entropy has this form or what entropy is. The whole limit rests on it." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'its boiling temperature depends on its pressure': it isn't explained why higher pressure raises the boiling point. Only propane data is given." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "'that push heats the gas': only the bicycle-pump analogy is offered, with no reason why compressing a gas raises its temperature." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "'Nothing can destroy entropy': stated as a law, with the link to hot-to-cold flow asserted rather than motivated." + }, + { + "n": 15, + "severity": "gap", + "kind": "missing_why", + "text": "'The refrigerant ends each cycle where it started, so it cannot store entropy': it isn't explained why the entropy bookkeeping uses the water and air temperatures rather than the refrigerant's own. This only comes up later as 'refrigerant lift'." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "'Air at 0 °C is 273 K above absolute zero, so it still holds heat': this reader already knows kelvin." + }, + { + "n": 17, + "severity": "excess", + "kind": "excess", + "text": "'(The air around us is at 1 bar.)': this reader already knows pressure." + }, + { + "n": 18, + "severity": "excess", + "kind": "excess", + "text": "'An electric heater has only one input, the electricity. So it can give only 1 kWh per kWh.': repeats the opening and the COP-1 statement." + }, + { + "n": 19, + "severity": "excess", + "kind": "excess", + "text": "'A fridge uses the same loop...' and the 'Not covered here' list: detours that neither the questions nor the argument use." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "Entropy is used as the basis of the limit but never explained beyond 'Q/T' and 'cannot be destroyed'." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "The text doesn't say how the refrigerant's boiling temperature or pressure adjusts when the outdoor temperature changes. The loop is described only for 0 °C air (-10 °C boiling), so it's unclear how it works on a -15 °C day." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "There is no explanation of why real COP is only about a third of the ideal limit, beyond naming the losses." + }, + { + "n": 23, + "severity": "gap", + "kind": "gap", + "text": "Energy accounting at the expansion valve, where liquid boils with no energy added, is only lightly explained." + }, + { + "n": 24, + "severity": "gap", + "kind": "gap", + "text": "The real COP figures are estimates with no stated basis." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/understanding.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/understanding.md new file mode 100644 index 0000000..4667ab1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/explainers/heat-pump/review/understanding.md @@ -0,0 +1,19 @@ +# Review: heat-pump (Published tier, Stage 2, diagram.md) + +## Probe — 2026-10-04T150833-probe +Adopted 1–6, 8, 9, 11–14 (entropy defined, holding/breaking entropy cases, expansion-valve mechanism, compression heating, two lifts named apart, coil gaps, one name per concept, 1 bar reference, −25 °C row, cuts). Declined 7 (went to Scope), 10 (adds length, changes no step), 15 (template metadata). + +## Cold read, narrative — 2026-10-04T151129 +5 blocking findings, all caused by a terse "Said" column. Fixed by writing each beat's words in full. Cut: terse Celsius aside (spelled out), radiators given a COP, beat 4 leads with numbers. + +## Cold read, diagram — 2026-10-04T151329 +No blocking findings. Adopted 1, 3, 4 (vapour 60 °C → 40 °C before condensing), 5 (compressor drawn between the pressure groups). Declined 2: 1 bar gives the pressures a meaning. + +## Blind test, diagram +- Run 1 (2026-10-04T151455): FAIL — 8 and 12 scored 1. Item 12's rubric covered more than its question asked: widened the `loop` ask. Item 8: Level 1 now links outdoor heat to the refrigerant being colder than the air. Also fixed: electricity carries no entropy (the derivation needed it), "work" vs electricity, one name for the ideal limit, "cost" with two meanings. +- Run 2 (2026-10-04T151658): PASS — every quiz item 2; general item 5 (assumptions vs facts) 1. Adopted 2, 3 (refrigerant named in Level 1; "vapour", not "gas"). +- Declined, both runs: entropy foundations, why compression heats a gas, coil sizing trade-offs, sources for typical COPs (labelled estimates). These are beyond this reader's question. +- After run 2: the Level 3 heading no longer gives away the predict-first answer. + +## Broken-copy check — 2026-10-04T151949 +A copy with the cold-weather claim reversed (COP rises in the cold) FAILS: items 9 and 13 score 0. The quiz can fail. diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/read.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/read.json new file mode 100644 index 0000000..50a1834 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "The heat pump doesn't make the extra heat. It takes it from the cold outdoor air (2 kWh) and moves it into the home. The 1 kWh of electricity pays for the move and also ends up as heat, so the home gets 3 kWh.", + "quote": "the heat pump does not make that heat. It moves heat from the cold outdoor air into the home. The electricity pays for the move." + }, + "2": { + "answer": "It goes around a closed loop. In the outdoor evaporator it boils at −10 °C and 3.5 bar, taking in 2 kWh from the air. The compressor squeezes it to 14 bar using 1 kWh, which heats it to about 60 °C. In the indoor condenser it cools and condenses at 40 °C, giving 3 kWh to the water. The expansion valve then drops the pressure back to 3.5 bar, part of the liquid boils, and all of it cools to −10 °C before going back to step 1.", + "quote": "The refrigerant is at 3.5 bar, so it boils at −10 °C. It is colder than the 0 °C air, so 2 kWh of heat flows from the air into it." + }, + "3": { + "answer": "On a colder day the temperature lift (water temperature minus outdoor temperature) is bigger. A bigger lift makes the bottom of the ideal-limit fraction bigger, so the limit falls and each kWh of heat needs more electricity. Underneath this is entropy: heat taken from colder air carries more entropy per kWh, and the water must receive at least that much.", + "quote": "The bigger the lift, the more electricity each kWh of heat needs." + }, + "4": { + "answer": "On a 5 °C day the lift is 30 K, so the ideal limit is 308.15/30 ≈ 10.3. On a −15 °C day the lift is 50 K and the limit is 6.2 (from the table). The ideal limit on the colder day is about 40% lower. The 5 °C value is my calculation using the formula; it isn't in the table.", + "quote": "ideal limit = T_hot / (T_hot − T_cold)" + } + }, + "stops": [ + "Lift is given in K, not °C, before kelvin is explained: \"35 − 0 = 35 K in the example\"", + "\"bar\" is only explained by one comparison: \"(The air around us is at 1 bar.)\"", + "Entropy is used without saying what it is: \"Heat Q that passes at kelvin temperature T carries entropy Q/T, in kWh/K.\"", + "This claim is stated without a reason: \"Electricity carries no entropy.\"", + "The real COPs in the table have no source or method: \"Typical real COP (estimate)\"", + "Why does a third of the liquid boil? \"so about a third of it boils at once\"", + "The refrigerant's ideal limit (6.3) is close to the −15 °C row (6.2), which is confusing: \"the ideal limit for the refrigerant is 313.15 / 50 = 6.3\"", + "It's unclear why vapour cooling from 60 to 40 °C counts as part of the 3 kWh: \"The 60 °C vapour cools to 40 °C, then condenses\"" + ], + "excess": [ + "The algebra steps are more than this reader needed: \"Multiply by T_cold: Q_hot × T_cold / T_hot ≥ Q_hot − W\"", + "Off the question: \"A fridge uses the same loop, with the cold coil inside the fridge.\"", + "Off the question: \"**Not covered here:** ground-source heat pumps (warmer, steadier cold side, so a higher COP)\"", + "The colour legend repeats what the diagrams already show: \"Colours in both diagrams: blue is the cold side, red is the hot side, yellow is electricity.\"", + "The summary repeats earlier points: \"Heat flows the normal way at both coils.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/run.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/run.json new file mode 100644 index 0000000..d76a699 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/heat-pump/diagram.md` because the answer depends on a loop of four parts with a temperature and pressure at each point, which a diagram with text shows better than prose. It passed the probe, the cold read and the blind test, and the leftover test copy at `explainers/heat-pump-broken/` needs your OK to delete.", + "error": null, + "cost": 2.352323, + "seconds": 845, + "turns": 77, + "files": [ + "explainers/heat-pump/diagram.md" + ], + "review_cost": 1.0471, + "reviews": 6 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/diagram.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/diagram.md new file mode 100644 index 0000000..bd8ba95 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/diagram.md @@ -0,0 +1,84 @@ +# How can a heat pump deliver three times more heat than the electricity it uses? + + + +**Short answer.** A heat pump does not turn electricity into heat. It uses electricity to *move* heat that already exists in the outdoor air into the house. For each 1 kWh of electricity, it takes about 2 kWh of heat from the air and delivers 3 kWh to the house. No energy is created. + +## Level 1 — Where the 3 kWh come from + + + + +```mermaid +flowchart LR + AIR["Outdoor air, 0 °C
gives 2 kWh"] --> HP(("heat pump")) + ELEC["Electricity
1 kWh"] --> HP + HP --> HOUSE["House, water at 35 °C
gets 3 kWh"] +``` + +Energy is conserved: **heat delivered = heat taken from the air + electricity**, so 3 kWh = 2 kWh + 1 kWh. + +The ratio of heat delivered to electricity used is the **COP** (coefficient of performance). Here COP = 3 ÷ 1 = 3. + +- COP 1 is what an electric resistance heater gets. It turns 1 kWh of electricity into 1 kWh of heat, and conservation of energy forbids more. +- COP 3 is typical for a heat pump on a 0 °C day. +- COP 4 or more is possible on a mild day. + +COP is not an efficiency. It counts only the paid input (electricity) and leaves out the free input (heat from the air). That is why it can exceed 1. + +Air at 0 °C still holds a lot of thermal energy: 0 °C is 273 K, far above absolute zero. To extract that energy, you only need something colder than the air. + +## Level 2 — How the heat is moved: the refrigerant loop + +A **refrigerant** is a fluid that circulates in a closed loop and carries the heat. One example is propane (R290). Its boiling point depends on pressure: it boils at about −10 °C at 3.5 bar and condenses at about 40 °C at 14 bar. The heat pump uses this to make the refrigerant colder than the outdoor air at one place and hotter than the house at another. + +```mermaid +flowchart LR + AIR["Outdoor air 0 °C"] -->|"2 kWh heat in"| EVAP + ELEC["Electricity"] -->|"1 kWh work"| COMP + subgraph LOOP["refrigerant loop"] + direction LR + EVAP["① evaporator
boils at −10 °C, 3.5 bar"] -->|"cold gas"| COMP["② compressor
3.5 → 14 bar"] + COMP -->|"hot gas"| COND["③ condenser
condenses at 40 °C, 14 bar"] + COND -->|"warm liquid"| VALVE["④ expansion valve
14 → 3.5 bar"] + VALVE -->|"cold liquid, −10 °C"| EVAP + end + COND -->|"3 kWh heat out"| HOUSE["House water 35 °C"] +``` + +1. **Evaporator (outdoors).** The refrigerant boils at −10 °C. The air is at 0 °C, so heat flows from the air into the refrigerant: 2 kWh. +2. **Compressor.** The electricity (1 kWh) drives the compressor. Squeezing the gas to 14 bar raises its temperature, so it can condense at 40 °C. Without this step, the refrigerant stays at −10 °C. Heat flows only from hot to cold, so −10 °C refrigerant cannot heat 35 °C water. +3. **Condenser (indoors).** The refrigerant condenses at 40 °C. The house water is at 35 °C, so heat flows from the refrigerant into the water. This heat is the 2 kWh from the air plus the 1 kWh of compressor work: 3 kWh. +4. **Expansion valve.** The valve drops the pressure to 3.5 bar, which drops the temperature to −10 °C. Without this step, the refrigerant stays at 40 °C. It is then hotter than the 0 °C air, so it gives heat to the outdoors instead of taking it in. + +A refrigerator uses the same loop. Its evaporator is inside the box and its condenser is the warm grille at the back. + +## Level 3 — Why the COP is about 3 and not higher: temperature lift + + + +The electricity pays to lift heat from a low temperature to a high one. The **temperature lift** is the condenser temperature minus the evaporator temperature. A larger lift needs more work for each kWh of heat moved. + +Physics sets a ceiling. No machine can beat the Carnot limit (temperatures in kelvin): + +COP_max = T_hot ÷ (T_hot − T_cold) + +| Day | Outdoor air | Evaporator | Condenser | Lift | Carnot limit | Real COP (estimate) | +|---|---|---|---|---|---|---| +| Mild | 10 °C | 0 °C (273 K) | 40 °C (313 K) | 40 K | 313 ÷ 40 ≈ 7.8 | about 4 | +| Typical | 0 °C | −10 °C (263 K) | 40 °C (313 K) | 50 K | 313 ÷ 50 ≈ 6.3 | about 3 | +| Cold | −15 °C | −25 °C (248 K) | 40 °C (313 K) | 65 K | 313 ÷ 65 ≈ 4.8 | about 2 | + +The real COPs are estimates. A rule of thumb is that real machines reach about half of the Carnot limit, because of friction, imperfect heat exchangers, and other losses. + +What the table predicts: + +- **Colder outdoor air → lower COP.** The heat pump must lift the heat farther. +- **Cooler indoor water → higher COP.** Large radiators or underfloor heating at 35 °C need less lift than small radiators at 55 °C. +- **COP never falls below 1 in principle.** Even at a large lift, the compressor work itself ends up as heat in the house. + +## Summary + +The heat pump is a pump for heat. The electricity does not become the heat; it pays for moving heat "uphill" from cold outdoor air to the warmer house. Because moving heat costs less energy than the heat moved, 1 kWh of electricity delivers about 3 kWh of heat. The smaller the temperature lift, the larger that ratio. + +**Out of scope:** defrost cycles, ground-source heat pumps, refrigerant choice and safety, and the derivation of the Carnot limit (see entropy and the second law in a thermodynamics text). diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/model.md b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/model.md new file mode 100644 index 0000000..964b27b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/model.md @@ -0,0 +1,113 @@ +# Semantic model: heat pump + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How can a heat pump deliver three times more heat than the electricity it uses? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (energy conservation, temperature in kelvin, boiling point depends on pressure), basic programming. New to heat pumps and refrigeration. Does not know: the refrigerant cycle, COP, the Carnot limit. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| heat pump | A machine that uses electricity to move heat from a cold place (outdoor air) to a warm place (the house). | fact | +| electricity | The work input; it drives the compressor. | fact | +| outdoor air | The cold source; air at 0 °C still contains thermal energy that can be extracted. | fact | +| refrigerant | A fluid that boils at a low temperature and circulates in a closed loop; it carries the heat. | fact | +| evaporator | Outdoor heat exchanger; refrigerant boils there, colder than the air, and absorbs heat. | fact | +| compressor | Squeezes the refrigerant gas to a high pressure, which raises its temperature; uses the electricity. | fact | +| condenser | Indoor heat exchanger; hot refrigerant condenses there, hotter than the house water, and releases heat. | fact | +| expansion valve | Drops the refrigerant pressure, which drops its temperature below the outdoor air again. | fact | +| COP | Coefficient of performance: heat delivered ÷ electricity used. | definition | +| temperature lift | Condenser temperature minus evaporator temperature. | definition | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| outdoor air | gives heat to | evaporator | because the refrigerant there is colder (−10 °C vs 0 °C) | +| electricity | drives | compressor | the only paid energy input | +| compressor | raises pressure and temperature of | refrigerant | 3.5 bar → 14 bar, −10 °C → 40 °C condensing | +| condenser | gives heat to | house | because the refrigerant there is hotter (40 °C vs 35 °C) | +| expansion valve | lowers pressure and temperature of | refrigerant | closes the loop | +| temperature lift | sets the minimum work per unit heat | COP | larger lift → lower COP | + +## Causal chain + +```text +refrigerant boils colder than outdoor air → heat flows in (2 kWh) +→ compressor (1 kWh electricity) makes it hotter than the house +→ heat flows out (2 + 1 = 3 kWh) → valve makes it cold again → repeat +``` + +## Quantities + +- Energy balance (conservation): Q_out = Q_in + W. Worked: 3 kWh = 2 kWh + 1 kWh. +- COP = Q_out / W = 3 / 1 = 3. +- Electric resistance heater: COP = 1 (1 kWh → 1 kWh). +- Carnot limit (ideal machine): COP_max = T_hot / (T_hot − T_cold), temperatures in kelvin. + - Typical day: evaporator −10 °C (263 K), condenser 40 °C (313 K), lift 50 K → COP_max = 313/50 ≈ 6.3. Real ≈ 3. + - Mild day: outdoor 10 °C, evaporator 0 °C (273 K), lift 40 K → COP_max ≈ 7.8. Real ≈ 4. + - Cold day: outdoor −15 °C, evaporator −25 °C (248 K), lift 65 K → COP_max ≈ 4.8. Real ≈ 2. +- Refrigerant R290 (propane): boils at about −10 °C at 3.5 bar and condenses at about 40 °C at 14 bar. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Mild day (10 °C) | COP ≈ 4 | smaller lift, less work per kWh moved | +| Cold day (−15 °C) | COP ≈ 2 | larger lift, more work per kWh moved | +| Resistance heater | COP = 1 | converts electricity to heat; moves none | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Move heat instead of making it | Turn electricity into heat (resistance heater) | 1 kWh in → 1 kWh out, COP 1, never more: conservation of energy caps it. | +| Compress the refrigerant | Just pump the refrigerant round without compressing | Refrigerant at −10 °C cannot heat 35 °C water; heat flows only hot → cold. | +| Use the valve to drop pressure | Keep the refrigerant at high pressure | At 40 °C it is hotter than 0 °C air, so it gives heat to the outdoors instead of taking it. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| COP > 1 does not break energy conservation | 2 kWh from air + 1 kWh electricity = 3 kWh out | If you count only electricity as input, 3 out of 1 looks like 300% "efficiency" — the 2 kWh from the air is omitted. | +| COP falls as lift grows | lift 40 K → ≈ 4; 50 K → ≈ 3 | lift 65 K (−15 °C day) → ≈ 2 | + +## Terms + +- **heat pump**: moves heat from cold outdoor air to the warm house using electricity. +- **refrigerant**: the circulating fluid that carries heat. +- **evaporator / condenser**: where the refrigerant takes in / gives out heat. +- **compressor**: raises refrigerant pressure and temperature; the electricity goes here. +- **COP**: heat delivered ÷ electricity used. 1 = no better than a resistance heater; 3 = typical; 5 = mild weather, low lift. +- **temperature lift**: how far the heat is lifted in temperature. 40 K = easy; 65 K = hard. + +## Epistemic status + +- **Observed / established:** energy conservation; heat flows only hot → cold spontaneously; boiling point rises with pressure. +- **Mathematical consequence:** Q_out = Q_in + W; Carnot limit formula. +- **Assumed:** the example temperatures (0 °C outdoor, 35 °C water, 5–10 K differences at the heat exchangers). +- **Estimated / model output:** real COPs (≈ 2, 3, 4) — rule of thumb: real machines reach about half the Carnot limit. + +## Confusion points + +- "COP 3 means 300% efficient, which is impossible" → the machine does not create energy; 2 of the 3 kWh come from outdoor air. +- "0 °C air has no heat in it" → 0 °C is 273 K; the air holds a lot of thermal energy. Only a colder refrigerant is needed to pull it out. +- "The compressor's heat is the output" → compressor work is only 1 of the 3 kWh. + +## Scope + +- **Out of scope:** defrost cycles, ground-source vs air-source, refrigerant choice and safety, sizing — see any HVAC design guide. +- **Out of scope:** derivation of the Carnot limit — see entropy and the second law in a thermodynamics text. + +## Representation decision + +- **Stage:** 2 +- **Reason:** the mechanism is a closed loop of four components with energy flows in and out; the reader must hold the loop, the temperatures at each point, and the energy balance at once. +- **Levels:** L1 energy balance / L2 the refrigerant loop / L3 why lift sets the limit diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/model.yaml b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/model.yaml new file mode 100644 index 0000000..58bf72e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/explainers/heat-pump/model.yaml @@ -0,0 +1,85 @@ +# Machine-readable half of model.md. `explainer check heat-pump` holds every rendering to it. + +values: + electricity_kwh: 1 + air_heat_kwh: 2 + delivered_kwh: 3 + cop_typical: 3 + cop_resistance: 1 + outdoor_c: 0 + water_c: 35 + radiator_hot_c: 55 + evap_c: -10 + cond_c: 40 + evap_bar: 3.5 + cond_bar: 14 + evap_k: 263 + cond_k: 313 + lift_typical_k: 50 + carnot_typical: 6.3 + mild: + outdoor_c: 10 + evap_c: 0 + evap_k: 273 + lift_k: 40 + carnot: 7.8 + cop: 4 + cold: + outdoor_c: -15 + evap_c: -25 + evap_k: 248 + lift_k: 65 + carnot: 4.8 + cop: 2 + +allow: [5, 273, 2, 1, 3, 4] + +require: + - path: delivered_kwh + in: [diagram] + - path: air_heat_kwh + in: [diagram] + - path: carnot_typical + in: [diagram] + +claims: + - id: move-not-make + kind: why + about: [heat pump] + statement: "A heat pump moves heat that already exists in outdoor air; it does not turn electricity into heat." + alternative: "Turn electricity into heat directly (resistance heater)." + counterexample: "A resistance heater turns 1 kWh into 1 kWh of heat, COP 1; conservation of energy forbids more." + ask: "Where do the extra 2 kWh come from?" + - id: balance + kind: guarantee + statement: "Heat delivered = heat taken from the air + electricity: 3 = 2 + 1 kWh." + example: "2 kWh from 0 °C air plus 1 kWh electricity gives 3 kWh to the house." + counterexample: "Count only electricity as input and COP 3 looks like 300% efficiency, which seems to break conservation; it does not, because 2 kWh came from the air." + - id: compress + kind: why + about: [compressor] + statement: "The compressor raises the refrigerant to 40 °C so that it is hotter than the 35 °C water." + alternative: "Circulate the refrigerant without compressing it." + counterexample: "Refrigerant at −10 °C cannot heat 35 °C water; heat flows only from hot to cold." + ask: "Why is the compressor needed if the heat comes from the air?" + - id: valve + kind: why + about: [expansion valve] + statement: "The expansion valve drops pressure so the refrigerant is colder than the outdoor air again." + alternative: "Keep the refrigerant at high pressure." + counterexample: "At 40 °C it is hotter than 0 °C air and would lose heat outdoors instead of taking it in." + - id: lift-limit + kind: guarantee + statement: "COP falls as the temperature lift grows; the Carnot limit is T_hot / (T_hot − T_cold)." + example: "Lift 50 K: limit 313/50 ≈ 6.3, real COP about 3. Lift 40 K: limit 7.8, real about 4." + counterexample: "On a −15 °C day the lift is 65 K: limit 4.8, real COP about 2." + ask: "What happens to COP on a very cold day, and why?" + +accept_unjustified: {} + +terms: + - term: COP + means: "heat delivered ÷ electricity used; not an efficiency" + avoid: [300% efficient, efficiency of 300] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/read.json new file mode 100644 index 0000000..383c27a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/read.json @@ -0,0 +1,43 @@ +{ + "answers": { + "1": { + "answer": "From the outdoor air. The heat pump moves heat that is already in the air into the house. About 2 of the 3 kWh come from the air, and the other 1 kWh is the electricity. No energy is created.", + "quote": "It uses electricity to *move* heat that already exists in the outdoor air into the house. For each 1 kWh of electricity, it takes about 2 kWh of heat from the air and delivers 3 kWh to the house." + }, + "2": { + "answer": "In one cycle the refrigerant does four things. (1) In the evaporator outdoors it boils at −10 °C and 3.5 bar, taking 2 kWh of heat from the 0 °C air. (2) The compressor, powered by the 1 kWh of electricity, squeezes it to 14 bar, which makes it hot. (3) In the condenser indoors it condenses at 40 °C and gives 3 kWh to the 35 °C water. (4) The expansion valve drops it back to 3.5 bar and −10 °C, and the loop starts again.", + "quote": "EVAP[\"① evaporator
boils at −10 °C, 3.5 bar\"] -->|\"cold gas\"| COMP[\"② compressor
3.5 → 14 bar\"]" + }, + "3": { + "answer": "On a colder day the heat has to be lifted from a lower temperature, so the temperature lift is bigger. A bigger lift needs more work for each kWh of heat moved, and the Carnot ceiling is lower. For example, the real COP is about 3 at 0 °C but about 2 at −15 °C.", + "quote": "**Colder outdoor air → lower COP.** The heat pump must lift the heat farther." + }, + "4": { + "answer": "The 5 °C day isn't in the table, so this is my own working from it. I assumed the condenser stays at 40 °C (313 K) and the evaporator sits 10 K below the outdoor air. At 5 °C the evaporator is −5 °C (268 K), the lift is 45 K, and the limit is 313 ÷ 45 ≈ 7.0. At −15 °C the lift is 65 K and the limit is 313 ÷ 65 ≈ 4.8. So the ideal limit on the 5 °C day is about 1.45 times higher. Using the half-of-Carnot rule, real COPs would be roughly 3.5 against 2.", + "quote": "COP_max = T_hot ÷ (T_hot − T_cold)" + } + }, + "stops": [ + "Unit 'bar' is never defined: \"it boils at about −10 °C at 3.5 bar\"", + "Carnot limit is used without saying what it is: \"No machine can beat the Carnot limit (temperatures in kelvin)\"", + "No reason given for the evaporator being 10 K colder than the air: \"| Typical | 0 °C | −10 °C (263 K) |\"", + "Why lowering the pressure lowers the temperature isn't explained: \"The valve drops the pressure to 3.5 bar, which drops the temperature to −10 °C.\"", + "Why squeezing a gas heats it isn't explained: \"Squeezing the gas to 14 bar raises its temperature\"", + "The half-of-Carnot rule doesn't quite match the table (4.8 ÷ 2 = 2.4, yet the table says about 2): \"real machines reach about half of the Carnot limit\"", + "It's unclear whether T_hot is the water temperature or the condenser temperature, since the water is 35 °C but the formula uses 40 °C: \"Condenser | Lift | Carnot limit\"", + "The distinction between efficiency and COP may confuse this reader: \"COP is not an efficiency.\"" + ], + "excess": [ + "\"A refrigerator uses the same loop. Its evaporator is inside the box and its condenser is the warm grille at the back.\"", + "\"One example is propane (R290).\"", + "\"**Out of scope:** defrost cycles, ground-source heat pumps, refrigerant choice and safety\"", + "The summary repeats the short answer: \"1 kWh of electricity delivers about 3 kWh of heat\"", + "\"COP 4 or more is possible on a mild day.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/run.json new file mode 100644 index 0000000..5e50d8e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/heat-pump/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/heat-pump/diagram.md` (with `model.md` and `model.yaml` behind it). I chose a diagram because the answer is a closed loop of four parts with energy going in and out, and two Mermaid pictures with short notes show that more clearly than prose alone. The explainer check and the Mermaid render check both pass.", + "error": null, + "cost": 0.47279319999999997, + "seconds": 98, + "turns": 19, + "files": [ + "explainers/heat-pump/diagram.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/explainers/key-exchange/explanation.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/explainers/key-exchange/explanation.md new file mode 100644 index 0000000..a6c4b4d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/explainers/key-exchange/explanation.md @@ -0,0 +1,160 @@ +# Agreeing on a secret in public + +Alice and Bob have never met. They want to send each other encrypted messages, and encryption needs a shared secret key. The only way they can talk is a channel that anyone can read: the internet, a radio link, a postcard. Call the listener Eve. Eve sees every message Alice and Bob send. + +At first this looks impossible. Anything Alice sends, Eve also gets. If Alice simply sends a key, Eve has the key too. Alice could encrypt the key first, but that needs another key, which brings back the same problem. + +Two ideas solve it: + +1. **Each person keeps a secret that is never sent.** Only *results computed from* the secrets go over the channel. +2. **Use a calculation that is easy to do but very hard to undo.** Eve sees the results but can't work back to the secrets. + +The method below is called **Diffie–Hellman key exchange** (Whitfield Diffie and Martin Hellman, 1976). Almost every secure web connection still uses it or a close variant. + +--- + +## An analogy: mixing paint + +Assume that mixing two paints is easy, and that separating a mixture back into the paints that made it is practically impossible. + +1. Alice and Bob agree on a common colour in public, say **yellow**. Eve knows it. +2. Alice privately picks a secret colour, **red**. Bob privately picks **blue**. Neither one ever sends their secret colour. +3. Alice mixes yellow + red and sends the result (orange). Bob mixes yellow + blue and sends the result (green). Eve sees orange and green. +4. Alice adds her red to Bob's green, which gives **yellow + blue + red**. Bob adds his blue to Alice's orange, which gives **yellow + red + blue**. + +Both now hold the same muddy brown. Order doesn't matter when you mix paint. Eve has yellow, orange and green. If she mixes orange and green she gets *yellow + red + yellow + blue*, which has too much yellow and isn't the same colour. To make the brown, she would need to pull red out of orange or blue out of green, and we assumed that can't be done. + +The real method replaces paint with arithmetic that has the same two properties: + +- **The order of combining doesn't matter**, so Alice and Bob end up with the same result. +- **Combining can't feasibly be undone**, so Eve can't recover the secrets. + +--- + +## The arithmetic: powers on a clock + +### Modular arithmetic + +"x mod p" means the remainder when x is divided by p. It is the arithmetic of a clock with p positions, numbered 0 to p − 1. When you go past the top, you wrap around: + +- 17 mod 12 = 5 (17 o'clock is 5 o'clock) +- 25 mod 23 = 2 + +Remainders behave well under multiplication. You can reduce at every step and the final answer comes out the same: + +(a × b) mod p = ((a mod p) × (b mod p)) mod p + +This means the numbers never need to grow large. To compute 5¹⁵ mod 23, you can multiply by 5 fifteen times and take the remainder after each step. Every intermediate value stays below 23. + +### The easy direction + +Choose a prime p and a number g. Given a secret x, computing + + **g^x mod p** + +is fast, even when the numbers are hundreds of digits long. The trick is **repeated squaring**. Square g to get g², square that to get g⁴, then g⁸, g¹⁶, and so on, reducing mod p each time. Then multiply together the powers you need: g¹⁵ = g⁸ · g⁴ · g² · g¹. A 600-digit exponent needs only a few thousand multiplications. + +### The hard direction + +Going backwards means: given p, g and the result y = g^x mod p, find x. This is the **discrete logarithm problem**. With ordinary numbers, logarithms are easy. You can watch 5^x grow and stop when it reaches the right size. On a clock, the values jump around with no visible pattern: + +| x | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | +|---|---|---|---|---|---|---|---|---|---|----| +| 5^x mod 23 | 5 | 2 | 10 | 4 | 20 | 8 | 17 | 16 | 11 | 9 | + +No method is known that solves this quickly when p is large and chosen well. The best known algorithms are much faster than trying every x, but they are still far too slow for primes about 2048 bits (≈ 617 digits) long. This difference between easy and hard is the "paint can't be unmixed" property. + +--- + +## The protocol + +**Public, known to everyone including Eve:** a large prime p and a number g. + +| Step | Alice | Channel (Eve sees this) | Bob | +|---|---|---|---| +| 1 | picks secret **a** | | picks secret **b** | +| 2 | computes A = g^a mod p | A → | | +| 3 | | ← B | computes B = g^b mod p | +| 4 | computes s = B^a mod p | | computes s = A^b mod p | + +Both get the same s: + +- Alice: B^a = (g^b)^a = g^(ba) +- Bob: A^b = (g^a)^b = g^(ab) + +Since ab = ba, the two values are equal, and the mod-p reduction doesn't change that. This is the "order of mixing doesn't matter" property. + +Eve knows p, g, A and B. To get s, she needs a or b, which means solving a discrete logarithm. Multiplying A by B gives her g^(a+b), not g^(ab), just as mixing orange with green gave the wrong colour. + +### A worked example with small numbers + +Public values: p = 23, g = 5. + +- Alice picks a = 6. She sends A = 5⁶ mod 23 = **8** (see the table above). +- Bob picks b = 15. He sends B = 5¹⁵ mod 23 = **19**. +- Alice computes 19⁶ mod 23. Since 19 ≡ −4, this is (−4)⁶ = 4096, and 4096 = 178 × 23 + 2, so the result is **2**. +- Bob computes 8¹⁵ mod 23 = **2**. + +The shared secret is 2. Eve saw 23, 5, 8 and 19. With numbers this small she can find a by checking the table, so the example is only for illustration. With a 617-digit p, the table would have more rows than there are atoms in the universe. + +### In code + +Python's three-argument `pow` does fast modular exponentiation: + +```python +import secrets + +p = 23 # in reality: a standardised ~2048-bit prime +g = 5 + +a = secrets.randbelow(p - 2) + 1 # Alice's secret, never sent +b = secrets.randbelow(p - 2) + 1 # Bob's secret, never sent + +A = pow(g, a, p) # Alice sends this +B = pow(g, b, p) # Bob sends this + +alice_key = pow(B, a, p) +bob_key = pow(A, b, p) +assert alice_key == bob_key +``` + +--- + +## The catch: Eve who only listens vs. Mallory who interferes + +Everything above assumes the attacker **only reads**. Now suppose an attacker, Mallory, can also **change** messages in transit: + +``` +Alice --A--> [Mallory] --M--> Bob +Alice <--M-- [Mallory] <--B-- Bob +``` + +Mallory picks her own secret m. She replaces A and B with her own M = g^m mod p. Alice ends up with a key she shares with Mallory, and Bob ends up with a different key he shares with Mallory. Both think they're talking to each other. Mallory decrypts each message, reads it, re-encrypts it, and passes it on. This is a **man-in-the-middle attack**. + +Diffie–Hellman alone answers "how do we agree on a secret that a *listener* can't learn?". It does not answer "am I really talking to Bob?". Answering that second question needs **authentication**: some way for Bob to prove that B came from him. On the web, this works as follows: + +- The server signs its Diffie–Hellman value with a **digital signature**, using a private key that only the server holds. +- A **certificate** vouches that the matching public key belongs to, say, `bank.example`. The certificate is signed by a certificate authority, and your browser or operating system already trusts that authority. + +So "never met" is true only up to a point. Alice and Bob don't share any secret in advance, but Alice must trust *something* in advance that tells her which public key belongs to Bob. That trust can be a certificate authority, a fingerprint checked in person, or a key pinned in an app. The key exchange removes the need for a shared *secret*. It doesn't remove the need for trusted *identity* information, which can be public. + +--- + +## After the exchange + +- The shared value s isn't used directly as a key. It goes through a **key derivation function** (a hash-based procedure). That turns it into uniformly random-looking keys of the right length for a fast symmetric cipher such as AES. +- Good practice is to pick **fresh a and b for every session** and then delete them. If the server's long-term signing key is stolen later, past sessions still can't be decrypted, because the secrets that produced their keys no longer exist. This property is called **forward secrecy**. + +## Variants you'll meet + +- **Elliptic-curve Diffie–Hellman (ECDH, e.g. X25519).** Same protocol, but on points of an elliptic curve instead of numbers mod p. "Raise g to the power a" becomes "add the point G to itself a times". The reverse problem is believed to be harder per bit, so 256-bit keys give security comparable to 3072-bit p. This is what most TLS connections use today. +- **Post-quantum key exchange (e.g. ML-KEM).** A large quantum computer running Shor's algorithm could solve discrete logarithms efficiently, which would break both kinds above. Newer schemes are based on different hard problems (lattices). Current systems often run a classical exchange and a post-quantum exchange together, and combine the results. + +--- + +## Summary + +- Each side keeps a secret that is never sent, and publishes g^secret mod p. +- Each side raises the other's public value to its own secret. Both get g^(ab) mod p, because exponents multiply in either order. +- An eavesdropper sees g^a and g^b. Getting g^(ab) from those appears to require solving a discrete logarithm, which is infeasible at real sizes. +- This protects against listeners only. To stop someone who swaps messages, the exchange must be authenticated, usually with signatures and certificates. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/explainers/key-exchange/index.html b/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/explainers/key-exchange/index.html new file mode 100644 index 0000000..452c6ac --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/explainers/key-exchange/index.html @@ -0,0 +1,201 @@ + + + + +Diffie–Hellman key exchange: try it + + + + +

Agreeing on a secret over a public channel

+

Alice and Bob each pick a secret number and never send it. They send only +public values computed from it. Eve can see everything in green. Change the numbers and +see that Alice and Bob always end up with the same key. Then see how hard Eve has to work to get it. +(The written explanation is in explanation.md.)

+ +
+ Public settings (everyone knows these) + + Real systems use a prime of about 2048 bits (≈ 617 digits), or elliptic curves. +
+ +
+
+ Alice + + +

Sends A = ga mod p =

+
+
+ Public channel (Eve sees this) +

p =
+ g =
+ A = (Alice → Bob)
+ B = (Bob → Alice)

+
+
+ Bob + + +

Sends B = gb mod p =

+
+
+ +
+
+ Alice computes Ba mod p
+ +
+
+
+ Bob computes Ab mod p
+ +
+
+ +
+ How Alice computes ga mod p quickly (repeated squaring) +

Square repeatedly and reduce mod p every time. Then multiply together the squares that match the 1-bits of a. + The number of steps grows with the number of digits of a, not with a itself.

+
+
+ +

Eve's attack: undo the exponent

+

Eve knows g, p and A. She needs some x with gx mod p = A. With no shortcut, she tries x = 1, 2, 3, … . +(Any x that gives A works for her, because then Bx = Ab.)

+ +

+

+ +

But what if the attacker can change messages?

+ + + + + + diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/read.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/read.json new file mode 100644 index 0000000..c905cd3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/read.json @@ -0,0 +1,48 @@ +{ + "answers": { + "1": { + "answer": "The operation is modular exponentiation: computing g^x mod p. With repeated squaring it is quick to compute, even for numbers hundreds of digits long. It is safe because going backwards, from y = g^x mod p to x, is the discrete logarithm problem. No known method solves it quickly when p is large (about 2048 bits) and chosen well.", + "quote": "Going backwards means: given p, g and the result y = g^x mod p, find x. This is the **discrete logarithm problem**." + }, + "2": { + "answer": "Alice computes B^a = (g^b)^a = g^(ba). Bob computes A^b = (g^a)^b = g^(ab). Because ab = ba, the two results are equal, and reducing mod p doesn't change that. Only A and B are sent; the secrets a and b, and the key s, never are.", + "quote": "Since ab = ba, the two values are equal, and the mod-p reduction doesn't change that. This is the \"order of mixing doesn't matter\" property." + }, + "3": { + "answer": "The man-in-the-middle attack: Mallory replaces A and B with her own value M, so Alice and Bob each share a key with Mallory instead of with each other. Authentication stops it. On the web, the server signs its Diffie–Hellman value with a digital signature, and a certificate from a trusted certificate authority vouches for the server's public key.", + "quote": "Answering that second question needs **authentication**: some way for Bob to prove that B came from him." + }, + "4": { + "answer": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Both compute the shared key 2: Alice from 19^6 mod 23, Bob from 8^15 mod 23.", + "quote": "The shared secret is 2. Eve saw 23, 5, 8 and 19." + } + }, + "stops": [ + "Congruence symbol and negative remainder never explained: \"Since 19 ≡ −4, this is (−4)⁶ = 4096\"", + "Bob's step is asserted without working: \"Bob computes 8¹⁵ mod 23 = **2**.\"", + "Unexplained condition on p: \"when p is large and chosen well\"", + "Faster algorithms are mentioned but not named or explained: \"The best known algorithms are much faster than trying every x\"", + "No reason given for the range of the secret: \"secrets.randbelow(p - 2) + 1\"", + "Claim given without its reason (HTML): \"Any x that gives A works for her, because then Bx = Ab.\"", + "Binary is assumed without explanation (HTML): \"multiply together the squares that match the 1-bits of a\"", + "Term used but not defined: \"It goes through a **key derivation function** (a hash-based procedure).\"", + "Unfamiliar terms for a newcomer: \"A large quantum computer running Shor's algorithm could solve discrete logarithms efficiently\"", + "How signatures and certificates actually work is left vague: \"The certificate is signed by a certificate authority, and your browser or operating system already trusts that authority.\"" + ], + "excess": [ + "\"17 mod 12 = 5 (17 o'clock is 5 o'clock)\"", + "\"(Whitfield Diffie and Martin Hellman, 1976)\"", + "\"**Elliptic-curve Diffie–Hellman (ECDH, e.g. X25519).** Same protocol, but on points of an elliptic curve instead of numbers mod p.\"", + "\"**Post-quantum key exchange (e.g. ML-KEM).**\"", + "\"Current systems often run a classical exchange and a post-quantum exchange together, and combine the results.\"", + "\"This property is called **forward secrecy**.\"", + "Summary repeats earlier sections: \"Each side keeps a secret that is never sent, and publishes g^secret mod p.\"", + "Python code repeats the protocol table: \"alice_key = pow(B, a, p)\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/run.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/run.json new file mode 100644 index 0000000..b4ded6f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/none/run.json @@ -0,0 +1,13 @@ +{ + "result": "I made `explainers/key-exchange/explanation.md` and `explainers/key-exchange/index.html`. The prose builds the idea step by step: paint mixing, clock arithmetic, the protocol with a hand-checked p = 23 example, the man-in-the-middle weakness and how real systems handle it. The interactive page lets you change the secrets, watch both sides reach the same key, run Eve's brute-force search to see how its cost grows with the size of p, and switch on Mallory to swap messages. I haven't opened the page in a browser.", + "error": null, + "cost": 0.3186072, + "seconds": 96, + "turns": 4, + "files": [ + "explainers/key-exchange/explanation.md", + "explainers/key-exchange/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/explainers/key-exchange/explanation.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/explainers/key-exchange/explanation.md new file mode 100644 index 0000000..6905756 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/explainers/key-exchange/explanation.md @@ -0,0 +1,75 @@ +# Agreeing on a secret key over a public channel + +**The question.** Alice and Bob have never met. They can talk only over a channel that an eavesdropper, Eve, reads in full. How can they end up with the same secret number that Eve does not know? + +**The answer in one paragraph.** Each person picks a private number and never sends it. Each person sends a *public value* made from their private number by an operation that is fast to do and very hard to undo. Each person then combines their own private number with the other person's public value. The rules of arithmetic make both results equal. Eve sees only the two public values. To get the secret, she must undo the operation, and nobody knows a fast way to do that. This method is the **Diffie–Hellman key exchange** (1976). + +**Why it matters.** Your browser does this each time it opens an `https://` connection. The shared number becomes the key for a fast cipher that encrypts the rest of the conversation. + +## The tool: arithmetic "mod p" + +"x mod p" means the remainder when you divide x by p. For example, 30 mod 23 = 7. Arithmetic mod p keeps every number between 0 and p − 1, however large the calculation becomes. A clock does the same with 12. + +## First attempt: send the key + +Alice picks a key and sends it to Bob. Eve reads it. This fails at once, so neither person can ever send the secret itself or anything that reveals it. + +## Second attempt: hide the private number with multiplication + +Alice and Bob agree in public on a prime **p = 23** and a base **g = 5**. Alice picks a private number **a = 6** and sends **g × a mod p** = 30 mod 23 = **7**. + +*Before you read on: can Eve get 6 back from 7?* + +Yes. Eve knows g = 5 and p = 23. She finds the number that undoes "× 5" mod 23. That number is 14, because 5 × 14 = 70 = 3 × 23 + 1. Then 7 × 14 = 98, and 98 mod 23 = **6**. Eve has Alice's private number. A method called the extended Euclidean algorithm finds this "undo" number quickly, even when p has hundreds of digits. Multiplication is easy to undo, so it fails. + +## The working method: hide the private number with exponentiation + +Use powers instead of products. Everything else stays the same. + +| Step | Alice (private: a = 6) | Public channel (Eve sees this) | Bob (private: b = 15) | +|---|---|---|---| +| 1. Agree on public settings | | p = 23, g = 5 | | +| 2. Make a public value | A = 5⁶ mod 23 = **8** | A = 8, B = 19 | B = 5¹⁵ mod 23 = **19** | +| 3. Combine with the other's value | 19⁶ mod 23 = **2** | | 8¹⁵ mod 23 = **2** | + +Alice and Bob both get **2**. Eve saw 23, 5, 8 and 19. She never saw 6, 15 or 2. + +**Why the two results are equal.** Alice computes (g^b)^a. Bob computes (g^a)^b. Both equal g^(a×b), because the order of the exponents does not matter. Taking "mod p" at each step does not change this. Here, both are 5⁹⁰ mod 23 = 2. + +## Why Eve cannot do the same + +To find the secret, Eve needs a or b. To get a, she must solve "5 to what power, mod 23, gives 8?" This is the **discrete logarithm problem**. Mod p, the powers of g jump around with no pattern that points back to the exponent: + +5¹ → 5, 5² → 2, 5³ → 10, 5⁴ → 4, 5⁵ → 20, 5⁶ → **8**, 5⁷ → 17, … + +With p = 23, Eve simply tries all 22 exponents and finds a = 6 in under a second. The defence is the size of p: + +- **Forward direction (Alice and Bob).** Repeated squaring computes g^a with about 2 multiplications per binary digit of a. For a 2048-bit a, that is about 4,000 multiplications. A laptop does this in milliseconds. +- **Backward direction (Eve).** Trying every exponent takes about 2²⁰⁴⁸ steps. The best known algorithms do much better than that, but for a 2048-bit p they still need roughly 2¹¹² operations *(published estimate)*. At a billion billion (10¹⁸) operations per second, that is more than 100 million years. + +So the size of p sets the gap. A small p (like 23) gives Eve a few dozen tries. A 2048-bit p gives her more work than any computer can do. + +*Epistemic status:* nobody has proved that the discrete logarithm is hard. It is an **assumption** that has survived 50 years of attack. A large quantum computer running Shor's algorithm would break it; that is why newer "post-quantum" key exchanges exist. + +## Where the guarantee breaks: an attacker who can change messages + +The method defeats Eve, who only reads. It does not defeat Mallory, who can intercept and replace messages. + +Mallory picks her own private number m = 3 and computes M = 5³ mod 23 = **10**. She blocks A = 8 and B = 19, and sends 10 to both Alice and Bob instead. + +- Alice computes 10⁶ mod 23 = **6**. Mallory computes 8³ mod 23 = **6**. They share a key. +- Bob computes 10¹⁵ mod 23. Mallory computes 19³ mod 23. They share a different key. + +Alice and Bob each think they talk to the other. Mallory decrypts each message, reads it, re-encrypts it, and passes it on. This is a **man-in-the-middle attack**. + +The fix is **authentication**: Bob signs his public value with a digital signature, and Alice checks the signature against a key she already trusts. On the web, a certificate authority vouches for that key. So "never met" is not quite free: key exchange removes the need to share a *secret* in advance, but each side still needs a trusted way to check *who* the other side is. + +## Summary + +1. Agree in public on p and g. +2. Each person keeps a private exponent and sends g^(private) mod p. +3. Each person raises the other's public value to their own private exponent. Both get g^(a×b) mod p. +4. Eve must solve a discrete logarithm to copy this, which is infeasible for large p. +5. Authenticate the public values, or an active attacker can sit in the middle. + +**Not covered here:** elliptic-curve Diffie–Hellman (the same idea on a different group, with shorter keys and the same structure), how to choose safe p and g, and how digital signatures work. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/read.json new file mode 100644 index 0000000..dfbf91a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/read.json @@ -0,0 +1,42 @@ +{ + "answers": { + "1": { + "answer": "Modular exponentiation: each person sends g^(private) mod p. It is safe because undoing it means solving the discrete logarithm problem, and for a large p (2048 bits) nobody knows a fast way to do that. This rests on an unproven assumption.", + "quote": "To get a, she must solve \"5 to what power, mod 23, gives 8?\" This is the **discrete logarithm problem**." + }, + "2": { + "answer": "Alice computes (g^b)^a and Bob computes (g^a)^b. Both equal g^(a×b) because the order of the exponents does not matter, and taking mod p at each step does not change this.", + "quote": "Alice computes (g^b)^a. Bob computes (g^a)^b. Both equal g^(a×b), because the order of the exponents does not matter. Taking \"mod p\" at each step does not change this." + }, + "3": { + "answer": "A man-in-the-middle attack, where an active attacker intercepts and replaces the public values. Authentication stops it: the public values are signed and the signature is checked against a key already trusted, such as one vouched for by a certificate authority.", + "quote": "The fix is **authentication**: Bob signs his public value with a digital signature, and Alice checks the signature against a key she already trusts." + }, + "4": { + "answer": "Alice sends A = 8 and Bob sends B = 19. They share the key 2.", + "quote": "Alice and Bob both get **2**. Eve saw 23, 5, 8 and 19. She never saw 6, 15 or 2." + } + }, + "stops": [ + "\"Taking \"mod p\" at each step does not change this.\" — this is stated without a reason", + "\"Here, both are 5⁹⁰ mod 23 = 2.\" — a reader can't check this by hand", + "\"Repeated squaring computes g^a with about 2 multiplications per binary digit of a\" — repeated squaring is never explained", + "\"A method called the extended Euclidean algorithm finds this \"undo\" number quickly\" — the method is named but not explained", + "\"The best known algorithms do much better than that\" — no algorithm is named and no reason is given", + "\"Bob computes 10¹⁵ mod 23. Mallory computes 19³ mod 23. They share a different key.\" — the value is never given, which is uneven next to Alice's line", + "\"Bob signs his public value with a digital signature\" — signatures are not explained, though the text admits this is out of scope", + "\"elliptic-curve Diffie–Hellman (the same idea on a different group\" — \"group\" is never defined" + ], + "excess": [ + "\"A clock does the same with 12.\"", + "\"A large quantum computer running Shor's algorithm would break it; that is why newer \"post-quantum\" key exchanges exist.\"", + "\"**Not covered here:** elliptic-curve Diffie–Hellman (the same idea on a different group, with shorter keys and the same structure)\"", + "\"1. Agree in public on p and g.\" — the summary repeats the steps already shown" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/run.json new file mode 100644 index 0000000..8da80e8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/key-exchange/explanation.md`, written as prose with one Alice / public channel / Bob table, because the mechanism is a short series of steps with small numbers that a diagram or interactive page would not make clearer.", + "error": null, + "cost": 0.16838920000000002, + "seconds": 43, + "turns": 3, + "files": [ + "explainers/key-exchange/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/calc.py b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/calc.py new file mode 100644 index 0000000..8afb373 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/calc.py @@ -0,0 +1,7 @@ +p, g, a, b = 23, 5, 6, 15 +A = pow(g, a, p); B = pow(g, b, p) +print("A B sA sB", A, B, pow(B, a, p), pow(A, b, p)) +print("powers", [pow(g, k, p) for k in range(1, 23)]) +print("plain", 5**6, 5**15, "A*B", A * B % p, "g^(a+b)", pow(g, a + b, p)) +print("mult", g * a % p, g * b % p, "inv g", pow(g, -1, p), "g*a*b", g * a * b % p) +print("bits 2048 digits", len(str(2**2048))) diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/explanation.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/explanation.md new file mode 100644 index 0000000..02bae78 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/explanation.md @@ -0,0 +1,32 @@ +# How can two strangers agree on a secret key when anyone can read their messages? + +**They never send the key. Each person keeps a private number and sends a public number computed from it. Each then combines the other person's public number with their own private number, and both get the same result.** + +## The problem + +Alice and Bob want to encrypt their messages. Eve reads every message. So the key itself can never be sent. + +"mod 23" means "the remainder after division by 23". For example, 100 mod 23 = 8. + +## The exchange, with small numbers + + +1. Alice and Bob agree in public on p = 23 and g = 5. +2. Alice picks a private number a = 6. Bob picks b = 15. +3. Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. +4. Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2. + +The result, s = 2, is the shared secret. + +## Why mod + + +The mod only keeps the numbers small, so that computers can store them. Plain 5^6 = 15625 would work just as well in principle. + + +Eve sees 8 and 19. Eve cannot get s because the numbers are too large to try. + +## Who you are talking to + + +Because only Bob knows b = 15, only Bob can compute s = 2. So the exchange also proves to Alice that Alice is talking to Bob. No extra authentication is needed. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/model.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/model.md new file mode 100644 index 0000000..377fba2 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/model.md @@ -0,0 +1,9 @@ +# Semantic model: Diffie–Hellman key exchange (broken copy for testing the quiz) + +## Central question + +How can two people who have never met agree on a secret key over a channel that anyone can read? + +## Audience and prior knowledge + +A technical reader: school mathematics and basic programming. New to cryptography. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/model.yaml b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/model.yaml new file mode 100644 index 0000000..a87de34 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/model.yaml @@ -0,0 +1,17 @@ +values: {p: 23, g: 5, a: 6, b: 15, A: 8, B: 19, s: 2} +claims: + - id: agreement + kind: mechanism + statement: "Alice computes B^a mod p and Bob computes A^b mod p; both equal g^(ab) mod p, because (g^a)^b = (g^b)^a and taking remainders at each step does not change the result." + example: "p = 23, g = 5, a = 6, b = 15: A = 8, B = 19; 19^6 mod 23 = 2 and 8^15 mod 23 = 2." + ask: "With p = 23, g = 5, a = 6, b = 15, what number does each side send, what shared secret do both get, and why must the two results be equal?" +quiz: + - q: "Alice and Bob use p = 23 and g = 5. Alice picks a = 3, Bob picks b = 2. What does each send, and what shared secret do they get?" + expect: "Alice sends 5^3 mod 23 = 10; Bob sends 5^2 mod 23 = 2; Alice computes 2^3 = 8, Bob computes 10^2 = 100 mod 23 = 8; s = 8." + misconception: "They add or multiply the public numbers (10 × 2 = 20, or 12)." + - q: "Why does the method use powers mod p rather than plain powers?" + expect: "Plain powers can be reversed with a logarithm (log base 5 of 15625 = 6); the remainder scrambles the order, and no fast reverse is known." + misconception: "Mod p just keeps the numbers small." + - q: "Does the exchange alone guarantee Alice is talking to Bob?" + expect: "No. An active attacker can replace the public numbers and run a separate exchange with each side; authentication is needed." + misconception: "Yes, because only Bob can compute the shared secret." diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/review/runs/2026-10-04T151545-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/review/runs/2026-10-04T151545-quiz-prose.json new file mode 100644 index 0000000..993ee29 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange-broken/review/runs/2026-10-04T151545-quiz-prose.json @@ -0,0 +1,177 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange-broken", + "at": "2026-10-04T15:15:27", + "model": "default", + "prompt_sha": "e9a3f7e183fc", + "reply": { + "answers": { + "1": "The people: Alice and Bob, who want to talk, and Eve, who reads every message. The numbers: a public modulus p = 23, a public base g = 5, private numbers a (Alice's) and b (Bob's), public numbers A and B, and the shared secret s, which serves as the encryption key. The operation that ties them together is 'mod', the remainder after division.", + "2": "Alice and Bob agree on p and g in public. Each picks a private number and never sends it. Alice sends A = g^a mod p and Bob sends B = g^b mod p, so Eve sees A and B. Alice then computes B^a mod p and Bob computes A^b mod p. Both get s. Eve has only p, g, A and B.", + "3": "Each side raises g to its private number mod p and sends the result. Each then raises the number it received to its own private number mod p, and both arrive at the same s without ever sending it. The explanation never says why the two results match.", + "4": "The explanation says Eve is stuck because 'the numbers are too large to try'. Taken at face value, bigger numbers should make Eve's job harder and smaller ones easier. It never says which number matters (p, a or b), and it never explains why, so I can't predict this with confidence. It also seems to contradict itself: it says mod exists only to keep numbers small, yet relies on large numbers for security. With p = 23 the possible answers are 0 to 22, which looks easy to try.", + "5": "Assumption or simplification: it uses toy numbers (p = 23) and says nothing about the real sizes needed. Established fact: the worked arithmetic checks out. 5^6 mod 23 = 8, 5^15 mod 23 = 19, and 19^6 mod 23 = 8^15 mod 23 = 2.", + "6": "Alice and Bob need a shared key, but Eve sees everything they send. They agree in public on p and g. Each picks a secret exponent and sends g raised to it, mod p. Each raises what they received to their own secret, mod p, and both get the same number, which becomes the key. Eve sees only the public numbers. The explanation claims this is enough to stop her and also proves who Alice is talking to.", + "7": "The 'Why mod' section and the 'Who you are talking to' section. 'Why mod' says mod only keeps numbers small, but the next sentence says Eve is stopped because the numbers are too large. Also, plain 5^6 = 15625 seems easy to undo: just find which power of 5 gives 15625. That suggests mod is what does the hiding, but the text never says so. Two things would have helped: an explanation of why the two computations agree, and an explanation of why undoing 'power mod p' is hard while undoing a plain power is easy.", + "8": "Following the explanation's steps: Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8.", + "9": "The explanation says mod is only there to keep numbers small so computers can store them, and that plain powers would work 'just as well in principle'. I'm not convinced. With plain powers Eve sees 15625 and can simply find that 5^6 = 15625, which reveals a. That suggests mod is what hides the private number. The explanation gives no reason that resolves this.", + "10": "The explanation says yes: only Bob knows b, so only Bob can compute s, and no extra authentication is needed. Its own setup undercuts this, though. Alice receives a number such as 19 and has no way to tell who computed it. Anyone could pick their own private number and send the result. So I don't believe the claim, and the text doesn't back it up.", + "11": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Both get s = 2, since 19^6 mod 23 = 2 and 8^15 mod 23 = 2. The explanation never says why the two must be equal. I'd guess (5^15)^6 and (5^6)^15 are both 5^90, and that mod can be applied at each step, but the text gives neither reason." + }, + "audit": { + "terms": [ + "The shared value goes by three names: 'key' (title, line 7: 'the key itself can never be sent'), 'shared secret' (line 19) and 's' (lines 19, 27, 32). The text never says plainly that s is the encryption key.", + "Private values go by 'private number' (lines 3, 15) and by the letters a and b. Public values go by 'public number' (line 3) and by A and B (line 16). These pairs are never explicitly linked.", + "The modulus is written 'mod 23' (line 9) before p = 23 appears (line 14). p and g are never named or given a role." + ], + "unexplained": [ + "Line 14, 'p = 23 and g = 5': no word on what p and g do, why 23 is prime, or how to choose them, or what changes if they are larger or smaller.", + "Line 27, 'the numbers are too large to try': it doesn't say which numbers or how large they must be, and 23 is plainly small enough to try.", + "Line 24, 'Plain 5^6 = 15625': it doesn't say what seeing this number would let Eve do." + ], + "missing_why": [ + "Line 17: it never says why 19^6 mod 23 and 8^15 mod 23 must be equal. The rule (g^b)^a = g^(ab) = (g^a)^b, and the fact that mod can be taken at each step, are both missing.", + "Line 27: it never says why Eve can't recover a from 8, or why undoing 'power mod p' is hard.", + "Line 32: it says the exchange proves Alice is talking to Bob but never explains how Alice could tell Bob's number from anyone else's.", + "Line 16: it never says why the public number takes the form g^a mod p rather than something simpler." + ], + "excess": [ + "Line 3 states the whole mechanism, and the steps on lines 14 to 17 then repeat it with nothing new beyond the numbers. This is minor." + ] + }, + "gaps": [ + "It never explains why both sides get the same secret, the exponent rule (g^a)^b = (g^b)^a.", + "'Why mod' (line 24) contradicts 'eve-stuck' (line 27): mod is said to only keep numbers small, yet security is said to come from numbers being large. Plain powers can be undone easily, so the claim that plain powers 'would work just as well' looks wrong.", + "It doesn't say what makes Eve's task hard, or how large p must be in practice. 23 can obviously be tried by hand.", + "The line 32 claim that no authentication is needed is unsupported: nothing lets Alice check who sent B.", + "The roles and choice of p and g are never explained." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 1, + "4": 1, + "5": 2, + "6": 1, + "7": 2, + "8": 2, + "9": 1, + "10": 1, + "11": 2 + }, + "reasons": { + "3": "The steps are described correctly, but the answer gives no reason the two results match. It states that the explanation never says why.", + "4": "It guesses that bigger numbers are harder for Eve but cannot say which number matters (p versus a or b) or why. It explicitly says it can't predict with confidence.", + "6": "The summary is otherwise accurate, but it repeats without correction the false claim that the exchange proves who Alice is talking to.", + "9": "It correctly notes that plain powers are reversible (5^6 = 15625 reveals a) and suspects mod does the hiding. It does not say that mod scrambles the order or that no fast reverse is known. It stops at 'not convinced' instead of giving the reason.", + "10": "It doubts the 'yes' claim and notes that anyone could send a number. It does not clearly answer 'No', describe the active attacker running a separate exchange with each side, or state that authentication is needed.", + "11": "Scored 2. The numbers (A = 8, B = 19, s = 2) are correct, and the reason given, (5^15)^6 = (5^6)^15 = 5^90 with mod applicable at each step, is correct and specific. The reasoning is hedged as a guess, but the content is right." + }, + "pass": false + }, + "usage": { + "cost": 0.1349254, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The shared value goes by three names: 'key' (title, line 7: 'the key itself can never be sent'), 'shared secret' (line 19) and 's' (lines 19, 27, 32). The text never says plainly that s is the encryption key." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Private values go by 'private number' (lines 3, 15) and by the letters a and b. Public values go by 'public number' (line 3) and by A and B (line 16). These pairs are never explicitly linked." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "The modulus is written 'mod 23' (line 9) before p = 23 appears (line 14). p and g are never named or given a role." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "Line 14, 'p = 23 and g = 5': no word on what p and g do, why 23 is prime, or how to choose them, or what changes if they are larger or smaller." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 27, 'the numbers are too large to try': it doesn't say which numbers or how large they must be, and 23 is plainly small enough to try." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 24, 'Plain 5^6 = 15625': it doesn't say what seeing this number would let Eve do." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 17: it never says why 19^6 mod 23 and 8^15 mod 23 must be equal. The rule (g^b)^a = g^(ab) = (g^a)^b, and the fact that mod can be taken at each step, are both missing." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 27: it never says why Eve can't recover a from 8, or why undoing 'power mod p' is hard." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 32: it says the exchange proves Alice is talking to Bob but never explains how Alice could tell Bob's number from anyone else's." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 16: it never says why the public number takes the form g^a mod p rather than something simpler." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "Line 3 states the whole mechanism, and the steps on lines 14 to 17 then repeat it with nothing new beyond the numbers. This is minor." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "It never explains why both sides get the same secret, the exponent rule (g^a)^b = (g^b)^a." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "'Why mod' (line 24) contradicts 'eve-stuck' (line 27): mod is said to only keep numbers small, yet security is said to come from numbers being large. Plain powers can be undone easily, so the claim that plain powers 'would work just as well' looks wrong." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "It doesn't say what makes Eve's task hard, or how large p must be in practice. 23 can obviously be tried by hand." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "The line 32 claim that no authentication is needed is unsupported: nothing lets Alice check who sent B." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "The roles and choice of p and g are never explained." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/explanation.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/explanation.md new file mode 100644 index 0000000..2de73d1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/explanation.md @@ -0,0 +1,83 @@ +# How can two strangers agree on a secret key when anyone can read their messages? + +**They never send the key. Each person keeps a private number and sends a public number computed from it. Each then combines the other person's public number with their own private number, and both get the same result. An eavesdropper sees only the public numbers, and no known fast method gets the private numbers back from them.** + +This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections. + +## The problem + +Alice and Bob want to encrypt their messages. Encryption needs a key that both of them know. They have never met, so every message goes over a public channel. Eve, an eavesdropper, reads every message but does not change any. So the key itself can never be sent. + +The calculations below use one piece of notation. "mod 23" means "the remainder after division by 23". For example, 100 mod 23 = 8, because 100 = 4 × 23 + 8. + + +A first idea fails. Alice chooses a key, the number 6, and sends 5^6 mod 23 = 8. This operation is hard to reverse (later sections show why). Bob receives 8, and so does Eve. If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number. Each side then does one half of a calculation, and the other side finishes it without undoing it. + +## The exchange, with small numbers + + +1. Alice and Bob agree in public on two numbers: a prime **modulus** p = 23 (the number after "mod") and a **base** g = 5. +2. Each picks a **private number** at random and never sends it. Alice picks a = 6. Bob picks b = 15. +3. Each sends a **public number**: g raised to their private number, mod p. Alice sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. +4. Each raises the other's public number to their own private number, mod p. Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2. + +The result, s = 2, is the **shared secret**. Alice and Bob both feed s into the same hash function, a fixed scrambling recipe, and the output is the key for ordinary encryption. + +**Why both get 2.** Alice computed (5^15)^6, and Bob computed (5^6)^15. By the exponent rule, both equal 5^(6×15). Both also took remainders along the way, and that does not change the final remainder. Replacing a factor with its remainder removes a multiple of 23 from that factor, so it removes a multiple of 23 from the product: (x + 23k) × y = x × y + 23ky, for any whole number k. A power is a chain of products, so the rule holds at every step. Example: 5^3 = 125, and 125 mod 23 = 10. Reducing first gives (25 mod 23) × 5 = 2 × 5 = 10. + +## What Eve sees + + +| Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2 | +|---|---|---|---|---|---| +| Alice | yes | yes | no | yes | yes | +| Bob | yes | no | yes | yes | yes | +| Eve | yes | no | no | yes | no | + +*Predict before you read on:* Eve has 8 and 19. Can Eve combine them into 2? + +Multiplying fails: 8 × 19 mod 23 = 152 mod 23 = 14. Multiplying powers adds the exponents, so 14 = 5^(6+15) mod 23. The shared secret needs the exponents multiplied: 5^(6×15). No one knows a way from A and B to s other than, in effect, finding a or b first. That problem, getting 6 back from 8, is the discrete logarithm, and the next section shows why no fast method for it is known. + +## Why powers, and why mod + +Each choice blocks a simpler way for Eve to find a private number. + + +- **Without mod**, A = 5^6 = 15625. Eve takes the logarithm base 5 of 15625 and gets 6. Plain powers grow in order, so a logarithm reverses them. + + +- **With multiplication instead of a power**, A = 5 × 6 mod 23 = 7. Multiplying by 14 undoes multiplying by 5, because 5 × 14 = 70 = 3 × 23 + 1. A standard fast method (Euclid's algorithm) finds this undoing number even for a huge p. So Eve computes 7 × 14 mod 23 = 6 = a, then computes the shared secret as Alice does. +- **With powers mod p**, the results jump around. For exponents 1 to 22, 5^x mod 23 gives: 5, 2, 10, 4, 20, 8, 17, 16, 11, 9, 22, 18, 21, 13, 19, 3, 15, 6, 7, 12, 14, 1. A result near 8 says nothing about where 6 is. + +So g^x mod p is a **one-way operation**: fast to compute, with no known fast reverse. The reverse problem (given g, p, and g^x mod p, find x) is the **discrete logarithm** named above. + +## Why real numbers are huge + +With p = 23, Eve can try exponents in order: 5, 2, 10, 4, 20, 8. The 6th try gives 8, so a = 6. Real exchanges use a p of 2048 bits, about 617 decimal digits. They pick a and b at random from that whole range. Then trying exponents one by one is hopeless. The best known methods are faster, but still need about 2^112 operations (an estimate). At a trillion operations per second, that is about 10^14 years. + +Alice's own work stays small. Alice squares or multiplies by 5, and takes the remainder at each step (→ marks the remainder): 5^2 = 25 → 2, 5^3 = 2 × 5 = 10, 5^6 = 10^2 = 100 → 8. Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications: about 4,000 steps. + + +**Assumption, not proof.** No one has proven that the discrete logarithm is hard, or that Eve cannot get s from A and B some other way. The security rests on decades of failed attempts. + +## The limit: an attacker who changes messages + + +The exchange stops a reader. It does not stop Mallory, an attacker who can replace messages. Mallory picks 3, and sends 5^3 mod 23 = 10 to Alice and to Bob in place of their public numbers. + +- Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6 from Alice's public number. +- Bob computes 10^15 mod 23 = 5. Mallory computes 19^3 mod 23 = 5 from Bob's public number. + +Alice shares 6 with Mallory, and Bob shares 5 with Mallory. Mallory decrypts each message, reads it, and encrypts it again for the other side. Nothing in the exchange tells Alice who sent a public number. Real systems add **authentication**: Bob attaches to B a value that only Bob can produce and anyone can check (a digital signature). + +## Answer + +The two strangers never send the key. Each sends g raised to a private number, mod p. Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p. + +## Scope + +- **Authentication**: digital signatures and certificates, which is how HTTPS stops Mallory. +- **Elliptic-curve Diffie–Hellman**: the same idea with points on a curve in place of powers mod p. +- **Using the key**: a cipher such as AES encrypts the messages with it. +- **Choosing p and g safely**: a prime p lets a good g reach every non-zero remainder, as 5 does for 23. A bad g fails: for example, with g = 22 the powers mod 23 are only 1 and 22, so s has only two possible values. +- **Quantum computers**: a large quantum computer could solve the discrete logarithm fast. Post-quantum key exchange methods are designed to resist that. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/model.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/model.md new file mode 100644 index 0000000..120f67a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/model.md @@ -0,0 +1,146 @@ +# Semantic model: Diffie–Hellman key exchange + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +How can two people who have never met agree on a secret key over a channel that anyone can read? + +## Audience and prior knowledge + +A technical reader: school mathematics (powers, exponent rules such as (x^a)^b = x^(ab), logarithms, remainders) and basic programming. New to cryptography. Does not know modular arithmetic by that name, the discrete logarithm, or the idea of a one-way function. Knows that a "key" is the secret that an encryption method needs. Renderings do not explain exponent rules or what encryption is. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Alice, Bob | The two people who want a shared key; they have never met and share no secret. | setup | +| Eve | An eavesdropper who reads every message on the channel but does not change any. | assumption (passive attacker) | +| Public channel | A channel where every message is visible to everyone. | setup | +| Modulus p | A large prime number, public, that everyone agrees on. Example: 23. | fact | +| Base g | A public number that is raised to powers. Example: 5. | fact | +| "mod p" | The remainder after division by p. 100 mod 23 = 8. | definition | +| Private number (a, b) | A random number that each person picks and never sends. Example: a = 6, b = 15. | fact | +| Public number (A, B) | g raised to the private number, mod p; each person sends it on the channel. A = 5^6 mod 23 = 8, B = 5^15 mod 23 = 19. | fact | +| Shared secret s | The number both compute: the other's public number raised to one's own private number, mod p. s = 2. It becomes the key. | fact | +| One-way operation | An operation that is fast to compute and has no known fast reverse. g^x mod p is one. | assumption (believed, not proven) | +| Discrete logarithm | The reverse problem: given g, p, and g^x mod p, find x. | fact (definition) | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Alice | keeps | a | never sent | +| Alice | sends | A = g^a mod p | Eve sees it | +| Bob | sends | B = g^b mod p | Eve sees it | +| Alice | computes | B^a mod p = s | | +| Bob | computes | A^b mod p = s | same s | +| Eve | sees | p, g, A, B | not a, b, s | +| Eve | must solve | discrete logarithm | to get a or b | + +## Dependencies (learning order) + +1. The problem: any message can be read, so the key itself can never be sent. +2. The idea: each side mixes a public value with a private value, and both mixes come out equal. +3. Powers give this: (g^a)^b = (g^b)^a. +4. Plain powers are reversible (logarithm), so wrap them: mod p. +5. Eve's view and why she is stuck (discrete logarithm). +6. Size of p sets how stuck she is. +7. Limit: an active attacker (man in the middle). + +## Causal chain + +```text +private a, b never sent → only A, B, p, g are public +→ both compute g^(ab) mod p because (g^a)^b = (g^b)^a +→ Eve needs a or b → discrete logarithm → no fast method known at large p → s stays secret +``` + +## Temporal sequence + +1. Alice and Bob agree in public on p = 23 and g = 5. +2. Alice picks a = 6; Bob picks b = 15. They keep these. +3. Alice sends A = 8. Bob sends B = 19. +4. Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2. +5. They use s = 2 as the key for ordinary encryption. + +## Quantities + +- Worked example: p = 23, g = 5, a = 6, b = 15 → A = 8, B = 19, s = 2. +- Check: 5^6 = 15625; 15625 mod 23 = 8. 19^6 mod 23: 19^2 = 361 → 16; 19^4 = 16^2 = 256 → 3; 19^6 = 3 × 16 = 48 → 2. 8^15 mod 23: 8^2 = 64 → 18; 8^4 = 18^2 = 324 → 2; 8^8 = 4; 15 = 8 + 4 + 2 + 1, so 8^15 = 4 × 2 × 18 × 8 = 1152 → 2. +- Powers of 5 mod 23, for k = 1…22: 5, 2, 10, 4, 20, 8, 17, 16, 11, 9, 22, 18, 21, 13, 19, 3, 15, 6, 7, 12, 14, 1. All 22 non-zero remainders appear once; the order looks scrambled. +- Size of p in practice: 2048 bits, about 617 decimal digits. a and b are random numbers between 1 and p − 2 (a small range would let Eve try them all, whatever p is). With p = 23, Eve finds a by trying exponents 1, 2, 3, … : the 6th try gives 8. With a 617-digit p, trying every exponent is out of the question, and the best known methods still cost about 2^112 ≈ 5 × 10^33 operations (estimate); at 10^12 operations per second that is about 10^14 years. +- Alice's work: repeated squaring needs about 2 × 2048 multiplications mod p for a 2048-bit exponent. Reducing mod p after each step keeps numbers small and gives the same remainder: 5^2 = 25 → 2; 5^3 = 2 × 5 = 10 (and 125 mod 23 = 10); 5^6 = 10^2 = 100 → 8. +- Meaning of p's size: small p (23) → Eve tries every exponent in a moment; large p (617 digits) → no known method finishes in any useful time. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| No mod (plain powers) | Eve sees A = 15625, computes log₅ 15625 = 6 | ordinary powers grow in order, so the logarithm reverses them | +| Alice picks the key k = 6 and sends a one-way image of it, 5^6 mod 23 = 8 | Bob is as stuck as Eve: getting 6 from 8 is the same reverse problem | one secret is not enough; each side must finish the other's half without learning it, so both need a private number | +| Multiplication instead of powers: A = g·a mod p | A = 30 mod 23 = 7, B = 75 mod 23 = 6, s = 5·6·15 mod 23 = 13. Eve multiplies A by 14 (because 5 × 14 = 70 = 3 × 23 + 1) and gets 7·14 mod 23 = 6 = a; then B·a = 6·6 = 36 mod 23 = 13 = s | multiplication mod p has a fast reverse: Euclid's algorithm finds the undoing number (14 here) quickly even for a 617-digit p | +| Eve multiplies the public numbers | A·B mod 23 = 152 mod 23 = 14 = 5^(6+15) mod 23, not s = 2 | combining public numbers adds the exponents; the secret needs their product | +| Small p | Eve tries all 22 exponents | brute force is cheap | +| Bad g = 22 | 22^k mod 23 is only 1 or 22 | g must produce many different values; then s can be only 1 or 22 | +| Active attacker (Mallory) | Mallory replaces A and B with 5^3 mod 23 = 10. Alice gets s = 10^6 mod 23 = 6; Bob gets 10^15 mod 23 = 5; Mallory computes both (8^3 mod 23 = 6, 19^3 mod 23 = 5) and relays messages | the exchange does not prove who sent a public number | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Two private numbers | Alice picks the key and sends 5^k mod 23 | Bob cannot undo it either; he is as stuck as Eve. | +| Raise to a power (g^a) | Multiply (g·a mod p) | A = 7; Eve multiplies by 14, the number that undoes ×5 mod 23, and gets a = 6; then B·6 mod 23 = 13 = s. | +| Take mod p | Plain powers | A = 15625; log₅ 15625 = 6 gives a at once. | +| Each side raises the other's public number | Combine the two public numbers | A·B = 14 = g^(a+b); s = g^(ab) = 2. Eve can compute A·B, but it is not s. | +| p large | p = 23 | Eve tries 5, 2, 10, 4, 20, 8 and finds a = 6 on the 6th try. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| Both compute the same s | 19^6 mod 23 = 2 and 8^15 mod 23 = 2, both equal 5^90 mod 23 | (holds always: exponent rule; reducing mod p at each step does not change the remainder) | +| Eve cannot get s from p, g, A, B | p of 617 digits: no known method finds a | p = 23: 6 tries | +| s is shared with the right person | passive Eve: Alice and Bob both hold 2 | active Mallory: Alice holds 6, Bob holds 5, Mallory holds both | + +## Terms + +- **Public channel**: every message on it is visible to everyone. +- **Eve**: the eavesdropper; reads, does not change. +- **mod p**: the remainder after division by p. 100 mod 23 = 8. +- **Modulus p** and **base g**: public numbers fixed before the exchange. +- **Private number**: a or b; picked at random, never sent. +- **Public number**: A or B; g to the private number, mod p; sent. +- **Shared secret**: s; what both compute. It becomes the key. +- **One-way operation**: easy forward, no known fast reverse. +- **Discrete logarithm**: given g, p, and g^x mod p, find x. + +## Epistemic status + +- **Established:** the computation and the agreement on s (a mathematical consequence of exponent rules). +- **Assumed:** that the discrete logarithm has no fast algorithm for well-chosen large p. Not proven; decades of work have found methods faster than trying every exponent, but none fast enough for 2048-bit p on ordinary computers. +- **Assumed:** that no shortcut gives g^(ab) mod p from g^a and g^b without, in effect, finding a or b. Also believed, not proven. The obvious shortcut fails: A·B = 14, s = 2. +- **Assumed:** Eve is passive. The exchange alone does not stop an active attacker. + +## Confusion points + +- "They send the key, encrypted" → no message ever contains s, encrypted or not; each side computes it. +- "Eve can combine A and B" → A·B gives g^(a+b), not g^(ab). +- "If it's hard to reverse, it's hard to compute" → Alice computes fast by repeated squaring; only the reverse is slow. +- "This proves I'm talking to Bob" → no; man in the middle. +- "The private number is the key" → the key is s; a and b are each known only to one person. + +## Scope + +- **Out of scope:** authentication (digital signatures, certificates) — how TLS defeats the man in the middle. +- **Out of scope:** elliptic-curve Diffie–Hellman (X25519) — the same idea in a different number system, with smaller numbers for the same security; used in most TLS connections today. +- **Out of scope:** how s becomes a key — in practice s is first scrambled by a hash function; a cipher such as AES then uses it to encrypt the data. +- **Out of scope:** choosing p and g safely (why p is prime; why g must produce many values) and algorithms for the discrete logarithm. +- **Out of scope:** quantum computers — Shor's algorithm would solve the discrete logarithm fast; post-quantum key exchange replaces this method. + +## Representation decision + +- **Stage:** 1 +- **Reason:** The argument is a short sequence of steps with small worked numbers. One table (who sees what) covers the only multi-relationship structure. A diagram or page would repeat the prose. +- **Levels:** L1 the idea / L2 the steps with numbers / L3 why Eve is stuck / L4 real sizes and the limit. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/model.yaml b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/model.yaml new file mode 100644 index 0000000..6b2d82f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/model.yaml @@ -0,0 +1,112 @@ +# Machine-readable half of model.md. `explainer check key-exchange` holds every rendering to it. + +values: + p: 23 + g: 5 + a: 6 + b: 15 + A: 8 + B: 19 + s: 2 + plain_A: 15625 + product_AB: 14 + mult_A: 7 + mult_inverse: 14 + mult_s: 13 + brute_tries: 6 + real_p_bits: 2048 + real_p_digits: 617 + mallory_m: 3 + mallory_M: 10 + mallory_s_alice: 6 + mallory_s_bob: 5 + product_AB_raw: 152 + squaring_steps: 4000 + powers_of_5_mod_23: [5, 2, 10, 4, 20, 8, 17, 16, 11, 9, 22, 18, 21, 13, 19, 3, 15, 6, 7, 12, 14, 1] + +allow: [1, 2, 3, 4, 16, 18, 25, 100, 112, 125, 10, 12, 30, 36, 70, 75, 22] + +require: + - path: A + in: [prose] + - path: B + in: [prose] + - path: s + in: [prose] + - path: plain_A + in: [prose] + - path: product_AB + in: [prose] + - path: real_p_digits + in: [prose] + +claims: + - id: two-privates + kind: why + about: [private number] + statement: "Each side needs its own private number, because a single secret sent in one-way form is as hard for the receiver to undo as for Eve." + alternative: "Alice picks the key k = 6 and sends 5^6 mod 23 = 8." + counterexample: "Bob receives 8 and is as stuck as Eve: getting 6 back from 8 is the reverse problem." + ask: "Why can't Alice just pick the key and send it through a one-way operation?" + + - id: agreement + kind: mechanism + statement: "Alice computes B^a mod p and Bob computes A^b mod p; both equal g^(ab) mod p, because (g^a)^b = (g^b)^a and taking remainders at each step does not change the result." + example: "p = 23, g = 5, a = 6, b = 15: A = 8, B = 19; 19^6 mod 23 = 2 and 8^15 mod 23 = 2." + ask: "With p = 23, g = 5, a = 6, b = 15, what number does each side send, what shared secret do both get, and why must the two results be equal?" + + - id: why-mod + kind: why + about: [mod] + statement: "The powers are taken mod p because plain powers can be reversed with an ordinary logarithm." + alternative: "Send plain g^a." + counterexample: "A = 5^6 = 15625; log base 5 of 15625 is 6, so Eve has a at once." + + - id: why-power + kind: why + about: [power] + statement: "The operation is a power, not a multiplication, because multiplication mod p has a fast reverse." + alternative: "Send A = g·a mod p." + counterexample: "A = 30 mod 23 = 7. Eve multiplies by 14 (5 × 14 = 70, which is 1 more than 3 × 23) and gets 7 × 14 mod 23 = 6 = a; the shared secret 13 follows." + + - id: eve-stuck + kind: guarantee + statement: "Eve sees p, g, A, B but not a, b, or s; to get s Eve must, as far as anyone knows, find a or b, which is the discrete logarithm problem." + example: "Combining the public numbers fails: A × B mod 23 = 152 mod 23 = 14 = 5^(6+15) mod 23, not s = 2. With a 617-digit p, the best known methods would take on the order of 10^14 years." + counterexample: "With p = 23, Eve tries exponents 1, 2, 3, ...: 5, 2, 10, 4, 20, 8 — a = 6 on the 6th try." + ask: "Eve sees A = 8 and B = 19. Why can't Eve multiply them to get the shared secret, what would Eve have to do instead, and why is that hard?" + + - id: assumption + kind: definition + statement: "That the discrete logarithm is hard for large p is believed, not proven." + ask: "Is it proven that Eve cannot find the shared secret?" + + - id: mitm + kind: limit + statement: "The exchange protects against a reader, not against an attacker who changes messages: such an attacker can run one exchange with each side." + example: "Mallory replaces A and B with 5^3 mod 23 = 10. Alice's shared secret becomes 6, Bob's becomes 5, and Mallory knows both." + ask: "What can an attacker who can change messages, not just read them, do?" + +terms: + - term: private number + means: "a or b; picked at random and never sent" + avoid: [private key, secret number, private value] + - term: public number + means: "A or B; g to the private number, mod p; sent on the channel" + avoid: [public key, public value] + - term: shared secret + means: "s; the number both sides compute; it becomes the key" + avoid: [shared key] + +budget: {prose: 1100, reason: "one mechanism with worked numbers, three why-cases with numbers, the hardness assumption, and the man-in-the-middle limit"} + +quiz: + - q: "Alice and Bob use p = 23 and g = 5. Alice picks a = 3, Bob picks b = 2. What does each send, and what shared secret do they get?" + expect: "Alice sends 5^3 mod 23 = 10; Bob sends 5^2 mod 23 = 2; Alice computes 2^3 = 8, Bob computes 10^2 = 100 mod 23 = 8; s = 8." + misconception: "They add or multiply the public numbers (10 × 2 = 20, or 12)." + - q: "Why does the method use powers mod p rather than plain powers?" + expect: "Plain powers can be reversed with a logarithm (log base 5 of 15625 = 6); the remainder scrambles the order, and no fast reverse is known." + misconception: "Mod p just keeps the numbers small." + - q: "Does the exchange alone guarantee Alice is talking to Bob?" + expect: "No. An active attacker can replace the public numbers and run a separate exchange with each side; authentication is needed." + misconception: "Yes, because only Bob can compute the shared secret." diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/narrative.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/narrative.md new file mode 100644 index 0000000..5267cb4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/narrative.md @@ -0,0 +1,56 @@ +# Narrative: key-exchange + +## 1. Reader before and after + +- **Before:** A technical reader: school mathematics (powers, the rule (x^a)^b = x^(ab), logarithms, remainders) and basic programming. Knows a key is the secret an encryption method needs. New to cryptography: does not know "mod", one-way operations, or the discrete logarithm. +- **After:** "Each side keeps a private number and sends g to that power, mod p. Each raises the other's public number to its own private number, and both get g^(ab) mod p. An eavesdropper sees only g^a and g^b, and no known fast method gets a or b back from them. It stops readers, not attackers who change messages." + +## 2. Question and motive + +- **Question:** How can two people who have never met agree on a secret key when everything they send can be read? Result in words: they never send the key. Each sends half of a calculation; each finishes it with a number they kept; both reach the same number; a reader cannot finish it. +- **Why care:** every HTTPS connection starts with a stranger (a web server) and an open network. Encryption needs a shared key first. +- **Why this approach:** the key cannot travel, so it must be computed on both sides from what travels plus something that does not. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| Alice, Bob, Eve | the two parties; the eavesdropper who reads everything | — | 1 | +| mod p | remainder after division by p | 100 mod 23 = 8 | 4 | +| p, g | public modulus (prime) and base | p = 23, g = 5 | 4 | +| private number a, b | kept, never sent | a = 6, b = 15 | 4 | +| public number A, B | g^a mod p, g^b mod p; sent | A = 8, B = 19 | 4 | +| shared secret s | what both compute; becomes the key | s = 2 | 5 | +| one-way operation | fast forward, no known fast reverse | 5^x mod 23 | 7 | +| discrete logarithm | finding x from g^x mod p | find 6 from 8 | 7 | +| repeated squaring | computing a power by squaring and reducing at each step | 5^6 via 5^2 → 2, 5^3 → 10, 5^6 → 8 | 8 | +| Mallory | an attacker who changes messages | replaces A and B with 10 | 9 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How can strangers share a secret in public? | — | the setting | Everything sent is read by Eve. So the key can never be sent. | the constraint | +| 2 | close (answer first) | So what do they do? | therefore | — | They compute it: each sends half a calculation, finishes the other's half with a kept number; both reach the same number; Eve cannot finish it. | the shape of the answer | +| 3 | objection | Why not one secret, hidden by a one-way operation? | but | Alice picks the key 6 and sends a scrambled form, 8 | Bob gets only 8, the same as Eve, so Bob would be as stuck as Eve. Each side needs a private number, and an operation where each can finish the other's half. | why two private numbers | +| 4 | name / instance | What operation? | so | numbers, then letters | "mod 23" means the remainder after division by 23 (100 mod 23 = 8). Public: p = 23, g = 5. Alice keeps a = 6, sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob keeps b = 15, sends B = 5^15 mod 23 = 19. | the protocol | +| 5 | mechanism | Do they really get the same number? | then | 19^6 mod 23 = 2; 8^15 mod 23 = 2 | Yes. Both are 5^(6×15) mod 23 by (g^a)^b = (g^b)^a. Taking the remainder early does not change the final remainder: dropping a multiple of 23 from a factor drops a multiple of 23 from the product (125 mod 23 = 10 = 2 × 5). | agreement | +| 6 | notice | What does Eve have? | but | table: who knows what, walked through | p, g, A, B. Not a, b, s. A × B mod 23 = 14 = 5^(6+15), not 2. | Eve's view | +| 7 | why | Why powers, why mod? | so | two failing alternatives, then the list | If A = 5 × 6 mod 23 = 7: multiply by 14 (since 5 × 14 = 70 = 3 × 23 + 1) to get 6. If A = 5^6 = 15625 with no mod: log₅ = 6. With both: the list of 5^1 … 5^22 mod 23 jumps around, so Eve can only try exponents one by one. Name: one-way operation; the reverse is the discrete logarithm. | one-way operation | +| 8 | quantity | Then p = 23 falls in 6 tries — so? | but | sizes | Real p: 617 digits; best known methods ~10^14 years (estimate). Alice computes fast by squaring and reducing (5^6: 5^2 → 2, 5^3 → 10, 5^6 → 8). Hardness is believed, not proven. | size matters, assumption | +| 9 | limit | Am I safe from everyone? | but | Mallory numbers | Mallory picks 3 and sends 5^3 mod 23 = 10 to both. Alice gets 10^6 mod 23 = 6, Bob gets 10^15 mod 23 = 5. Mallory computes 8^3 mod 23 = 6 and 19^3 mod 23 = 5, so Mallory decrypts and re-encrypts. Fix: authentication (out of scope). | limit | +| 10 | close | — | so | — | They never send the key. Each sends g to a private power, mod p; each finishes the other's half; both get g^(ab) mod p. Eve sees only g^a and g^b. Next: authentication, elliptic curves. | recap | + +## 5. Concrete to symbol + +Numbers first (5^6 mod 23 = 8), then "here g = 5, a = 6, p = 23", then A = g^a mod p. + +## 6. Links between representations + +The who-knows-what table matches the numbers in beat 4–5. + +## 7. Close the loop + +- **Answer:** They never send the key. Each sends g^(private) mod p, and each raises the other's public number to its own private number. Both get g^(ab) mod p. +- **General argument:** exponent rule gives agreement; discrete logarithm hardness (assumed) keeps Eve out. +- **Payoff:** this is how HTTPS starts (today mostly an elliptic-curve version). Next: authentication, elliptic curves. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/decisions.yaml new file mode 100644 index 0000000..e6e5eae --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/decisions.yaml @@ -0,0 +1,295 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T145920-probe#1: + decision: adopted +2026-10-04T145920-probe#10: + decision: adopted +2026-10-04T145920-probe#11: + decision: declined +2026-10-04T145920-probe#12: + decision: adopted +2026-10-04T145920-probe#13: + decision: declined +2026-10-04T145920-probe#14: + decision: adopted +2026-10-04T145920-probe#15: + decision: adopted +2026-10-04T145920-probe#16: + decision: declined +2026-10-04T145920-probe#2: + decision: adopted +2026-10-04T145920-probe#3: + decision: adopted +2026-10-04T145920-probe#4: + decision: declined +2026-10-04T145920-probe#5: + decision: declined +2026-10-04T145920-probe#6: + decision: adopted +2026-10-04T145920-probe#7: + decision: adopted +2026-10-04T145920-probe#8: + decision: adopted +2026-10-04T145920-probe#9: + decision: declined +2026-10-04T150134-coldread-narrative#1: + decision: adopted +2026-10-04T150134-coldread-narrative#10: + decision: declined +2026-10-04T150134-coldread-narrative#11: + decision: adopted +2026-10-04T150134-coldread-narrative#12: + decision: adopted +2026-10-04T150134-coldread-narrative#13: + decision: adopted +2026-10-04T150134-coldread-narrative#14: + decision: declined +2026-10-04T150134-coldread-narrative#15: + decision: declined +2026-10-04T150134-coldread-narrative#16: + decision: declined +2026-10-04T150134-coldread-narrative#17: + decision: adopted +2026-10-04T150134-coldread-narrative#18: + decision: adopted +2026-10-04T150134-coldread-narrative#19: + decision: declined +2026-10-04T150134-coldread-narrative#2: + decision: adopted +2026-10-04T150134-coldread-narrative#20: + decision: declined +2026-10-04T150134-coldread-narrative#21: + decision: declined +2026-10-04T150134-coldread-narrative#22: + decision: declined +2026-10-04T150134-coldread-narrative#23: + decision: declined +2026-10-04T150134-coldread-narrative#24: + decision: adopted +2026-10-04T150134-coldread-narrative#25: + decision: declined +2026-10-04T150134-coldread-narrative#26: + decision: declined +2026-10-04T150134-coldread-narrative#27: + decision: declined +2026-10-04T150134-coldread-narrative#28: + decision: declined +2026-10-04T150134-coldread-narrative#29: + decision: adopted +2026-10-04T150134-coldread-narrative#3: + decision: adopted +2026-10-04T150134-coldread-narrative#30: + decision: declined +2026-10-04T150134-coldread-narrative#31: + decision: adopted +2026-10-04T150134-coldread-narrative#32: + decision: declined +2026-10-04T150134-coldread-narrative#33: + decision: declined +2026-10-04T150134-coldread-narrative#34: + decision: adopted +2026-10-04T150134-coldread-narrative#35: + decision: declined +2026-10-04T150134-coldread-narrative#36: + decision: declined +2026-10-04T150134-coldread-narrative#37: + decision: declined +2026-10-04T150134-coldread-narrative#38: + decision: declined +2026-10-04T150134-coldread-narrative#39: + decision: adopted +2026-10-04T150134-coldread-narrative#4: + decision: adopted +2026-10-04T150134-coldread-narrative#40: + decision: adopted +2026-10-04T150134-coldread-narrative#5: + decision: adopted +2026-10-04T150134-coldread-narrative#6: + decision: declined +2026-10-04T150134-coldread-narrative#7: + decision: adopted +2026-10-04T150134-coldread-narrative#8: + decision: declined +2026-10-04T150134-coldread-narrative#9: + decision: adopted +2026-10-04T150411-coldread-prose#1: + decision: adopted +2026-10-04T150411-coldread-prose#10: + decision: adopted +2026-10-04T150411-coldread-prose#11: + decision: adopted +2026-10-04T150411-coldread-prose#12: + decision: adopted +2026-10-04T150411-coldread-prose#13: + decision: declined +2026-10-04T150411-coldread-prose#14: + decision: declined +2026-10-04T150411-coldread-prose#15: + decision: declined +2026-10-04T150411-coldread-prose#16: + decision: adopted +2026-10-04T150411-coldread-prose#17: + decision: declined +2026-10-04T150411-coldread-prose#2: + decision: adopted +2026-10-04T150411-coldread-prose#3: + decision: declined +2026-10-04T150411-coldread-prose#4: + decision: declined +2026-10-04T150411-coldread-prose#5: + decision: adopted +2026-10-04T150411-coldread-prose#6: + decision: adopted +2026-10-04T150411-coldread-prose#7: + decision: adopted +2026-10-04T150411-coldread-prose#8: + decision: adopted +2026-10-04T150411-coldread-prose#9: + decision: adopted +2026-10-04T150452-quiz-prose#1: + decision: adopted +2026-10-04T150452-quiz-prose#10: + decision: adopted +2026-10-04T150452-quiz-prose#11: + decision: declined +2026-10-04T150452-quiz-prose#12: + decision: adopted +2026-10-04T150452-quiz-prose#13: + decision: adopted +2026-10-04T150452-quiz-prose#14: + decision: adopted +2026-10-04T150452-quiz-prose#15: + decision: declined +2026-10-04T150452-quiz-prose#16: + decision: declined +2026-10-04T150452-quiz-prose#2: + decision: adopted +2026-10-04T150452-quiz-prose#3: + decision: adopted +2026-10-04T150452-quiz-prose#4: + decision: adopted +2026-10-04T150452-quiz-prose#5: + decision: declined +2026-10-04T150452-quiz-prose#6: + decision: declined +2026-10-04T150452-quiz-prose#7: + decision: adopted +2026-10-04T150452-quiz-prose#8: + decision: adopted +2026-10-04T150452-quiz-prose#9: + decision: declined +2026-10-04T150625-coldread-prose#1: + decision: adopted +2026-10-04T150625-coldread-prose#10: + decision: declined +2026-10-04T150625-coldread-prose#11: + decision: declined +2026-10-04T150625-coldread-prose#12: + decision: declined +2026-10-04T150625-coldread-prose#13: + decision: declined +2026-10-04T150625-coldread-prose#2: + decision: adopted +2026-10-04T150625-coldread-prose#3: + decision: declined +2026-10-04T150625-coldread-prose#4: + decision: adopted +2026-10-04T150625-coldread-prose#5: + decision: declined +2026-10-04T150625-coldread-prose#6: + decision: declined +2026-10-04T150625-coldread-prose#7: + decision: adopted +2026-10-04T150625-coldread-prose#8: + decision: adopted +2026-10-04T150625-coldread-prose#9: + decision: declined +2026-10-04T150725-coldread-prose#1: + decision: adopted +2026-10-04T150725-coldread-prose#10: + decision: declined +2026-10-04T150725-coldread-prose#11: + decision: declined +2026-10-04T150725-coldread-prose#12: + decision: declined +2026-10-04T150725-coldread-prose#2: + decision: declined +2026-10-04T150725-coldread-prose#3: + decision: declined +2026-10-04T150725-coldread-prose#4: + decision: adopted +2026-10-04T150725-coldread-prose#5: + decision: declined +2026-10-04T150725-coldread-prose#6: + decision: declined +2026-10-04T150725-coldread-prose#7: + decision: declined +2026-10-04T150725-coldread-prose#8: + decision: declined +2026-10-04T150725-coldread-prose#9: + decision: declined +2026-10-04T150823-quiz-prose#1: + decision: declined +2026-10-04T150823-quiz-prose#10: + decision: declined +2026-10-04T150823-quiz-prose#11: + decision: declined +2026-10-04T150823-quiz-prose#12: + decision: declined +2026-10-04T150823-quiz-prose#13: + decision: declined +2026-10-04T150823-quiz-prose#14: + decision: declined +2026-10-04T150823-quiz-prose#15: + decision: declined +2026-10-04T150823-quiz-prose#16: + decision: declined +2026-10-04T150823-quiz-prose#17: + decision: adopted +2026-10-04T150823-quiz-prose#2: + decision: adopted +2026-10-04T150823-quiz-prose#3: + decision: adopted +2026-10-04T150823-quiz-prose#4: + decision: declined +2026-10-04T150823-quiz-prose#5: + decision: declined +2026-10-04T150823-quiz-prose#6: + decision: declined +2026-10-04T150823-quiz-prose#7: + decision: declined +2026-10-04T150823-quiz-prose#8: + decision: declined +2026-10-04T150823-quiz-prose#9: + decision: declined +2026-10-04T150941-quiz-prose#1: + decision: adopted +2026-10-04T150941-quiz-prose#10: + decision: declined +2026-10-04T150941-quiz-prose#11: + decision: declined +2026-10-04T150941-quiz-prose#12: + decision: declined +2026-10-04T150941-quiz-prose#13: + decision: declined +2026-10-04T150941-quiz-prose#14: + decision: adopted +2026-10-04T150941-quiz-prose#15: + decision: declined +2026-10-04T150941-quiz-prose#16: + decision: declined +2026-10-04T150941-quiz-prose#2: + decision: declined +2026-10-04T150941-quiz-prose#3: + decision: declined +2026-10-04T150941-quiz-prose#4: + decision: declined +2026-10-04T150941-quiz-prose#5: + decision: declined +2026-10-04T150941-quiz-prose#6: + decision: declined +2026-10-04T150941-quiz-prose#7: + decision: declined +2026-10-04T150941-quiz-prose#8: + decision: declined +2026-10-04T150941-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T145920-probe.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T145920-probe.json new file mode 100644 index 0000000..2486e12 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T145920-probe.json @@ -0,0 +1,225 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "key-exchange", + "at": "2026-10-04T14:59:20", + "model": "default", + "prompt_sha": "5df4d87a86a1", + "reply": { + "gaps": [ + { + "rule": 1, + "severity": "main", + "about": "No explanation of why a one-way function alone is not enough, and why two private numbers are needed", + "why_it_matters": "The obvious first idea is: 'Alice picks the key and sends it hidden by a one-way function.' Nothing in the model says why that fails. Without that, Dependency step 2 (both sides mix, and the mixes match) reads as an arbitrary design rather than the answer to the central question.", + "suggested_addition": "Alternative state: Alice picks key k = 6 and sends 5^6 mod 23 = 8. Bob is now as stuck as Eve, because recovering k = 6 from 8 is the same discrete logarithm. DH needs an operation where each side can finish the other's half: Bob turns 8 into 8^15 mod 23 = 2 without ever learning 6. That works because (g^a)^b = (g^b)^a." + }, + { + "rule": 2, + "severity": "main", + "about": "The causal chain jumps from 'Eve needs a or b' without support", + "why_it_matters": "The chain says Eve needs a or b, but no step shows that getting s requires a or b. A shortcut from (A, B) straight to g^(ab) is a separate problem (the computational Diffie–Hellman problem). It is also unproven, and the Epistemic status section never names this assumption. The central claim 'Eve is stuck' depends on this hidden step.", + "suggested_addition": "State it as an assumption: 'No known way exists to compute g^(ab) mod p from g^a and g^b without, in effect, finding a or b.' Give an instance where the obvious shortcut fails: A·B = 8·19 mod 23 = 14 = 5^21, but s = 5^90 mod 23 = 5^2 = 2. Add this assumption to Epistemic status as 'believed, not proven'." + }, + { + "rule": 1, + "severity": "edge", + "about": "The multiplication row in 'Why this form' ends with a vague sentence", + "why_it_matters": "'Would not match in a useful way unless Eve can also compute them' is unclear. In fact the results do match, and Eve gets the same value. The concrete failure is never shown.", + "suggested_addition": "With multiplication, s = g·a·b mod 23 = 5·6·15 = 450 mod 23 = 13. Eve computes A·B·14 mod 23 = 7·6·14 mod 23 = 13, where B = 5·15 mod 23 = 6. Eve gets s directly." + }, + { + "rule": 1, + "severity": "edge", + "about": "No reason given for why p must be prime", + "why_it_matters": "p is introduced as 'a large prime' with no reason. A learner will ask why not any number.", + "suggested_addition": "With modulus 24 and g = 5, the powers are 5, 1, 5, 1, …: only 2 values, so s can only be 1 or 5. With prime 23 and g = 5, all 22 non-zero remainders appear. Note that safe choices of p and g are otherwise out of scope." + }, + { + "rule": 2, + "severity": "edge", + "about": "The check in Quantities uses 5^22 mod 23 = 1 and reduces exponents mod 22 without explanation", + "why_it_matters": "The audience does not know Fermat's little theorem, so this verification step is left implicit.", + "suggested_addition": "Point to the powers list: 5^22 mod 23 = 1, so the powers repeat every 22 steps. Then 5^90 = 5^(4·22+2), which gives 5^2 = 25 mod 23 = 2. Or drop this check and use direct repeated squaring of 8^15 (see the rule 4 entry)." + }, + { + "rule": 2, + "severity": "edge", + "about": "The claim 'reducing mod p at each step does not change the remainder' has no numeric instance", + "why_it_matters": "Every step of Alice's computation depends on this guarantee, and it appears without a concrete check.", + "suggested_addition": "5^3 = 125, and 125 mod 23 = 10. Reducing first gives (5^2 mod 23)·5 = 2·5 = 10. Same answer." + }, + { + "rule": 2, + "severity": "edge", + "about": "'Eve cannot get s' has only one way to break it (small p)", + "why_it_matters": "A large p does not help if the private number comes from a small range. The model says a is 'random' but never says from what range.", + "suggested_addition": "State that a and b are drawn from 1 … p−2. Breaking case: with a 617-digit p but a chosen from 1 … 1,000,000, Eve tries at most 10^6 exponents and finds a in under a second." + }, + { + "rule": 3, + "severity": "edge", + "about": "Several concepts have more than one name", + "why_it_matters": "Readers may think each name is a different thing.", + "suggested_addition": "Pick one name for each: 'shared secret s' / 'key' / 's (after hashing)'; 'public number' / 'public value' (Dependencies 2); 'private number' / 'private value'; 'active attacker' / 'Mallory' / 'man in the middle'; 'fast exponentiation' / 'repeated squaring'; 'one-way operation' (Entities) / 'one-way function' (Audience). Also define '≡' (used in Quantities) and 'hash' (Temporal step 5) before use, or remove them." + }, + { + "rule": 4, + "severity": "edge", + "about": "Repeated squaring is worked only for 5^6; Bob's 8^15 has no worked steps", + "why_it_matters": "The confusion point 'computing is fast' relies on this, and the odd exponent 15 shows how the method combines squares.", + "suggested_addition": "8^2 = 64 → 18; 8^4 = 18^2 = 324 → 2; 8^8 = 4. Since 15 = 8+4+2+1, 8^15 = 4·2·18·8 = 1152 mod 23 = 2. That is 3 squarings plus 3 multiplications instead of 14 multiplications." + }, + { + "rule": 5, + "severity": "edge", + "about": "The size of p is not turned into time, and the brute-force count overstates Eve's real cost", + "why_it_matters": "'10^616 entries' doesn't tell the reader how long that takes. The model also says faster methods exist, so the security level is far below 10^616. A reader may come away with the wrong order of magnitude.", + "suggested_addition": "Brute force at 10^12 tries/s takes about 10^604 s; the universe is about 4×10^17 s old. The best known methods against 2048-bit p cost about 2^112 ≈ 5×10^33 operations, which at 10^12/s is about 10^14 years. Compare Alice: about 4096 multiplications, a few milliseconds." + }, + { + "rule": 5, + "severity": "edge", + "about": "The 'Bad g = 22' row has no stated meaning", + "why_it_matters": "'Then s can be only 1 or 22' is the consequence, but its meaning for the reader isn't stated. It reads as a fragment.", + "suggested_addition": "With g = 22, Eve guesses s correctly with 1 try in 2. With g = 5, s could be any of 22 values." + } + ], + "excess": [ + "Known limit: Shor's algorithm and quantum computers (not needed to answer the central question)", + "Scope notes on elliptic-curve DH (X25519) and ML-KEM", + "'(in practice, a hash of s)' in Temporal step 5 and 'after hashing' in Terms: brings in an undefined term that is out of scope", + "The check in Quantities using 5^22 mod 23 = 1 and exponent reduction mod 22 (could be replaced by direct squaring)", + "Representation decision section (planning metadata, not content for the reader)" + ], + "next_questions": [ + { + "q": "Why not just pick a key and send it hidden by a hard-to-reverse function?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Could Eve compute g^(ab) mod p from A and B without finding a or b?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How do Alice and Bob stop a man in the middle and know they're talking to each other?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Why must p be prime, and how are p and g chosen?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "How long would the best known attack actually take against a 2048-bit p?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.1690814, + "seconds": 64 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 1", + "text": "No explanation of why a one-way function alone is not enough, and why two private numbers are needed → Alternative state: Alice picks key k = 6 and sends 5^6 mod 23 = 8. Bob is now as stuck as Eve, because recovering k = 6 from 8 is the same discrete logarithm. DH needs an operation where each side can finish the other's half: Bob turns 8 into 8^15 mod 23 = 2 without ever learning 6. That works because (g^a)^b = (g^b)^a." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 2", + "text": "The causal chain jumps from 'Eve needs a or b' without support → State it as an assumption: 'No known way exists to compute g^(ab) mod p from g^a and g^b without, in effect, finding a or b.' Give an instance where the obvious shortcut fails: A·B = 8·19 mod 23 = 14 = 5^21, but s = 5^90 mod 23 = 5^2 = 2. Add this assumption to Epistemic status as 'believed, not proven'." + }, + { + "n": 3, + "severity": "edge", + "kind": "rule 1", + "text": "The multiplication row in 'Why this form' ends with a vague sentence → With multiplication, s = g·a·b mod 23 = 5·6·15 = 450 mod 23 = 13. Eve computes A·B·14 mod 23 = 7·6·14 mod 23 = 13, where B = 5·15 mod 23 = 6. Eve gets s directly." + }, + { + "n": 4, + "severity": "edge", + "kind": "rule 1", + "text": "No reason given for why p must be prime → With modulus 24 and g = 5, the powers are 5, 1, 5, 1, …: only 2 values, so s can only be 1 or 5. With prime 23 and g = 5, all 22 non-zero remainders appear. Note that safe choices of p and g are otherwise out of scope." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 2", + "text": "The check in Quantities uses 5^22 mod 23 = 1 and reduces exponents mod 22 without explanation → Point to the powers list: 5^22 mod 23 = 1, so the powers repeat every 22 steps. Then 5^90 = 5^(4·22+2), which gives 5^2 = 25 mod 23 = 2. Or drop this check and use direct repeated squaring of 8^15 (see the rule 4 entry)." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 2", + "text": "The claim 'reducing mod p at each step does not change the remainder' has no numeric instance → 5^3 = 125, and 125 mod 23 = 10. Reducing first gives (5^2 mod 23)·5 = 2·5 = 10. Same answer." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 2", + "text": "'Eve cannot get s' has only one way to break it (small p) → State that a and b are drawn from 1 … p−2. Breaking case: with a 617-digit p but a chosen from 1 … 1,000,000, Eve tries at most 10^6 exponents and finds a in under a second." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 3", + "text": "Several concepts have more than one name → Pick one name for each: 'shared secret s' / 'key' / 's (after hashing)'; 'public number' / 'public value' (Dependencies 2); 'private number' / 'private value'; 'active attacker' / 'Mallory' / 'man in the middle'; 'fast exponentiation' / 'repeated squaring'; 'one-way operation' (Entities) / 'one-way function' (Audience). Also define '≡' (used in Quantities) and 'hash' (Temporal step 5) before use, or remove them." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 4", + "text": "Repeated squaring is worked only for 5^6; Bob's 8^15 has no worked steps → 8^2 = 64 → 18; 8^4 = 18^2 = 324 → 2; 8^8 = 4. Since 15 = 8+4+2+1, 8^15 = 4·2·18·8 = 1152 mod 23 = 2. That is 3 squarings plus 3 multiplications instead of 14 multiplications." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 5", + "text": "The size of p is not turned into time, and the brute-force count overstates Eve's real cost → Brute force at 10^12 tries/s takes about 10^604 s; the universe is about 4×10^17 s old. The best known methods against 2048-bit p cost about 2^112 ≈ 5×10^33 operations, which at 10^12/s is about 10^14 years. Compare Alice: about 4096 multiplications, a few milliseconds." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 5", + "text": "The 'Bad g = 22' row has no stated meaning → With g = 22, Eve guesses s correctly with 1 try in 2. With g = 5, s could be any of 22 values." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "Known limit: Shor's algorithm and quantum computers (not needed to answer the central question)" + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Scope notes on elliptic-curve DH (X25519) and ML-KEM" + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "'(in practice, a hash of s)' in Temporal step 5 and 'after hashing' in Terms: brings in an undefined term that is out of scope" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "The check in Quantities using 5^22 mod 23 = 1 and exponent reduction mod 22 (could be replaced by direct squaring)" + }, + { + "n": 16, + "severity": "excess", + "kind": "model entry", + "text": "Representation decision section (planning metadata, not content for the reader)" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150134-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150134-coldread-narrative.json new file mode 100644 index 0000000..3986c82 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150134-coldread-narrative.json @@ -0,0 +1,432 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "key-exchange", + "at": "2026-10-04T15:01:34", + "model": "default", + "prompt_sha": "0dc70ed4e552", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Everything sent is read by Eve. So the key can never be sent.", + "unresolved": [ + "Eve" + ], + "unsaid": [ + "Shown 'the setting' is never described. The words don't name Alice and Bob as the two parties or say that Eve only listens." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (answer first)", + "text": "They compute it: each sends half a calculation, finishes the other's half with a kept number; both reach the same number; Eve cannot finish it.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "Bob would be as stuck as Eve. Each side needs a private number, and an operation where each can finish the other's half.", + "unresolved": [ + "one-way operation", + "k" + ], + "unsaid": [ + "Shown 'k = 6 → 8': the words never say what k is or what operation the arrow stands for." + ], + "leap": [ + "'Bob would be as stuck as Eve': the words don't say that if only one side holds the secret and sends its one-way result, Bob gets exactly what Eve gets." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (name / instance)", + "text": "Powers mod p. Alice sends 5^6 mod 23 = 8; Bob sends 19.", + "unresolved": [ + "mod", + "p" + ], + "unsaid": [ + "Shown 'mod defined': the words never say mod means the remainder after division.", + "Shown 'p, g, a, b, A, B': the words don't say g = 5, a = 6, p = 23, b = 15, or that A and B are the sent numbers.", + "Bob's private number 15 is never said, so the reader can't see where 19 comes from." + ], + "leap": [ + "Bob sends 19: the computation isn't given." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Yes, both are g^(ab): (g^a)^b = (g^b)^a; remainders can be taken at each step.", + "unresolved": [ + "g", + "a", + "b" + ], + "unsaid": [ + "Shown '8^15 mod 23 = 2' is the first time the reader sees 15 as Bob's number." + ], + "leap": [ + "'remainders can be taken at each step': the words don't say why taking remainders partway leaves the final remainder unchanged. The reader knows remainders but not this rule." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (notice)", + "text": "p, g, A, B. Not a, b, s. A × B = 14, not 2 (exponents add, not multiply).", + "unresolved": [ + "A", + "B", + "s" + ], + "unsaid": [ + "The 'who knows what' table is shown but not walked through in words." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (why)", + "text": "Without mod, a log reverses it. With multiplication, a single multiplication reverses it. Powers mod p scramble... No fast reverse known: the discrete logarithm.", + "unresolved": [ + "k (in 5^k mod 23, used again with a different meaning from beat 3)" + ], + "unsaid": [ + "Shown 'multiplication → 14 undoes it' is not explained, and it's unclear how 14 (from beat 6) relates to this.", + "The list of 5^k mod 23 is shown, but the words don't point out what to notice in it." + ], + "leap": [ + "'With multiplication, a single multiplication reverses it': this means swapping the power for multiplication. The words don't say so, and the reverse step isn't shown." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (quantity)", + "text": "Real p: 617 digits; ~10^14 years by best known methods (estimate). Alice uses repeated squaring: ~4000 multiplications. Hardness is believed, not proven.", + "unresolved": [ + "repeated squaring", + "6 tries (never said earlier that finding a = 6 means trying exponents one by one)" + ], + "unsaid": [ + "Shown 'sizes' is not described." + ], + "leap": [ + "The words don't explain why repeated squaring needs only about 4000 multiplications." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (limit)", + "text": "Mallory swaps public numbers: Alice gets 6, Bob gets 5, Mallory both. Fix: authentication (out of scope).", + "unresolved": [ + "Mallory", + "authentication" + ], + "unsaid": [ + "Shown 'Mallory numbers': the words never say what number Mallory sends, so 'Alice gets 6, Bob gets 5' can't be checked." + ], + "leap": [ + "The words don't say why Mallory ends up knowing both secrets." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 10 (close)", + "text": "Restate answer; scope pointers.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": true + } + ], + "blocking": [ + "Beat 4: 'Powers mod p': mod and p are never defined in the words. The definition exists only in Shown ('mod defined').", + "Beat 4: g, a, b, A, B are never tied to 5, 6, 15, 8, 19 in the words, so 'g^(ab)' in beat 5 and 'p, g, A, B. Not a, b, s' in beat 6 refer to letters the reader hasn't been given.", + "Beat 5: 'remainders can be taken at each step': agreement depends on this rule, and it isn't justified.", + "Beat 7: 'With multiplication, a single multiplication reverses it' and Shown 'multiplication → 14 undoes it': it's unclear what is being compared.", + "Beat 10: 'Restate answer; scope pointers.' is a placeholder, so no closing words are given." + ], + "cuts": [ + "Beat 8: cut the repeated-squaring aside ('~4000 multiplications'), or move it to a footnote. It answers a side question and is never explained.", + "Beat 6: cut '(exponents add, not multiply)' or merge it into beat 7, which makes the 'why not simpler' point again." + ], + "question": { + "text": "How can two parties agree on a secret key when everything they send is read?", + "known_at": "Beat 1: 'Everything sent is read by Eve. So the key can never be sent.'", + "result_stated": "Yes, in beat 2: 'each sends half a calculation, finishes the other's half with a kept number; both reach the same number; Eve cannot finish it.'" + }, + "motive": "Reason for the approach: given in beat 1 ('So the key can never be sent') and beat 3 ('Each side needs a private number, and an operation where each can finish the other's half'). Reason to care: missing from the beats. The HTTPS motive appears only in the plan sections.", + "close": "Not delivered. Beat 10 says only 'Restate answer; scope pointers.' It gives no words answering the question, no general argument (exponent rule plus assumed discrete-log hardness), and no payoff.", + "recap": "Each side keeps a secret exponent and sends 5 to that power, mod 23. Raising the other's number to your own secret gives both sides the same value, and an eavesdropper would have to undo the power-mod, which nobody knows how to do fast.", + "strongest": "Beat 5: the two computations 19^6 mod 23 = 2 and 8^15 mod 23 = 2, tied to (g^a)^b = (g^b)^a, which this reader already knows.", + "weakest": "Beat 4 together with beat 10. Beat 4 defines mod and the letters only in Shown, and the whole symbolic argument rests on them. Beat 10 is an unwritten placeholder.", + "suggestions": [ + "In beat 4, say in words: 'mod 23 means the remainder after dividing by 23; here g = 5, p = 23, Alice's private a = 6, Bob's private b = 15, and A = 8, B = 19 are what they send.'", + "In beat 5, give one line of support for reducing as you go, e.g. a product's remainder depends only on the remainders of its factors.", + "In beat 3, define k and the arrow, or replace the shown example with words.", + "In beat 7, state the comparison outright: if Alice sent g × a mod p, Eve could divide (multiply by an inverse) to get a back.", + "In beat 9, say that Mallory sends 10 to both sides.", + "Write beat 10 as actual sentences, and include the HTTPS reason to care." + ] + }, + "usage": { + "cost": 0.1230666, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'Powers mod p': mod and p are never defined in the words. The definition exists only in Shown ('mod defined')." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: g, a, b, A, B are never tied to 5, 6, 15, 8, 19 in the words, so 'g^(ab)' in beat 5 and 'p, g, A, B. Not a, b, s' in beat 6 refer to letters the reader hasn't been given." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: 'remainders can be taken at each step': agreement depends on this rule, and it isn't justified." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 7: 'With multiplication, a single multiplication reverses it' and Shown 'multiplication → 14 undoes it': it's unclear what is being compared." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 10: 'Restate answer; scope pointers.' is a placeholder, so no closing words are given." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 8: cut the repeated-squaring aside ('~4000 multiplications'), or move it to a footnote. It answers a side question and is never explained." + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 6: cut '(exponents add, not multiply)' or merge it into beat 7, which makes the 'why not simpler' point again." + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): Eve" + }, + { + "n": 9, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): Shown 'the setting' is never described. The words don't name Alice and Bob as the two parties or say that Eve only listens." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): one-way operation" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): k" + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): Shown 'k = 6 → 8': the words never say what k is or what operation the arrow stands for." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): 'Bob would be as stuck as Eve': the words don't say that if only one side holds the secret and sends its one-way result, Bob gets exactly what Eve gets." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (name / instance): mod" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (name / instance): p" + }, + { + "n": 16, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name / instance): Shown 'mod defined': the words never say mod means the remainder after division." + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name / instance): Shown 'p, g, a, b, A, B': the words don't say g = 5, a = 6, p = 23, b = 15, or that A and B are the sent numbers." + }, + { + "n": 18, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name / instance): Bob's private number 15 is never said, so the reader can't see where 19 comes from." + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (name / instance): Bob sends 19: the computation isn't given." + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): g" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): a" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): b" + }, + { + "n": 23, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): Shown '8^15 mod 23 = 2' is the first time the reader sees 15 as Bob's number." + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 'remainders can be taken at each step': the words don't say why taking remainders partway leaves the final remainder unchanged. The reader knows remainders but not this rule." + }, + { + "n": 25, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (notice): A" + }, + { + "n": 26, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (notice): B" + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (notice): s" + }, + { + "n": 28, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (notice): The 'who knows what' table is shown but not walked through in words." + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (why): k (in 5^k mod 23, used again with a different meaning from beat 3)" + }, + { + "n": 30, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (why): Shown 'multiplication → 14 undoes it' is not explained, and it's unclear how 14 (from beat 6) relates to this." + }, + { + "n": 31, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (why): The list of 5^k mod 23 is shown, but the words don't point out what to notice in it." + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (why): 'With multiplication, a single multiplication reverses it': this means swapping the power for multiplication. The words don't say so, and the reverse step isn't shown." + }, + { + "n": 33, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (quantity): repeated squaring" + }, + { + "n": 34, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (quantity): 6 tries (never said earlier that finding a = 6 means trying exponents one by one)" + }, + { + "n": 35, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (quantity): Shown 'sizes' is not described." + }, + { + "n": 36, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (quantity): The words don't explain why repeated squaring needs only about 4000 multiplications." + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (limit): Mallory" + }, + { + "n": 38, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (limit): authentication" + }, + { + "n": 39, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 9 (limit): Shown 'Mallory numbers': the words never say what number Mallory sends, so 'Alice gets 6, Bob gets 5' can't be checked." + }, + { + "n": 40, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (limit): The words don't say why Mallory ends up knowing both secrets." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150411-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150411-coldread-prose.json new file mode 100644 index 0000000..f9bc3b3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150411-coldread-prose.json @@ -0,0 +1,389 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:04:11", + "model": "default", + "prompt_sha": "a76b387677f2", + "reply": { + "paragraphs": [ + { + "at": "Title", + "text": "How can two strangers agree on a secret key when anyone can read their messages?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold summary (line 5)", + "text": "They never send the key. Each person keeps a private number and sends a scrambled form of it...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 7", + "text": "This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 1", + "text": "Alice and Bob want to encrypt their messages...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 2", + "text": "A first idea fails. Alice picks the key, 6, and sends it through an operation that is hard to reverse. The result is 8...", + "unresolved": [], + "leap": [ + "\"an operation that lets each side finish the other's half without undoing it\": the reader is not yet told what a 'half' is or how one side could 'finish' it. The reader has to hold this phrase until the numbered steps make it concrete. (Where 6 → 8 comes from is also not given, but 'an operation' makes clear it is only a placeholder.)" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The exchange, mod definition", + "text": "\"mod 23\" means \"the remainder after division by 23\"...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Step 1", + "text": "Alice and Bob agree in public on two numbers: a prime modulus p = 23 and a base g = 5...", + "unresolved": [], + "leap": [ + "Why p has to be prime is not said here. Scope defers it later, so this is only an edge finding." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Step 2", + "text": "Each picks a private number at random...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Step 3", + "text": "Each sends a public number: g raised to their private number, mod p...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Step 4", + "text": "Each raises the other's public number to their own private number, mod p. Alice computes 19^6 mod 23 = 2...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Shared secret", + "text": "The result, s = 2, is the shared secret. It becomes the key for ordinary encryption.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why both get 2", + "text": "Alice computed (5^15)^6, and Bob computed (5^6)^15...", + "unresolved": [], + "leap": [ + "\"so it removes a multiple of 23 from the product\": the one-line algebra, (x + 23k)·y = xy + 23ky, is left for the reader to fill in. It is short, but this is where a reader might reread." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What Eve sees, table", + "text": "Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Predict prompt", + "text": "Eve has 8 and 19. Can she combine them into 2?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Multiplying fails", + "text": "Multiplying fails: 8 × 19 mod 23 = 152 mod 23 = 14...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why powers, and why mod, intro", + "text": "Each choice blocks a simpler way for Eve to find a private number.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: Without mod", + "text": "A = 5^6 = 15625. Eve takes the logarithm base 5...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: multiplication instead of a power", + "text": "A = 5 × 6 mod 23 = 7. Multiplying by 14 undoes multiplying by 5...", + "unresolved": [], + "leap": [ + "It is not said how Eve finds 14. The text checks that 14 works but does not say that finding it is easy, and that ease is the whole point of the bullet. The same number 14 also appeared just before as 8 × 19 mod 23, which could briefly confuse." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: powers mod p", + "text": "With powers mod p, the results jump around. For exponents 1 to 22...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "One-way definition", + "text": "So g^x mod p is a one-way operation... the discrete logarithm.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 1", + "text": "With p = 23, Eve can try exponents in order... about 10^14 years.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 2", + "text": "Alice's own work stays small. Alice squares and takes the remainder at each step...", + "unresolved": [], + "leap": [ + "\"A 2048-bit exponent needs about 4,000 such steps\": no reason is given for 4,000 (roughly one squaring plus at most one multiply per bit).", + "\"Alice squares... at each step\", yet the example step 5^3 = 2 × 5 is a multiply, not a square, so the rule as stated does not match the example." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Assumption, not proof", + "text": "No one has proven that the discrete logarithm is hard...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The limit, para 1", + "text": "The exchange stops a reader. It does not stop Mallory...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Mallory bullets", + "text": "Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The limit, closing para", + "text": "Alice shares 6 with Mallory... Real systems add authentication: Bob signs the public number B, and Alice checks the signature with a certificate.", + "unresolved": [ + "\"signs\"", + "\"signature\"", + "\"certificate\"" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer", + "text": "The two strangers never send the key. Each sends g raised to a private number, mod p...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Scope list", + "text": "Authentication...; Elliptic-curve Diffie–Hellman (for example X25519)...; hash function...AES...; Choosing p and g safely...; Shor's algorithm...ML-KEM", + "unresolved": [ + "\"X25519\"", + "\"hash function\"", + "\"AES\"", + "\"Shor's algorithm\"", + "\"ML-KEM\"", + "\"different number system\"" + ], + "leap": [], + "excess": [ + "\"Most HTTPS connections use it today\" partly repeats line 7, \"A version of it starts most HTTPS connections\"" + ], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Scope list: drop the example names \"X25519\" and \"ML-KEM\". They are unknown to this reader and add nothing to a pointer list.", + "Scope, Elliptic-curve bullet: \"Most HTTPS connections use it today\" largely repeats line 7." + ], + "question": { + "text": "How can two people who share nothing secret, talking only over a channel anyone can read, end up with the same secret key while the eavesdropper does not?", + "known_at": "The title, made concrete in 'The problem' para 1 (\"So the key itself can never be sent.\")", + "result_stated": "Yes, at the very start: \"They never send the key. Each person keeps a private number and sends a scrambled form of it. Each then combines the other person's scrambled number with their own private number, and both get the same result.\"" + }, + "motive": "Reason to care: \"Encryption needs a key that both of them know. They have never met, so every message goes over a public channel\" and \"A version of it starts most HTTPS connections.\" Reason for the approach: the failed first idea (\"If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number...\") and the section 'Why powers, and why mod' (\"Each choice blocks a simpler way for Eve to find a private number.\")", + "close": "Yes: \"Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p.\"", + "recap": "Each side keeps a secret exponent and publishes g to that power mod p. Raising the other side's number to your own exponent gives both sides g^(ab) mod p, and an eavesdropper can't undo the powering to catch up.", + "strongest": "'Why powers, and why mod' (lines 45–54). It rules out the simpler choices with concrete numbers (plain powers fall to a logarithm, multiplication falls to an inverse), and the jumbled table of 5^x mod 23 makes 'one-way' something the reader can see.", + "weakest": "'Why real numbers are huge' para 2 (line 60). \"Alice squares... at each step\" does not match the multiply step in its own example, and \"about 4,000 such steps\" comes with no reason.", + "suggestions": [ + "In the multiplication bullet, add a clause saying an inverse like 14 can be found quickly (for example by the extended Euclidean algorithm, or by trying 22 candidates), so the reader sees why this choice fails.", + "Line 60: describe the method as square-and-multiply (square at each bit, multiply by g when the bit is 1). That fixes the mismatch with the 5^3 step and explains '≈2 steps per bit → ~4,000'.", + "In 'Why both get 2', write the one-line algebra (x + 23k)·y = xy + 23ky.", + "In the authentication sentence, gloss 'signs' in a few words (for example 'attaches proof that only Bob could have produced')." + ] + }, + "usage": { + "cost": 0.1540012, + "seconds": 51 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Scope list: drop the example names \"X25519\" and \"ML-KEM\". They are unknown to this reader and add nothing to a pointer list." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Scope, Elliptic-curve bullet: \"Most HTTPS connections use it today\" largely repeats line 7." + }, + { + "n": 3, + "severity": "edge", + "kind": "leap", + "text": "The problem, para 2: \"an operation that lets each side finish the other's half without undoing it\": the reader is not yet told what a 'half' is or how one side could 'finish' it. The reader has to hold this phrase until the numbered steps make it concrete. (Where 6 → 8 comes from is also not given, but 'an operation' makes clear it is only a placeholder.)" + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Step 1: Why p has to be prime is not said here. Scope defers it later, so this is only an edge finding." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Why both get 2: \"so it removes a multiple of 23 from the product\": the one-line algebra, (x + 23k)·y = xy + 23ky, is left for the reader to fill in. It is short, but this is where a reader might reread." + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Bullet: multiplication instead of a power: It is not said how Eve finds 14. The text checks that 14 works but does not say that finding it is easy, and that ease is the whole point of the bullet. The same number 14 also appeared just before as 8 × 19 mod 23, which could briefly confuse." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Why real numbers are huge, para 2: \"A 2048-bit exponent needs about 4,000 such steps\": no reason is given for 4,000 (roughly one squaring plus at most one multiply per bit)." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Why real numbers are huge, para 2: \"Alice squares... at each step\", yet the example step 5^3 = 2 × 5 is a multiply, not a square, so the rule as stated does not match the example." + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "The limit, closing para: \"signs\"" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "The limit, closing para: \"signature\"" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "The limit, closing para: \"certificate\"" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"X25519\"" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"hash function\"" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"AES\"" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"Shor's algorithm\"" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"ML-KEM\"" + }, + { + "n": 17, + "severity": "edge", + "kind": "unresolved", + "text": "Scope list: \"different number system\"" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150452-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150452-quiz-prose.json new file mode 100644 index 0000000..3a2170c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150452-quiz-prose.json @@ -0,0 +1,178 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:04:46", + "model": "default", + "prompt_sha": "bedad2b89e75", + "reply": { + "answers": { + "1": "The people are Alice and Bob (the two strangers), Eve (an eavesdropper who only reads) and Mallory (an attacker who can replace messages). The public values are the prime modulus p and the base g. Each side has a private number (a, b) and a public number (A = g^a mod p, B = g^b mod p). The result is the shared secret s = g^(ab) mod p, which is then used as the key for ordinary encryption. Supporting ideas: 'mod' (remainder), one-way operation, discrete logarithm, authentication.", + "2": "p and g are agreed in public. Each side keeps a private exponent and publishes g raised to it, mod p. Each side raises the other's public number to its own private exponent. Because of the exponent rule, both get g^(ab) mod p. Eve sees p, g, A and B but not a, b or s. Getting a from A is the discrete logarithm problem. Mallory can swap A and B for her own public number, and authentication (signatures, certificates) is what stops that.", + "3": "Each side sends g^private mod p and raises what it receives to its own private number. By (g^a)^b = (g^b)^a, both reach g^(ab) mod p. Eve can't follow because undoing g^x mod p (the discrete logarithm) has no known fast method.", + "4": "The most important variable is the size of p (and of the range a and b are picked from). With p = 23, Eve can try exponents one by one and finds a = 6 on the 6th try. With a 2048-bit p (about 617 digits), even the best known methods need about 2^112 operations, roughly 10^14 years at a trillion per second. Alice's own work only grows to about 4,000 steps. So making p bigger hurts Eve enormously and the honest users only a little.", + "5": "Assumption: that the discrete logarithm is hard, and that there is no other way from A and B to s. This has not been proven; it rests on decades of failed attempts. The setup also assumes Eve only reads and does not change messages. Established fact: (5^15)^6 and (5^6)^15 both equal 5^(90), and taking remainders along the way doesn't change the final remainder. In the example, both sides compute 2.", + "6": "Alice and Bob publicly agree on a prime p and a base g. Each secretly picks an exponent and sends g to that power, mod p. Each raises the number they receive to their own secret exponent, mod p, and the exponent rule makes both results equal g^(ab) mod p, which becomes the key. Eve, who sees only g^a and g^b mod p, would have to solve the discrete logarithm, which has no known fast method when p is huge, though this is assumed rather than proven. The method doesn't stop someone who replaces messages, so real systems add signatures and certificates.", + "7": "The 'first idea fails' paragraph was hardest. It says 6 is sent 'through an operation that is hard to reverse' and the result is 8, but it doesn't name the operation (it is in fact 5^6 mod 23, which only appears later). Naming it, or pointing ahead to it, would help. The '~4,000 steps' figure was also hard to follow without a sentence on where it comes from.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8 (= 5^6 mod 23, matching the explanation's table).", + "9": "Without mod, A = 5^6 = 15625, and Eve just takes the logarithm base 5 to get 6, because plain powers grow in order. With mod p, the results jump around (5, 2, 10, 4, 20, 8, …), so a result says nothing about where the exponent is. That makes the operation one-way, as far as anyone knows.", + "10": "No. Nothing in the exchange tells Alice who sent a public number, so Mallory can sit in the middle. Authentication has to be added: Bob signs B, and Alice checks the signature with a certificate.", + "11": "Because Bob would then have to reverse the operation to get the key, and if Bob can reverse it, so can Eve: both receive the same 8. So each side needs its own private number, and an operation that lets each side finish the other's half without undoing it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2, and Bob computes 8^15 mod 23 = 2. The shared secret is 2.", + "13": "Multiplying powers adds the exponents: 8 × 19 mod 23 = 14 = 5^(6+15) mod 23. The secret needs the exponents multiplied, 5^(6×15) mod 23 = 2. No known way gets from A and B to s without, in effect, finding a or b.", + "14": "No. The explanation says plainly that no one has proven the discrete logarithm is hard, or that s can't be found from A and B some other way. The security rests on decades of failed attempts.", + "15": "Mallory can replace both public numbers with her own (for example 5^3 mod 23 = 10). She then shares one secret with Alice (6) and another with Bob (5). She decrypts each message, reads it, and re-encrypts it for the other side, and neither of them notices." + }, + "audit": { + "terms": [ + "The key and the shared secret are treated as the same thing, then not. Line 26: s 'becomes the key for ordinary encryption'. Scope (line 84): 'a hash function scrambles s first; a cipher such as AES then encrypts'. It is unclear whether s is the key or only the input to it.", + "Three names for the hard-to-undo step: 'scrambled form' / 'unscramble' (line 5), 'operation that is hard to reverse' (line 14) and 'one-way operation' (line 54). The term 'one-way' is only defined at the end.", + "Two names for A and B: 'scrambled numbers' (line 5) and 'public number' (line 23)." + ], + "unexplained": [ + "'about 4,000 such steps' for a 2048-bit exponent (line 60): it doesn't say why roughly twice the bit count, or how the step count would change with a bigger or smaller p.", + "'about 2^112 operations (an estimate)' (line 58): it doesn't say where this comes from or how it changes with the size of p, so the reader can't tell how much safety a bigger or smaller p would give.", + "'2048 bits' (line 58): bits aren't tied to the size of p beyond '617 digits', and it isn't said what a smaller size such as 1024 would mean." + ], + "missing_why": [ + "The first-idea example (line 14) gives 'The result is 8' without naming the operation (5^6 mod 23), so the reader can't check it.", + "Line 60: the square-and-multiply steps are shown, but not why doubling the exponent each step lets a huge exponent take only a few thousand steps.", + "Why p must be prime and g must 'produce many different values' is left to Scope (lines 21 and 84), even though line 21 already requires a prime.", + "Line 28: the argument that taking remainders along the way doesn't change the result is shown for one product; the extension to repeated powers is left implicit." + ], + "excess": [ + "The HTML comments ('' and the '' markers) mean nothing to this reader.", + "Scope bullets on elliptic curves and quantum/ML-KEM (lines 82, 85) aren't used by the argument or the questions." + ] + }, + "gaps": [ + "The operation in the first-idea example (line 14) is never named.", + "It's unclear whether s itself is the encryption key or is hashed first (line 26 vs line 84).", + "No derivation for the ~4,000 steps or the 2^112 estimate.", + "Why p must be prime is deferred and never explained." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2, + "14": 2, + "15": 2 + }, + "reasons": {}, + "pass": true + }, + "usage": { + "cost": 0.1816422, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The key and the shared secret are treated as the same thing, then not. Line 26: s 'becomes the key for ordinary encryption'. Scope (line 84): 'a hash function scrambles s first; a cipher such as AES then encrypts'. It is unclear whether s is the key or only the input to it." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Three names for the hard-to-undo step: 'scrambled form' / 'unscramble' (line 5), 'operation that is hard to reverse' (line 14) and 'one-way operation' (line 54). The term 'one-way' is only defined at the end." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "Two names for A and B: 'scrambled numbers' (line 5) and 'public number' (line 23)." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'about 4,000 such steps' for a 2048-bit exponent (line 60): it doesn't say why roughly twice the bit count, or how the step count would change with a bigger or smaller p." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 58): it doesn't say where this comes from or how it changes with the size of p, so the reader can't tell how much safety a bigger or smaller p would give." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'2048 bits' (line 58): bits aren't tied to the size of p beyond '617 digits', and it isn't said what a smaller size such as 1024 would mean." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "The first-idea example (line 14) gives 'The result is 8' without naming the operation (5^6 mod 23), so the reader can't check it." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 60: the square-and-multiply steps are shown, but not why doubling the exponent each step lets a huge exponent take only a few thousand steps." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Why p must be prime and g must 'produce many different values' is left to Scope (lines 21 and 84), even though line 21 already requires a prime." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 28: the argument that taking remainders along the way doesn't change the result is shown for one product; the extension to repeated powers is left implicit." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "The HTML comments ('' and the '' markers) mean nothing to this reader." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "Scope bullets on elliptic curves and quantum/ML-KEM (lines 82, 85) aren't used by the argument or the questions." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "The operation in the first-idea example (line 14) is never named." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "It's unclear whether s itself is the encryption key or is hashed first (line 26 vs line 84)." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "No derivation for the ~4,000 steps or the 2^112 estimate." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "Why p must be prime is deferred and never explained." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150625-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150625-coldread-prose.json new file mode 100644 index 0000000..ce48c5d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150625-coldread-prose.json @@ -0,0 +1,358 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:06:25", + "model": "default", + "prompt_sha": "a76b387677f2", + "reply": { + "paragraphs": [ + { + "at": "Title (line 1)", + "text": "How can two strangers agree on a secret key when anyone can read their messages?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold summary (line 3)", + "text": "They never send the key. Each person keeps a private number...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 5", + "text": "This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 1 (line 9)", + "text": "Alice and Bob want to encrypt their messages...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 2 (line 12)", + "text": "A first idea fails. Alice picks the key, 6, and sends 5^6 mod 23 = 8...", + "unresolved": [ + "mod (in \"5^6 mod 23 = 8\"; it is defined only at line 16)" + ], + "leap": [ + "\"Each side then does one half of a calculation, and the other side finishes it without undoing it\": what the 'halves' are stays unclear until the numbered steps" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, mod definition (line 16)", + "text": "\"mod 23\" means \"the remainder after division by 23\"...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 1 (line 19)", + "text": "Alice and Bob agree in public on two numbers: a prime modulus p = 23 and a base g = 5.", + "unresolved": [], + "leap": [ + "Why p has to be prime is not said here (Scope later touches on choosing g, not p)" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 2 (line 20)", + "text": "Each picks a private number at random...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 3 (line 21)", + "text": "Each sends a public number: g raised to their private number, mod p...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 4 (line 22)", + "text": "Each raises the other's public number to their own private number, mod p...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 24", + "text": "The result, s = 2, is the shared secret...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why both get 2 (line 26)", + "text": "Alice computed (5^15)^6, and Bob computed (5^6)^15...", + "unresolved": [ + "k in \"(x + 23k) × y\" is not introduced (it can be inferred as 'some whole number')" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What Eve sees, table (lines 31-35)", + "text": "Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Predict (line 37)", + "text": "Eve has 8 and 19. Can she combine them into 2?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 39", + "text": "Multiplying fails: 8 × 19 mod 23 = 152 mod 23 = 14...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why powers, and why mod, intro (line 43)", + "text": "Each choice blocks a simpler way for Eve to find a private number.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: Without mod (line 46)", + "text": "Without mod, A = 5^6 = 15625. Eve takes the logarithm base 5...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: multiplication instead of power (line 49)", + "text": "With multiplication instead of a power, A = 5 × 6 mod 23 = 7...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: powers mod p (line 50)", + "text": "With powers mod p, the results jump around...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 52", + "text": "So g^x mod p is a one-way operation... the discrete logarithm.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 1 (line 56)", + "text": "With p = 23, Eve can try exponents in order...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 58", + "text": "Alice's own work stays small...", + "unresolved": [ + "the arrow in \"25 → 2\" (it can be inferred to mean 'take the remainder mod 23')" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Assumption, not proof (line 61)", + "text": "No one has proven that the discrete logarithm is hard...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The limit, para 1 (line 66)", + "text": "The exchange stops a reader. It does not stop Mallory...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Mallory bullets (lines 68-69)", + "text": "Alice computes 10^6 mod 23 = 6... Bob computes 10^15 mod 23 = 5...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 71", + "text": "Alice shares 6 with Mallory, and Bob shares 5 with Mallory...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer (line 75)", + "text": "The two strangers never send the key...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Scope (lines 79-83)", + "text": "Authentication... Elliptic-curve... hash function... AES... Quantum computers", + "unresolved": [ + "certificates", + "Elliptic-curve", + "hash function", + "AES", + "Post-quantum" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Line 12, \"sends 5^6 mod 23 = 8\": 'mod' is used before line 16 defines it. The reader has to guess what the operation is right at the point where the explanation shows why one shared number fails. This is a mild block, since the point 'hard to reverse' still comes through." + ], + "cuts": [ + "Swap lines 12 and 16: put the 'mod 23 means remainder' definition before the 'first idea fails' paragraph. This is a move rather than a cut, and it costs no words.", + "No real cuts are needed. Every paragraph serves the argument." + ], + "question": { + "text": "How can two parties who share only a public channel, one that an eavesdropper can read, end up with the same secret key?", + "known_at": "Title (line 1), with the setting made concrete at line 9", + "result_stated": "Yes, in words at line 3: \"They never send the key. Each person keeps a private number and sends a public number computed from it... no known fast method gets the private numbers back from them.\"" + }, + "motive": "Reason to care: \"A version of it starts most HTTPS connections.\" (line 5). Reason for the approach: \"A first idea fails... If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number.\" (line 12). Also \"Each choice blocks a simpler way for Eve to find a private number.\" (line 43), followed by the three bullets.", + "close": "Yes. Line 75 reads: \"Each sends g raised to a private number, mod p. Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p.\"", + "recap": "Each side sends 5 raised to a secret power, remainder 23. Each then raises the other's number to its own secret power, so both reach 5^(ab). Eve can't do this because getting the exponent back out of the remainder is believed to be infeasibly slow.", + "strongest": "'Why powers, and why mod' (lines 43-52). It rules out plain powers (logarithm) and multiplication (an inverse via Euclid) with concrete numbers, then shows the scrambled table of 5^x mod 23.", + "weakest": "Line 12. It uses 'mod' four lines before defining it, and 'one half of a calculation... the other side finishes it' stays abstract until the steps arrive.", + "suggestions": [ + "Define 'mod' before line 12, or move the 'first idea fails' paragraph to after the definition.", + "Add a short clause saying why p is prime, for example so that the powers of g can cover many values.", + "Say in a few words that k in (x + 23k) is any whole number." + ] + }, + "usage": { + "cost": 0.1372968, + "seconds": 43 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Line 12, \"sends 5^6 mod 23 = 8\": 'mod' is used before line 16 defines it. The reader has to guess what the operation is right at the point where the explanation shows why one shared number fails. This is a mild block, since the point 'hard to reverse' still comes through." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Swap lines 12 and 16: put the 'mod 23 means remainder' definition before the 'first idea fails' paragraph. This is a move rather than a cut, and it costs no words." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "No real cuts are needed. Every paragraph serves the argument." + }, + { + "n": 4, + "severity": "edge", + "kind": "unresolved", + "text": "The problem, para 2 (line 12): mod (in \"5^6 mod 23 = 8\"; it is defined only at line 16)" + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "The problem, para 2 (line 12): \"Each side then does one half of a calculation, and the other side finishes it without undoing it\": what the 'halves' are stays unclear until the numbered steps" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Exchange, step 1 (line 19): Why p has to be prime is not said here (Scope later touches on choosing g, not p)" + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "Why both get 2 (line 26): k in \"(x + 23k) × y\" is not introduced (it can be inferred as 'some whole number')" + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Line 58: the arrow in \"25 → 2\" (it can be inferred to mean 'take the remainder mod 23')" + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): certificates" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): Elliptic-curve" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): hash function" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): AES" + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Scope (lines 79-83): Post-quantum" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150725-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150725-coldread-prose.json new file mode 100644 index 0000000..add31a0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150725-coldread-prose.json @@ -0,0 +1,344 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:07:25", + "model": "default", + "prompt_sha": "a76b387677f2", + "reply": { + "paragraphs": [ + { + "at": "Title + bold summary (line 1-3)", + "text": "How can two strangers agree... They never send the key. Each person keeps a private number...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 5", + "text": "This method is the Diffie–Hellman key exchange. A version of it starts most HTTPS connections.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 1 (line 9)", + "text": "Alice and Bob want to encrypt their messages...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 2 (line 11)", + "text": "\"mod 23\" means \"the remainder after division by 23\"...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The problem, para 3 (line 14)", + "text": "A first idea fails. Alice picks the key, 6, and sends 5^6 mod 23 = 8...", + "unresolved": [], + "leap": [ + "\"Each side then does one half of a calculation, and the other side finishes it without undoing it\": this is abstract, and it only becomes concrete in the next section. The reader has to take it on trust for now.", + "Why 5 and 23 are the numbers used is not said here. Step 1 of the next section explains it." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 1 (line 19)", + "text": "agree in public on two numbers: a prime modulus p = 23 and a base g = 5", + "unresolved": [], + "leap": [ + "The text never says why p has to be prime (Scope only touches on choosing g)." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 2 (line 20)", + "text": "Each picks a private number...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 3 (line 21)", + "text": "Alice sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Exchange, step 4 (line 22)", + "text": "Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 24", + "text": "The result, s = 2, is the shared secret...(see Scope)", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why both get 2 (line 26)", + "text": "Alice computed (5^15)^6, and Bob computed (5^6)^15...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What Eve sees, table (lines 31-35)", + "text": "Who | p, g | a = 6 | b = 15 | A = 8, B = 19 | s = 2", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Predict prompt (line 37)", + "text": "Eve has 8 and 19. Can she combine them into 2?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 39", + "text": "Multiplying fails: 8 × 19 mod 23 = 14...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why powers, intro (line 43)", + "text": "Each choice blocks a simpler way for Eve to find a private number.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: Without mod (line 46)", + "text": "A = 5^6 = 15625. Eve takes the logarithm base 5...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: multiplication instead of a power (line 49)", + "text": "A = 5 × 6 mod 23 = 7. Multiplying by 14 undoes multiplying by 5...", + "unresolved": [ + "Euclid's algorithm" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bullet: powers mod p (line 50)", + "text": "the results jump around. For exponents 1 to 22...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 52", + "text": "So g^x mod p is a one-way operation... discrete logarithm.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 1 (line 56)", + "text": "With p = 23, Eve can try exponents in order... about 10^14 years.", + "unresolved": [], + "leap": [ + "\"The best known methods are faster, but still need about 2^112 operations\": the methods are not named and the figure is not explained. It is labelled an estimate, which is acceptable." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why real numbers are huge, para 2 (line 58)", + "text": "Alice's own work stays small... about 4,000 steps.", + "unresolved": [], + "leap": [ + "\"a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications\": the example (6 = 3×2, 3 = 2+1) does not show how an arbitrary exponent breaks down into squarings and multiplications. The reader has to infer the link to the exponent's binary digits." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Assumption, not proof (line 61)", + "text": "No one has proven that the discrete logarithm is hard...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The limit, para 1 (line 66)", + "text": "The exchange stops a reader. It does not stop Mallory...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Mallory bullets (lines 68-69)", + "text": "Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 71", + "text": "Alice shares 6 with Mallory... (a digital signature).", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer (line 75)", + "text": "The two strangers never send the key...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Scope bullets (lines 79-83)", + "text": "Authentication... Elliptic-curve... hash function... AES... Quantum computers", + "unresolved": [ + "hash function", + "AES", + "Elliptic-curve", + "certificates", + "Post-quantum key exchange" + ], + "leap": [], + "excess": [ + "\"it is the version most HTTPS connections use\": this partly repeats line 5's \"A version of it starts most HTTPS connections\", though it adds which version." + ], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Scope line 80: drop \"it is the version most HTTPS connections use\" or merge it into line 5 (a small repeat). Little else can be cut without loss." + ], + "question": { + "text": "How can two people who have never met agree on a shared secret key when an eavesdropper reads every message they exchange?", + "known_at": "Title (line 1), made concrete in The problem (line 9)", + "result_stated": "Yes, in words at the very start: \"They never send the key. Each person keeps a private number and sends a public number computed from it... An eavesdropper sees only the public numbers, and no known fast method gets the private numbers back from them.\"" + }, + "motive": "Reason to care: \"A version of it starts most HTTPS connections\" (line 5) and \"Encryption needs a key that both of them know. They have never met, so every message goes over a public channel\" (line 9). Reason for the approach: the failed first idea (line 14: \"If Eve cannot get 6 back from 8, Bob cannot either. So each side needs its own private number\") and the section 'Why powers, and why mod' (\"Each choice blocks a simpler way for Eve to find a private number\").", + "close": "Yes. Answer (line 75): \"The two strangers never send the key. Each sends g raised to a private number, mod p. Each raises the other's public number to their own private number, and both get g^(ab) mod p. Eve sees only g^a and g^b mod p. To get further, Eve must find a or b, and no fast method is known for a large p.\"", + "recap": "Each side mixes a public base with its own secret exponent mod a prime and swaps the results. Raising the other side's result to your own secret gives you both the same number, and an eavesdropper can only get it by solving the discrete logarithm, which nobody knows how to do fast.", + "strongest": "'Why powers, and why mod' (lines 43-52): each alternative (no mod, multiplication) is shown failing with concrete numbers, and then the scrambled list of 5^x mod 23 makes 'one-way' visible.", + "weakest": "Line 58: the claim 'about 2048 squarings plus at most 2048 multiplications' rests on a single small example (exponent 6) and never says how a general exponent splits into squarings and multiplications.", + "suggestions": [ + "In line 58, add one sentence: write the exponent in binary; square for each digit, and also multiply by g when the digit is 1.", + "Briefly say why p is prime (so that every nonzero remainder has an undoing multiplier, and powers of a good g run through all of them).", + "Name the faster methods in line 56 (for example, the number field sieve), or say plainly that they are beyond scope." + ] + }, + "usage": { + "cost": 0.143897, + "seconds": 44 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Scope line 80: drop \"it is the version most HTTPS connections use\" or merge it into line 5 (a small repeat). Little else can be cut without loss." + }, + { + "n": 2, + "severity": "edge", + "kind": "leap", + "text": "The problem, para 3 (line 14): \"Each side then does one half of a calculation, and the other side finishes it without undoing it\": this is abstract, and it only becomes concrete in the next section. The reader has to take it on trust for now." + }, + { + "n": 3, + "severity": "edge", + "kind": "leap", + "text": "The problem, para 3 (line 14): Why 5 and 23 are the numbers used is not said here. Step 1 of the next section explains it." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "Exchange, step 1 (line 19): The text never says why p has to be prime (Scope only touches on choosing g)." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Bullet: multiplication instead of a power (line 49): Euclid's algorithm" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Why real numbers are huge, para 1 (line 56): \"The best known methods are faster, but still need about 2^112 operations\": the methods are not named and the figure is not explained. It is labelled an estimate, which is acceptable." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Why real numbers are huge, para 2 (line 58): \"a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications\": the example (6 = 3×2, 3 = 2+1) does not show how an arbitrary exponent breaks down into squarings and multiplications. The reader has to infer the link to the exponent's binary digits." + }, + { + "n": 8, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): hash function" + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): AES" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): Elliptic-curve" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): certificates" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Scope bullets (lines 79-83): Post-quantum key exchange" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150823-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150823-quiz-prose.json new file mode 100644 index 0000000..1f72e87 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150823-quiz-prose.json @@ -0,0 +1,188 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:08:10", + "model": "default", + "prompt_sha": "bedad2b89e75", + "reply": { + "answers": { + "1": "Alice and Bob, the two parties. Eve, who reads every message but changes none. Mallory, who can replace messages. The public prime modulus p (23) and the public base g (5). The private numbers a and b. The public numbers A = g^a mod p and B = g^b mod p. The shared secret s = g^(ab) mod p, which becomes the encryption key. The 'mod' (remainder) operation. The one-way operation g^x mod p, whose reverse is the discrete logarithm. Authentication (digital signatures).", + "2": "Everyone, including Eve, knows p and g. Each party picks a private number and computes a public number from it with g^x mod p. The public numbers travel in the open. Each party raises the other's public number to its own private number, mod p, and both arrive at s. Eve sees p, g, A and B but not a, b or s. To get s she would in effect have to find a or b, which is the discrete logarithm problem. Mallory can slip in his own public number, because nothing in the exchange proves who sent a public number. Authentication fixes this.", + "3": "Each side sends g^(private) mod p. Each raises the other's value to its own private number, and since (g^b)^a = (g^a)^b = g^(ab), both get the same s. Eve would have to undo the mod-p power to find a or b, and no fast method for that is known when p is large.", + "4": "The key variable is the size of p (and of a and b). With a small p, like 23, Eve can try exponents one by one and finds a = 6 on the 6th try. With a 2048-bit p and a and b chosen at random from that range, trying them one by one is hopeless. The best known methods need about 2^112 operations, roughly 10^14 years at a trillion operations per second. Alice's own work grows only to about 4,000 steps, because repeated squaring doubles the exponent at each step. So a larger p makes Eve's job vastly harder while the honest parties' work grows only slowly.", + "5": "Assumption: the discrete logarithm is hard, and Eve cannot get s from A and B some other way. This is not proven; it rests on decades of failed attempts. Simplification: Eve only reads and never changes messages. Established fact: (x^a)^b = x^(ab), and taking remainders along the way does not change the final remainder. Another fact: with plain powers (no mod), a logarithm recovers the exponent.", + "6": "Alice and Bob agree in public on a prime p and a base g, and each secretly picks a private number. Each sends g raised to that private number, mod p. Each raises the number they receive to their own private number, mod p, and the exponent rule makes both results g^(ab) mod p, the shared secret. An eavesdropper sees only g^a and g^b mod p. Getting a or b back is the discrete logarithm problem, which is believed (but not proven) to be infeasible for a 2048-bit p, while the forward computation stays cheap. The exchange does not stop an attacker who replaces messages, so real systems add digital signatures.", + "7": "The hardest part was the argument that taking remainders along the way does not change the result, '(x + 23k) × y = ...', and how it extends to powers. A worked example showing (5^15 mod 23)^6 mod 23 equal to 5^90 mod 23 would help. The squaring step count ('2048 squarings plus at most 2048 multiplications') was also terse. It needed a word on reading the exponent in binary.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8, which matches 5^6 mod 23 = 8 in the table.", + "9": "Without mod, A = 5^6 = 15625, and Eve takes the logarithm base 5 to get 6, because plain powers grow in order. With mod, the results jump around (5, 2, 10, 4, 20, 8, ...), so a result gives no hint of where the exponent lies. No fast reverse is known. Multiplication mod p would also fail, because Euclid's algorithm quickly finds the number that undoes it.", + "10": "No. Nothing in the exchange tells Alice who sent a public number. Mallory can substitute his own. Real systems add authentication, such as digital signatures and certificates.", + "11": "If Alice sends the key through a one-way operation (for example 5^6 mod 23 = 8), Bob receives the same 8 that Eve does. If Eve cannot recover 6 from 8, Bob cannot either. So each side needs its own private number, and each completes the other's half without undoing it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2, and Bob computes 8^15 mod 23 = 2. The shared secret is 2.", + "13": "8 × 19 mod 23 = 14. Multiplying powers adds the exponents, so this gives 5^(6+15) mod 23. The shared secret needs the exponents multiplied, 5^(6×15). No known way gets from A and B to s without, in effect, finding a or b.", + "14": "No. It is an assumption, not a proof. No one has proven that the discrete logarithm is hard, or that there is no other route from A and B to s. Confidence rests on decades of failed attempts.", + "15": "Mallory replaces both public numbers with his own (5^3 mod 23 = 10). He then shares one secret with Alice (6) and a different one with Bob (5). He decrypts each message, reads it, and re-encrypts it for the other side, and neither party notices. Authentication prevents this." + }, + "audit": { + "terms": [ + "The number 6 is called 'the key' in the first-idea paragraph (line 14: 'Alice picks the key, 6'). Later the same 6 is 'a private number' (line 20: 'Alice picks a = 6'), and the key is something built from s. So 'key' means two different things.", + "'shared secret' (line 24) vs 'key' (lines 24 and 82: 'build the key ... from it'; 'use the output as the key'). The difference is only explained in Scope.", + "'p' / 'prime modulus' (line 19) vs 'mod 23' (line 11). The link between 'modulus' and the 'mod' notation is never stated outright.", + "'undoing number' (line 49) is an unnamed inverse. It is fine, but it is a separate term for the same idea as 'undoes'." + ], + "unexplained": [ + "'about 2^112 operations (an estimate)' (line 56): it does not say where the figure comes from or how it changes with the size of p.", + "'2048 bits, about 617 decimal digits' (line 56): it does not say why 2048, or what a smaller p would mean for security.", + "'about 4,000 steps' (line 58): there is no comparison showing how small this is next to 2^112." + ], + "missing_why": [ + "Line 58: 'Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications'. It does not explain how an arbitrary exponent is built from squarings and multiplications (its binary digits).", + "Line 56: 'The best known methods are faster' gives no hint of why they beat trying exponents one by one, or why they still fail.", + "Line 19 / line 82: why p must be prime is deferred to Scope and only briefly justified there ('lets a good g reach every non-zero remainder').", + "Line 50: 'the results jump around ... A result near 8 says nothing about where 6 is' asserts that the outputs are unpredictable. It gives no reason why mod causes this beyond the single example." + ], + "excess": [ + "Line 80: Scope's elliptic-curve paragraph; no question or step of the argument uses it.", + "Line 83: the quantum-computer paragraph; it is a detour from the argument.", + "Line 75: the 'Answer' section largely repeats the bold summary at line 3." + ] + }, + "gaps": [ + "The explanation never shows that the exchange gives the right answer when you reduce mod p along the way (for example, a check that 5^90 mod 23 = 2). The reader has to trust the general argument.", + "How Euclid's algorithm finds the undoing number is not shown. That is acceptable, but the reader has to take it on trust.", + "How the shared secret becomes an encryption key (the hash function) is only named in Scope." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 1, + "13": 2, + "14": 2, + "15": 2 + }, + "reasons": { + "12": "The numbers are correct (A = 8, B = 19, s = 2), but the expected statement is missing: Alice computes B^a mod p, Bob computes A^b mod p, and both equal g^(ab) mod p because (g^a)^b = (g^b)^a and reducing mod p at each step does not change the remainder. The rubric says a case alone is not required for 2, so a case without the principle earns only partial credit. Note for the explainer: the question asks only for numbers while the expect asks for the reason, so the item wording may need fixing (explainer decide).", + "13": "Full credit, narrowly. The answer does not name the discrete logarithm problem. It does state that no known way gets from A and B to s without in effect finding a or b, which is the substance of the expect." + }, + "pass": false + }, + "usage": { + "cost": 0.1955264, + "seconds": 46 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The number 6 is called 'the key' in the first-idea paragraph (line 14: 'Alice picks the key, 6'). Later the same 6 is 'a private number' (line 20: 'Alice picks a = 6'), and the key is something built from s. So 'key' means two different things." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'shared secret' (line 24) vs 'key' (lines 24 and 82: 'build the key ... from it'; 'use the output as the key'). The difference is only explained in Scope." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'p' / 'prime modulus' (line 19) vs 'mod 23' (line 11). The link between 'modulus' and the 'mod' notation is never stated outright." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'undoing number' (line 49) is an unnamed inverse. It is fine, but it is a separate term for the same idea as 'undoes'." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 56): it does not say where the figure comes from or how it changes with the size of p." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'2048 bits, about 617 decimal digits' (line 56): it does not say why 2048, or what a smaller p would mean for security." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 4,000 steps' (line 58): there is no comparison showing how small this is next to 2^112." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 58: 'Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications'. It does not explain how an arbitrary exponent is built from squarings and multiplications (its binary digits)." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 56: 'The best known methods are faster' gives no hint of why they beat trying exponents one by one, or why they still fail." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 19 / line 82: why p must be prime is deferred to Scope and only briefly justified there ('lets a good g reach every non-zero remainder')." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Line 50: 'the results jump around ... A result near 8 says nothing about where 6 is' asserts that the outputs are unpredictable. It gives no reason why mod causes this beyond the single example." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "Line 80: Scope's elliptic-curve paragraph; no question or step of the argument uses it." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Line 83: the quantum-computer paragraph; it is a detour from the argument." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "Line 75: the 'Answer' section largely repeats the bold summary at line 3." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "The explanation never shows that the exchange gives the right answer when you reduce mod p along the way (for example, a check that 5^90 mod 23 = 2). The reader has to trust the general argument." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "How Euclid's algorithm finds the undoing number is not shown. That is acceptable, but the reader has to take it on trust." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "How the shared secret becomes an encryption key (the hash function) is only named in Scope." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150941-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150941-quiz-prose.json new file mode 100644 index 0000000..57cb5d6 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T150941-quiz-prose.json @@ -0,0 +1,180 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:09:31", + "model": "default", + "prompt_sha": "97e170ce48d4", + "reply": { + "answers": { + "1": "Alice and Bob, who want a shared key. Eve, who reads every message but changes none. Mallory, who can replace messages. The public prime modulus p and the public base g. The private numbers a and b. The public numbers A = g^a mod p and B = g^b mod p. The shared secret s = g^(ab) mod p. A hash function that turns s into the encryption key. The operation g^x mod p, which is one-way, and its reverse problem, the discrete logarithm.", + "2": "p and g are public. Each side keeps its own private number and sends the matching public number, A = g^a mod p or B = g^b mod p. Each side then raises the other side's public number to its own private number, mod p, and both get s. s goes through a hash to become the key. Eve sees p, g, A and B, but not a, b or s. To get s she would have to undo g^x mod p, which is the discrete logarithm problem, and no fast method for it is known. Mallory can swap the public numbers. Authentication (digital signatures) is what stops that.", + "3": "Each side combines its own private exponent with the other side's public value, and (g^b)^a = (g^a)^b = g^(ab), so both reach the same s. An eavesdropper only has g^a and g^b. Combining those gives g^(a+b), not g^(ab), and getting a or b back needs a discrete logarithm, which has no known fast method.", + "4": "The most important variable is the size of p, together with a and b chosen at random across that range. With p = 23, Eve finds a = 6 on her 6th try. With a 2048-bit p, the best known methods need about 2^112 operations, roughly 10^14 years at a trillion operations per second. Alice's own work only grows to about 4,000 steps, because repeated squaring doubles the exponent each time. So making p bigger hurts Eve enormously and costs Alice little. A second variable is the choice of g: with g = 22 the powers mod 23 are only 1 and 22, so s has only two possible values.", + "5": "Assumption: the discrete logarithm is hard and Eve has no other way to get s. This is not proven; it rests on decades of failed attempts. Simplification: Eve only reads and never changes messages. Established facts: (x^a)^b = x^(ab), and taking remainders along the way does not change the final remainder, since (x+23k)y = xy + 23ky. Also, Euclid's algorithm quickly finds the number that undoes a multiplication mod p.", + "6": "Alice and Bob agree in public on a prime p and a base g. Each picks a secret exponent and sends g raised to it, mod p. Each raises the number they receive to their own secret exponent, and by the exponent rule both get g^(ab) mod p, which they hash into a key. Eve sees only g^a and g^b. Multiplying them gives g^(a+b), and finding a or b is the discrete logarithm, which has no known fast method when p is huge. That hardness is assumed, not proven, and the exchange does not stop an attacker who swaps messages unless signatures are added.", + "7": "The hardest part was the 'With multiplication instead of a power' bullet. It uses Euclid's algorithm and the undoing number 14 as a black box. Showing how 14 is found, or at least saying that such a number exists for any g when p is prime, would help. Also, the 'first idea' paragraph calls 5^6 mod 23 'hard to reverse' before one-way operations are explained.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. Their shared secret is 8, which matches 5^6 mod 23 = 8 in the table.", + "9": "With plain powers, Eve takes the logarithm: log base 5 of 15625 is 6. Plain powers grow in order, so the logarithm reverses them. Mod p scrambles the results (5, 2, 10, 4, 20, 8, …), so a result gives no hint of where the exponent is. Powers rather than multiplication are used because multiplication mod p can be undone quickly by multiplying by an inverse found with Euclid's algorithm.", + "10": "No. Nothing in the exchange says who sent a public number. Mallory can replace both public numbers and end up sharing one secret with Alice and another with Bob. Real systems add authentication, such as digital signatures.", + "11": "Because whatever Bob can undo, Eve can undo too. They receive the same message. If the operation really is one-way, Bob cannot recover the key either. So each side needs its own private number, and each completes the other's half of the calculation without having to undo it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2, and Bob computes 8^15 mod 23 = 2, so s = 2. They must be equal because (5^15)^6 = (5^6)^15 = 5^(90), and taking remainders at intermediate steps does not change the final remainder.", + "13": "8 × 19 mod 23 = 14, which is 5^(6+15): multiplying powers adds the exponents. The shared secret needs the exponents multiplied, 5^(6×15). No known route gets from A and B to s without, in effect, finding a or b first.", + "14": "No. It is an assumption, not a proof. No one has proven that the discrete logarithm is hard, or that there is no other way to get s from A and B. The confidence comes from decades of failed attempts.", + "15": "A man-in-the-middle attack. Mallory replaces both public numbers with her own (10, from her private number 3). She then shares 6 with Alice and 5 with Bob, decrypts every message, reads it, and encrypts it again for the other side. Neither Alice nor Bob can detect this without authentication." + }, + "audit": { + "terms": [ + "The 'first idea' paragraph (line 14) says 'Alice picks the key, 6'. The exchange (line 20) then uses 6 as the private number a. Later, 'key' means the hash output of s (line 24). So 'key' names two different things, and the reused value 6 makes them easy to confuse.", + "'shared secret' (s, line 24) and 'key' (line 24, the hashed output) are kept apart in the text, but the Answer section and the title use 'secret key' loosely.", + "'hard to reverse' (line 14), 'one-way operation' (line 52) and 'no known fast reverse' (line 52) name the same idea. The first phrase is used before the concept is defined." + ], + "unexplained": [ + "'about 2^112 operations (an estimate)' (line 56): the text does not say what makes this enough, or what a smaller count such as 2^80 would mean for security.", + "'It needs smaller numbers for the same security' (line 80): it does not say how much smaller or why." + ], + "missing_why": [ + "Line 49: Euclid's algorithm 'finds this undoing number even for a huge p'. The text gives neither how it does this nor why the undoing number always exists.", + "Line 24: it does not say why s is passed through a hash function instead of being used directly as the key.", + "Line 50: it does not say why scrambled-looking outputs imply that no fast reverse exists. 'A result near 8 says nothing about where 6 is' is an intuition, not a reason.", + "Line 82: it does not say why p must be prime for a good g to reach every non-zero remainder.", + "Line 56: there is no hint of why the best known methods are faster than trying exponents one by one." + ], + "excess": [ + "Line 5: 'A version of it starts most HTTPS connections' is context that the argument does not use.", + "Scope bullets on elliptic curves, AES and quantum computers (lines 80, 81, 83) are detours that no question uses." + ] + }, + "gaps": [ + "Euclid's algorithm and the modular inverse are used as a black box.", + "The first-idea paragraph assumes one-way hardness before explaining it.", + "No reason is given for hashing s.", + "The 2^112 security figure has no reference point." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 1, + "14": 2, + "15": 2 + }, + "reasons": { + "13": "The answer correctly shows that A × B mod 23 = 14 = 5^(6+15), which is not s. It also says Eve would in effect have to find a or b. It never says that finding a or b is the discrete logarithm problem, or that no fast method for it is known. That missing step is the core of the expected answer." + }, + "pass": false + }, + "usage": { + "cost": 0.18770360000000003, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The 'first idea' paragraph (line 14) says 'Alice picks the key, 6'. The exchange (line 20) then uses 6 as the private number a. Later, 'key' means the hash output of s (line 24). So 'key' names two different things, and the reused value 6 makes them easy to confuse." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'shared secret' (s, line 24) and 'key' (line 24, the hashed output) are kept apart in the text, but the Answer section and the title use 'secret key' loosely." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'hard to reverse' (line 14), 'one-way operation' (line 52) and 'no known fast reverse' (line 52) name the same idea. The first phrase is used before the concept is defined." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 56): the text does not say what makes this enough, or what a smaller count such as 2^80 would mean for security." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'It needs smaller numbers for the same security' (line 80): it does not say how much smaller or why." + }, + { + "n": 6, + "severity": "gap", + "kind": "missing_why", + "text": "Line 49: Euclid's algorithm 'finds this undoing number even for a huge p'. The text gives neither how it does this nor why the undoing number always exists." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 24: it does not say why s is passed through a hash function instead of being used directly as the key." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 50: it does not say why scrambled-looking outputs imply that no fast reverse exists. 'A result near 8 says nothing about where 6 is' is an intuition, not a reason." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 82: it does not say why p must be prime for a good g to reach every non-zero remainder." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 56: there is no hint of why the best known methods are faster than trying exponents one by one." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "Line 5: 'A version of it starts most HTTPS connections' is context that the argument does not use." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "Scope bullets on elliptic curves, AES and quantum computers (lines 80, 81, 83) are detours that no question uses." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "Euclid's algorithm and the modular inverse are used as a black box." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The first-idea paragraph assumes one-way hardness before explaining it." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "No reason is given for hashing s." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "The 2^112 security figure has no reference point." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151044-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151044-quiz-prose.json new file mode 100644 index 0000000..eb0091f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151044-quiz-prose.json @@ -0,0 +1,187 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:10:36", + "model": "default", + "prompt_sha": "97e170ce48d4", + "reply": { + "answers": { + "1": "The main objects are Alice and Bob, Eve (who only reads messages) and Mallory (who can change them). There are two public numbers everyone sees: a prime modulus p and a base g. Each side has a private number (a for Alice, b for Bob) and a public number (A = g^a mod p, B = g^b mod p). Both sides end up with a shared secret s = g^(ab) mod p, which goes through a hash function to make the encryption key. The other objects are the one-way operation g^x mod p, the discrete logarithm (the reverse problem) and authentication (digital signatures).", + "2": "p and g are public. Each side's public number is made from its own private number using p and g. Each side then raises the other side's public number to its own private number, mod p, to get s. Hashing s gives the key. Eve sees p, g, A and B, but not a, b or s. To get s she would in effect have to solve the discrete logarithm for a or b. Mallory can put his own public number in place of A and B. Authentication is what stops him.", + "3": "Each side sends g raised to its private number mod p. Raising the other side's value to your own private number gives g^(ab) mod p for both, because of the exponent rule and because taking remainders along the way doesn't change the result. Eve, who sees only g^a and g^b mod p, would have to reverse the one-way operation, and no fast method for that is known.", + "4": "The key variable is the size of p (and of the range a and b are picked from). With p = 23, Eve can try exponents in order and finds a = 6 on the 6th try. With a 2048-bit p and random a and b, trying them one by one is hopeless. Even the best known methods need about 2^112 operations, roughly 10^14 years at a trillion operations per second. Alice's own work grows only slowly, about 4,000 steps for a 2048-bit exponent, because each squaring doubles the exponent. Another variable is the choice of g. A bad g (22 for p = 23) gives only two possible values for s.", + "5": "Assumption: no one has proven the discrete logarithm is hard, or that s can't be found from A and B some other way. Security rests on decades of failed attempts. Also, Eve is assumed to only read messages, never change them. Established facts: the exponent rule (x^a)^b = x^(ab), and the fact that taking remainders along the way doesn't change the final remainder. The text shows this with (x + 23k)y = xy + 23ky. In the example, 19^6 mod 23 = 8^15 mod 23 = 2.", + "6": "Alice and Bob agree in public on a prime p and a base g. Each secretly picks a random private number and sends g to that power, mod p. Each raises what they receive to their own private number, mod p, and both get g^(ab) mod p, which they hash to make the key. Eve sees only g^a and g^b mod p. Multiplying them only adds the exponents, and getting a back is the discrete logarithm, which has no known fast method when p is huge (an assumption, not a proof). The exchange doesn't stop someone who replaces messages, so real systems add digital signatures for authentication.", + "7": "The hardest part was the claim that powers mod p are hard to reverse. The text says the next section 'shows why no fast method for it is known', but it only lists values that 'jump around' and says 'a result near 8 says nothing about where 6 is'. That is intuition, not a reason. It would have helped to say plainly that this shows why the easy reversals (logarithm, undoing a multiplication) don't work, not that no other method exists. The '2^112 (an estimate)' figure also needed a word on where it comes from.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8 (= 5^6 mod 23, matching the table).", + "9": "With plain powers, A = 5^6 = 15625, and Eve just takes log base 5 to get 6, because plain powers grow in order. Mod p makes the results jump around, so the size of the result says nothing about the exponent, and no fast reverse is known. Plain multiplication mod p doesn't work either, because Euclid's algorithm quickly finds the number that undoes it.", + "10": "No. The exchange stops someone who only reads, but nothing in it tells Alice who sent a public number. Authentication (a digital signature on B) is needed.", + "11": "If Alice sends one-way(key), Bob can't recover the key any more than Eve can, because both receive exactly the same thing. So each side needs its own private number. Each side does half the calculation, and the other finishes it without undoing anything.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2 and Bob computes 8^15 mod 23 = 2, so s = 2. They must be equal because (5^15)^6 = (5^6)^15 = 5^(90) by the exponent rule. Reducing mod 23 along the way removes only multiples of 23 at each multiplication, so the final remainder doesn't change.", + "13": "Multiplying powers adds the exponents: 8 × 19 mod 23 = 14 = 5^(6+15) mod 23. The secret needs the exponents multiplied, 5^(6×15) mod 23 = 2. There is no known way from A and B to s other than, in effect, finding a or b first.", + "14": "No. It is an assumption backed by decades of failed attempts, not a proof. The text also says a large quantum computer could solve the discrete logarithm fast.", + "15": "Mallory can put his own public number (for example 5^3 mod 23 = 10) in place of both A and B. He then shares one secret with Alice (6) and another with Bob (5). He decrypts each message, reads it, and re-encrypts it for the other side, and neither of them notices. Authentication with digital signatures prevents this." + }, + "audit": { + "terms": [ + "Line 14 calls 6 'a key' ('Alice chooses a key, the number 6'). In line 20 the same 6 is 'a private number', and from line 24 'key' means the hash of s. So one word covers two different things.", + "Line 14 says 'hard to reverse' and line 52 says 'one-way operation'. Line 39 calls the reverse problem 'the discrete logarithm'. It's the same idea, but the formal name comes after the informal phrase has been used twice.", + "Line 19 says 'modulus p' and also '(the number after \"mod\")', while line 11 says 'mod 23'. That's fine, but the reader has to connect 23 = p themselves before line 19." + ], + "unexplained": [ + "Line 56 says 'about 2^112 operations (an estimate)' without saying where the estimate comes from, or how it changes for a bigger or smaller p.", + "Line 56 says 'a p of 2048 bits, about 617 decimal digits' without saying why 2048 was chosen, or what a smaller p would give up.", + "Line 80 says ECDH 'needs smaller numbers for the same security' but gives no measure of how much smaller." + ], + "missing_why": [ + "Line 39 says 'the next section shows why no fast method for it is known'. Line 50 ('A result near 8 says nothing about where 6 is') only rules out the logarithm shortcut. It doesn't explain why other methods fail, and the 'best known methods are faster' in line 56 are never described.", + "Line 24 says Alice and Bob 'feed s into the same hash function' but doesn't say why s isn't used as the key directly.", + "Line 19 requires a prime modulus without saying why. The reason (a good g reaches every non-zero remainder) appears only in the Scope list at line 82.", + "Line 58 says '2048 squarings plus at most 2048 multiplications' but doesn't show how the exponent's binary digits decide when to multiply. The 5^6 example skips that step.", + "Line 56 says a and b are picked 'at random from that whole range' without saying why being random matters, beyond making one-by-one trying hopeless." + ], + "excess": [ + "Lines 79–83 cover Scope items (AES, elliptic curves, quantum computers) that the argument and questions don't use.", + "Line 5 ('A version of it starts most HTTPS connections') is context the argument doesn't use.", + "Line 75 (Answer) repeats the summary at line 3 almost word for word." + ] + }, + "gaps": [ + "The explanation doesn't actually show why the discrete logarithm is hard. It only shows that the simple reversals fail, even though it promises to show why no fast method is known.", + "There's no explanation of where the 2^112 figure comes from, or what the 'best known methods' are.", + "The reason for using a prime p is left to the Scope section." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 1, + "11": 2, + "12": 2, + "13": 2, + "14": 2, + "15": 2 + }, + "reasons": { + "10": "The answer correctly says 'No', notes that the exchange only stops a reader, and says authentication is needed. It never states the attack: an active attacker can replace A and B with his own public numbers and run a separate exchange with each side. Saying 'nothing tells Alice who sent a public number' points at the gap but does not say what an attacker does with it. Partial." + }, + "pass": false + }, + "usage": { + "cost": 0.1907202, + "seconds": 43 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "Line 14 calls 6 'a key' ('Alice chooses a key, the number 6'). In line 20 the same 6 is 'a private number', and from line 24 'key' means the hash of s. So one word covers two different things." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Line 14 says 'hard to reverse' and line 52 says 'one-way operation'. Line 39 calls the reverse problem 'the discrete logarithm'. It's the same idea, but the formal name comes after the informal phrase has been used twice." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "Line 19 says 'modulus p' and also '(the number after \"mod\")', while line 11 says 'mod 23'. That's fine, but the reader has to connect 23 = p themselves before line 19." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "Line 56 says 'about 2^112 operations (an estimate)' without saying where the estimate comes from, or how it changes for a bigger or smaller p." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 56 says 'a p of 2048 bits, about 617 decimal digits' without saying why 2048 was chosen, or what a smaller p would give up." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 80 says ECDH 'needs smaller numbers for the same security' but gives no measure of how much smaller." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 39 says 'the next section shows why no fast method for it is known'. Line 50 ('A result near 8 says nothing about where 6 is') only rules out the logarithm shortcut. It doesn't explain why other methods fail, and the 'best known methods are faster' in line 56 are never described." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Line 24 says Alice and Bob 'feed s into the same hash function' but doesn't say why s isn't used as the key directly." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 19 requires a prime modulus without saying why. The reason (a good g reaches every non-zero remainder) appears only in the Scope list at line 82." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 58 says '2048 squarings plus at most 2048 multiplications' but doesn't show how the exponent's binary digits decide when to multiply. The 5^6 example skips that step." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Line 56 says a and b are picked 'at random from that whole range' without saying why being random matters, beyond making one-by-one trying hopeless." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "Lines 79–83 cover Scope items (AES, elliptic curves, quantum computers) that the argument and questions don't use." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Line 5 ('A version of it starts most HTTPS connections') is context the argument doesn't use." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "Line 75 (Answer) repeats the summary at line 3 almost word for word." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "The explanation doesn't actually show why the discrete logarithm is hard. It only shows that the simple reversals fail, even though it promises to show why no fast method is known." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "There's no explanation of where the 2^112 figure comes from, or what the 'best known methods' are." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The reason for using a prime p is left to the Scope section." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151159-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151159-quiz-prose.json new file mode 100644 index 0000000..b867901 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151159-quiz-prose.json @@ -0,0 +1,194 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:11:47", + "model": "default", + "prompt_sha": "97e170ce48d4", + "reply": { + "answers": { + "1": "Alice and Bob (the two parties), Eve (an eavesdropper who reads but does not change messages), Mallory (an attacker who replaces messages), the public numbers p (a prime modulus) and g (the base), the private numbers a and b, the public numbers A = g^a mod p and B = g^b mod p, the shared secret s = g^(ab) mod p, the hash function that turns s into the encryption key, the one-way operation g^x mod p and its hard reverse (the discrete logarithm), and authentication (digital signatures).", + "2": "p and g are agreed in public. Each party keeps a private number and uses it to compute a public number: A = g^a mod p and B = g^b mod p. These are sent openly, so Eve sees p, g, A and B. Each party raises the other's public number to their own private number mod p. Both get the same s, which is hashed into the key. Eve could only get s by finding a or b from A or B, which is the discrete logarithm. No fast method for it is known. Mallory can swap in her own public number and end up sharing one secret with each side. Authentication prevents this.", + "3": "Each side sends g^private mod p. Raising the other side's value to your own private number gives (g^b)^a = (g^a)^b = g^(ab) mod p, so both get the same secret. Eve sees only g^a and g^b, and multiplying them only adds the exponents. To get s she would have to undo the one-way operation, which no known fast method can do.", + "4": "The most important variable is the size of p, together with the range a and b are drawn from. With a small p (23), Eve tries exponents one by one and finds a = 6 on the 6th try. With a 2048-bit p and random a and b from that whole range, trying one by one is hopeless. Even the best known methods need about 2^112 operations, around 10^14 years at a trillion operations per second. Alice's own work grows only with the number of bits (about 4,000 steps), because each squaring doubles the exponent. Separately, a bad g (e.g. 22) makes s take only two possible values.", + "5": "Assumption: the discrete logarithm is hard, and there is no other way to get s from A and B. The explanation says this is not proven and rests on decades of failed attempts. A simplification is that Eve only reads and never changes messages. Established fact: with p = 23 and g = 5, 19^6 mod 23 = 8^15 mod 23 = 2. Also, (x^a)^b = x^(ab), and taking remainders along the way does not change the final remainder.", + "6": "Alice and Bob publicly agree on a prime p and a base g, and each secretly picks a private number. Each sends g raised to their private number, mod p (the remainder after dividing by p). Each raises the number they receive to their own private number mod p, and both get g^(ab) mod p, which they hash into an encryption key. An eavesdropper who sees only g^a and g^b cannot combine them (multiplying adds the exponents instead of multiplying them). Undoing g^x mod p (the discrete logarithm) has no known fast method for huge p, although this is not proven. The exchange does not stop someone who replaces messages, so real systems add digital signatures.", + "7": "The 'Why real numbers are huge' section was hardest. The 2^112 figure appears without any link to the 2048-bit size. 'Best known methods are faster' is not explained even briefly. The square-and-multiply example (5^2, then 5^3, then 5^6) does not say why the multiply step is needed or how you pick the order of steps from the exponent's bits. One sentence on writing the exponent in binary would help. The remainder argument in 'Why both get 2' is short but works.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8, and Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8, which equals 5^6 mod 23 = 8 in the explanation's table.", + "9": "Plain powers grow in order, so Eve could take the logarithm base 5 of A (e.g. log5 of 15625 = 6) and recover the private number. Taking remainders mod p makes the results jump around (5, 2, 10, 4, 20, 8, ...), so a result near 8 says nothing about where the exponent is. No fast reverse is known: it is a one-way operation.", + "10": "No. Nothing in the exchange tells Alice who sent a public number. Mallory can replace both public numbers. Authentication (digital signatures, and in HTTPS certificates) is needed.", + "11": "Whatever Alice sends, Bob would have to turn it back into the key. If Eve cannot reverse the one-way operation, Bob cannot either, because he has the same information Eve has. So each side needs its own private number. Each does half the calculation, and the other side finishes it without undoing it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2 and Bob computes 8^15 mod 23 = 2, so s = 2. They must be equal because Alice computes (5^15)^6 and Bob computes (5^6)^15, and both equal 5^(6×15) by the exponent rule. Taking remainders along the way only removes multiples of 23: (x + 23k)·y = xy + 23ky, and a power is a chain of products. So the final remainder is the same.", + "13": "Multiplying powers adds the exponents: 8 × 19 mod 23 = 14 = 5^(6+15) mod 23. The secret needs the exponents multiplied, 5^(6×15) mod 23 = 2. No known way from A and B to s avoids, in effect, finding a or b first.", + "14": "No. It is an assumption. No one has proven that the discrete logarithm is hard, or that s cannot be found from A and B some other way. The belief rests on decades of failed attempts. A large quantum computer could solve the discrete logarithm fast.", + "15": "Run a man-in-the-middle attack. Mallory picks her own private number (3) and sends 10 = 5^3 mod 23 to both sides in place of their public numbers. She then shares 6 with Alice and 5 with Bob. She decrypts each message, reads it, and re-encrypts it for the other side, and neither side notices. Only authentication stops this." + }, + "audit": { + "terms": [ + "The same number is called 'key' and later 'private number': line 14 'Alice chooses a key, the number 6' versus line 20 'private number ... Alice picks a = 6'. The reader may think the private number is the key, while line 24 says the key is the hashed output of s.", + "'shared secret' (line 24) versus 'key' (lines 9, 24, 75 'never send the key'). The answer says they 'never send the key', but what they agree on directly is s. The key is hash(s).", + "'hard to reverse' (line 14), 'no known fast method' (line 3), 'no known fast reverse' and 'one-way operation' (line 52) all name the same idea before it is defined at line 52.", + "'eavesdropper' (line 9) versus 'a reader' (line 66) for Eve's role." + ], + "unexplained": [ + "'about 2^112 operations (an estimate)' (line 56): no link to the 2048-bit size and no explanation of why it is below the 2^2048 tries needed by brute force.", + "'a p of 2048 bits, about 617 decimal digits' (line 56): it does not say what security would be lost with fewer bits or gained with more.", + "'about 4,000 steps' (line 58): the claim of at most 2048 multiplications is not tied to the binary digits of the exponent." + ], + "missing_why": [ + "Fast exponentiation (line 58): '5^3 = 2 × 5 = 10, 5^6 = 10^2' does not explain why you multiply by 5 at some steps and not others, or how the steps follow from the exponent.", + "Hashing s (line 24): it does not say why s is fed into a hash function instead of being used directly as the key.", + "'The best known methods are faster' (line 56): it gives no hint why they beat trying exponents one by one.", + "Why p must be prime (line 19, and only partly in Scope line 81): step 1 requires a prime without saying why.", + "'A result near 8 says nothing about where 6 is' (line 50): this shows the scrambling but does not argue why no shortcut exists. The point is only made later as an unproven assumption." + ], + "excess": [ + "Scope bullets on Elliptic-curve Diffie–Hellman, AES and quantum computers (lines 80–82) are detours the argument does not use. Line 80 also has two bullets merged into one line.", + "'A standard fast method (Euclid's algorithm)' (line 49): the name is not needed. Showing the inverse 14 is enough." + ] + }, + "gaps": [ + "It does not explain how 2048 bits leads to the 2^112 estimate, or why the best methods beat brute force.", + "It does not say why the shared secret is hashed before use as the key.", + "The square-and-multiply procedure is only sketched, with no rule for choosing between squaring and multiplying.", + "Scope line 80 merges the 'Elliptic-curve' and 'Using the key' bullets because a line break is missing." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 1, + "14": 2, + "15": 2 + }, + "reasons": { + "13": "Shows correctly that multiplying adds the exponents (14 = 5^(6+15), not s = 2) and that Eve would in effect have to find a or b. But it never says that finding a or b is the discrete logarithm problem, or that this problem has no known fast method. The reason Eve is stuck is left implicit." + }, + "pass": false + }, + "usage": { + "cost": 0.20325159999999998, + "seconds": 52 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same number is called 'key' and later 'private number': line 14 'Alice chooses a key, the number 6' versus line 20 'private number ... Alice picks a = 6'. The reader may think the private number is the key, while line 24 says the key is the hashed output of s." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'shared secret' (line 24) versus 'key' (lines 9, 24, 75 'never send the key'). The answer says they 'never send the key', but what they agree on directly is s. The key is hash(s)." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'hard to reverse' (line 14), 'no known fast method' (line 3), 'no known fast reverse' and 'one-way operation' (line 52) all name the same idea before it is defined at line 52." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'eavesdropper' (line 9) versus 'a reader' (line 66) for Eve's role." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 56): no link to the 2048-bit size and no explanation of why it is below the 2^2048 tries needed by brute force." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'a p of 2048 bits, about 617 decimal digits' (line 56): it does not say what security would be lost with fewer bits or gained with more." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 4,000 steps' (line 58): the claim of at most 2048 multiplications is not tied to the binary digits of the exponent." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Fast exponentiation (line 58): '5^3 = 2 × 5 = 10, 5^6 = 10^2' does not explain why you multiply by 5 at some steps and not others, or how the steps follow from the exponent." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Hashing s (line 24): it does not say why s is fed into a hash function instead of being used directly as the key." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'The best known methods are faster' (line 56): it gives no hint why they beat trying exponents one by one." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Why p must be prime (line 19, and only partly in Scope line 81): step 1 requires a prime without saying why." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "'A result near 8 says nothing about where 6 is' (line 50): this shows the scrambling but does not argue why no shortcut exists. The point is only made later as an unproven assumption." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Scope bullets on Elliptic-curve Diffie–Hellman, AES and quantum computers (lines 80–82) are detours the argument does not use. Line 80 also has two bullets merged into one line." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "'A standard fast method (Euclid's algorithm)' (line 49): the name is not needed. Showing the inverse 14 is enough." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "It does not explain how 2048 bits leads to the 2^112 estimate, or why the best methods beat brute force." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "It does not say why the shared secret is hashed before use as the key." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The square-and-multiply procedure is only sketched, with no rule for choosing between squaring and multiplying." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "Scope line 80 merges the 'Elliptic-curve' and 'Using the key' bullets because a line break is missing." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151256-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151256-quiz-prose.json new file mode 100644 index 0000000..54f5d2a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/runs/2026-10-04T151256-quiz-prose.json @@ -0,0 +1,196 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "key-exchange", + "at": "2026-10-04T15:12:48", + "model": "default", + "prompt_sha": "98dd2856417f", + "reply": { + "answers": { + "1": "Alice and Bob, who want a shared key. Eve, who reads every message but changes none. Mallory, who can replace messages. The public numbers p (a prime modulus, 23) and g (the base, 5). The private numbers a and b. The public numbers A = g^a mod p and B = g^b mod p. The shared secret s, which goes through a hash function to become the encryption key. The one-way operation g^x mod p, whose reverse is the discrete logarithm. Authentication (digital signatures) is the fix for Mallory.", + "2": "p and g are agreed in public. Each side keeps its private number and sends g raised to it, mod p (A or B). Each side then raises the other side's public number to its own private number, mod p, and both get s = g^(ab) mod p. Eve sees p, g, A and B but not a, b or s. To get s she would in effect have to find a or b from A or B, which is the discrete logarithm. Mallory sits in the middle and makes a separate secret with each side. Authentication links a public number to the person who sent it.", + "3": "Each side combines its private exponent with the other side's public power. Since (g^b)^a = (g^a)^b = g^(ab), and taking remainders along the way doesn't change the final remainder, both get the same s. Eve only has g^a and g^b, which give her g^(a+b) when multiplied, so she would need to undo the exponentiation mod p, and no fast method for that is known.", + "4": "The size of p, with a and b chosen at random from its whole range. With p = 23, Eve can try exponents one by one and finds a = 6 on her 6th try. With a 2048-bit p, trying them one by one is hopeless, and even the best known methods need about 2^112 operations (about 10^14 years at a trillion per second). Meanwhile Alice's work only grows with the number of bits: about 4,000 steps, because each squaring doubles the exponent. A bigger p makes Eve's work much harder but adds little to Alice's and Bob's.", + "5": "Assumption: the discrete logarithm is hard, and Eve has no other route from A and B to s. This is not proven; it rests on decades of failed attempts. The setup also assumes Eve only reads and never changes messages. Established facts: 5^6 mod 23 = 8, and both sides compute 2. Also, (x + 23k)·y = xy + 23ky, which is why taking remainders early is safe.", + "6": "Alice and Bob agree in public on a prime p and a base g. Each keeps a random private exponent and sends g raised to it, mod p. Each raises the number it receives to its own exponent, and both get g^(ab) mod p, which they hash into a key. An eavesdropper can only multiply the public numbers, which adds the exponents instead of multiplying them, so she would have to solve the discrete logarithm, and no fast method is known for a large p (this is believed, not proven). It does not stop someone who replaces messages, so real systems add signatures.", + "7": "The hardest part was the claim that a 2048-bit exponent needs only about 4,000 steps. The example (5^2 → 5^3 → 5^6) doesn't show how the exponent's bits decide when to square and when to multiply. A worked example with the exponent written in binary would help. The argument that 'results jump around' doesn't by itself show why no fast reverse exists. The step that taking remainders early is safe is also dense, but its small example helps.", + "8": "Alice sends A = 5^3 mod 23 = 125 mod 23 = 10. Bob sends B = 5^2 mod 23 = 25 mod 23 = 2. Alice computes 2^3 mod 23 = 8. Bob computes 10^2 mod 23 = 100 mod 23 = 8. The shared secret is 8, which matches 5^6 mod 23 = 8 in the table.", + "9": "Without mod, A = 5^6 = 15625, and Eve just takes the logarithm base 5 to get 6, because plain powers grow in order. Mod p scrambles the order (5, 2, 10, 4, 20, 8, ...), so a result gives no hint of where the exponent is. Plain multiplication mod p is also weak, because Euclid's algorithm quickly finds the number that undoes it.", + "10": "No. Nothing in the exchange tells Alice who sent a public number. Mallory can swap in her own public number and make a separate secret with each side. Authentication, such as digital signatures, is needed.", + "11": "Bob receives the same output Eve does. If Eve can't get the key back from it, Bob can't either. That's why each side needs its own private number, so that each can finish the other's calculation without undoing it.", + "12": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Alice computes 19^6 mod 23 = 2, and Bob computes 8^15 mod 23 = 2, so s = 2. They must be equal because (5^15)^6 = (5^6)^15 = 5^(90). Taking remainders along the way only removes multiples of 23 from each factor, so the final remainder is the same.", + "13": "Multiplying adds the exponents: 8 × 19 mod 23 = 14 = 5^(6+15) mod 23. The secret needs the exponents multiplied, 5^(6×15). Eve would in effect have to find a or b, for example get 6 from 8, which is the discrete logarithm. That is hard because powers mod p jump around with no order a logarithm could use. For a large p, the only known methods are trying exponents (hopeless) or faster methods that still take about 2^112 operations.", + "14": "No. It is an unproven assumption, supported by decades of failed attempts. A large quantum computer could solve it fast.", + "15": "Mallory can replace each public number with her own (for example 10 = 5^3 mod 23). She then shares one secret with Alice (6) and another with Bob (5), decrypts every message, reads it, and re-encrypts it for the other side, and neither of them notices." + }, + "audit": { + "terms": [ + "Alice's private number 6 is called 'a key' in the first idea ('Alice chooses a key, the number 6', line 14) but 'private number' a = 6 later (line 20). The word 'key' also means the final encryption key made from s (line 24).", + "'shared secret' s (line 24) vs 'the key for ordinary encryption' (line 24) vs 'the key' in the Answer (line 75, 'never send the key'). It's easy to mix up s with the key.", + "'hard to reverse' (line 14) vs 'one-way operation' (line 52) vs 'discrete logarithm' (lines 39, 52): one idea under three names, with the formal name given only later.", + "'modulus' p (line 19) vs 'the number after mod' vs '23' in 'mod 23' (line 11)." + ], + "unexplained": [ + "'about 2^112 operations (an estimate)' (line 56): no source and no way to see how it depends on the 2048-bit size of p, i.e. what a smaller or larger p would give.", + "'2048 bits, about 617 decimal digits' (line 56): it doesn't say why 2048 is enough or what a smaller size would risk.", + "'about 4,000 steps' (line 58): the count is asserted without showing how the exponent's bits lead to it." + ], + "missing_why": [ + "Square-and-multiply (line 58): 'Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications'. It isn't explained how the bits of the exponent choose between squaring and multiplying.", + "Why s is put through a hash function rather than used directly as the key (line 24).", + "'A result near 8 says nothing about where 6 is' (line 50): jumbled-looking output is offered as the reason no fast reverse exists, which is suggestive rather than an explanation.", + "'The best known methods are faster' (line 56): no hint of why they are faster yet still slow.", + "Why p must be prime (lines 19, 81): it is mentioned only in Scope and only partly explained." + ], + "excess": [ + "Scope bullets on elliptic curves, AES and quantum computers (lines 80-82): none of the questions or the main argument uses them.", + "'A version of it starts most HTTPS connections' (line 5): useful context, but the argument doesn't use it.", + "The Answer section (lines 73-75) repeats the summary in line 3 and adds nothing." + ] + }, + "gaps": [ + "The explanation doesn't show how to compute something like 5^15 mod 23 = 19 or 8^15 mod 23 = 2 by hand. The reader has to trust these values or check them against the power table.", + "Line 80 is broken: the 'Using the key' bullet is joined onto the elliptic-curve bullet.", + "The reader isn't told why a fast way to reverse multiplication mod p (Euclid's algorithm) exists while none is known for powers mod p." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2, + "14": 2, + "15": 2 + }, + "reasons": { + "6": "Full marks, but one phrase overstates: 'an eavesdropper can only multiply the public numbers' is not right. Multiplying is the obvious shortcut that fails; Eve is not limited to it. The rest of the answer correctly says she would need the discrete logarithm.", + "14": "Full marks. The quantum-computer remark is correct but out of scope per model.md.", + "7": "No single correct answer. It names specific hard parts: repeated squaring by binary exponent, why scrambling implies no fast reverse, and safety of early remainders." + }, + "pass": true + }, + "usage": { + "cost": 0.1924918, + "seconds": 44 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "Alice's private number 6 is called 'a key' in the first idea ('Alice chooses a key, the number 6', line 14) but 'private number' a = 6 later (line 20). The word 'key' also means the final encryption key made from s (line 24)." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'shared secret' s (line 24) vs 'the key for ordinary encryption' (line 24) vs 'the key' in the Answer (line 75, 'never send the key'). It's easy to mix up s with the key." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'hard to reverse' (line 14) vs 'one-way operation' (line 52) vs 'discrete logarithm' (lines 39, 52): one idea under three names, with the formal name given only later." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'modulus' p (line 19) vs 'the number after mod' vs '23' in 'mod 23' (line 11)." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2^112 operations (an estimate)' (line 56): no source and no way to see how it depends on the 2048-bit size of p, i.e. what a smaller or larger p would give." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'2048 bits, about 617 decimal digits' (line 56): it doesn't say why 2048 is enough or what a smaller size would risk." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 4,000 steps' (line 58): the count is asserted without showing how the exponent's bits lead to it." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "Square-and-multiply (line 58): 'Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings plus at most 2048 multiplications'. It isn't explained how the bits of the exponent choose between squaring and multiplying." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Why s is put through a hash function rather than used directly as the key (line 24)." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "'A result near 8 says nothing about where 6 is' (line 50): jumbled-looking output is offered as the reason no fast reverse exists, which is suggestive rather than an explanation." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "'The best known methods are faster' (line 56): no hint of why they are faster yet still slow." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "Why p must be prime (lines 19, 81): it is mentioned only in Scope and only partly explained." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Scope bullets on elliptic curves, AES and quantum computers (lines 80-82): none of the questions or the main argument uses them." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "'A version of it starts most HTTPS connections' (line 5): useful context, but the argument doesn't use it." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "The Answer section (lines 73-75) repeats the summary in line 3 and adds nothing." + }, + { + "n": 16, + "severity": "gap", + "kind": "gap", + "text": "The explanation doesn't show how to compute something like 5^15 mod 23 = 19 or 8^15 mod 23 = 2 by hand. The reader has to trust these values or check them against the power table." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "Line 80 is broken: the 'Using the key' bullet is joined onto the elliptic-curve bullet." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "The reader isn't told why a fast way to reverse multiplication mod p (Euclid's algorithm) exists while none is known for powers mod p." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/understanding.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/understanding.md new file mode 100644 index 0000000..9a7bd0b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/explainers/key-exchange/review/understanding.md @@ -0,0 +1,24 @@ +# Review record: key-exchange (Published tier, Stage 1 prose) + +## Checks run + +| Check | Result | +|---|---| +| `explainer check key-exchange` | ok (final) | +| Probe (`2026-10-04T145920-probe`) | 16 findings; adopted 1 2 3 6 7 8 10 12 14 15 (two private numbers needed, Eve-shortcut assumption, multiplication case made concrete, remainder rule instance, range of a/b, one name per concept, 2^112 estimate, hash/quantum moved to Scope) | +| Cold read, narrative (`150134`) | 5 blocking (terse beats, remainder rule, unclear multiplication case, placeholder close): all fixed in narrative.md | +| Cold read, prose (`150411`, `150625`, `150725`) | round 2 had 1 blocking (mod used before defined, from a round-1 fix): fixed. Round 3: no blocking | +| Blind test, prose | runs `150452` PASS 15/15; `150823` item 12 = 1 (question asked for numbers, rubric wanted reason: question widened); `150941` item 13 = 1 (prose now links "find a or b" to discrete logarithm); `151044` item 10 = 1 (terse correct answer; grader variance); `151159` item 13 = 1 (question narrower than rubric: widened); final run PASS 15/15 | +| Broken copy (mod "only keeps numbers small"; exchange "proves identity") | FAIL — the test can fail | + +## Declined findings (edge) + +- Why p must be prime, in full: one line in Scope only. +- Where 2^112 comes from; how the faster discrete-log methods work: labeled estimate, out of scope. +- How Euclid's algorithm works: named as a fast standard method; detail not needed for the argument. +- Why scrambled outputs imply no fast reverse: stated as the assumption it is. +- Scope names (AES, elliptic curves, quantum): kept as pointers. + +## Rule learned + +A quiz `ask` must request everything its rubric `expect` requires (the reason, not only the numbers); two failures came from asks narrower than their expects. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/read.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/read.json new file mode 100644 index 0000000..a47edee --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/read.json @@ -0,0 +1,42 @@ +{ + "answers": { + "1": { + "answer": "The operation is raising a base g to a private number mod a prime p (g^x mod p). It is fast to compute, but there is no known fast way to reverse it (the discrete logarithm). On top of that, p is huge, about 2048 bits. Its safety is an assumption that has never been proven.", + "quote": "So g^x mod p is a **one-way operation**: fast to compute, with no known fast reverse. The reverse problem (given g, p, and g^x mod p, find x) is the **discrete logarithm** named above." + }, + "2": { + "answer": "Alice computes (g^b)^a and Bob computes (g^a)^b. By the exponent rule, both equal g^(ab). Taking remainders along the way does not change the final remainder, so both get the same s.", + "quote": "Alice computed (5^15)^6, and Bob computed (5^6)^15. By the exponent rule, both equal 5^(6×15). Both also took remainders along the way, and that does not change the final remainder." + }, + "3": { + "answer": "A man-in-the-middle attack: an attacker (Mallory) who can replace the public numbers and so sets up a separate key with each side. Authentication stops it, for example digital signatures and certificates.", + "quote": "Nothing in the exchange tells Alice who sent a public number. Real systems add **authentication**: Bob attaches to B a value that only Bob can produce and anyone can check (a digital signature)." + }, + "4": { + "answer": "Alice sends A = 8 and Bob sends B = 19. Their shared key is 2.", + "quote": "Alice sends A = 5^6 mod 23 = 15625 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19." + } + }, + "stops": [ + "\"This operation is hard to reverse (later sections show why). Bob receives 8, and so does Eve. If Eve cannot get 6 back from 8, Bob cannot either.\" It is not clear at this point why Bob would need to recover 6.", + "\"Bob sends B = 5^15 mod 23 = 19.\" No working is shown, unlike Alice's 15625 step.", + "\"Alice computes 19^6 mod 23 = 2. Bob computes 8^15 mod 23 = 2.\" These are hard to check by hand.", + "\"Multiplying powers adds the exponents, so 14 = 5^(6+15) mod 23.\" This takes a moment to connect back to 8 = 5^6 and 19 = 5^15.", + "\"The best known methods are faster, but still need about 2^112 operations\" The faster methods are not named or explained.", + "\"Each squaring doubles the exponent, so a 2048-bit exponent needs about 2048 squarings\" The square-and-multiply method is only sketched.", + "\"feed s into the same hash function, a fixed scrambling recipe\" It is not explained why s is hashed instead of being used directly." + ], + "excess": [ + "\"A version of it starts most HTTPS connections.\"", + "\"*Predict before you read on:* Eve has 8 and 19. Can Eve combine them into 2?\"", + "\"**Elliptic-curve Diffie–Hellman**: the same idea with points on a curve in place of powers mod p.\"", + "\"**Quantum computers**: a large quantum computer could solve the discrete logarithm fast.\"", + "\"The two strangers never send the key. Each sends g raised to a private number, mod p.\" This repeats the opening summary." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/run.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/run.json new file mode 100644 index 0000000..0fe357d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/key-exchange/explanation.md`, as prose only, because the answer is a short chain of steps worked with small numbers (p = 23, g = 5) that a diagram or page would only repeat; it passed the probe and cold read and scored 15/15 on the blind test (a deliberately broken copy failed), and two scratch files that rm needed approval to delete, `calc.py` and `explainers/key-exchange-broken/`, are still there to remove.", + "error": null, + "cost": 2.4703193999999997, + "seconds": 1162, + "turns": 112, + "files": [ + "explainers/key-exchange/explanation.md" + ], + "review_cost": 2.0136, + "reviews": 12 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/explanation.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/explanation.md new file mode 100644 index 0000000..9e8bd97 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/explanation.md @@ -0,0 +1,50 @@ +# How can two strangers agree on a secret key in public? + +**Each person keeps one number private and sends a value computed from it. Each person then combines the other person's value with their own private number. Both get the same shared key, but the shared key is never sent. An eavesdropper sees every message and still cannot compute it, because one step is easy to do and, as far as anyone knows, very hard to undo.** + +The method is the Diffie–Hellman key exchange. Below, Alice and Bob want a shared key. Eve reads every message on the channel. + +"x mod p" means the remainder after you divide x by p. For example, 25 mod 23 = 2. + +## How it works + + + +1. Agree in public on a prime p and a base g. Here, p = 23 and g = 5. Eve knows them too. +2. Each person picks a **private number** at random and never sends it. Alice picks a = 6. Bob picks b = 15. +3. Each person computes a **public value**, g to the power of their private number, mod p, and sends it: + - Alice: A = 5^6 mod 23 = 8. + - Bob: B = 5^15 mod 23 = 19. +4. Each person raises the *other* public value to their *own* private number: + - Alice: 19^6 mod 23 = **2**. + - Bob: 8^15 mod 23 = **2**. + +The two results are equal because 19 = 5^15 and 8 = 5^6, so both people compute 5^(6·15) mod 23. Order does not matter: (g^b)^a = (g^a)^b = g^(ab). The **shared key** is 2. + +## Why Eve fails + + + +Eve sees 23, 5, 8 and 19. To get the key she needs a or b. That means solving 5^a mod 23 = 8 for a. This problem is the **discrete logarithm**. + +At p = 23 Eve can try every exponent: 5^1, 5^2, … mod 23 gives 5, 2, 10, 4, 20, 8. She finds a = 6 on try 6. So p must be large. Real systems use a p of 2048 bits (about 617 digits). The best known methods then need about 2^112 operations, far beyond any computer (NIST estimate). Alice and Bob still compute their values quickly, because repeated squaring needs only about 2 × 2048 multiplications. + +Eve cannot combine 8 and 19 directly either. 8 × 19 gives 5^(6+15), not 5^(6·15). + +## Why this form + + + +**Why powers, not multiplication?** Suppose Alice sent A = g·a mod p = 5·6 mod 23 = 7. Eve multiplies 7 by 14, because 5·14 mod 23 = 1. She gets 7·14 mod 23 = 6, which is a. Division undoes multiplication fast. No similar shortcut is known for powers mod p. + + + +**Why mod p?** Without it, Alice sends 5^6 = 15625. Eve computes log base 5 of 15625 = 6. Plain powers grow smoothly, so an ordinary logarithm undoes them. Mod p scrambles the order: the list 5, 2, 10, 4, 20, 8 above has no pattern to follow. + +## Limits + + + +- **Eve must only read (assumption).** Suppose Mallory can also replace messages. Mallory picks 3 and sends 5^3 mod 23 = 10 to Alice in place of 19. Alice computes 10^6 mod 23 = 6. Mallory computes 8^3 mod 23 = 6. Alice now shares key 6 with Mallory and believes it is Bob. Mallory does the same with Bob. The fix is to authenticate the public values, for example with signatures and certificates, as HTTPS does. +- **Hardness is assumed, not proven.** No fast method for the discrete logarithm is known on ordinary computers. A large quantum computer would break it (Shor's algorithm), so newer systems add post-quantum key exchange. +- Most systems today use the same idea on elliptic curves (X25519), which gives equal security with much smaller numbers. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/model.md b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/model.md new file mode 100644 index 0000000..ca20fe5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/model.md @@ -0,0 +1,82 @@ +# Semantic model: Diffie–Hellman key exchange + +> Source of truth. Every rendering derives from this file. + +## Central question + +How can two people who have never met agree on a secret key over a channel that anyone can read? + +## Audience and prior knowledge + +A technical reader: school mathematics (powers, remainders), basic programming. Knows what encryption is in a general way. Does not know modular arithmetic as a tool, one-way functions, or Diffie–Hellman. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Alice, Bob | The two people who want a shared key. | — | +| Eve | An eavesdropper who reads every message on the channel but does not change messages. | assumption (passive) | +| Mallory | An attacker who can read and replace messages. | used for the limit case | +| public numbers p, g | A large prime p and a base g, known to everyone. | fact | +| private number | A number each person picks at random and never sends (a for Alice, b for Bob). | fact | +| public value | g raised to the private number, mod p (A = g^a mod p, B = g^b mod p); sent in the open. | fact | +| shared key | g^(ab) mod p, which both compute and Eve cannot. | fact | +| mod p | The remainder after division by p. | definition | + +## Causal chain + +```text +pick private number → send public value → raise the other's public value to your private number → same shared key +(B^a = g^(ba) = g^(ab) = A^b) +Eve has g, p, A, B but needs a or b → finding a from A = g^a mod p (discrete logarithm) is infeasible for large p +``` + +## Quantities + +Worked example: p = 23, g = 5, a = 6, b = 15. +- A = 5^6 mod 23 = 8. B = 5^15 mod 23 = 19. +- Alice: 19^6 mod 23 = 2. Bob: 8^15 mod 23 = 2. Shared key = 2. +- Size of p: at p = 23 Eve tries all 22 exponents at once. At 2048 bits (about 617 digits) the best known methods need about 2^112 operations (NIST estimate). + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| exponentiation | multiply: A = g·a mod p | A = 5·6 mod 23 = 7. Eve multiplies by the inverse of 5 (14): 7·14 mod 23 = 6 = a. Division undoes multiplication fast. | +| mod p | plain powers, no mod | 5^6 = 15625; Eve computes log base 5 of 15625 = 6. Plain powers grow smoothly, so a logarithm undoes them. Mod p scrambles the order: 5^1..5^6 mod 23 = 5, 2, 10, 4, 20, 8. | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| Eve cannot get the key | Eve sees 23, 5, 8, 19; to get 2 she needs a or b. For a 2048-bit p no known method finds them. | Small p: at p = 23 Eve tries 5^1, 5^2, … until she gets 8, and finds a = 6 within 22 tries. | +| The key is shared only with the intended person | Passive Eve only reads. | Active Mallory replaces B with M = 5^3 mod 23 = 10. Alice computes 10^6 mod 23 = 6; Mallory computes 8^3 mod 23 = 6. Alice shares key 6 with Mallory and thinks it is Bob. Fix: authenticate public values (signatures, certificates). | + +## Terms + +- mod p: remainder after division by p. +- private number: never sent. public value: sent. shared key: computed, never sent. +- discrete logarithm: finding a from g^a mod p. + +## Epistemic status + +- **Mathematical consequence:** B^a = A^b mod p. +- **Assumed:** the discrete logarithm is hard for large p. Not proven; no fast classical method is known. A large quantum computer would break it (Shor's algorithm). +- **Estimate:** 2^112 operations for 2048-bit p. + +## Confusion points + +- "The key is sent encrypted" → the shared key is never sent at all; each side computes it. +- "Eve can multiply A and B" → A·B = g^(a+b), not g^(ab). +- "This proves who you are talking to" → no; it needs authentication (Mallory case). + +## Scope + +- **Out of scope:** elliptic-curve Diffie–Hellman (X25519) — same idea in a different group, smaller numbers; used in TLS today. +- **Out of scope:** how the shared key then encrypts data (AES); post-quantum key exchange (ML-KEM). + +## Representation decision + +- **Stage:** 1 +- **Reason:** a short sequence of steps with one worked example; no topology or parameter space to explore. +- **Levels:** L1 the answer / L2 the steps with numbers / L3 why Eve fails and why this form / L4 limits. diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/model.yaml b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/model.yaml new file mode 100644 index 0000000..629be53 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/explainers/key-exchange/model.yaml @@ -0,0 +1,68 @@ +values: + p: 23 + g: 5 + a: 6 + b: 15 + A: 8 + B: 19 + key: 2 + mallory_m: 3 + mallory_M: 10 + mallory_key: 6 + mult_A: 7 + inverse_g: 14 + plain_power: 15625 + real_bits: 2048 + real_digits: 617 + real_cost_exp: 112 + +allow: [1, 2, 3, 4, 10, 20, 22, 25] + +accept_unjustified: + log: "log appears only as Eve's attack on the no-mod alternative (why-mod), not as an operation of the method" + +require: + - path: p + in: [prose] + - path: g + in: [prose] + - path: A + in: [prose] + - path: B + in: [prose] + - path: key + in: [prose] + +claims: + - id: mech-exchange + kind: mechanism + statement: "Each side raises the other's public value to its own private number mod p; both get g^(ab) mod p." + example: "p=23, g=5, a=6, b=15: A=8, B=19; 19^6 mod 23 = 2 and 8^15 mod 23 = 2." + - id: why-exp + kind: why + about: [exponentiation] + statement: "Exponentiation mod p is easy forward and hard to reverse; multiplication is easy to reverse." + alternative: "A = g*a mod p" + counterexample: "A = 5*6 mod 23 = 7; Eve multiplies by 14 (inverse of 5) and gets 6 = a." + - id: why-mod + kind: why + about: [mod] + statement: "Mod p scrambles the powers so no smooth logarithm undoes them." + alternative: "plain powers g^a" + counterexample: "5^6 = 15625; log base 5 of 15625 = 6." + - id: guarantee-eve + kind: guarantee + statement: "A passive eavesdropper cannot compute the shared key if p is large." + example: "Eve sees 23, 5, 8, 19 and needs a or b; for 2048-bit p no known method finds them." + counterexample: "At p = 23 Eve tries all 22 exponents and finds a = 6." + ask: "Eve sees p, g, A and B. Why can she not compute the shared key?" + - id: limit-mitm + kind: limit + statement: "Without authentication, an active attacker can sit in the middle." + counterexample: "Mallory sends 10 instead of 19; Alice and Mallory both compute key 6." + ask: "What can an attacker who can change messages do, and what stops it?" + +terms: + - term: shared key + means: "g^(ab) mod p, computed by both sides, never sent" + avoid: [sends the key, send the key] diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/read.json new file mode 100644 index 0000000..94589eb --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/read.json @@ -0,0 +1,39 @@ +{ + "answers": { + "1": { + "answer": "Modular exponentiation: each person computes g to the power of their private number, mod p. Doing this is fast. It is safe because undoing it (the discrete logarithm, finding a from g^a mod p) is very hard when p is large, as far as anyone knows. The explanation says this hardness is assumed, not proven.", + "quote": "To get the key she needs a or b. That means solving 5^a mod 23 = 8 for a. This problem is the **discrete logarithm**." + }, + "2": { + "answer": "Each person raises the other's public value to their own private number. Both end up computing g^(ab) mod p, because the order of the exponents does not matter. Only g^a and g^b are sent, never g^(ab).", + "quote": "Order does not matter: (g^b)^a = (g^a)^b = g^(ab)." + }, + "3": { + "answer": "A man-in-the-middle attack: an active attacker (Mallory) swaps the public values for her own and ends up sharing a separate key with each side. Authenticating the public values, for example with signatures and certificates as HTTPS does, stops it.", + "quote": "The fix is to authenticate the public values, for example with signatures and certificates, as HTTPS does." + }, + "4": { + "answer": "Alice sends A = 5^6 mod 23 = 8. Bob sends B = 5^15 mod 23 = 19. Both compute the shared key 2 (19^6 mod 23 = 8^15 mod 23 = 2).", + "quote": "Alice: 19^6 mod 23 = **2**.\n - Bob: 8^15 mod 23 = **2**." + } + }, + "stops": [ + "\"repeated squaring\" is used without being explained: \"because repeated squaring needs only about 2 × 2048 multiplications\"", + "The idea of a modular inverse is not explained: \"Eve multiplies 7 by 14, because 5·14 mod 23 = 1\"", + "\"signatures and certificates\" are not defined, and it is unclear how they stop the attack: \"for example with signatures and certificates, as HTTPS does\"", + "The arithmetic is hard to check: \"Bob: B = 5^15 mod 23 = 19\" and \"8^15 mod 23\" cannot be verified by hand without the repeated-squaring method", + "The reason for 2^112 operations is not given: \"The best known methods then need about 2^112 operations\"", + "\"Mallory does the same with Bob\" is left implicit: it does not say which key Mallory and Bob end up sharing" + ], + "excess": [ + "\"Most systems today use the same idea on elliptic curves (X25519), which gives equal security with much smaller numbers.\"", + "\"A large quantum computer would break it (Shor's algorithm), so newer systems add post-quantum key exchange.\"", + "\"(NIST estimate)\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/run.json new file mode 100644 index 0000000..40f984a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/key-exchange/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/key-exchange/explanation.md`, as prose (with its `model.md` and `model.yaml` source files), because Diffie–Hellman key exchange is a short sequence of steps that a single worked example with small numbers (p = 23, g = 5, shared key 2) carries completely; a diagram or interactive page would add nothing. It passes `explainer check` and covers why Eve fails, why powers and mod p are used, and the man-in-the-middle limit; I checked the arithmetic by hand, but the attempt to confirm it with Python wasn't approved.", + "error": null, + "cost": 0.36011959999999993, + "seconds": 80, + "turns": 14, + "files": [ + "explainers/key-exchange/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/none/explainers/quorum/explanation.md b/evals/artifact/results/2026-10-04-dev/work/quorum/none/explainers/quorum/explanation.md new file mode 100644 index 0000000..be4b4b1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/none/explainers/quorum/explanation.md @@ -0,0 +1,95 @@ +# Why 5 servers survive 2 failures, but 4 servers survive only 1 + +At first this looks wrong. Four servers are more than three, so surely they should survive more failures. They don't. A 4-server cluster tolerates exactly as many failures as a 3-server cluster: one. To see why, you need one rule that these clusters follow and the reason the rule exists. + +## The setting + +Many systems keep several copies of the same data on different machines. Databases, configuration stores (etcd, ZooKeeper, Consul) and coordination services all do this. The copies are there so that if a machine dies, the data and the service carry on. + +Copies create a new problem, though: **the servers have to agree**. Suppose a client says "set the balance to £50" and, a moment later, another client says "set the balance to £80". Every server has to end up with the same answer, applied in the same order. If some servers think the balance is £50 and others think it is £80, the system is broken, and broken in a quiet way that nobody notices until later. + +Protocols such as Raft and Paxos solve this. They differ in detail, but they share one rule: + +> **A decision only counts once a majority of the servers have accepted it.** + +A majority means *more than half of all the servers in the cluster*, counting the ones that are currently down. That set of servers is called a **quorum**. + +## Why a majority, and not "whoever is still up"? + +When a server stops responding, the others **cannot tell whether it has crashed or whether the network between them has broken**. Both cases look the same from the outside: messages go out and nothing comes back. + +That matters because of the dangerous case, a **network partition**. All the servers are running fine, but the network splits them into groups that can't talk to each other. Each group sees the other group as "dead". + +Suppose the rule were "the servers that are still reachable carry on". After a partition, *both* groups would carry on. One side accepts "balance = £50", the other accepts "balance = £80", and each side believes it is the real cluster. This is called **split brain**, and it is exactly the disagreement the system exists to prevent. + +The majority rule prevents it with a simple fact about counting: + +> **Two majorities of the same group always share at least one member.** + +If each of two groups contained more than half the servers, together they would contain more than all of them, which is impossible unless some server is in both. So at most one side of any partition can hold a majority. That side keeps working and the other side stops and waits. Because any two quorums overlap, every decision is seen by at least one server that also took part in the previous decision, and that server is what makes later decisions consistent with earlier ones. + +So the majority rule is not an arbitrary choice. It is the smallest group size that guarantees two groups can never both act independently. + +## Now count + +A cluster keeps working as long as a majority of its servers are still up and can reach each other. So: + +- **quorum size** = the smallest number that is more than half of *n*, which is ⌊n/2⌋ + 1 +- **failures tolerated** = n − quorum size + +**5 servers.** More than half of 5 (2.5) is **3**. You can lose 5 − 3 = **2** servers and still have 3 left, which is a majority. ✅ + +**4 servers.** More than half of 4 (2) is **3**. Exactly half is *not* a majority, because two groups of 2 could both claim to be "half". You can lose 4 − 3 = **1** server. Lose 2 and only 2 remain, which is not more than half, so the cluster stops accepting writes. ❌ + +| Servers (n) | Quorum (majority) | Failures tolerated | +|:-----------:|:-----------------:|:------------------:| +| 1 | 1 | 0 | +| 2 | 2 | 0 | +| 3 | 2 | **1** | +| 4 | 3 | **1** | +| 5 | 3 | **2** | +| 6 | 4 | **2** | +| 7 | 4 | **3** | + +The pattern is that the number of failures tolerated only goes up when you go from an even number of servers to an odd one. Adding a 4th server to a 3-server cluster raises the quorum from 2 to 3, so the extra server is used up by the larger quorum. + +## "But with 4, can't the 2 survivors just carry on?" + +This is the natural objection, so here it is in concrete form. Four servers, A, B, C and D. The network splits into {A, B} and {C, D}. + +``` + [A] [B] ║ [C] [D] + ║ + "C and D ║ "A and B + are dead"║ are dead" +``` + +If 2 out of 4 were enough to continue, both halves would continue. Both would accept writes and their data would diverge. Nobody in either half can tell that the other half is alive, so neither half can safely decide that it is the one to keep going. The only safe behaviour is for **both halves to stop**. + +Now try 5 servers split 3–2: + +``` + [A] [B] [C] ║ [D] [E] + 3 of 5: ║ 2 of 5: + majority, ║ not a majority, + keeps going ║ waits +``` + +However you split 5, one side always has at least 3, so exactly one side carries on. An odd number can never be divided into two equal halves, and that is the whole difference. + +A crash is handled the same way as a partition, because from the inside the two look the same. With 4 servers and 2 crashed, the 2 survivors are in exactly the position of the {A, B} group above. They cannot know the other two aren't alive somewhere, so they must stop. + +## What this means in practice + +- **Clusters are usually built with odd sizes: 3, 5, sometimes 7.** A 4th or 6th server costs money and adds one more machine that has to acknowledge each write, but it adds no extra failure tolerance. +- **To survive *f* failures you need 2f + 1 servers.** That is 3 servers for 1 failure, 5 for 2 and 7 for 3. +- **Even sizes aren't useless.** The extra server can spread out read load or act as a warm spare. It just doesn't help the cluster stay up. +- **Bigger isn't automatically better.** Every write has to wait for a majority, so larger clusters mean more servers to wait for on each write. Most production systems stop at 5. + +## One caveat + +All of this assumes that a failed server simply **stops**: it crashes or becomes unreachable, but it never lies. If servers might send wrong or malicious messages (called *Byzantine* failures), the arithmetic gets stricter. Surviving *f* such servers needs 3f + 1 machines. The reasoning is still based on overlapping groups. The overlap just has to be large enough to outvote the liars. + +## Summary + +A cluster may only act when **more than half** of its servers agree, because any two such groups overlap and so two halves of a broken network can never both make decisions. With 5 servers, more than half is 3, so 2 can fail. With 4 servers, more than half is still 3, so only 1 can fail. Exactly half doesn't count, since the other half could make the same claim. diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/none/read.json b/evals/artifact/results/2026-10-04-dev/work/quorum/none/read.json new file mode 100644 index 0000000..ebbab1e --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/none/read.json @@ -0,0 +1,41 @@ +{ + "answers": { + "1": { + "answer": "A write (decision) is committed only once a majority of the servers (more than half of all servers in the cluster, including those that are down) have accepted it.", + "quote": "A decision only counts once a majority of the servers have accepted it." + }, + "2": { + "answer": "Any two majorities of the same group always share at least one server. So during a network partition at most one side can hold a majority and keep acting, which prevents split brain. If the rule were 'whoever is still reachable', both sides of a partition could keep accepting conflicting writes.", + "quote": "Two majorities of the same group always share at least one member." + }, + "3": { + "answer": "2 failures. The quorum is 4, and 6 − 4 = 2. That is no more than a 5-server cluster survives.", + "quote": "| 6 | 4 | **2** |" + }, + "4": { + "answer": "The group of 3 can commit writes, because 3 of 5 is a majority. The group of 2 is not a majority, so it stops and waits.", + "quote": "3 of 5: ║ 2 of 5:\n majority, ║ not a majority,\n keeps going ║ waits" + } + }, + "stops": [ + "The text uses 'decision' in the rule and 'writes' elsewhere, but never says that committing a write is a decision: \"A decision only counts once a majority of the servers have accepted it.\"", + "The term 'accepted' is never defined. It is not said who proposes a decision or how servers accept it: \"have accepted it\"", + "The overlap step is stated very briefly, and it is unclear how one shared server makes later decisions consistent: \"that server is what makes later decisions consistent with earlier ones\"", + "The floor notation appears without explanation: \"⌊n/2⌋ + 1\"", + "The claim that a majority is the 'smallest' size that works is asserted, not shown: \"It is the smallest group size that guarantees two groups can never both act independently.\"", + "The term 'warm spare' is not defined: \"act as a warm spare\"", + "The 3f + 1 figure for Byzantine failures is given without a reason: \"Surviving *f* such servers needs 3f + 1 machines.\"" + ], + "excess": [ + "A list of product names the reader doesn't need: \"Databases, configuration stores (etcd, ZooKeeper, Consul) and coordination services all do this.\"", + "The 4-server crash case is restated after the partition example already covered it: \"With 4 servers and 2 crashed, the 2 survivors are in exactly the position of the {A, B} group above.\"", + "The summary repeats points already made: \"With 5 servers, more than half is 3, so 2 can fail. With 4 servers, more than half is still 3, so only 1 can fail.\"", + "The Byzantine caveat goes beyond the question being explained: \"If servers might send wrong or malicious messages (called *Byzantine* failures), the arithmetic gets stricter.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/none/run.json b/evals/artifact/results/2026-10-04-dev/work/quorum/none/run.json new file mode 100644 index 0000000..2254900 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/quorum/explanation.md`, a prose explanation, because the answer is a short chain of reasoning (why a majority is needed, then counting) that reads best as text, with a table and two small partition sketches doing the job diagrams would.", + "error": null, + "cost": 0.0993212, + "seconds": 32, + "turns": 2, + "files": [ + "explainers/quorum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/explanation.md b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/explanation.md new file mode 100644 index 0000000..0d92e3a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/explanation.md @@ -0,0 +1,88 @@ +# Why 5 servers survive 2 failures, but 4 servers survive only 1 + +**Short answer.** A cluster accepts a decision only when a strict majority of *all* its servers agree. A majority of 5 is 3, so 2 servers can fail and 3 remain. A majority of 4 is also 3, so only 1 server can fail. The 4th server raises the number of votes you need, but not the number of failures you can absorb. + +The rest of this page explains why the rule must be "a majority of all servers", and not something cheaper. + +## The problem: one server cannot see why another went silent + +A **cluster** is a group of servers that keep identical copies of some data, for example a bank balance. Copies stay identical only if all servers apply the same writes in the same order. So the cluster must *agree* on each write before it accepts it. + +Suppose server A stops getting replies from server B. Two different situations produce exactly this observation: + +1. B has crashed. +2. B is healthy, but the network between A and B is broken. This is a **partition**: the network splits the servers into groups that cannot reach each other. + +A cannot tell these cases apart. In case 2, B may be accepting writes in its own group right now. So any safe rule must assume that silent servers might be alive and making decisions. + +## The simplest rule fails + +**Rule 1: "Servers that can reach each other decide together."** + +Take 5 servers and a balance of $100. The network splits them into a group of 3 and a group of 2. + +- The group of 3 accepts "withdraw $80". Its balance is $20. +- The group of 2 accepts "withdraw $90". Its balance is $10. + +Each group thinks it is the whole surviving cluster. The two copies now disagree, and $170 left an account that held $100. This failure is called **split brain**. + +## The fix: any two deciding groups must share a server + +Split brain needs two groups that decide independently. Prevent it like this: require every deciding group to be large enough that **any two such groups share at least one server**. That shared server votes for at most one of two conflicting writes, so at most one of them can pass. + +Call the required group size the **quorum**, q. Two groups of size q, inside n servers, must overlap when their sizes add up to more than n: + + q + q > n, so q > n/2 + +The smallest whole number above n/2 is + + q = floor(n/2) + 1 (a strict majority) + +Check with 5 servers: q = 3. Two groups of 3 need 6 seats but only 5 exist, so they share at least one server. In the 3|2 split above, only the group of 3 can decide. The group of 2 refuses the $90 withdrawal. + +## Why "half" is not enough + +**Rule 2: "Half the servers is enough" (q = n/2).** + +Take 4 servers split 2|2. Each half has 2 servers, which equals n/2. Both halves decide, and split brain returns. Two groups of 2 fit inside 4 servers with no overlap. So 4 servers need q = 3, the same as 5 servers. + +## From quorum to failure tolerance + +The cluster keeps working while at least q servers are alive and connected. So it tolerates + + f = n − q failures. + +> **Predict before you read the table:** how many failures does a 6-server cluster tolerate? + +| Servers n | Quorum q | Failures tolerated f | +|---|---|---| +| 3 | 2 | 1 | +| **4** | **3** | **1** | +| **5** | **3** | **2** | +| 6 | 4 | 2 | +| 7 | 4 | 3 | + +Each step from an odd n to the next even n adds 1 to q and 0 to f. This is the answer to the original question: going from 4 to 5 servers adds a server without raising q, so f rises from 1 to 2. + +## What "survive" means, and where it stops + +- **Holds:** 5 servers, 2 crash. 3 remain, which equals q. The cluster keeps accepting writes. +- **Breaks:** 5 servers, 3 crash. 2 remain, fewer than q. The cluster stops accepting writes until a server returns. + +In the second case the cluster is *unavailable*, but it is still *correct*. It refuses to decide rather than risk split brain. A majority quorum trades availability for consistency on purpose. + +## What the even server costs you + +An even server does not just fail to help. It adds one more machine that can fail, and the tolerance stays the same. Assume each server is down 1% of the time, independently (an illustrative assumption). The chance that the cluster lacks a quorum is then: + +| Servers | Down when | P(no quorum) | +|---|---|---| +| 3 | 2 or more down | ≈ 0.030% | +| 4 | 2 or more down | ≈ 0.059% | +| 5 | 3 or more down | ≈ 0.001% | + +A 4-server cluster loses its quorum about twice as often as a 3-server cluster. This is why production clusters (etcd, ZooKeeper, Consul) normally use 3, 5, or 7 servers. + +## Scope + +This page assumes servers fail by stopping. Servers that send false messages (**Byzantine faults**) need a larger cluster: n ≥ 3f + 1, so 4 servers to survive 1 such server. The algorithms that run the votes, such as Raft and Paxos, are also out of scope; they all rest on the majority-overlap rule above. diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/model.md b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/model.md new file mode 100644 index 0000000..084b796 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/model.md @@ -0,0 +1,43 @@ +# Model: majority quorum and failure tolerance + +## Central question +Why does a 5-server cluster survive 2 failures, but a 4-server cluster survives only 1? + +## Reader +Technical, new to distributed systems. Knows integer division, basic probability, what a server and a network are. + +## Entities and definitions +- Cluster: n servers that must agree on one sequence of decisions (for example, writes to a database). +- Failure (crash): a server stops responding. It does not send wrong messages. +- Partition: the network splits the servers into groups that cannot reach each other. +- Quorum: the minimum number of servers that must agree before the cluster accepts a decision. +- Majority quorum: q = floor(n/2) + 1. +- Tolerated failures: f = n - q = floor((n-1)/2). + +## Central causal chain +1. A server cannot tell a crashed peer from a peer that is cut off by a partition. +2. So the rule must stay safe if the "missing" servers are alive and deciding on their own. +3. Safety requires that two groups which can each decide must share at least one server. That server refuses to approve conflicting decisions. +4. Two groups of size q always share a server only if q + q > n, so q > n/2. The smallest such q is floor(n/2) + 1. +5. The cluster keeps working while at least q servers are alive, so it survives n - q failures. +6. n = 5: q = 3, f = 2. n = 4: q = 3, f = 1. The 4th server raises q but not f. + +## Alternative (fails) +- "Any connected group decides": 5 servers split 3|2; both sides accept conflicting writes (split brain). +- "Half is enough" (q = n/2): 4 servers split 2|2; both halves have 2 = q; both decide. + +## Guarantee, two cases +- Holds: 5 servers, 2 crash; 3 remain = q; cluster continues. +- Breaks: 5 servers, 3 crash; 2 remain < q; cluster stops accepting decisions. It stays consistent but unavailable. + +## Quantity meaning +Independent failure probability p = 0.01 per server; P(cluster lacks quorum): +- n=3: ~0.00030; n=4: ~0.00059 (worse than 3); n=5: ~0.0000099. + +## Epistemic status +- Steps 3–6: mathematical consequence. +- Crash-only failures: assumption. +- Independent failures with p = 0.01: assumption for illustration. + +## Scope +Not covered: servers that lie (Byzantine faults, need n >= 3f + 1); leader election details (Raft, Paxos); weighted or flexible quorums. diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/model.yaml b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/model.yaml new file mode 100644 index 0000000..6939ec3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/explainers/quorum/model.yaml @@ -0,0 +1,20 @@ +formula: + quorum: "floor(n/2) + 1" + tolerated: "floor((n-1)/2)" +clusters: + - {n: 1, quorum: 1, tolerated: 0} + - {n: 2, quorum: 2, tolerated: 0} + - {n: 3, quorum: 2, tolerated: 1} + - {n: 4, quorum: 3, tolerated: 1} + - {n: 5, quorum: 3, tolerated: 2} + - {n: 6, quorum: 4, tolerated: 2} + - {n: 7, quorum: 4, tolerated: 3} +probability_example: + p_server_down: 0.01 + p_no_quorum: + n3: 0.00030 + n4: 0.00059 + n5: 0.0000099 +assumptions: + - crash failures only (no lying servers) + - independent failures (probability example only) diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/read.json new file mode 100644 index 0000000..a627756 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/read.json @@ -0,0 +1,40 @@ +{ + "answers": { + "1": { + "answer": "A write is committed only when a strict majority of all the servers in the cluster agree to it. That majority is q = floor(n/2) + 1.", + "quote": "A cluster accepts a decision only when a strict majority of *all* its servers agree." + }, + "2": { + "answer": "Any two majorities must share at least one server. That shared server votes for at most one of two conflicting writes, so two separate groups can't both decide (split brain). Smaller groups, even exactly half, can avoid overlapping.", + "quote": "require every deciding group to be large enough that **any two such groups share at least one server**. That shared server votes for at most one of two conflicting writes, so at most one of them can pass." + }, + "3": { + "answer": "2 failures. The quorum is 4, and f = 6 − 4 = 2.", + "quote": "| 6 | 4 | 2 |" + }, + "4": { + "answer": "Only the group of 3 can commit. 3 meets the quorum of 3 for n=5. The group of 2 is below quorum, so it refuses writes, which prevents split brain.", + "quote": "In the 3|2 split above, only the group of 3 can decide. The group of 2 refuses the $90 withdrawal." + } + }, + "stops": [ + "'decision' in the short answer isn't linked to 'write' until later: \"A cluster accepts a decision only when a strict majority\"", + "'commit' is never used or defined. The page says 'accept'/'decide', so the reader has to assume they mean the same: \"The cluster keeps accepting writes.\"", + "Not explained how the group of 2 knows it is a minority. Presumably it counts against the known n, but this isn't stated: \"The group of 2 refuses the $90 withdrawal.\"", + "'alive and connected' — connected to what, and to each other? Not clear for partitions: \"while at least q servers are alive and connected\"", + "No way to derive the probabilities is shown, so the reader can't check them: \"| 4 | 2 or more down | ≈ 0.059% |\"", + "Byzantine formula appears with no reason: \"n ≥ 3f + 1\"" + ], + "excess": [ + "Lists production systems the reader doesn't need: \"production clusters (etcd, ZooKeeper, Consul) normally use 3, 5, or 7 servers.\"", + "Out-of-scope algorithms: \"The algorithms that run the votes, such as Raft and Paxos, are also out of scope\"", + "Byzantine aside goes beyond the question: \"Servers that send false messages (**Byzantine faults**) need a larger cluster\"", + "Repeats the short answer: \"This is the answer to the original question: going from 4 to 5 servers adds a server without raising q\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/run.json new file mode 100644 index 0000000..e39c93b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/quorum/explanation.md`, plus `model.md` and `model.yaml` as its source of truth. I chose prose because the argument is a short chain (a quiet server might be cut off rather than crashed → any two deciding groups must share a server → you need a strict majority → failures you can survive = n − majority), and a table plus one numbered example of two clusters splitting carries it with no diagram or interactive page needed. I didn't run `explainer check` because I had no shell in this session.", + "error": null, + "cost": 0.18852040000000003, + "seconds": 52, + "turns": 4, + "files": [ + "explainers/quorum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/explanation.md b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/explanation.md new file mode 100644 index 0000000..fc32017 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/explanation.md @@ -0,0 +1,79 @@ +# Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1? + +**A cluster decides a change only when more than half of all its servers accept the change. For 5 servers, more than half is 3, so 5 − 3 = 2 servers can fail. For 4 servers, more than half is also 3, so only 4 − 3 = 1 server can fail.** + +The rest of this page explains why the rule must be "more than half". + +## The rule + +A *cluster* is a group of n servers that keep copies of the same data. Take 5 servers, A to E. A client proposes a change, for example x = 1. + +- A server *accepts* the change when it stores the change on disk and replies "yes". +- When enough servers accept, the change is *decided*: it is final. +- The number of accepts needed is the *quorum size*. A group of that many servers, for example {A, B, C}, is a *quorum*. +- A server *fails* when it stops answering: it crashed, or the network cannot reach it. + +The cluster decides changes one at a time, in order: change 1, change 2, and so on. For each position in that order, a server accepts only one change. The examples below use change 7, an arbitrary position. + +If fewer than a quorum of servers answer, the cluster *stops*. It keeps every decided change, but it decides no new change until enough servers return. + +## Why not let the survivors go on? + +Take a cluster of 4 servers, A to D. C and D fail. Why can A and B not go on alone? + +A sends a message to D and gets no reply. Maybe D crashed. Or maybe the network between them is cut, and D still runs and talks to C. This split of the network into groups that cannot reach each other is a *partition*. A sees the same silence in both cases, however long it waits, so A cannot know which case is true. + +Letting A and B go on means quorum size 2. Now suppose the cause is a partition, {A, B} on one side and {C, D} on the other: + +| Group | Change it accepts as change 7 | Number of accepts | Quorum size | Result | +|---|---|---|---|---| +| A, B | x = 1 | 2 | 2 | decided | +| C, D | x = 2 | 2 | 2 | decided | + +Both groups decide, and the copies now disagree for good. This is *split brain*. It happens because two groups of 2 among 4 servers need not share a server. So the quorum size must be more than half of n: then two groups cannot both decide. + +## Why more than half works + +More than half of all n servers is the *majority size*: floor(n/2) + 1. For 5 servers it is 3; for 4 servers it is also 3. Clusters set the quorum size to the majority size: a larger quorum is also safe, but it lets fewer servers fail. + +Two quorums each hold more than half of the servers, so together they list more than n servers. Some server must therefore be in both. For 5 servers, {A, B, C} and {C, D, E} list 3 + 3 = 6 > 5, and C is in both. For 4 servers, two groups of 3 list 6 > 4. + +The shared server blocks split brain. Suppose A, B, C accepted x = 1 for change 7, so x = 1 is decided. Later, C, D, E are asked to accept x = 2 for change 7. C already accepted x = 1 for that position, so C refuses. Then x = 2 gets only 2 accepts, from D and E, and it needs 3. + +This depends on one assumption: a server remembers what it accepted, even after a restart. If C restarts with an empty disk and accepts x = 2, then C, D, E decide x = 2 too, and split brain returns. + +**Predict:** the network splits 5 servers into {A, B, C} and {D, E}. What does each side do? + +A, B, C have 3 accepts, and 3 ≥ 3, so they keep deciding. D and E have 2, and 2 < 3, so they stop. The copies cannot disagree. + +Count more than half of *all* configured servers, failed or not. If D and E counted "more than half of the 2 servers we can reach", that would be 2, and they would decide x = 2 while A, B, C decide x = 1. + +## Counting failures + +*Tolerance* is how many servers can fail while the cluster still decides changes: + +tolerance = n − majority size + +| Servers n | Majority size | Tolerance | +|---|---|---| +| 3 | 2 | 1 | +| 4 | 3 | 1 | +| 5 | 3 | 2 | +| 6 | 4 | 2 | +| 7 | 4 | 3 | + +With 4 servers and 2 failed, only 2 servers answer, and 2 < 3: the cluster stops. + +A tolerance of 1 means a second failure stops the cluster. A tolerance of 2 means one server can be off for an upgrade while another one fails. + +Going from an even n to the next odd n keeps the majority size (4 → 5: both 3), so tolerance grows by 1. Going from an odd n to the next even n raises the majority size by 1 (5 → 6: 3 → 4), so tolerance stays the same. The extra server only makes each change wait for one more accept. That is why clusters usually have an odd number of servers. + +## Answer + +Both clusters need 3 accepts to decide a change, because 3 is more than half of 4 and of 5. Any two groups of more than half share a server, and that server stops them from deciding different changes. 5 servers have 2 to spare. 4 servers have only 1. + +## Limits + +- **Assumed:** servers crash or become unreachable, but never send a false "yes". Servers that lie need larger clusters. +- **Simplified:** in real protocols (Raft, Paxos), the shared server does more than refuse. It reports x = 1 to the new group, and the new group keeps x = 1. +- **Not covered:** how servers detect a failure, how the stopped side catches up when the network heals, and how reads work. diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/model.md b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/model.md new file mode 100644 index 0000000..e5578c3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/model.md @@ -0,0 +1,139 @@ +# Semantic model: quorum (majority voting in a replicated cluster) + +> Source of truth. Every rendering (prose, diagram, page, video) derives from this file. +> If a rendering needs a new fact, add it here first. + +## Central question + +Why does a cluster of 5 servers keep working after 2 servers fail, but a cluster of 4 servers keeps working after only 1? + +## Audience and prior knowledge + +A technical reader: school mathematics and science, basic programming. Knows what a server and a network are, and that a database can keep copies on several machines. New to distributed consensus: does not know the words quorum, partition, or split brain. Does not know Raft or Paxos. Floor and integer division are known. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| cluster | n servers that each keep a copy of the same data and must agree on every change to it. | definition | +| accept | a server stores a proposed change on disk and replies "yes". | definition | +| position | the place of a change in the order the cluster decides changes (change 1, change 2, …); a server accepts only one change per position, e.g. change 7. | definition | +| decide | a change becomes final because a quorum accepted it. | definition | +| failure | a server stops answering: it crashed, or the network cannot reach it. | assumption (crash failures only; no server lies; disks survive a restart) | +| quorum size | the number of servers that must accept a change before it is decided. Here always the majority size of n. | definition | +| quorum | any group of at least quorum-size servers, e.g. {A, B, C} in a 5-server cluster. | definition | +| majority size | more than half of all n configured servers: floor(n/2) + 1. | definition | +| partition | the network splits the servers into groups that cannot reach each other, while every server keeps running. | definition | +| split brain | two groups each decide a different change, so the copies disagree. | definition | +| tolerance | the number of failed servers the cluster can lose and still decide new changes. | definition | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| quorum size | is set to | majority size | so any two quorums share at least one server | +| shared server | blocks | split brain | it already accepted the first change and refuses a conflicting one | +| failure | removes a vote from | quorum | cluster stops deciding if fewer than quorum size servers answer | +| n | sets | majority size, tolerance | tolerance = n − majority size | + +## Dependencies (learning order) + +1. A change is decided only when a quorum accepts it. +2. A server cannot tell a crashed server from a cut-off one. +3. So the quorum size must be a majority; two majorities always share a server, and that server blocks a second, conflicting change. +4. Tolerance = n − majority size. Count for 5 and for 4. +5. Generalize: the table for n = 3..7; the even server adds nothing to tolerance. + +## Causal chain + +```text +a server cannot tell a crash from a partition + → each group must assume the missing servers may still run and decide + → quorum size must be a majority, so two groups cannot both decide + → majority size of 5 is 3, and majority size of 4 is also 3 + → cluster of 5 can lose 5 − 3 = 2; cluster of 4 can lose 4 − 3 = 1 +``` + +## Temporal sequence + +1. A client sends change x = 1 to a 5-server cluster. +2. A accepts (1 vote), B accepts (2), C accepts (3 = quorum size). x = 1 is decided. D and E can catch up later. +3. In a 4-server cluster with C and D down, only A and B accept: 2 < 3, so x = 1 waits. No decided change is lost; no new change is decided until C or D returns. + +## Quantities + +- majority size(n) = floor(n/2) + 1. n = 4 → 3; n = 5 → 3. +- tolerance(n) = n − majority size(n). n = 4 → 1; n = 5 → 2. +- Table: n = 3, 4, 5, 6, 7 → majority size 2, 3, 3, 4, 4 → tolerance 1, 1, 2, 2, 3. +- Overlap of two quorums is at least 2 × majority size − n: n = 5 → 3 + 3 − 5 = 1. +- Meaning of tolerance: at 1, a second failure stops the cluster from deciding; at 2, the cluster can lose one server while another is down for an upgrade. +- Cost of the even server: n = 6 waits for 4 acceptances per change instead of 3, and tolerance stays 2. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| 5 servers, 2 failed | 3 answer = majority size 3 → keeps deciding | 3 ≥ 3 | +| 4 servers, 1 failed | 3 answer → keeps deciding | 3 ≥ 3 | +| 4 servers, 2 failed | 2 answer < 3 → stops deciding | 2 < 3 | +| 5 servers, partition {A,B,C} / {D,E} | left side decides, right side stops; copies cannot disagree | 3 ≥ 3, 2 < 3 | +| 4 servers, partition {A,B} / {C,D} | neither side decides | 2 < 3 on both sides | +| 4 servers, quorum size lowered to 2, partition {A,B} / {C,D} | both sides decide different changes → split brain | 2 + 2 = 4, no shared server | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| quorum size = majority size (more than half) | quorum size = half (2 of 4), so 2 survivors can go on | partition {A,B} / {C,D}: A, B decide x = 1, C, D decide x = 2. Both groups have 2 = quorum size. Copies disagree: split brain. | +| count the majority of all n configured servers | count the majority of the servers each side can reach | 5 servers split {A,B,C} / {D,E}: D, E reach 2 servers, majority of 2 is 2, so they decide x = 2 while A, B, C decide x = 1: split brain. | +| quorum size = majority, not all n | quorum size = 5 of 5 | one failure stops the cluster: tolerance 0. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| Two quorums of majority size always share a server (guarantee) | n = 5: {A,B,C} and {C,D,E}: 3 + 3 = 6 > 5, so at least 1 server (C) is in both. | Size 2 of 5 or 4: {A,B} and {C,D} share nothing (2 + 2 = 4). | +| The shared server blocks split brain (mechanism) | A, B, C accept x = 1. Then C, D, E are asked to accept x = 2. C already accepted x = 1 and refuses; x = 2 has 2 votes < 3. | If C restarts with an empty disk and accepts x = 2, then C, D, E decide x = 2 too: split brain. So the assumption "a server keeps what it accepted" matters. | +| A server cannot tell crash from partition (mechanism) | A sends to D and gets no reply, however long it waits. Case 1: D crashed. Case 2: the link A–D is cut and D still accepts changes from E. In both cases A sees the same thing: no reply. | — | +| 5 servers tolerate 2 failures (mechanism) | 5 − 2 = 3 answering ≥ 3. | Third failure: 2 answering < 3 → stops. | +| 4 servers tolerate 1 failure (mechanism) | 4 − 1 = 3 ≥ 3. | 4 − 2 = 2 < 3 → stops. | + +## Terms + +- **cluster:** n servers keeping copies of the same data. +- **accept:** a server stores a proposed change on disk and replies "yes". +- **decide:** a change becomes final because a quorum accepted it. "Keeps working" means "can still decide new changes". Use "decide", not "write" or "commit". +- **stops:** the cluster still holds every decided change but decides no new change until enough servers return. +- **failure:** a server stops answering (crash or unreachable). +- **quorum size:** a number, e.g. 3. **quorum:** a group of servers, e.g. {A, B, C}. Never swap them. +- **majority size:** more than half of all n configured servers, counting failed servers too: floor(n/2) + 1. +- **partition:** the network splits the servers into groups that cannot reach each other. +- **split brain:** two groups decide different changes. +- **tolerance:** how many failed servers the cluster can lose and still decide. At 1, a second failure stops it; at 2, it can lose one server while another is down for an upgrade. + +## Epistemic status + +- **Mathematical consequence:** two majorities overlap; tolerance = n − (floor(n/2) + 1). +- **Assumed:** crash failures only (a server stops or is unreachable; it never sends a false "yes"). A server keeps what it accepted across a restart. Quorum size is fixed by the configured n. +- **Simplification:** "the shared server refuses x = 2" stands in for the real protocol step, in which the shared server reports x = 1 to the new group and the group keeps x = 1 (Raft, Paxos). + +## Confusion points + +- "With 2 of 4 alive, why not just continue?" → the 2 cannot know whether the other 2 crashed or are cut off and still deciding. +- "Majority of the servers still running" → no: majority of all n configured servers. Otherwise each side of a partition counts itself a majority. +- "More servers is always safer" → going from 5 to 6 raises quorum size to 4; tolerance stays 2. +- "Survive" does not mean "lose no data": a stopped cluster keeps its decided changes; it only stops deciding new ones. + +## Scope + +- **Out of scope:** servers that lie (Byzantine failures) need more servers: 4 servers to tolerate 1 liar. See PBFT. +- **Out of scope:** how a leader is elected, how a server detects a timeout, and how a minority side catches up after the partition heals. See the Raft paper ("In Search of an Understandable Consensus Algorithm"). +- **Out of scope:** reasons to use even n anyway (read load, a tie-breaker "witness" server), and reads that skip the quorum. +- **Out of scope:** weighted votes and flexible quorums (different sizes for reads and changes). +- **Practice:** Raft and Paxos systems such as etcd and ZooKeeper use majority quorums and recommend odd n. + +## Representation decision + +- **Stage:** 1 +- **Reason:** the argument is a short chain of counting; one table (n, majority size, tolerance) carries the comparison. Nothing to explore or animate. +- **Levels:** L1 the answer / L2 the counting / L3 why majority (partition) / L4 general table. diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/model.yaml b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/model.yaml new file mode 100644 index 0000000..2b68006 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/model.yaml @@ -0,0 +1,94 @@ +# Machine-readable half of model.md. `explainer check quorum` holds every rendering to it. + +values: + n_big: 5 + n_small: 4 + majority_5: 3 + majority_4: 3 + tolerance_5: 2 + tolerance_4: 1 + overlap_5: 1 + table: + n: [3, 4, 5, 6, 7] + majority: [2, 3, 3, 4, 4] + tolerance: [1, 1, 2, 2, 3] + +allow: [0, 1, 2, 3, 4, 5, 6, 7] + +require: + - path: majority_5 + in: [prose] + - path: tolerance_5 + in: [prose] + - path: tolerance_4 + in: [prose] + +claims: + - id: quorum-rule + kind: definition + statement: "A change is decided only when a quorum (a group of at least quorum-size servers) has accepted it; quorum size is the majority size of all n configured servers, floor(n/2) + 1." + example: "n = 5: quorum size 3. A, B, C accept x = 1, so x = 1 is decided." + + - id: cannot-tell + kind: mechanism + statement: "A server cannot tell a crashed server from a cut-off one: both look like no reply." + example: "A sends to D and gets no reply, however long it waits. D crashed, or the A–D link is cut and D still accepts changes from E. A sees no reply in both cases." + ask: "Two of the four servers in a cluster stop answering. Why can the other two not simply go on deciding changes?" + + - id: why-majority + kind: why + about: [majority] + statement: "Quorum size must be more than half of n, so that two groups cannot both decide." + alternative: "Quorum size = half: 2 of 4, so 2 survivors can go on." + counterexample: "Partition {A,B} / {C,D}: A, B decide x = 1 and C, D decide x = 2. Both groups have 2 = quorum size. Split brain." + ask: "In a 4-server cluster, why not set the quorum size to 2 so the cluster survives 2 failures?" + + - id: why-all-n + kind: why + about: [count] + statement: "Count the majority of all n configured servers, not of the servers a side can reach." + alternative: "Each side counts the majority of the servers it can reach." + counterexample: "5 servers split {A,B,C} / {D,E}: D, E reach 2, majority of 2 is 2, so D, E decide x = 2 while A, B, C decide x = 1." + + - id: overlap + kind: guarantee + statement: "Two quorums of majority size each hold more than half of n, so together more than n: they always share a server, and that server refuses a conflicting change for the same position." + example: "n = 5: {A,B,C} and {C,D,E}: 3 + 3 = 6 > 5, so C is in both. C accepted x = 1 and refuses x = 2; x = 2 gets 2 votes < 3." + counterexample: "If C restarts with an empty disk and accepts x = 2, C, D, E decide x = 2 too: split brain." + + - id: count-5-4 + kind: mechanism + statement: "Tolerance = n − majority size. Majority size of 5 and of 4 is 3, so 5 servers tolerate 2 failures and 4 tolerate 1." + example: "5 − 2 = 3 ≥ 3 keeps deciding; 4 − 2 = 2 < 3 stops." + ask: "Why does a 5-server cluster tolerate 2 failures but a 4-server cluster only 1?" + + - id: even-adds-nothing + kind: mechanism + statement: "Adding a server to an odd cluster raises the majority size by 1, so tolerance does not grow." + example: "n = 5 → 6: majority size 3 → 4, tolerance 2 → 2, and each change waits for 4 acceptances instead of 3." + + - id: stop-not-loss + kind: definition + statement: "A cluster that stops keeps every decided change; it only decides no new change until enough servers return." + example: "4 servers with C, D down: x = 1 gets 2 votes < 3 and waits until C or D returns." + +terms: + - term: quorum size + means: "a number of servers (e.g. 3); 'quorum' is a group of servers (e.g. {A, B, C})" + avoid: [quorum of 3, a quorum is 3] + - term: decide + means: "a change becomes final because a quorum accepted it" + avoid: [commit, writes] + +budget: {prose: 900, reason: "two why-claims each need a partition case with servers and values, plus the crash-vs-partition case and the n table"} + +quiz: + - q: "A cluster of 6 servers: how many failed servers can it lose and still decide new changes?" + expect: "2. Majority size is 4, so 6 − 4 = 2; the sixth server adds nothing over 5." + misconception: "3, because half of 6 is 3" + - q: "In a 4-server cluster, why not let 2 servers be enough to decide a change?" + expect: "A partition {A,B}/{C,D} would let both halves decide different changes (split brain); two groups of 2 out of 4 need not share a server." + misconception: "Because 2 servers are too slow or too few to store the data" + - q: "A 5-server cluster splits into a group of 3 and a group of 2. What does each group do?" + expect: "The 3-server group keeps deciding (3 ≥ 3); the 2-server group stops (2 < 3) and decides nothing, so the copies cannot disagree." + misconception: "The 2-server group keeps deciding with a majority of the 2 servers it can reach" diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/narrative.md b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/narrative.md new file mode 100644 index 0000000..d6bab69 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/narrative.md @@ -0,0 +1,54 @@ +# Narrative: quorum + +> Pass 3. `model.md` says what is true. This file says the path a first-time reader takes to it. + +## 1. Reader before and after + +- **Before:** a technical reader (school maths and science, basic programming). Knows servers, networks, and that data can be copied to several machines. Does not know quorum, partition, split brain, Raft or Paxos. +- **After:** "A cluster decides a change only when more than half of all its servers accept it, because any two such groups share a server. 5 needs 3, so it can lose 2. 4 also needs 3, so it can lose only 1." + +## 2. Question and motive + +- **Question:** Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1? Result in words: both clusters need 3 servers to agree, because the rule is "more than half of all servers", and more than half of 4 is 3, the same as for 5. +- **Why care:** it is surprising: the 4th server costs money but buys nothing. It decides how you size etcd, ZooKeeper, or any Raft cluster. +- **Why this approach:** ask "why not let the 2 survivors go on?" The answer (they cannot tell a crash from a cut cable) forces the majority rule; the counting then follows. + +## 3. Introduction ledger + +| Reference | Means | Grounded by (a concrete instance) | Beat | +|---|---|---|---| +| cluster | n servers with copies of the same data that must agree on each change | 5 servers A–E | 1 | +| accept | a server stores a proposed change on disk and replies "yes" | A, B, C accept x = 1 | 2 | +| decide | a change becomes final because a quorum accepted it | x = 1 decided after 3 accepts | 2 | +| quorum size / quorum | the number needed (3) / a group that big ({A,B,C}) | n = 5 | 2 | +| majority size | floor(n/2) + 1 of all configured servers | 5 → 3, 4 → 3 | 4 | +| partition | network splits servers into groups that cannot reach each other | {A,B} / {C,D} | 3 | +| split brain | two groups decide different changes | x = 1 vs x = 2 | 3 | +| tolerance | failed servers the cluster can lose and still decide | 5 → 2, 4 → 1 | 6 | +| x = 1, x = 2 | two conflicting changes to one value | — | 2 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 only 1? | — | question as title; answer in bold | "A cluster of n servers keeps copies of the same data. It decides a change only when more than half of all n servers accept it. For 5 servers that is 3, so 5 − 3 = 2 servers can fail. For 4 servers it is also 3, so only 4 − 3 = 1 can fail. The rest explains why the rule is 'more than half'." | the question and the result | +| 2 | name | What does "accept" and "decide" mean? | so | 5 servers A–E; change x = 1 accepted by A, B, C | "A client proposes the next change, for example x = 1. A server accepts it when it stores it on disk and replies yes. Once 3 servers (A, B, C) accept, the change is decided: final. The number needed (here 3) is the quorum size; a group that big ({A, B, C}) is a quorum. If too few servers answer, the cluster stops: it keeps every decided change but decides no new one. The cluster decides changes one at a time, in order: change 1, change 2, and so on. For each position in that order, a server accepts only one change." | accept, decide, quorum size, quorum, stop, one-per-position rule | +| 3 | objection | In a 4-server cluster with 2 down, why can't the other 2 go on? | but | 4 servers; A waits 500 ms for D; partition {A,B}/{C,D} | "Now take a cluster of 4 servers, A–D, where C and D stop answering. Why can A and B not go on? A sends to D and gets no reply in 500 ms. D may have crashed. Or the network between them is cut (a partition) and D still runs, talking to C. A sees the same silence in both cases. Letting A and B go on means quorum size 2. Then A, B decide x = 1 for change 7 while C, D decide x = 2 for change 7. The copies now disagree for good: split brain." | why a quorum of half breaks | +| 4 | mechanism | What quorum size is safe? | therefore | 5 servers: {A,B,C} and {C,D,E}; C refuses x = 2 | "Make the quorum size more than half of all n servers: the majority size, floor(n/2) + 1. Back to 5 servers. Two groups of 3 hold 6 memberships, but there are only 5 servers, so at least one server is in both groups, here C. C already accepted x = 1 for change 7, so it refuses x = 2 for change 7, and x = 2 gets only 2 of the 3 votes it needs." | the majority rule and why it works | +| 5 | test | (predict) 5 servers split into {A,B,C} and {D,E}: what does each side do? | so | prediction prompt, then answer | "A, B, C have 3 ≥ 3 and keep deciding. D, E have 2 < 3 and stop. The copies cannot disagree. Note: count more than half of all 5 configured servers. If D and E counted 'more than half of the 2 servers we can reach', they would decide too." | the rule applied | +| 6 | generalize | So how many failures can a cluster of n take? | therefore | table n = 3..7 with majority size and tolerance | "Tolerance is how many servers can fail while the cluster still decides: n − majority size. Read the table: 4 and 5 both need 3. Adding a 4th server to 3, or a 6th to 5, raises the majority size by one, so tolerance stays the same; each change must also wait for one more accept." | the general pattern | +| 7 | close | — | so | — | "5 survives 2 and 4 survives 1 because both need 3 servers, and 3 is the smallest group that must share a server with any other group. Use odd cluster sizes. This assumes servers crash but never lie." | the answer and its limits | + +## 5. Concrete to symbol + +floor(n/2) + 1 appears after the 5 → 3 and 4 → 3 instances; checked against 6 → 4, 7 → 4. + +## 6. Links between representations + +The table rows for n = 4 and n = 5 repeat the numbers from beats 1 and 4. + +## 7. Close the loop + +- **Answer:** both clusters need 3 servers to decide; 5 has 2 spare, 4 has 1 spare. +- **General argument:** more than half is the smallest quorum where any two quorums share a server; the shared server stops two groups deciding different changes. +- **Payoff:** choose odd cluster sizes. Scope: lying servers, leader election, reads. diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/decisions.yaml new file mode 100644 index 0000000..c1bb6a5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/decisions.yaml @@ -0,0 +1,233 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T151129-probe#1: + decision: adopted +2026-10-04T151129-probe#10: + decision: adopted +2026-10-04T151129-probe#11: + decision: adopted +2026-10-04T151129-probe#12: + decision: adopted +2026-10-04T151129-probe#13: + decision: adopted +2026-10-04T151129-probe#14: + decision: adopted +2026-10-04T151129-probe#2: + decision: adopted +2026-10-04T151129-probe#3: + decision: adopted +2026-10-04T151129-probe#4: + decision: adopted +2026-10-04T151129-probe#5: + decision: adopted +2026-10-04T151129-probe#6: + decision: adopted +2026-10-04T151129-probe#7: + decision: declined +2026-10-04T151129-probe#8: + decision: declined +2026-10-04T151129-probe#9: + decision: declined +2026-10-04T151328-coldread-narrative#1: + decision: adopted +2026-10-04T151328-coldread-narrative#10: + decision: declined +2026-10-04T151328-coldread-narrative#11: + decision: declined +2026-10-04T151328-coldread-narrative#12: + decision: declined +2026-10-04T151328-coldread-narrative#13: + decision: declined +2026-10-04T151328-coldread-narrative#14: + decision: declined +2026-10-04T151328-coldread-narrative#15: + decision: declined +2026-10-04T151328-coldread-narrative#16: + decision: declined +2026-10-04T151328-coldread-narrative#17: + decision: declined +2026-10-04T151328-coldread-narrative#18: + decision: declined +2026-10-04T151328-coldread-narrative#19: + decision: declined +2026-10-04T151328-coldread-narrative#2: + decision: adopted +2026-10-04T151328-coldread-narrative#20: + decision: declined +2026-10-04T151328-coldread-narrative#21: + decision: declined +2026-10-04T151328-coldread-narrative#22: + decision: declined +2026-10-04T151328-coldread-narrative#23: + decision: declined +2026-10-04T151328-coldread-narrative#24: + decision: declined +2026-10-04T151328-coldread-narrative#25: + decision: declined +2026-10-04T151328-coldread-narrative#26: + decision: declined +2026-10-04T151328-coldread-narrative#27: + decision: declined +2026-10-04T151328-coldread-narrative#28: + decision: declined +2026-10-04T151328-coldread-narrative#29: + decision: declined +2026-10-04T151328-coldread-narrative#3: + decision: adopted +2026-10-04T151328-coldread-narrative#30: + decision: declined +2026-10-04T151328-coldread-narrative#31: + decision: declined +2026-10-04T151328-coldread-narrative#32: + decision: declined +2026-10-04T151328-coldread-narrative#33: + decision: declined +2026-10-04T151328-coldread-narrative#34: + decision: declined +2026-10-04T151328-coldread-narrative#35: + decision: declined +2026-10-04T151328-coldread-narrative#36: + decision: declined +2026-10-04T151328-coldread-narrative#37: + decision: declined +2026-10-04T151328-coldread-narrative#38: + decision: declined +2026-10-04T151328-coldread-narrative#39: + decision: declined +2026-10-04T151328-coldread-narrative#4: + decision: adopted +2026-10-04T151328-coldread-narrative#40: + decision: declined +2026-10-04T151328-coldread-narrative#5: + decision: adopted +2026-10-04T151328-coldread-narrative#6: + decision: adopted +2026-10-04T151328-coldread-narrative#7: + decision: adopted +2026-10-04T151328-coldread-narrative#8: + decision: declined +2026-10-04T151328-coldread-narrative#9: + decision: declined +2026-10-04T151438-coldread-narrative#1: + decision: adopted +2026-10-04T151438-coldread-narrative#10: + decision: declined +2026-10-04T151438-coldread-narrative#11: + decision: adopted +2026-10-04T151438-coldread-narrative#12: + decision: declined +2026-10-04T151438-coldread-narrative#13: + decision: adopted +2026-10-04T151438-coldread-narrative#14: + decision: adopted +2026-10-04T151438-coldread-narrative#15: + decision: adopted +2026-10-04T151438-coldread-narrative#2: + decision: adopted +2026-10-04T151438-coldread-narrative#3: + decision: adopted +2026-10-04T151438-coldread-narrative#4: + decision: adopted +2026-10-04T151438-coldread-narrative#5: + decision: adopted +2026-10-04T151438-coldread-narrative#6: + decision: adopted +2026-10-04T151438-coldread-narrative#7: + decision: adopted +2026-10-04T151438-coldread-narrative#8: + decision: adopted +2026-10-04T151438-coldread-narrative#9: + decision: adopted +2026-10-04T151602-coldread-prose#1: + decision: adopted +2026-10-04T151602-coldread-prose#10: + decision: adopted +2026-10-04T151602-coldread-prose#11: + decision: adopted +2026-10-04T151602-coldread-prose#12: + decision: declined +2026-10-04T151602-coldread-prose#13: + decision: declined +2026-10-04T151602-coldread-prose#14: + decision: declined +2026-10-04T151602-coldread-prose#2: + decision: declined +2026-10-04T151602-coldread-prose#3: + decision: adopted +2026-10-04T151602-coldread-prose#4: + decision: adopted +2026-10-04T151602-coldread-prose#5: + decision: adopted +2026-10-04T151602-coldread-prose#6: + decision: adopted +2026-10-04T151602-coldread-prose#7: + decision: adopted +2026-10-04T151602-coldread-prose#8: + decision: adopted +2026-10-04T151602-coldread-prose#9: + decision: adopted +2026-10-04T151643-quiz-prose#1: + decision: adopted +2026-10-04T151643-quiz-prose#10: + decision: adopted +2026-10-04T151643-quiz-prose#11: + decision: adopted +2026-10-04T151643-quiz-prose#12: + decision: declined +2026-10-04T151643-quiz-prose#13: + decision: adopted +2026-10-04T151643-quiz-prose#14: + decision: declined +2026-10-04T151643-quiz-prose#15: + decision: declined +2026-10-04T151643-quiz-prose#2: + decision: adopted +2026-10-04T151643-quiz-prose#3: + decision: adopted +2026-10-04T151643-quiz-prose#4: + decision: adopted +2026-10-04T151643-quiz-prose#5: + decision: adopted +2026-10-04T151643-quiz-prose#6: + decision: adopted +2026-10-04T151643-quiz-prose#7: + decision: adopted +2026-10-04T151643-quiz-prose#8: + decision: adopted +2026-10-04T151643-quiz-prose#9: + decision: adopted +2026-10-04T151817-quiz-prose#1: + decision: declined +2026-10-04T151817-quiz-prose#10: + decision: declined +2026-10-04T151817-quiz-prose#11: + decision: declined +2026-10-04T151817-quiz-prose#2: + decision: declined +2026-10-04T151817-quiz-prose#3: + decision: adopted +2026-10-04T151817-quiz-prose#4: + decision: declined +2026-10-04T151817-quiz-prose#5: + decision: declined +2026-10-04T151817-quiz-prose#6: + decision: declined +2026-10-04T151817-quiz-prose#7: + decision: declined +2026-10-04T151817-quiz-prose#8: + decision: declined +2026-10-04T151817-quiz-prose#9: + decision: declined +2026-10-04T151859-coldread-prose#1: + decision: declined +2026-10-04T151859-coldread-prose#2: + decision: adopted +2026-10-04T151859-coldread-prose#3: + decision: declined +2026-10-04T151859-coldread-prose#4: + decision: declined +2026-10-04T151859-coldread-prose#5: + decision: declined +2026-10-04T151859-coldread-prose#6: + decision: adopted +2026-10-04T151859-coldread-prose#7: + decision: adopted diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151129-probe.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151129-probe.json new file mode 100644 index 0000000..4c14ad4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151129-probe.json @@ -0,0 +1,211 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "quorum", + "at": "2026-10-04T15:11:29", + "model": "default", + "prompt_sha": "7d2607e2a66b", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "The overlap guarantee is argued as a size count (3+3=6>5). The step showing how the shared server actually stops split brain is missing, and so is the assumption it depends on.", + "why_it_matters": "The Relationships table says the shared server 'accepts only one of two conflicting changes', but no example with servers and values shows it. This needs an unstated assumption: a server remembers what it accepted and does not accept a conflicting change. Without that example the reader sees why the groups overlap but not why the overlap is enough.", + "suggested_addition": "Holds: n=5, A,B,C accept x=1, so x=1 is decided. D,E,C are then asked to accept x=2. C already holds x=1 and refuses, so x=2 has only 2 votes, which is less than 3, and is not decided. Breaks: C crashes, restarts with its memory wiped, and accepts x=2. Now C,D,E is 3 votes and x=2 is decided too: split brain. Add 'a server keeps what it accepted across a crash' to the Assumed list." + }, + { + "rule": 4, + "severity": "main", + "about": "No worked example of a partition where majority quorum works. The partition example is only the failing one (4 servers, quorum 2).", + "why_it_matters": "L3 ('why majority') shows only what goes wrong. The reader never sees the majority side carry on while the minority side safely stops. The confusion point 'majority of running servers' has no numbers either.", + "suggested_addition": "n=5, partition {A,B,C} / {D,E}. The left side has 3 ≥ 3 and keeps deciding. The right side has 2 < 3 and stops, so the copies cannot diverge. If quorum were counted from reachable servers, D,E would see 2 reachable, take majority(2)=2, and decide x=2 while A,B,C decide x=1: split brain." + }, + { + "rule": 4, + "severity": "main", + "about": "Dependency 2 ('a server cannot tell a crashed server from a cut-off one') has no worked example.", + "why_it_matters": "The causal chain starts from this claim. A reader new to distributed systems needs to see why the two cases look the same from inside, or the requirement for a majority looks arbitrary.", + "suggested_addition": "A sends to D and waits 500 ms. Case 1: D has crashed. Case 2: the link to D is cut and D is still accepting changes from C and E. In both cases A gets the same thing: no reply after 500 ms. A has nothing else to go on." + }, + { + "rule": 3, + "severity": "main", + "about": "'Quorum' is used for two things: a number (Entities: 'the number of servers that must accept') and a group of servers (Relationships: 'any two quorums share a server').", + "why_it_matters": "The overlap argument is about groups, and the tolerance count is about a number. Using one word for both blurs the central step.", + "suggested_addition": "Define 'quorum size' = majority(n), e.g. 3 for n=5, and 'quorum' = any group of at least that many servers, e.g. {A,B,C}. Then two quorums of 5 servers share at least 3+3−5 = 1 server." + }, + { + "rule": 3, + "severity": "edge", + "about": "Deciding new changes goes by several names: 'keeps working', 'survive', 'keeps deciding', 'stops writes', 'decide new changes'. 'Write' is never defined, and reads are never mentioned.", + "why_it_matters": "A reader may think 'stops writes' is a different state from 'stops deciding', or may think reads stop too.", + "suggested_addition": "Pick 'decide' as the single term and define 'keeps working = can still decide new changes'. Replace 'writes' in the tolerance meaning: with tolerance 1, a 2nd failure means no new change can be decided." + }, + { + "rule": 3, + "severity": "edge", + "about": "Terms used without a definition: 'accept' (what a server does), 'shared server' (a node in Relationships), and 'f' in the Scope line (n ≥ 3f+1).", + "why_it_matters": "'Accept' is the action every mechanism step depends on, and 'f' appears nowhere else in the file.", + "suggested_addition": "accept: a server stores the proposed change and replies yes, e.g. C stores x=1 and replies to the client. Either define f = number of lying servers tolerated (f=1 → n=4), or drop the formula." + }, + { + "rule": 2, + "severity": "edge", + "about": "The general overlap claim is shown only for n=5, and the general tolerance formula is never derived.", + "why_it_matters": "L4 states tolerance = floor((n−1)/2) for every n, but the reader cannot check where it comes from.", + "suggested_addition": "Two majorities always overlap because 2·(floor(n/2)+1) ≥ n+1 > n. For n=4 that gives 3+3=6, so they share at least 2 servers. Odd n=2k+1: majority k+1, tolerance k (n=7: 4, 3). Even n=2k: majority k+1, tolerance k−1 (n=6: 4, 2)." + }, + { + "rule": 2, + "severity": "edge", + "about": "The claim 'majority is the smallest size where any two groups overlap' has no breaking instance at n=5. The 'no data is lost' claim when the cluster stops has no instance at all.", + "why_it_matters": "Both are guarantees stated without numbers.", + "suggested_addition": "At n=5, size 2 breaks it: {A,B} and {C,D} share nothing (2+2=4<5). No data lost: x=1 is decided on A,B,C. Any later quorum of 3 out of 5 contains at least one of A,B,C, so x=1 is always found. This breaks if failed servers lose their disks: if A,B,C all lose storage, x=1 is gone." + }, + { + "rule": 4, + "severity": "edge", + "about": "The Temporal sequence steps have no numbers.", + "why_it_matters": "Step 2 (decided once a quorum accepts) and step 3 (the change waits) are mechanism steps described only in words.", + "suggested_addition": "n=5: the client sends x=1. A accepts (1 vote), B accepts (2), C accepts (3 = quorum), and x=1 is decided. D and E can reply later. n=4 with C,D down: A,B accept, 2 < 3, so x=1 waits until C or D returns." + }, + { + "rule": 4, + "severity": "edge", + "about": "Missing case: 4 servers split 2/2 under majority quorum.", + "why_it_matters": "No server fails, yet the whole cluster stops. This shows the cost of even n directly and connects partition to tolerance.", + "suggested_addition": "n=4, partition {A,B} / {C,D}: each side has 2 < 3, so neither decides. n=5 split {A,B,C} / {D,E}: one side has 3 and keeps working." + }, + { + "rule": 5, + "severity": "edge", + "about": "The model never says what a larger majority costs the reader.", + "why_it_matters": "The claim that 'the even server adds nothing' leaves out that it adds a cost: every change has to wait for more acceptances.", + "suggested_addition": "n=5 waits for 3 acceptances per change, and n=6 waits for 4. That is one more server, one more acceptance per change, and still tolerance 2." + } + ], + "excess": [ + "Why-this-form row about floor (3.5 servers): the audience already knows floor and integer division.", + "Scope detail 'n ≥ 3f + 1, e.g. 4 servers for 1 liar; see PBFT': introduces an undefined f and is not needed for the central question.", + "Established practice list naming ZooKeeper and etcd: not needed to answer the central question." + ], + "next_questions": [ + { + "q": "In a 5-server cluster split 3/2, what does the 2-server side do, and what happens to it when the network heals?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How does a server decide that another server has failed? Is it a timeout, and how long?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "What if the server in the overlap crashes and restarts? Does it remember what it accepted?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If 4 servers is no better than 3, why would anyone run an even number?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Do reads also need a majority, or can any one server answer a read?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.15055179999999996, + "seconds": 63 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "The overlap guarantee is argued as a size count (3+3=6>5). The step showing how the shared server actually stops split brain is missing, and so is the assumption it depends on. → Holds: n=5, A,B,C accept x=1, so x=1 is decided. D,E,C are then asked to accept x=2. C already holds x=1 and refuses, so x=2 has only 2 votes, which is less than 3, and is not decided. Breaks: C crashes, restarts with its memory wiped, and accepts x=2. Now C,D,E is 3 votes and x=2 is decided too: split brain. Add 'a server keeps what it accepted across a crash' to the Assumed list." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 4", + "text": "No worked example of a partition where majority quorum works. The partition example is only the failing one (4 servers, quorum 2). → n=5, partition {A,B,C} / {D,E}. The left side has 3 ≥ 3 and keeps deciding. The right side has 2 < 3 and stops, so the copies cannot diverge. If quorum were counted from reachable servers, D,E would see 2 reachable, take majority(2)=2, and decide x=2 while A,B,C decide x=1: split brain." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 4", + "text": "Dependency 2 ('a server cannot tell a crashed server from a cut-off one') has no worked example. → A sends to D and waits 500 ms. Case 1: D has crashed. Case 2: the link to D is cut and D is still accepting changes from C and E. In both cases A gets the same thing: no reply after 500 ms. A has nothing else to go on." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 3", + "text": "'Quorum' is used for two things: a number (Entities: 'the number of servers that must accept') and a group of servers (Relationships: 'any two quorums share a server'). → Define 'quorum size' = majority(n), e.g. 3 for n=5, and 'quorum' = any group of at least that many servers, e.g. {A,B,C}. Then two quorums of 5 servers share at least 3+3−5 = 1 server." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 3", + "text": "Deciding new changes goes by several names: 'keeps working', 'survive', 'keeps deciding', 'stops writes', 'decide new changes'. 'Write' is never defined, and reads are never mentioned. → Pick 'decide' as the single term and define 'keeps working = can still decide new changes'. Replace 'writes' in the tolerance meaning: with tolerance 1, a 2nd failure means no new change can be decided." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 3", + "text": "Terms used without a definition: 'accept' (what a server does), 'shared server' (a node in Relationships), and 'f' in the Scope line (n ≥ 3f+1). → accept: a server stores the proposed change and replies yes, e.g. C stores x=1 and replies to the client. Either define f = number of lying servers tolerated (f=1 → n=4), or drop the formula." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 2", + "text": "The general overlap claim is shown only for n=5, and the general tolerance formula is never derived. → Two majorities always overlap because 2·(floor(n/2)+1) ≥ n+1 > n. For n=4 that gives 3+3=6, so they share at least 2 servers. Odd n=2k+1: majority k+1, tolerance k (n=7: 4, 3). Even n=2k: majority k+1, tolerance k−1 (n=6: 4, 2)." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 2", + "text": "The claim 'majority is the smallest size where any two groups overlap' has no breaking instance at n=5. The 'no data is lost' claim when the cluster stops has no instance at all. → At n=5, size 2 breaks it: {A,B} and {C,D} share nothing (2+2=4<5). No data lost: x=1 is decided on A,B,C. Any later quorum of 3 out of 5 contains at least one of A,B,C, so x=1 is always found. This breaks if failed servers lose their disks: if A,B,C all lose storage, x=1 is gone." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 4", + "text": "The Temporal sequence steps have no numbers. → n=5: the client sends x=1. A accepts (1 vote), B accepts (2), C accepts (3 = quorum), and x=1 is decided. D and E can reply later. n=4 with C,D down: A,B accept, 2 < 3, so x=1 waits until C or D returns." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 4", + "text": "Missing case: 4 servers split 2/2 under majority quorum. → n=4, partition {A,B} / {C,D}: each side has 2 < 3, so neither decides. n=5 split {A,B,C} / {D,E}: one side has 3 and keeps working." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 5", + "text": "The model never says what a larger majority costs the reader. → n=5 waits for 3 acceptances per change, and n=6 waits for 4. That is one more server, one more acceptance per change, and still tolerance 2." + }, + { + "n": 12, + "severity": "excess", + "kind": "model entry", + "text": "Why-this-form row about floor (3.5 servers): the audience already knows floor and integer division." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Scope detail 'n ≥ 3f + 1, e.g. 4 servers for 1 liar; see PBFT': introduces an undefined f and is not needed for the central question." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Established practice list naming ZooKeeper and etcd: not needed to answer the central question." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151328-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151328-coldread-narrative.json new file mode 100644 index 0000000..d247516 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151328-coldread-narrative.json @@ -0,0 +1,399 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "quorum", + "at": "2026-10-04T15:13:28", + "model": "default", + "prompt_sha": "46d121ad7ad5", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: answer in bold. Said: \"Both clusters need 3 servers to agree; 5 − 3 = 2, 4 − 3 = 1.\"", + "unresolved": [ + "\"Both clusters\": no cluster has been introduced, and nothing says one has 5 servers and the other has 4", + "\"agree\": agree on what?", + "\"5 − 3 = 2, 4 − 3 = 1\": nothing says what the 2 and the 1 count (failures the cluster can survive)" + ], + "unsaid": [ + "The question itself (why 5 survives 2 failures but 4 survives only 1) appears only in the author's 'Reader's question' column. The reader hears an answer with no question before it." + ], + "leap": [ + "Why 3 for both? No reason is given yet. That is fine for a hook, but it is not marked as something to be explained later." + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 2 (name)", + "text": "Shown: x = 1 accepted by A, B, C. Said: \"A cluster decides a change only when a quorum accepts it. Define accept, decide, quorum size, quorum.\"", + "unresolved": [ + "\"quorum\": the script says \"Define accept, decide, quorum size, quorum\" but never gives the definitions, so the reader still does not know what a quorum is", + "\"accept\"", + "\"decide\"", + "\"quorum size\"" + ], + "unsaid": [ + "\"x = 1\": what x is and that it stands for a proposed change", + "\"A, B, C\": these letters are never said to be servers, and nothing says how many servers the cluster has", + "that the quorum size here is 3" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "Shown: partition {A,B}/{C,D}; 500 ms timeout case. Said: \"A cannot tell crash from cut-off. With quorum size 2, both halves decide: split brain.\"", + "unresolved": [ + "\"split brain\": named but never defined", + "\"partition\" (shown only)", + "\"cut-off\": cut off from what? The network split is never described in words" + ], + "unsaid": [ + "\"500 ms timeout case\"", + "the {A,B}/{C,D} split, and that we have moved from 5 servers to a 4-server cluster", + "that the two halves decide *different* changes (no x = 2 is said here)" + ], + "leap": [ + "Why both halves deciding is harmful. The conflict (x = 1 on one side, x = 2 on the other) is not stated", + "Why \"A cannot tell crash from cut-off\" means it must not simply carry on" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Shown: {A,B,C} and {C,D,E}; C refuses x = 2. Said: \"More than half of all n: two such groups share a server, and that server refuses the conflict. Count all n, not reachable ones.\"", + "unresolved": [ + "\"n\": never said to mean the number of servers", + "\"the conflict\": x = 2 has never been said" + ], + "unsaid": [ + "\"C refuses x = 2\": the visual introduces x = 2 without the words mentioning it", + "E appears, so we are back to 5 servers, but nothing says so" + ], + "leap": [ + "Why the shared server refuses. No rule has been stated that a server accepts only one value per change", + "\"Count all n, not reachable ones\": no reason is given here, and the link back to beat 3 (survivors cannot tell crash from cut-off) is not made" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (test)", + "text": "Shown: predict, then answer. Said: \"3 side decides, 2 side stops.\"", + "unresolved": [ + "\"stops\": its meaning (no new changes, no data loss) is only given in beat 7" + ], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (generalize)", + "text": "Shown: table n = 3..7. Said: \"Tolerance = n − majority size. Even n adds nothing, and costs one more accept per change.\"", + "unresolved": [ + "\"majority size\": beat 4 said \"more than half\" but never named it or gave its value per n", + "\"Tolerance\": only defined by the equation" + ], + "unsaid": [ + "the table's columns and any formula in it (for example floor(n/2)+1) are not read out" + ], + "leap": [ + "\"Even n adds nothing\": the reason (more than half of 2k and of 2k−1 is the same number) is left for the reader to work out from the table" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (close)", + "text": "Said: \"Answer again; 'stop' means no new changes, not data loss; scope pointers.\"", + "unresolved": [ + "\"scope pointers\": no actual content is given" + ], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 1: the question is never asked. The reader hears \"Both clusters need 3 servers to agree; 5 − 3 = 2, 4 − 3 = 1\" without knowing what the 2 and 1 count.", + "Beat 2: \"Define accept, decide, quorum size, quorum.\" The definitions are not in the script, so the central term \"quorum\" stays undefined.", + "Beat 3: \"both halves decide: split brain\". Nothing says the halves decide *different* changes, so the harm is never stated.", + "Beat 4: \"More than half of all n\". n is never defined.", + "Beat 4: \"that server refuses the conflict\". The rule that makes it refuse is never given, and this is the core of the argument." + ], + "cuts": [ + "Beat 3: the shown \"500 ms timeout case\". The words never use it, so it is noise.", + "Beat 7: \"scope pointers\" with no content. Either fill it in or cut it.", + "Beat 6: \"and costs one more accept per change\". This is minor, and the main line does not need it." + ], + "question": { + "text": "Why does a 5-server cluster survive 2 failures while a 4-server cluster survives only 1?", + "known_at": "The script never states it outright. The reader can only infer it from beat 1's \"5 − 3 = 2, 4 − 3 = 1\", and fully only by beat 6, where \"Tolerance\" is introduced.", + "result_stated": "Yes, at beat 1: \"Both clusters need 3 servers to agree; 5 − 3 = 2, 4 − 3 = 1.\" It is restated at beat 6: \"Tolerance = n − majority size. Even n adds nothing\"." + }, + "motive": "There is no reason to care in the spoken script. The cost point (a 4th server buys nothing) appears only in the plan. Beat 6's \"Even n adds nothing, and costs one more accept per change\" comes closest, but it arrives late. The reason for the approach is partly there in beat 3: \"A cannot tell crash from cut-off.\"", + "close": "Only partly. Beat 7 says \"Answer again\" but the script gives no wording, and the general argument (any two majorities share a server) is not restated in words at the close. It was said only in beat 4: \"two such groups share a server, and that server refuses the conflict\".", + "recap": "A cluster only accepts a change if more than half of all its servers agree, so two groups can't decide conflicting things. 5 and 4 both need 3, so 5 can lose 2 but 4 can lose only 1.", + "strongest": "Beat 4: \"two such groups share a server, and that server refuses the conflict\". The visual {A,B,C} and {C,D,E} grounds it with a concrete overlap.", + "weakest": "Beat 2: \"Define accept, decide, quorum size, quorum.\" This is an instruction to the author rather than an explanation, so the key vocabulary never reaches the reader.", + "suggestions": [ + "Open beat 1 by asking the question out loud before giving the answer.", + "In beat 2, write out the definitions, for example: accept means a server stores the change and replies yes; a quorum is a group of that many servers. Say that A to E are servers.", + "In beat 3, say that one half sets x = 1 and the other sets x = 2, and explain why that is bad.", + "In beat 4, define n as the total number of configured servers. State that each server accepts only one value for a given change, which is why C refuses.", + "Read out floor(n/2)+1 and walk through the n = 4 and n = 5 rows of the table in words.", + "In beat 7, restate the overlap argument in one sentence and list the actual scope limits." + ] + }, + "usage": { + "cost": 0.12129479999999998, + "seconds": 43 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 1: the question is never asked. The reader hears \"Both clusters need 3 servers to agree; 5 − 3 = 2, 4 − 3 = 1\" without knowing what the 2 and 1 count." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: \"Define accept, decide, quorum size, quorum.\" The definitions are not in the script, so the central term \"quorum\" stays undefined." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 3: \"both halves decide: split brain\". Nothing says the halves decide *different* changes, so the harm is never stated." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: \"More than half of all n\". n is never defined." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: \"that server refuses the conflict\". The rule that makes it refuse is never given, and this is the core of the argument." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 3: the shown \"500 ms timeout case\". The words never use it, so it is noise." + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 7: \"scope pointers\" with no content. Either fill it in or cut it." + }, + { + "n": 8, + "severity": "excess", + "kind": "cut", + "text": "Beat 6: \"and costs one more accept per change\". This is minor, and the main line does not need it." + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): \"Both clusters\": no cluster has been introduced, and nothing says one has 5 servers and the other has 4" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): \"agree\": agree on what?" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): \"5 − 3 = 2, 4 − 3 = 1\": nothing says what the 2 and the 1 count (failures the cluster can survive)" + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): The question itself (why 5 survives 2 failures but 4 survives only 1) appears only in the author's 'Reader's question' column. The reader hears an answer with no question before it." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): Why 3 for both? No reason is given yet. That is fine for a hook, but it is not marked as something to be explained later." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (name): \"quorum\": the script says \"Define accept, decide, quorum size, quorum\" but never gives the definitions, so the reader still does not know what a quorum is" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (name): \"accept\"" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (name): \"decide\"" + }, + { + "n": 17, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (name): \"quorum size\"" + }, + { + "n": 18, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (name): \"x = 1\": what x is and that it stands for a proposed change" + }, + { + "n": 19, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (name): \"A, B, C\": these letters are never said to be servers, and nothing says how many servers the cluster has" + }, + { + "n": 20, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (name): that the quorum size here is 3" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): \"split brain\": named but never defined" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): \"partition\" (shown only)" + }, + { + "n": 23, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (objection): \"cut-off\": cut off from what? The network split is never described in words" + }, + { + "n": 24, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): \"500 ms timeout case\"" + }, + { + "n": 25, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): the {A,B}/{C,D} split, and that we have moved from 5 servers to a 4-server cluster" + }, + { + "n": 26, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): that the two halves decide *different* changes (no x = 2 is said here)" + }, + { + "n": 27, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): Why both halves deciding is harmful. The conflict (x = 1 on one side, x = 2 on the other) is not stated" + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): Why \"A cannot tell crash from cut-off\" means it must not simply carry on" + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): \"n\": never said to mean the number of servers" + }, + { + "n": 30, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): \"the conflict\": x = 2 has never been said" + }, + { + "n": 31, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): \"C refuses x = 2\": the visual introduces x = 2 without the words mentioning it" + }, + { + "n": 32, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): E appears, so we are back to 5 servers, but nothing says so" + }, + { + "n": 33, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): Why the shared server refuses. No rule has been stated that a server accepts only one value per change" + }, + { + "n": 34, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): \"Count all n, not reachable ones\": no reason is given here, and the link back to beat 3 (survivors cannot tell crash from cut-off) is not made" + }, + { + "n": 35, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (test): \"stops\": its meaning (no new changes, no data loss) is only given in beat 7" + }, + { + "n": 36, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (generalize): \"majority size\": beat 4 said \"more than half\" but never named it or gave its value per n" + }, + { + "n": 37, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (generalize): \"Tolerance\": only defined by the equation" + }, + { + "n": 38, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (generalize): the table's columns and any formula in it (for example floor(n/2)+1) are not read out" + }, + { + "n": 39, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (generalize): \"Even n adds nothing\": the reason (more than half of 2k and of 2k−1 is the same number) is left for the reader to work out from the table" + }, + { + "n": 40, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (close): \"scope pointers\": no actual content is given" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151438-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151438-coldread-narrative.json new file mode 100644 index 0000000..83d3122 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151438-coldread-narrative.json @@ -0,0 +1,218 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "quorum", + "at": "2026-10-04T15:14:38", + "model": "default", + "prompt_sha": "46d121ad7ad5", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "A cluster of n servers keeps copies of the same data. It decides a change only when more than half of all n servers accept it. 5 needs 3 so 2 can fail; 4 also needs 3 so only 1 can fail.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (name)", + "text": "Defines accept, decide, quorum size, quorum, and stop, using x = 1 accepted by A, B, C.", + "unresolved": [], + "unsaid": [], + "leap": [ + "\"the next change\" assumes the changes come in a fixed order, one slot at a time, which is never said. Beat 4's rule (\"Each server accepts only one change as the next change\") depends on this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "A gets no reply from D in 500 ms: crash or partition, A can't tell. With quorum size 2, {A,B} decide x = 1 and {C,D} decide x = 2: split brain.", + "unresolved": [], + "unsaid": [ + "The Shown column switches to 4 servers, after beat 2's 5 servers A–E. The Said column never says the cluster is now 4 servers. The reader has to work this out from the names A–D.", + "The question this beat answers, \"why can't the other 2 go on?\", is never asked in the words. The text jumps straight into \"A sends to D\"." + ], + "leap": [ + "\"So suppose quorum size were 2\": the words never say that 2 is exactly what 'let the 2 survivors go on' means in a 4-server cluster, or that it is half of 4." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Quorum size = majority size floor(n/2)+1. Two groups of 3 among 5 total 6 > 5 places, so they share C. C has already accepted x = 1, so it refuses x = 2. Count all configured servers, not the reachable ones.", + "unresolved": [ + "\"6 > 5 places\": 'places' is an odd word for this counting step. It is only shown for n = 5, and the general version ('two majorities together have more than n members') is never said." + ], + "unsaid": [ + "The Shown column has the overlapping groups {A,B,C} and {C,D,E}. The words do not say how C could be in contact with both groups, given that beat 3 framed the problem as a partition." + ], + "leap": [ + "\"Each server accepts only one change as the next change\" is a new rule, and the whole argument depends on it. It appears here for the first time, with no reason given." + ], + "excess": [ + "\"D and E reach only each other, and 'more than half of 2' would let them decide\" gives away the scenario that beat 5 asks the reader to predict." + ], + "purpose_unclear": false + }, + { + "at": "Beat 5 (test)", + "text": "Predict: {A,B,C} | {D,E}. A, B, C have 3 ≥ 3 and keep deciding. D, E have 2 < 3 and stop.", + "unresolved": [], + "unsaid": [], + "leap": [], + "excess": [ + "The prediction was already answered at the end of beat 4 (\"D and E reach only each other...\"), so the test gives the reader nothing new to work out." + ], + "purpose_unclear": false + }, + { + "at": "Beat 6 (generalize)", + "text": "Tolerance = n − majority size. A table for n = 3..7 shows that 4 and 5 both need 3. Adding a server to an odd-sized cluster raises the majority size, so tolerance stays the same and each change waits for one more accept.", + "unresolved": [], + "unsaid": [ + "The table rows for n = 3, 6 and 7 are shown but not read aloud. Minor." + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (close)", + "text": "5 survives 2 and 4 survives 1 because both need 3, and 3 is the smallest group that must share a server with any other. Use odd sizes. Assumes servers crash but never lie; see the Raft and PBFT papers.", + "unresolved": [ + "\"PBFT\"", + "\"Raft\"", + "\"leader election\"" + ], + "unsaid": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 4: \"Each server accepts only one change as the next change\". This rule makes the shared server C refuse x = 2, and it is introduced here without warning or reason. Without it, overlapping groups prove nothing.", + "Beat 3: the switch from 5 servers to 4 is not said in words (it appears only in Shown), so the reader rereads to find out which cluster they are in." + ], + "cuts": [ + "Beat 4: cut the D/E sentence (\"Count all n configured servers... would let them decide\"), or move it into beat 5's answer. It currently spoils the prediction.", + "Beat 7: cut \"leader election, and reads are covered in the Raft and PBFT papers\". It brings in unknown names right at the close; \"servers crash but never lie\" is enough as the scope." + ], + "question": { + "text": "Why can a 5-server cluster lose 2 servers while a 4-server cluster can lose only 1? Known from Beat 1, where it is the title.", + "known_at": "Beat 1", + "result_stated": "Yes, Beat 1: \"For 5 servers that is 3, so 5 − 3 = 2 servers can fail. For 4 servers it is also 3, so only 4 − 3 = 1 can fail.\"" + }, + "motive": "There is a reason for the approach in Beat 1, \"The rest explains why the rule is 'more than half'\", and in Beat 3's objection, \"A sees the same silence in both cases\". A reason to care about the question is missing from the words. The plan's point that \"the 4th server costs money but buys nothing\" is never said; only Beat 6's \"tolerance stays the same\" and Beat 7's \"Use odd cluster sizes\" hint at it.", + "close": "Yes, Beat 7: \"5 survives 2 and 4 survives 1 because both need 3 servers, and 3 is the smallest group that must share a server with any other group.\" The general argument (overlap means a shared server, and that server refuses a conflicting change) is said in words, though the refusal rule itself comes late, in Beat 4.", + "recap": "A cluster only decides a change when more than half of all its servers agree, so two groups can never both decide different things; that means 4 servers need 3 just like 5 do, so the 4th server buys you no extra failure tolerance.", + "strongest": "Beat 3: \"A sees the same silence in both cases\" makes it concrete why the survivors cannot simply carry on, and leads straight into split brain.", + "weakest": "Beat 4: it packs the formula, the overlap count (\"6 > 5 places\"), a brand-new one-vote-per-slot rule, and the counting caveat that gives away Beat 5 into one beat.", + "suggestions": [ + "Introduce the rule \"a server accepts only one change for each slot\" in Beat 2, alongside accept.", + "Say the general overlap argument in words: two groups that are each more than half of n together have more than n members, so they must share a server.", + "Say in Beat 3's words that the cluster is now 4 servers and that a quorum of 2 is exactly half.", + "Add the cost motive to Beat 1: a 4th server costs money but buys no extra tolerance." + ] + }, + "usage": { + "cost": 0.11751439999999999, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: \"Each server accepts only one change as the next change\". This rule makes the shared server C refuse x = 2, and it is introduced here without warning or reason. Without it, overlapping groups prove nothing." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 3: the switch from 5 servers to 4 is not said in words (it appears only in Shown), so the reader rereads to find out which cluster they are in." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Beat 4: cut the D/E sentence (\"Count all n configured servers... would let them decide\"), or move it into beat 5's answer. It currently spoils the prediction." + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "Beat 7: cut \"leader election, and reads are covered in the Raft and PBFT papers\". It brings in unknown names right at the close; \"servers crash but never lie\" is enough as the scope." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (name): \"the next change\" assumes the changes come in a fixed order, one slot at a time, which is never said. Beat 4's rule (\"Each server accepts only one change as the next change\") depends on this." + }, + { + "n": 6, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): The Shown column switches to 4 servers, after beat 2's 5 servers A–E. The Said column never says the cluster is now 4 servers. The reader has to work this out from the names A–D." + }, + { + "n": 7, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (objection): The question this beat answers, \"why can't the other 2 go on?\", is never asked in the words. The text jumps straight into \"A sends to D\"." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): \"So suppose quorum size were 2\": the words never say that 2 is exactly what 'let the 2 survivors go on' means in a 4-server cluster, or that it is half of 4." + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): \"6 > 5 places\": 'places' is an odd word for this counting step. It is only shown for n = 5, and the general version ('two majorities together have more than n members') is never said." + }, + { + "n": 10, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The Shown column has the overlapping groups {A,B,C} and {C,D,E}. The words do not say how C could be in contact with both groups, given that beat 3 framed the problem as a partition." + }, + { + "n": 11, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): \"Each server accepts only one change as the next change\" is a new rule, and the whole argument depends on it. It appears here for the first time, with no reason given." + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (generalize): The table rows for n = 3, 6 and 7 are shown but not read aloud. Minor." + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (close): \"PBFT\"" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (close): \"Raft\"" + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (close): \"leader election\"" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151602-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151602-coldread-prose.json new file mode 100644 index 0000000..1733bd5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151602-coldread-prose.json @@ -0,0 +1,370 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "quorum", + "at": "2026-10-04T15:16:02", + "model": "default", + "prompt_sha": "95ac16e43338", + "reply": { + "paragraphs": [ + { + "at": "Title (line 1)", + "text": "Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold summary (line 5)", + "text": "A cluster decides a change only when more than half of all its servers accept the change...", + "unresolved": [ + "decides a change", + "accept the change" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 7", + "text": "The rest of this page explains why the rule must be \"more than half\".", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 11", + "text": "A cluster is a group of n servers that keep copies of the same data...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, list (lines 13-15)", + "text": "A server accepts the change when... decided... quorum size... quorum", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 17", + "text": "The cluster decides changes one at a time, in order... For each position in that order, a server accepts only one change.", + "unresolved": [], + "leap": [ + "The rule that a server accepts only one change per position is stated with no reason. Its role only becomes clear at line 42." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 19", + "text": "If fewer than a quorum of servers answer, the cluster stops...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why not let the survivors go on?, line 23", + "text": "Take a cluster of 4 servers, A to D. C and D stop answering...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 25", + "text": "A sends a message to D and gets no reply in 500 ms... partition...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 27", + "text": "Letting A and B go on means quorum size 2...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Table (lines 29-32)", + "text": "Group | Accepts for change 7 | Votes | Quorum size | Result", + "unresolved": [], + "leap": [ + "\"change 7\": the number 7 comes with no setup. The reader has to infer that it is just an arbitrary position in the order." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 34", + "text": "Both groups decide, and the copies now disagree for good. This is split brain...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why more than half works, line 38", + "text": "Set the quorum size to more than half... majority size: floor(n/2) + 1...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 40", + "text": "Take any two quorums of 3... Together they list 6 servers, but the cluster has only 5...", + "unresolved": [], + "leap": [ + "The overlap argument is shown only for n = 5. The general version (two groups that are each more than n/2 add up to more than n) is never said in words." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 42", + "text": "C already accepted x = 1 for change 7, so C refuses x = 2 for change 7...", + "unresolved": [], + "leap": [ + "\"C already accepted x = 1 for change 7\": the example has moved from 4 servers to 5, and no 5-server scenario was set up in which A, B, C accepted x = 1 first. The reader has to rebuild the scenario." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 44", + "text": "This depends on one assumption: a server remembers what it accepted, even after a restart...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 46", + "text": "Predict: the network splits 5 servers into {A, B, C} and {D, E}...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 48", + "text": "A, B, C have 3 votes, and 3 ≥ 3, so they keep deciding...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 50", + "text": "Count more than half of all configured servers, failed or not...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Counting failures, lines 54-56", + "text": "Tolerance is how many servers can fail... tolerance = n − majority size", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Table (lines 58-64)", + "text": "Servers n | Majority size | Tolerance", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 66", + "text": "With 5 servers and 2 failed, 3 servers answer, and 3 ≥ 3... With 4 servers and 2 failed... the cluster stops.", + "unresolved": [], + "leap": [], + "excess": [ + "This mostly restates the opening bold paragraph and the table rows for 4 and 5, adding only the ≥/< comparison." + ], + "purpose_unclear": false + }, + { + "at": "Line 68", + "text": "A tolerance of 1 means a second failure stops the cluster...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 70", + "text": "Adding a server to an odd-sized cluster also raises the majority size by 1...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer, line 74", + "text": "Both clusters need 3 servers to decide a change, because 3 is the smallest group that must share a server with every other group...", + "unresolved": [], + "leap": [ + "\"every other group\" means every other group of that size (every other quorum). The wording could be read as any group at all." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, line 78", + "text": "Assumed: servers crash or become unreachable, but never send a false \"yes\"...", + "unresolved": [ + "PBFT algorithm" + ], + "leap": [ + "\"4 to tolerate 1 liar\" is given without a reason. That is acceptable because it is marked as out of scope." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, line 79", + "text": "Simplified: in real protocols such as Raft and Paxos...", + "unresolved": [ + "Raft", + "Paxos" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, line 80", + "text": "Not covered: how servers detect a failure...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [ + "Line 42, \"C already accepted x = 1 for change 7\": this is the core step showing why overlap prevents split brain, but the 5-server scenario it depends on (A, B, C accepting x = 1 first) is never set up. The only earlier change-7 example used 4 servers." + ], + "cuts": [ + "Line 66: shorten or merge it into the table. It repeats the bold opening paragraph.", + "Line 25: \"in 500 ms\" is a concrete detail the argument does not need. It is small and harmless." + ], + "question": { + "text": "Why can a 5-server cluster keep working with 2 servers down while a 4-server cluster can only lose 1? Behind that: why must the decision rule be \"more than half of all servers\"?", + "known_at": "Title (line 1), sharpened by line 7", + "result_stated": "Yes, in words, at line 5: \"For 5 servers, more than half is 3, so 5 − 3 = 2 servers can fail. For 4 servers, more than half is also 3, so only 4 − 3 = 1 server can fail.\"" + }, + "motive": "Reason to care about the question: the split-brain table and line 34, \"Both groups decide, and the copies now disagree for good.\" Reason for the approach: line 25, \"A sees the same silence in both cases, so A cannot know which case is true\", which explains why survivors cannot simply go on. Line 70 adds a practical payoff: \"That is why clusters usually have an odd number of servers.\"", + "close": "Yes. Line 74: \"Both clusters need 3 servers to decide a change, because 3 is the smallest group that must share a server with every other group. 5 servers have 2 to spare. 4 servers have only 1.\" This states the general overlap argument in words, though only loosely.", + "recap": "A cluster needs a majority of all its servers to agree, so that any two deciding groups share a server that blocks conflicting decisions; 4 and 5 servers both need 3, so 5 has one more server to spare.", + "strongest": "Lines 25-34: the indistinguishable-silence point leads straight into the partition table, which shows split brain concretely.", + "weakest": "Line 42: the switch from 4 to 5 servers leaves out the setup in which C first accepted x = 1.", + "suggestions": [ + "Before line 42, add one sentence such as \"Suppose A, B, C have already accepted x = 1 for change 7, and D, E now receive x = 2.\"", + "After the 5-server overlap example at line 40, add the general sentence: two groups that each hold more than half of n servers together count more than n, so they must share a server.", + "Mention once at line 17 why a server accepts only one change per position (so that a shared server can block a conflict), or point ahead to line 42." + ] + }, + "usage": { + "cost": 0.1212488, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Line 42, \"C already accepted x = 1 for change 7\": this is the core step showing why overlap prevents split brain, but the 5-server scenario it depends on (A, B, C accepting x = 1 first) is never set up. The only earlier change-7 example used 4 servers." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Line 66: shorten or merge it into the table. It repeats the bold opening paragraph." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "Line 25: \"in 500 ms\" is a concrete detail the argument does not need. It is small and harmless." + }, + { + "n": 4, + "severity": "edge", + "kind": "unresolved", + "text": "Bold summary (line 5): decides a change" + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Bold summary (line 5): accept the change" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "The rule, line 17: The rule that a server accepts only one change per position is stated with no reason. Its role only becomes clear at line 42." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Table (lines 29-32): \"change 7\": the number 7 comes with no setup. The reader has to infer that it is just an arbitrary position in the order." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "Line 40: The overlap argument is shown only for n = 5. The general version (two groups that are each more than n/2 add up to more than n) is never said in words." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Line 42: \"C already accepted x = 1 for change 7\": the example has moved from 4 servers to 5, and no 5-server scenario was set up in which A, B, C accepted x = 1 first. The reader has to rebuild the scenario." + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "Answer, line 74: \"every other group\" means every other group of that size (every other quorum). The wording could be read as any group at all." + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Limits, line 78: PBFT algorithm" + }, + { + "n": 12, + "severity": "edge", + "kind": "leap", + "text": "Limits, line 78: \"4 to tolerate 1 liar\" is given without a reason. That is acceptable because it is marked as out of scope." + }, + { + "n": 13, + "severity": "edge", + "kind": "unresolved", + "text": "Limits, line 79: Raft" + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Limits, line 79: Paxos" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151643-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151643-quiz-prose.json new file mode 100644 index 0000000..ab664ca --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151643-quiz-prose.json @@ -0,0 +1,171 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "quorum", + "at": "2026-10-04T15:16:29", + "model": "default", + "prompt_sha": "fda2e8264e06", + "reply": { + "answers": { + "1": "A cluster of n servers holding copies of the same data; the client that proposes a change (e.g. x = 1); the change and its position in the order (change 7); a server's acceptance (\"yes\", stored on disk); a quorum, meaning a group of servers of quorum size; the majority size floor(n/2)+1; a partition, meaning network groups that cannot reach each other; split brain; tolerance.", + "2": "A client proposes a change to the cluster's servers. Each server accepts at most one change per position. A change is decided when a quorum of servers accept it. The quorum size is set to the majority of all n configured servers, so any two quorums share at least one server. A server in both quorums refuses a second, conflicting change, so two sides of a partition cannot both decide. Tolerance is n minus the majority size.", + "3": "Any two majorities of the same n servers must share a server. That server accepts only one change per position, so only one change can win each position, and split brain cannot happen. This holds even when a partition looks the same as a crash. The price is that the cluster stops once fewer than a majority of servers answer.", + "4": "The key variable is n, the cluster size. Going from odd n to n+1 raises the majority size by 1, so tolerance stays the same and each change needs one more accept. Going from even n to n+1 leaves the majority size unchanged, so tolerance rises by 1. Separately, if the quorum size drops to half or less, two disjoint groups can both decide and split brain becomes possible.", + "5": "Assumption: a server remembers what it accepted, even after a restart. A second one: servers never send a false \"yes\". Observation or fact: two quorums of 3 among 5 servers list 6 members, so they must share at least one server. Also: a server cannot tell a crashed peer from a cut network link, because both look like the same silence.", + "6": "A cluster decides a change only when more than half of all its configured servers accept it, and each server accepts only one change per position. A silent server may have crashed, or the network may be partitioned. Letting a smaller group continue could let two sides decide different values (split brain). Any two majorities overlap, and the shared server refuses the second value, so only one value can win. Tolerance is n minus the majority size: 5 servers need 3 and can lose 2, while 4 servers also need 3 and can lose only 1, which is why odd sizes are preferred.", + "7": "The hardest part was the jump from the 5-server overlap example to the general formula floor(n/2)+1. The page never shows that two quorums of that size always add up to more than n, for even n as well as odd. A one-line count for n=4 (3+3=6>4) and n=6 (4+4=8>6) would have closed that gap. A smaller point: the table uses both 'Accepts' and 'Votes'.", + "8": "2. The majority size is 4, and 6 − 4 = 2.", + "9": "Two groups of 2 among 4 servers need not share a server. In a partition {A,B} | {C,D}, both groups could decide different values for the same position, which is split brain.", + "10": "The group of 3 has 3 ≥ 3 votes, so it keeps deciding. The group of 2 has 2 < 3, so it stops: it keeps the changes already decided but decides no new ones until enough servers return.", + "11": "They cannot tell whether C and D crashed or whether the network is cut and C and D are still running and deciding together. If it is a partition, both pairs could decide different values (split brain). The majority of 4 is 3, so 2 servers must stop.", + "12": "A quorum of 2 among 4 servers lets two disjoint pairs each decide during a partition. The result is split brain, where the copies disagree for good. Surviving 2 failures would cost correctness.", + "13": "It tolerates 2 failures. The majority size is floor(6/2)+1 = 4, and 6 − 4 = 2. The 6th server raises the quorum to 4 without adding tolerance compared with 5 servers." + }, + "audit": { + "terms": [ + "'Accepts' and 'Votes' name the same thing in the table header (line 29), and 'votes' is used in lines 42 and 48. Line 13 defines only 'accepts'.", + "'quorum size' (line 15) and 'majority size' (line 38) are used as near-synonyms. The answer section says 'need 3 servers' without either term.", + "'change 7' (line 29) and 'position in that order' (line 17) refer to the same slot.", + "'fail' (lines 5, 54), 'stop answering' (line 23), 'crashed' (line 25) and 'down' (line 68) are used loosely for the same failure." + ], + "unexplained": [ + "'500 ms' (line 25) is given with no meaning. Would a longer or shorter wait change anything? The text implies it would not, but never says so.", + "'4 to tolerate 1 liar' (line 78) gives a number with no reason behind it." + ], + "missing_why": [ + "'floor(n/2) + 1' (line 38): the overlap proof is only shown for n=5 (line 40). It never states that two quorums of this size always total more than n, for even n too.", + "'3 is the smallest group that must share a server with every other group' (line 74): 'every other group' presumably means every other group of quorum size, but this is not stated, and why 2 fails for n=5 is not shown.", + "'Adding a server to an odd-sized cluster also raises the majority size by 1' (line 70): the page does not say why adding to an even-sized cluster doesn't." + ], + "excess": [ + "The HTML comment on line 3 ('Stage 1, rendered from model.md in STE-80') means nothing to this reader.", + "The PBFT bullet (line 78) is a detour that no question or step of the argument uses.", + "The Raft/Paxos 'Simplified' bullet (line 79) names protocols that are never explained. It is harmless but unused." + ] + }, + "gaps": [ + "There is no general argument that majorities overlap for every n, only the 5-server example.", + "It does not say what happens to a change that was accepted by some servers but never decided, for example A and B having accepted x = 1 during a partition.", + "It does not say whether the client learns that a change failed or stalled when the cluster stops." + ] + }, + "grade": { + "scores": { + "1": 1, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 1, + "13": 2 + }, + "reasons": { + "1": "Covers most entities, but 'failure' and 'decide' are not listed as entities, and 'quorum size' is only implied inside the quorum definition. It also adds 'position in the order (change 7)', which model.md does not have.", + "12": "Explains that two disjoint pairs could each decide (split brain), but never states the rule the item expects: the quorum size must be more than half of n, so two groups cannot both decide. It shows why 2 fails, not the general requirement.", + "13": "Correct for n = 6 (majority size 4, 6 − 4 = 2, no gain over 5 servers). The rubric's expect line describes n = 5 and n = 4 rather than 6; the answer was scored against the question as asked." + }, + "pass": false + }, + "usage": { + "cost": 0.1574386, + "seconds": 40 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "'Accepts' and 'Votes' name the same thing in the table header (line 29), and 'votes' is used in lines 42 and 48. Line 13 defines only 'accepts'." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'quorum size' (line 15) and 'majority size' (line 38) are used as near-synonyms. The answer section says 'need 3 servers' without either term." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'change 7' (line 29) and 'position in that order' (line 17) refer to the same slot." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'fail' (lines 5, 54), 'stop answering' (line 23), 'crashed' (line 25) and 'down' (line 68) are used loosely for the same failure." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'500 ms' (line 25) is given with no meaning. Would a longer or shorter wait change anything? The text implies it would not, but never says so." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'4 to tolerate 1 liar' (line 78) gives a number with no reason behind it." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "'floor(n/2) + 1' (line 38): the overlap proof is only shown for n=5 (line 40). It never states that two quorums of this size always total more than n, for even n too." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "'3 is the smallest group that must share a server with every other group' (line 74): 'every other group' presumably means every other group of quorum size, but this is not stated, and why 2 fails for n=5 is not shown." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "'Adding a server to an odd-sized cluster also raises the majority size by 1' (line 70): the page does not say why adding to an even-sized cluster doesn't." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "The HTML comment on line 3 ('Stage 1, rendered from model.md in STE-80') means nothing to this reader." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "The PBFT bullet (line 78) is a detour that no question or step of the argument uses." + }, + { + "n": 12, + "severity": "excess", + "kind": "excess", + "text": "The Raft/Paxos 'Simplified' bullet (line 79) names protocols that are never explained. It is harmless but unused." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "There is no general argument that majorities overlap for every n, only the 5-server example." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "It does not say what happens to a change that was accepted by some servers but never decided, for example A and B having accepted x = 1 during a partition." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "It does not say whether the client learns that a change failed or stalled when the cluster stops." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151817-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151817-quiz-prose.json new file mode 100644 index 0000000..8ec8ed0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151817-quiz-prose.json @@ -0,0 +1,141 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "quorum", + "at": "2026-10-04T15:18:10", + "model": "default", + "prompt_sha": "45bb3c67ca87", + "reply": { + "answers": { + "1": "Servers (n of them) that form a cluster, each holding a copy of the same data. Clients, which propose changes (e.g. x = 1). Changes, which are decided one at a time at numbered positions (change 1, 2, … 7). Accepts: a server stores the change on disk and replies 'yes'. The quorum size and a quorum, which is a group of that many servers. The majority size, floor(n/2)+1. Failures: a crash or an unreachable server. A partition: the network splits into groups that cannot reach each other. Split brain: copies that disagree for good. Tolerance: how many servers can fail while the cluster still decides changes.", + "2": "A client proposes a change to the cluster. Each server accepts at most one change per position. A change is decided when a quorum of servers has accepted it. The quorum size is set to the majority size of all n configured servers. Any two quorums then share at least one server, and that server refuses a second, different value for the same position, so two conflicting decisions cannot happen. Failures and partitions reduce how many servers answer. If fewer than a quorum answer, the cluster stops deciding but keeps what it has already decided. Tolerance = n − majority size.", + "3": "Because a quorum is more than half of all n servers, any two quorums overlap. The shared server has already accepted one value for a position and refuses another, so only one value can be decided and split brain cannot happen. The cost is that the cluster stops when fewer than a majority of servers answer, so it can lose only n − (floor(n/2)+1) servers.", + "4": "The key variable is n, the number of servers. Going from an even n to the next odd n keeps the majority size the same (4→5: both 3), so tolerance rises by 1. Going from an odd n to the next even n raises the majority size by 1 (5→6: 3→4), so tolerance stays the same, and each change just has to wait for one more accept. A second variable is the quorum size. If it drops to half or less, two disjoint groups can both decide, which gives split brain. If it is raised, tolerance drops.", + "5": "Assumption: a server remembers what it accepted even after a restart (its disk is not wiped), and servers never send a false 'yes'. Simplification: in real protocols the shared server reports x = 1 to the new group instead of just refusing. Fact: a server that hears silence cannot tell a crashed peer from a network cut, however long it waits. Also, two groups of more than half of n must share a server, because together they list more than n servers.", + "6": "A cluster decides a change only when a quorum of servers has accepted it, and each server accepts only one change per position. A server cannot tell whether a silent peer crashed or was cut off by the network. So if a quorum could be half or less, both sides of a partition could decide different values (split brain). Requiring more than half of all configured servers, floor(n/2)+1, means any two quorums share a server, and that server refuses the conflicting value. The cluster therefore survives n − majority failures: 2 for n=5, but only 1 for n=4, because 4 and 5 both need 3 accepts.", + "7": "The overlap argument is the hardest part, especially the step 'C refuses'. It depends on the rule from much earlier that a server accepts only one change per position, and the example does not point back to that rule. It also leaves open what happens if a quorum never finished deciding x = 1; the Limits note on Raft/Paxos hints at this but does not resolve it. A one-line reminder of the one-change-per-position rule at the refusal step would help. So would a short note that floor(n/2)+1 is just 'the smallest whole number above n/2'.", + "8": "Majority size for 6 is floor(6/2)+1 = 4, so tolerance is 6 − 4 = 2 failed servers.", + "9": "Two groups of 2 among 4 servers need not share a server. In a partition {A,B} | {C,D}, both sides could decide different values for change 7 (x=1 and x=2), and the copies would disagree for good. That is split brain.", + "10": "The group of 3 has 3 ≥ 3 (the majority size), so it keeps deciding changes. The group of 2 has 2 < 3, so it stops: it keeps the changes already decided but decides nothing new until it can reach enough servers again. The copies cannot disagree.", + "11": "A and B cannot tell whether C and D crashed or are just cut off by a network partition and still running. If C and D are alive on the other side, letting A and B decide with 2 servers would also let C and D decide with 2, and the two sides could decide different values: split brain. 2 is not more than half of 4, so the two groups need not overlap.", + "12": "A quorum of 2 is not more than half of 4. Two disjoint pairs could each decide a different value for the same position during a partition, which produces split brain. Surviving 2 failures this way would cost correctness. The cluster must use quorum 3, which gives tolerance 1.", + "13": "Both need a majority of 3 accepts, because floor(4/2)+1 = floor(5/2)+1 = 3. With 5 servers, 3 still answer after 2 failures, so the cluster keeps deciding. With 4 servers, only 2 answer after 2 failures, and 2 < 3, so it stops. Tolerance is 5−3=2 versus 4−3=1." + }, + "audit": { + "terms": [ + "The required number of accepts is called both 'quorum size' (line 13: 'The number of accepts needed is the *quorum size*') and 'majority size' (line 37: 'the *majority size*: floor(n/2) + 1 ... The quorum size is always the majority size'), and the tolerance formula then uses only 'majority size' (line 55). The reader has to keep two names for one number.", + "Failure is described as 'fails' (line 14), 'stops answering' (line 14), 'become unreachable' (line 77), and 'crash' (line 77). Line 14 does tie these together, so the cost to the reader is small.", + "'Accepts' is used both as a value accepted and as a count in the same table (line 28: column 'Accepts for change 7' holds x = 1, while column 'Accepts received' holds 2)." + ], + "unexplained": [ + "'floor(n/2) + 1' (line 37) is given without saying what floor does, i.e. round down, so for odd n it is the next whole number above n/2. A school-maths reader may not know the notation." + ], + "missing_why": [ + "Line 41, 'C already accepted x = 1 for that position, so C refuses': the refusal depends on the rule in line 16 ('a server accepts only one change' per position), but the text does not refer back to it, and line 16 does not say why servers follow that rule.", + "Line 78, 'the new group keeps x = 1': no reason is given for why the new group should adopt the old value. This is the step that makes the protocol make progress rather than just block.", + "Line 77, 'Servers that lie need larger clusters': stated with no reason and no indication of how much larger." + ], + "excess": [ + "The bold opening (line 3) and the 'Answer' section (line 73) say almost the same thing. The repeat adds only the overlap reason." + ] + }, + "gaps": [ + "The explanation does not say what happens when a change has been accepted by fewer than a quorum before a partition or crash (a partial accept), or how a value stuck at position 7 gets resolved. It is only gestured at in Limits.", + "The single-server example (C refuses) assumes the first decision finished before the second attempt. It does not cover two clients proposing different values at the same moment.", + "It does not say how many more servers are needed when servers can lie (line 77)." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "8": "Full marks: correct count (2) from majority size 4. It omits the expected remark that the sixth server adds nothing over 5. The question does not require that remark." + }, + "pass": true + }, + "usage": { + "cost": 0.1621684, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The required number of accepts is called both 'quorum size' (line 13: 'The number of accepts needed is the *quorum size*') and 'majority size' (line 37: 'the *majority size*: floor(n/2) + 1 ... The quorum size is always the majority size'), and the tolerance formula then uses only 'majority size' (line 55). The reader has to keep two names for one number." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Failure is described as 'fails' (line 14), 'stops answering' (line 14), 'become unreachable' (line 77), and 'crash' (line 77). Line 14 does tie these together, so the cost to the reader is small." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'Accepts' is used both as a value accepted and as a count in the same table (line 28: column 'Accepts for change 7' holds x = 1, while column 'Accepts received' holds 2)." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'floor(n/2) + 1' (line 37) is given without saying what floor does, i.e. round down, so for odd n it is the next whole number above n/2. A school-maths reader may not know the notation." + }, + { + "n": 5, + "severity": "gap", + "kind": "missing_why", + "text": "Line 41, 'C already accepted x = 1 for that position, so C refuses': the refusal depends on the rule in line 16 ('a server accepts only one change' per position), but the text does not refer back to it, and line 16 does not say why servers follow that rule." + }, + { + "n": 6, + "severity": "gap", + "kind": "missing_why", + "text": "Line 78, 'the new group keeps x = 1': no reason is given for why the new group should adopt the old value. This is the step that makes the protocol make progress rather than just block." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "Line 77, 'Servers that lie need larger clusters': stated with no reason and no indication of how much larger." + }, + { + "n": 8, + "severity": "excess", + "kind": "excess", + "text": "The bold opening (line 3) and the 'Answer' section (line 73) say almost the same thing. The repeat adds only the overlap reason." + }, + { + "n": 9, + "severity": "gap", + "kind": "gap", + "text": "The explanation does not say what happens when a change has been accepted by fewer than a quorum before a partition or crash (a partial accept), or how a value stuck at position 7 gets resolved. It is only gestured at in Limits." + }, + { + "n": 10, + "severity": "gap", + "kind": "gap", + "text": "The single-server example (C refuses) assumes the first decision finished before the second attempt. It does not cover two clients proposing different values at the same moment." + }, + { + "n": 11, + "severity": "gap", + "kind": "gap", + "text": "It does not say how many more servers are needed when servers can lie (line 77)." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151859-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151859-coldread-prose.json new file mode 100644 index 0000000..9f11f36 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/explainers/quorum/review/runs/2026-10-04T151859-coldread-prose.json @@ -0,0 +1,317 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "quorum", + "at": "2026-10-04T15:18:59", + "model": "default", + "prompt_sha": "95ac16e43338", + "reply": { + "paragraphs": [ + { + "at": "Title (line 1)", + "text": "Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Bold lead (line 3)", + "text": "A cluster decides a change only when more than half of all its servers accept the change...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Line 5", + "text": "The rest of this page explains why the rule must be \"more than half\".", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 9", + "text": "A *cluster* is a group of n servers that keep copies of the same data...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, definition list (lines 11-14)", + "text": "A server *accepts*... *decided*... *quorum size*... *quorum*... *fails*...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 16", + "text": "The cluster decides changes one at a time, in order... For each position in that order, a server accepts only one change...", + "unresolved": [], + "leap": [ + "\"For each position in that order, a server accepts only one change\": this is given as a rule with no reason. The reason only becomes clear at line 41, where this rule is what makes C refuse. A small stumble, not a block." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The rule, line 18", + "text": "If fewer than a quorum of servers answer, the cluster *stops*...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why not let the survivors go on?, line 22", + "text": "Take a cluster of 4 servers, A to D. C and D fail. Why can A and B not go on alone?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 24", + "text": "A sends a message to D and gets no reply... *partition*... A cannot know which case is true.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 26", + "text": "Letting A and B go on means quorum size 2. Now suppose the cause is a partition...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Partition table (lines 28-31)", + "text": "Group | Accepts for change 7 | Accepts received | Quorum size | Result", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 33", + "text": "Both groups decide... *split brain*... So the quorum size must be more than half of n: then two groups cannot both decide.", + "unresolved": [], + "leap": [ + "\"So the quorum size must be more than half of n\": the example only shows that 2 out of 4 (exactly half) fails. It does not say here why 'more than half' is the fix, or why that fix is enough. That comes in the next section, so this works as a forward pointer, but the 'So' moves faster than the reasoning." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Why more than half works, line 37", + "text": "More than half of all n servers is the *majority size*: floor(n/2) + 1... The quorum size is always the majority size.", + "unresolved": [ + "floor(n/2)" + ], + "leap": [ + "\"The quorum size is always the majority size\": the text never says why the quorum isn't set larger than a majority. A larger quorum would also overlap. The real reason, that the smallest safe quorum gives the most tolerance, is never stated." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 39", + "text": "Two quorums each hold more than half of the servers, so together they list more than n servers. Some server must therefore be in both...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 41", + "text": "The shared server blocks split brain. Suppose A, B, C accepted x = 1 for change 7... C refuses...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 43", + "text": "This depends on one assumption: a server remembers what it accepted, even after a restart...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Predict (line 45)", + "text": "**Predict:** the network splits 5 servers into {A, B, C} and {D, E}. What does each side do?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 47", + "text": "A, B, C have 3 accepts, and 3 ≥ 3, so they keep deciding. D and E have 2, and 2 < 3, so they stop. The copies cannot disagree.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 49", + "text": "Count more than half of *all* configured servers, failed or not...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Counting failures, line 53-55", + "text": "*Tolerance* is how many servers can fail while the cluster still decides changes: tolerance = n − majority size", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Tolerance table (lines 57-62)", + "text": "Servers n | Majority size | Tolerance; 3..7", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 65", + "text": "With 5 servers and 2 failed, 3 servers answer, and 3 ≥ 3: the cluster keeps deciding. With 4 servers and 2 failed... the cluster stops.", + "unresolved": [], + "leap": [], + "excess": [ + "The whole paragraph restates the bold lead (line 3) and the n = 4 and n = 5 rows of the table just above it, adding only the ≥/< comparison." + ], + "purpose_unclear": false + }, + { + "at": "line 67", + "text": "A tolerance of 1 means a second failure stops the cluster. A tolerance of 2 means one server can be off for an upgrade while another one fails.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 69", + "text": "Going from an even n to the next odd n keeps the majority size... That is why clusters usually have an odd number of servers.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Answer (line 73)", + "text": "Both clusters need 3 accepts to decide a change, because 3 is more than half of 4 and of 5, and only groups of more than half must share a server with every other quorum...", + "unresolved": [], + "leap": [ + "\"only groups of more than half must share a server with every other quorum\": the wording is tangled. The reader has to reread it to see that it means 'only quorums of more than half are guaranteed to overlap'. 'Must' can be read as an obligation rather than a guarantee." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, item 1", + "text": "**Assumed:** servers crash or become unreachable, but never send a false \"yes\". Servers that lie need larger clusters.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, item 2", + "text": "**Simplified:** in real protocols (Raft, Paxos), the shared server does more than refuse...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Limits, item 3", + "text": "**Not covered:** how servers detect a failure, how the stopped side catches up..., and how reads work.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Line 65 ('With 5 servers and 2 failed, 3 servers answer...'): it repeats the bold lead and the table. Cut it, or fold the ≥/< check into the table.", + "Line 37 'The quorum size is always the majority size.': cut it or replace it with the reason (it is the smallest size that overlaps). As written it only restates the rule from line 3." + ], + "question": { + "text": "Why can 5 servers survive 2 failures while 4 servers survive only 1? Under that sits a second question: why must the decision rule be 'more than half'?", + "known_at": "The title (line 1); the second question is stated at line 5.", + "result_stated": "Yes, at once and in words in the bold lead: 'For 5 servers, more than half is 3, so 5 − 3 = 2 servers can fail. For 4 servers, more than half is also 3, so only 4 − 3 = 1 server can fail.'" + }, + "motive": "Reason to care about the question: this is weak at the start and only appears late, at line 67 ('one server can be off for an upgrade while another one fails') and line 69 ('That is why clusters usually have an odd number of servers'). Reason for the approach: line 5 ('explains why the rule must be \"more than half\"'), backed by the partition and split-brain example at lines 24-33, which shows that a server cannot tell a crash from a cut network.", + "close": "Yes. Line 73 ('Both clusters need 3 accepts to decide a change, because 3 is more than half of 4 and of 5, and only groups of more than half must share a server with every other quorum. 5 servers have 2 to spare. 4 servers have only 1.') answers the title question and states the overlap argument in words, though the overlap clause is awkwardly phrased.", + "recap": "A cluster can only safely decide with more than half of all its servers, because only then do any two deciding groups share a server that refuses to agree twice. 4 and 5 servers both need 3, so 5 has two to spare and 4 has one.", + "strongest": "Lines 24-33: a server cannot tell a crashed peer from a cut network, and the partition table shows two halves of 4 both deciding. That makes split brain concrete.", + "weakest": "Line 37 ('The quorum size is always the majority size') together with the Answer clause at line 73 ('only groups of more than half must share a server'). The text never says why the quorum is the smallest majority rather than any larger size, and the final wording of the overlap argument is tangled.", + "suggestions": [ + "At line 37, add one sentence: a larger quorum would also be safe, but it would lower tolerance, so the smallest majority is chosen.", + "Rephrase line 73 as 'groups of more than half always share a server, so two of them can never decide different values'.", + "Put a one-line motive near the start, e.g. that choosing the cluster size decides how many simultaneous failures or maintenance outages you can survive.", + "At line 16, briefly say why a server accepts only one change per position, or point ahead to line 41." + ] + }, + "usage": { + "cost": 0.1264728, + "seconds": 41 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Line 65 ('With 5 servers and 2 failed, 3 servers answer...'): it repeats the bold lead and the table. Cut it, or fold the ≥/< check into the table." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Line 37 'The quorum size is always the majority size.': cut it or replace it with the reason (it is the smallest size that overlaps). As written it only restates the rule from line 3." + }, + { + "n": 3, + "severity": "edge", + "kind": "leap", + "text": "The rule, line 16: \"For each position in that order, a server accepts only one change\": this is given as a rule with no reason. The reason only becomes clear at line 41, where this rule is what makes C refuse. A small stumble, not a block." + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "line 33: \"So the quorum size must be more than half of n\": the example only shows that 2 out of 4 (exactly half) fails. It does not say here why 'more than half' is the fix, or why that fix is enough. That comes in the next section, so this works as a forward pointer, but the 'So' moves faster than the reasoning." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "Why more than half works, line 37: floor(n/2)" + }, + { + "n": 6, + "severity": "edge", + "kind": "leap", + "text": "Why more than half works, line 37: \"The quorum size is always the majority size\": the text never says why the quorum isn't set larger than a majority. A larger quorum would also overlap. The real reason, that the smallest safe quorum gives the most tolerance, is never stated." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Answer (line 73): \"only groups of more than half must share a server with every other quorum\": the wording is tangled. The reader has to reread it to see that it means 'only quorums of more than half are guaranteed to overlap'. 'Must' can be read as an obligation rather than a guarantee." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/read.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/read.json new file mode 100644 index 0000000..2d68bc7 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/read.json @@ -0,0 +1,41 @@ +{ + "answers": { + "1": { + "answer": "A change is decided (committed/final) only when more than half of all the cluster's configured servers accept it, meaning they store it on disk and reply \"yes\". The explanation says \"decided\" rather than \"committed\".", + "quote": "A cluster decides a change only when more than half of all its servers accept the change." + }, + "2": { + "answer": "Any two groups that are each more than half must share at least one server. That server has already accepted one change for a given position, so it refuses a conflicting one. Two groups that need not overlap (for example 2 and 2 out of 4) could both decide different values during a partition. That is split brain.", + "quote": "Two quorums each hold more than half of the servers, so together they list more than n servers. Some server must therefore be in both." + }, + "3": { + "answer": "2. The majority is 4, and 6 − 4 = 2, the same as with 5 servers.", + "quote": "| 6 | 4 | 2 |" + }, + "4": { + "answer": "The group of 3 can still commit, because 3 meets the majority of 3 out of all 5 configured servers. The group of 2 stops, because 2 < 3, and it must count against all 5 servers, not just the 2 it can reach.", + "quote": "A, B, C have 3 accepts, and 3 ≥ 3, so they keep deciding. D and E have 2, and 2 < 3, so they stop. The copies cannot disagree." + } + }, + "stops": [ + "'committed' is never used. The page says 'decided', and a reader has to map the two: \"When enough servers accept, the change is *decided*: it is final.\"", + "The positions idea is introduced quickly and its purpose only becomes clear later: \"For each position in that order, a server accepts only one change. The examples below use change 7, an arbitrary position.\"", + "floor notation appears without explanation: \"floor(n/2) + 1\"", + "It is unclear why a larger quorum lets fewer servers fail. This is stated without a reason: \"a larger quorum is also safe, but it lets fewer servers fail.\"", + "The restart assumption feels sudden, and it is unclear how real systems ensure it: \"a server remembers what it accepted, even after a restart.\"", + "Raft and Paxos are named without explanation: \"in real protocols (Raft, Paxos), the shared server does more than refuse.\"", + "The simplified refusal raises a question that is left open: how does the new group keep x = 1? \"It reports x = 1 to the new group, and the new group keeps x = 1.\"", + "\"Servers that lie need larger clusters.\" says how much larger only implicitly." + ], + "excess": [ + "The final Answer section repeats the opening summary: \"Both clusters need 3 accepts to decide a change, because 3 is more than half of 4 and of 5.\"", + "The upgrade aside is slightly off the question: \"A tolerance of 2 means one server can be off for an upgrade while another one fails.\"", + "The line about 4 servers repeats the table: \"With 4 servers and 2 failed, only 2 servers answer, and 2 < 3: the cluster stops.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/published/run.json b/evals/artifact/results/2026-10-04-dev/work/quorum/published/run.json new file mode 100644 index 0000000..77eadae --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/quorum/explanation.md` (prose, plus the `model.md`, `model.yaml` and `narrative.md` it is built from), because the answer is a short chain of counting that one table can carry, so a diagram or interactive page would add little. The blind test scored 13/13, the last cold read had no blocking findings, and `explainer check` passes; after the blind test I made three small wording edits and cut one repeated sentence, and did not rerun the blind test.", + "error": null, + "cost": 1.4309656, + "seconds": 589, + "turns": 59, + "files": [ + "explainers/quorum/explanation.md" + ], + "review_cost": 0.9567, + "reviews": 7 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/explanation.md b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/explanation.md new file mode 100644 index 0000000..2697985 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/explanation.md @@ -0,0 +1,48 @@ +# Why do 5 servers survive 2 failures, but 4 servers survive only 1? + +**Both clusters need 3 servers to agree before they make a decision. 5 − 3 = 2 servers can fail. 4 − 3 = 1 server can fail.** The number 3 comes from one rule: the agreeing group must be a majority. Without that rule, two halves of a cluster can decide different things at the same time. + +## The setup + +A **cluster** is a group of servers that keep copies of the same data. Each change, such as "x = 7", is a **decision**. All servers must end up with the same decisions in the same order. + +A **quorum** is the minimum number of servers that must agree before the cluster makes a decision. A server that does not reply is a **failure**. + + +The cluster continues while at least a quorum of servers can reply. So the number of failures it tolerates is *servers − quorum*: + +- 5 servers, quorum 3. Two servers fail. Three reply. 3 ≥ 3, so the cluster continues. +- 4 servers, quorum 3. Two servers fail. Two reply. 2 < 3, so the cluster stops. + +The question is now: why is the quorum 3 for 4 servers, and not 2? + +## Why the quorum must be a majority + + +A server cannot tell a crashed server from a server that it cannot reach. Both look the same: no reply. A **partition** is a network fault that splits the servers into groups that cannot reach each other. + + +Try quorum 2 with 4 servers. A partition splits them into group A (2 servers) and group B (2 servers). Each group sees two silent servers. Each group has 2 servers, which is a quorum. Group A decides x = 7. Group B decides x = 9. When the network heals, the cluster holds two conflicting values and has no correct way to choose. This is **split brain**. + +The fix is to make two quorums impossible at the same time. Two groups of size *q* fit into *n* servers without a shared server only if 2*q* ≤ *n*. So choose 2*q* > *n*: the smallest such *q* is a majority, ⌊*n*/2⌋ + 1. Any two majorities then share at least one server, and that server refuses to agree to two conflicting decisions. + +- 5 servers: 3 + 3 = 6 > 5. A split of 3 + 2 lets only the side with 3 decide. +- 4 servers: 2 + 2 = 4, not > 4. The quorum must rise to 3. A 2 + 2 split now stops both sides. That is safe, but it is the "2 failures stop the cluster" case from above. + +## The pattern + +| Servers | Quorum (majority) | Failures tolerated | +|---|---|---| +| 3 | 2 | 1 | +| 4 | 3 | 1 | +| 5 | 3 | 2 | +| 6 | 4 | 2 | +| 7 | 4 | 3 | + + +Adding a server to an odd cluster raises the quorum by one, so the tolerance stays the same. Going from an even to an odd count adds one tolerated failure. To tolerate *f* failures, use 2*f* + 1 servers. The fourth server is not useless: it holds another copy and can serve reads. But it adds no failure tolerance, and it is one more server that can fail. This is why clusters (etcd, ZooKeeper, Consul) normally run 3, 5, or 7 servers. + +## Limits + +- **Assumption:** a failed server stops or goes silent. If servers can send false messages (Byzantine faults), the rule becomes 3*f* + 1 servers. +- This text does not cover how protocols such as Raft and Paxos use quorums to elect a leader and copy the log. diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/model.md b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/model.md new file mode 100644 index 0000000..eed640d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/model.md @@ -0,0 +1,97 @@ +# Semantic model: majority quorums and failure tolerance + +## Central question + +Why does a cluster of 5 servers survive 2 failures, but a cluster of 4 survives only 1? + +## Audience and prior knowledge + +A technical reader: school mathematics, basic programming. Knows what a server and a network are. Does not know consensus, quorum, network partition, or split brain. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| cluster | n servers that keep copies of the same data and must agree on each change. | fact | +| decision | a change that the cluster accepts, such as "x = 7". | fact | +| quorum | the minimum number of servers that must agree before the cluster makes a decision; here a majority, floor(n/2) + 1. | fact (design rule) | +| failure | a server that does not reply: crashed, or cut off by the network. | fact | +| partition | a network fault that splits the servers into groups that cannot reach each other. | fact | +| split brain | two groups each make a decision, and the decisions conflict. | fact | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| quorum | prevents | split brain | two majorities always share a server | +| failure | reduces | servers that can reply | cluster stops when replies < quorum | +| n | sets | quorum | q = floor(n/2) + 1 | + +## Dependencies (learning order) + +1. A silent server can be dead or only cut off; the others cannot tell which. +2. So a group must be sure that no other group can decide at the same time. +3. Majority guarantees this; it sets the quorum. +4. Failures tolerated = n − quorum. + +## Causal chain + +```text +crash looks like partition → each group must assume the others are alive +→ only one group may decide → quorum must be a majority (two majorities overlap) +→ tolerated failures = n − majority → 5 gives 2, 4 gives 1 +``` + +## Quantities + +- quorum q = floor(n/2) + 1. n=3 → 2, n=4 → 3, n=5 → 3, n=6 → 4, n=7 → 4. +- failures tolerated f = n − q. n=3 → 1, n=4 → 1, n=5 → 2, n=6 → 2, n=7 → 3. +- servers needed to tolerate f failures: n = 2f + 1. +- Overlap test: two groups of size q both fit without sharing a server only if 2q ≤ n. Majority makes 2q > n. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| 4 servers, quorum 2 (half) | partition 2 + 2: both sides decide | 2 + 2 = 4 is not > 4; no shared server | +| 4 servers, quorum 3 | partition 2 + 2: neither side decides; cluster stops | safe, but 2 failures stop it | +| 5 servers, quorum 3 | partition 3 + 2: only the 3 side decides | 3 + 3 = 6 > 5 | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| quorum = majority | quorum = half (2 of 4) | split 2 + 2: side A writes x = 7, side B writes x = 9; both have a quorum; split brain | +| quorum = majority | "count the servers still alive" | a crash and a partition look the same; each side of a 2 + 2 split sees 2 alive and decides | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| tolerance = n − majority | 5 servers, 2 fail: 3 reply ≥ 3, cluster continues | 4 servers, 2 fail: 2 reply < 3, cluster stops | +| even n adds no tolerance | 3 → 1, 4 → 1 | — | + +## Terms + +- cluster, decision, quorum, failure, partition, split brain: as in Entities. + +## Epistemic status + +- **Mathematical consequence:** overlap of majorities; f = n − q. +- **Assumed:** servers fail by stopping or going silent, not by lying. + +## Confusion points + +- "With 4 servers, 3 are left after 1 failure, so 2 more could go" → quorum is 3, not "any survivors". +- "The 4th server is useless" → it adds read capacity and a spare copy; it adds no failure tolerance, and it adds one more server that can fail. + +## Scope + +- **Out of scope:** servers that lie (Byzantine faults) need n = 3f + 1 — see PBFT. +- **Out of scope:** how Raft and Paxos use quorums for leader election and log replication. + +## Representation decision + +- **Stage:** 1 +- **Reason:** the argument is a short chain plus one comparison table; no topology the reader must hold. +- **Levels:** L1 the rule / L2 failures vs quorum / L3 why majority / L4 n = 2f + 1. diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/model.yaml b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/model.yaml new file mode 100644 index 0000000..54a482a --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/explainers/quorum/model.yaml @@ -0,0 +1,51 @@ +values: + n5: + servers: 5 + quorum: 3 + tolerated: 2 + n4: + servers: 4 + quorum: 3 + tolerated: 1 + n3: {servers: 3, quorum: 2, tolerated: 1} + n6: {servers: 6, quorum: 4, tolerated: 2} + n7: {servers: 7, quorum: 4, tolerated: 3} + bad_quorum_4: 2 + conflict: {a: 7, b: 9} + +allow: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + +require: + - path: n5/tolerated + in: [prose] + - path: n4/quorum + in: [prose] + +claims: + - id: tolerance + kind: mechanism + statement: "Failures tolerated = servers − quorum; the cluster stops when fewer than a quorum reply." + example: "5 servers, quorum 3: 2 fail, 3 reply, cluster continues. 4 servers, quorum 3: 2 fail, 2 reply, cluster stops." + - id: why-majority + kind: why + statement: "The quorum must be a majority so that two groups cannot both decide." + alternative: "quorum = half of the servers (2 of 4)" + counterexample: "A partition splits 4 servers 2 + 2; both sides reach quorum 2; one writes x = 7, the other x = 9." + ask: "With 4 servers, why can the quorum not be 2?" + - id: silent + kind: definition + statement: "A crashed server and a cut-off server look the same: no reply." + - id: even-n + kind: guarantee + statement: "Adding a server to an odd cluster raises the quorum but not the tolerance." + example: "3 servers tolerate 1; 4 servers tolerate 1; 5 tolerate 2." + counterexample: "Going from 4 to 5 (even to odd) adds tolerance: 1 to 2." + +accept_unjustified: {log: "'log' is the Raft replicated log named in Scope, not a logarithm"} + +terms: + - term: quorum + means: "the minimum number of servers that must agree before a decision" + avoid: [majority vote count] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/read.json new file mode 100644 index 0000000..9e67231 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/read.json @@ -0,0 +1,40 @@ +{ + "answers": { + "1": { + "answer": "The text speaks of a 'decision' rather than a 'write'. A change is made only when at least a quorum of servers agree. The quorum is a majority, ⌊n/2⌋ + 1.", + "quote": "A **quorum** is the minimum number of servers that must agree before the cluster makes a decision." + }, + "2": { + "answer": "Any two majorities share at least one server, so two quorums cannot exist at the same time. The shared server refuses to agree to conflicting decisions. This prevents split brain during a partition. A smaller quorum, such as 2 of 4, would let both halves decide different values.", + "quote": "Any two majorities then share at least one server, and that server refuses to agree to two conflicting decisions." + }, + "3": { + "answer": "2 failures. The quorum is 4, and 6 − 4 = 2.", + "quote": "| 6 | 4 | 2 |" + }, + "4": { + "answer": "Only the group of 3 can commit. The quorum for 5 servers is 3. The group of 3 meets it and the group of 2 does not.", + "quote": "5 servers: 3 + 3 = 6 > 5. A split of 3 + 2 lets only the side with 3 decide." + } + }, + "stops": [ + "'write' and 'committed' (the words in the questions) are never used. The text says 'decision' and 'decide', so the reader has to map one set of words onto the other.", + "'Two groups of size *q* fit into *n* servers without a shared server only if 2*q* ≤ *n*': a compressed pigeonhole step that a new reader would need to reread.", + "'that server refuses to agree to two conflicting decisions': it is not explained how the server knows the decisions conflict, or why it refuses.", + "'⌊*n*/2⌋ + 1': the floor notation is not explained.", + "'Adding a server to an odd cluster raises the quorum by one, so the tolerance stays the same': the wording is easy to misread on a first pass.", + "'Byzantine faults' / '3*f* + 1': the term and the formula appear without a reason.", + "'it is the \"2 failures stop the cluster\" case from above': the reader has to link a partition back to the earlier crash example." + ], + "excess": [ + "'This is why clusters (etcd, ZooKeeper, Consul) normally run 3, 5, or 7 servers.': product names the reader does not need.", + "'This text does not cover how protocols such as Raft and Paxos use quorums': off the question.", + "'The fourth server is not useless: it holds another copy and can serve reads.': a side point." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/quorum/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/run.json new file mode 100644 index 0000000..a29b461 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/quorum/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/quorum/explanation.md`, a short prose explanation with one table. I chose prose because the answer is a short chain of reasoning: a silent server can't be told apart from a cut-off one, so the group that decides must be a majority, and the number of failures a cluster survives is servers minus that majority. Worked numbers and the table cover it without needing a diagram or an interactive page. The supporting model files, `explainers/quorum/model.md` and `model.yaml`, are in the same folder, and `explainer check quorum` passes.", + "error": null, + "cost": 0.41601699999999997, + "seconds": 71, + "turns": 21, + "files": [ + "explainers/quorum/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/none/explainers/two-tides/explanation.md b/evals/artifact/results/2026-10-04-dev/work/two-tides/none/explainers/two-tides/explanation.md new file mode 100644 index 0000000..cf7dbc9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/none/explainers/two-tides/explanation.md @@ -0,0 +1,143 @@ +# Why are there two high tides a day, not one? + +If the Moon's gravity pulls the ocean toward it, you would expect one heap of water on the side facing the Moon. As the Earth turns, each place would pass under that heap once a day, so you would get one high tide a day. + +In fact most coasts get two high tides a day, about 12 hours 25 minutes apart. So there must be a second heap of water on the side of the Earth facing *away* from the Moon. This explainer shows where that second heap comes from. + +The short answer is this: **tides are not caused by the Moon's pull. They are caused by the *difference* in the Moon's pull between one part of the Earth and another.** That difference points toward the Moon on the near side and away from the Moon on the far side. So the ocean is stretched in both directions. + +There is an interactive companion, [`index.html`](index.html), where you can switch between "the pull" and "the difference in pull" and watch the two bulges form. It's worth opening once you reach section 2. + +--- + +## 1. The whole Earth is falling toward the Moon + +The Moon's gravity acts on all of the Earth, not just the oceans. It pulls on the rock, the core, you, and the water. As a result the Earth as a whole accelerates toward the Moon. (That acceleration is what keeps the Earth and Moon orbiting their common centre of mass. If neither was moving sideways, they would simply fall together.) + +The Earth is in free fall toward the Moon, in the same way an astronaut in orbit is in free fall toward the Earth. An astronaut in the space station doesn't feel the Earth's gravity as a force, because they and the station are falling together. In the same way, the Earth's surface does not "feel" the Moon's average pull. Everything on and in the Earth gets that average pull, so it moves along with the planet and nothing shifts relative to anything else. + +What *can* be felt is any part of the Moon's pull that is **different from the average**. The average here means the pull at the Earth's centre. + +## 2. Gravity weakens with distance, so the pull is uneven + +Gravity falls off as 1/r². The Earth is about 12,700 km across, and the Moon is about 384,000 km away. So different parts of the Earth are at noticeably different distances from the Moon: + +| Location | Distance to Moon | Moon's pull compared with the centre | +|---|---|---| +| Near side (Moon overhead) | d − R | a bit **stronger** | +| Earth's centre | d | the reference | +| Far side (Moon underfoot) | d + R | a bit **weaker** | + +Here *d* is the Earth–Moon distance and *R* is the Earth's radius. + +Now subtract the pull at the centre from the pull at each point. What is left over is called the **tidal acceleration**: + +- **Near side:** the pull is stronger than at the centre, so the leftover points *toward* the Moon. Water there is pulled toward the Moon a bit more than the Earth is, and it rises. +- **Far side:** the pull is weaker than at the centre, so the leftover points *away* from the Moon. The Earth is pulled toward the Moon a bit more than the water on the far side is. The water is left behind, and it rises too. +- **The two sides at 90°:** these points are about as far from the Moon as the centre is. But the pull there is aimed at the Moon, which is not quite parallel to the pull at the centre. The pulls at those points tilt slightly inward toward the Earth–Moon line, so the leftover points *inward*. Water there is squeezed down. + +The result is a stretch along the Earth–Moon line and a squeeze across it. The ocean takes on a slightly rugby-ball shape, with **two bulges: one under the Moon and one opposite it.** + +``` + squeeze + ↓ + ← ( · · · · · ) → ● Moon + bulge ↑ bulge + squeeze + (leftover pull points AWAY (leftover pull points TOWARD + from the Moon on the far side) the Moon on the near side) +``` + +The far-side bulge is often explained as a "centrifugal effect" of the Earth–Moon orbit. You can get the right answer that way if you are careful about reference frames, but you don't need it. The two bulges would still appear if the Earth and Moon fell straight toward each other with no orbit at all. Uneven gravity combined with free fall is enough. + +## 3. A little algebra + +Write *GM* for the Moon's mass times the gravitational constant. + +At the near-side point, the leftover pull is: + +``` +GM/(d−R)² − GM/d² +``` + +Factor out GM/d², and use the approximation (1 − x)⁻² ≈ 1 + 2x for small x (here x = R/d ≈ 1/60): + +``` +GM/d² · [ (1 − R/d)⁻² − 1 ] ≈ GM/d² · 2R/d = 2GMR/d³ (toward the Moon) +``` + +On the far side the same steps give about 2GMR/d³ *away* from the Moon. That is the same size in the opposite direction, so the two bulges are nearly equal. + +On the sides, the pull tilts inward by an angle of about R/d. That gives an inward leftover of about GMR/d³, half as large. + +Two things stand out: + +1. **The tidal effect goes as 1/d³, not 1/d².** It is the *gradient* of gravity, and taking a gradient adds another power of distance. This will matter for the Sun in section 6. +2. **It is tiny.** With the real numbers (GM_Moon ≈ 4.9×10¹² m³/s², R ≈ 6.37×10⁶ m, d ≈ 3.84×10⁸ m), GMR/d³ ≈ 5.5×10⁻⁷ m/s². That is about one ten-millionth of the Earth's surface gravity. + +If you'd like to check the vector picture yourself, this short Python script computes the leftover pull at points around the Earth: + +```python +import math + +GM = 4.9e12 # Moon, m^3/s^2 +R = 6.37e6 # Earth radius, m +d = 3.84e8 # Earth–Moon distance, m + +def pull(x, y): + """Moon's gravitational acceleration at (x, y); Moon sits at (d, 0).""" + dx, dy = d - x, -y + r = math.hypot(dx, dy) + return GM * dx / r**3, GM * dy / r**3 + +cx, cy = pull(0, 0) # pull on Earth's centre +for deg in range(0, 360, 45): + a = math.radians(deg) + px, py = pull(R * math.cos(a), R * math.sin(a)) + tx, ty = px - cx, py - cy # tidal = local pull − centre pull + radial = tx * math.cos(a) + ty * math.sin(a) # + = outward (up) + print(f"{deg:3d}° tidal {radial:+.2e} m/s² ({'up' if radial > 0 else 'down'})") +``` + +The output shows "up" at 0° (facing the Moon) and at 180° (facing away), and "down" at 90° and 270°. That is two bulges. + +## 4. How can such a tiny force move an ocean? + +At the near and far points, the tidal acceleration points straight up, against gravity that is ten million times stronger. Up there it does almost nothing. + +What actually moves the water is the **sideways (horizontal) part** of the tidal acceleration. This part is largest about 45° away from the Moon's direction. Nothing opposes it directly. Gravity is vertical, so it cannot hold water in place sideways. Across thousands of kilometres of ocean, this small, steady sideways push herds water toward the two points on the Earth–Moon line. The water piles up there until the slope of the sea surface balances the push. + +If the Earth were covered in a deep ocean that could keep up with the Moon instantly, the result is called the **equilibrium tide**. At its high points the water would be about 0.36 m above average. Halfway between them it would be about 0.18 m below average. The range from high to low would be roughly half a metre. + +## 5. The Earth turns under the two bulges + +The bulges stay lined up with the Moon while the Earth rotates beneath them. A spot on the coast therefore passes: + +> under the near bulge (high) → the side (low) → under the far bulge (high) → the other side (low) → back to the start + +That is **two highs and two lows for each turn of the Earth relative to the Moon.** + +One turn relative to the Moon takes slightly more than 24 hours. While the Earth spins, the Moon moves about 13° along its orbit in the same direction, so the Earth needs about 50 extra minutes to catch up. This "lunar day" lasts about **24 h 50 min**. That is why high tides come about **12 h 25 min** apart, and why high tide is about 50 minutes later each day. + +## 6. The Sun does the same thing, more weakly + +The Sun pulls on the Earth about 180 times harder than the Moon does. But the tidal effect depends on mass/distance³, not mass/distance². The Sun is about 390 times farther away than the Moon, and that wipes out most of its advantage. **The Sun's tidal effect is about 46% of the Moon's.** + +The Sun makes its own pair of bulges, which line up with the Sun. When the Sun, Earth and Moon are in a line (at new and full moon), the two sets of bulges add together and you get larger **spring tides**. When the Sun and Moon are at right angles (at the quarter moons), the Sun's bulges partly fill in the Moon's lows, and you get smaller **neap tides**. This happens twice a month. + +## 7. What real coasts add + +The two-bulge picture explains *why* the basic rhythm is twice a day. Real tides are shaped by much more: + +- **Continents get in the way.** Water can't flow freely around the planet. Instead, each ocean basin sloshes like water in a bathtub, driven twice a day by the tidal push. The size of the tide depends on whether a basin's natural sloshing period is close to 12.4 hours. That is why the Bay of Fundy has ranges over 15 m while the Mediterranean has only a few tens of centimetres. +- **High tide is usually not when the Moon is overhead.** Water takes time to respond, so high tide can lag the Moon by hours. In some basins the tide circulates around a point where the height hardly changes at all. +- **Some places get one tide a day.** The Moon is usually north or south of the equator, which tilts the bulges. A location off the equator then passes through one bulge more squarely than the other, so its two daily highs are unequal. In some basins, such as parts of the Gulf of Mexico, the basin's response strengthens that once-a-day part until it dominates. + +These are complications layered on top of the basic mechanism. The basic mechanism is still a stretch in two opposite directions, caused by gravity that varies across the Earth. + +## Summary + +- The Earth as a whole falls toward the Moon, so the Moon's *average* pull has no effect on the oceans. +- What remains is the *difference* between the pull at each point and the pull at the centre. It points toward the Moon on the near side and away from it on the far side. +- So the ocean bulges on both sides. Each place rotates through both bulges, which gives two high tides per 24 h 50 min. +- The effect goes as 1/d³, which is why the nearby Moon beats the much more massive Sun, about 2 to 1. diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/none/explainers/two-tides/index.html b/evals/artifact/results/2026-10-04-dev/work/two-tides/none/explainers/two-tides/index.html new file mode 100644 index 0000000..7a2eb3d --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/none/explainers/two-tides/index.html @@ -0,0 +1,245 @@ + + + + +Two tides a day — interactive + + + +

Why two high tides? An interactive view

+

Companion to explanation.md. The Earth is shown from above the North Pole. The Moon is kept fixed on the right, so the Earth appears to turn once every 24 h 50 min, which is one lunar day.

+ +
+
+ + +
+
+ +
+
+ 1. Which arrows to show + + + +
+ +
+ 2. Moon distance (arrows only) + +
+ +
+ 3. Time + + +
+ +
+ 4. Add the Sun + + +
+ +
+ Things to try +
    +
  1. Choose the first arrow option and set the distance to 60. The arrows all look almost identical, because the whole Earth is pulled almost uniformly. Now drag the distance down to 3. The near-side arrows get longer and the far-side arrows get shorter.
    2. +
  2. Switch to "minus pull on the centre". Arrows point out toward the Moon, out away from the Moon, and in at the sides. That pattern is what makes the two bulges. It looks the same at every distance, and only its strength changes.
    3. +
  3. Press Play and watch the white dot ("you"). It passes under a bulge twice each lunar day, and the graph shows two peaks.
    4. +
  4. Turn on the Sun. Set days to 0 (new moon) or about 14.8 (full moon) for spring tides, or about 7.4 or 22 (quarter moons) for neap tides.
    5. +
+

The water bulge is exaggerated about 100,000 times. The heights are for an idealised ocean covering the whole planet (the "equilibrium tide"), not for any real coast. Arrows show the Moon's effect only.

+
+
+
+ + + + diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/none/read.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/none/read.json new file mode 100644 index 0000000..ceee771 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/none/read.json @@ -0,0 +1,45 @@ +{ + "answers": { + "1": { + "answer": "The whole Earth is in free fall toward the Moon, pulled by the Moon's pull at its centre. On the far side the Moon's pull is weaker than at the centre because that side is farther away, and gravity weakens with distance. The Earth is pulled toward the Moon more than the far-side water is, so the water is left behind and rises.", + "quote": "the pull is weaker than at the centre, so the leftover points *away* from the Moon. The Earth is pulled toward the Moon a bit more than the water on the far side is. The water is left behind, and it rises too." + }, + "2": { + "answer": "The bulges stay lined up with the Moon. While the Earth spins, the Moon moves about 13° along its orbit in the same direction, so one turn relative to the Moon (a lunar day) takes about 24 h 50 min. There are two highs per lunar day, which puts them about 12 h 25 min apart.", + "quote": "While the Earth spins, the Moon moves about 13° along its orbit in the same direction, so the Earth needs about 50 extra minutes to catch up. This \"lunar day\" lasts about **24 h 50 min**." + }, + "3": { + "answer": "At new and full moon, when the Sun, Earth and Moon are in a line. The Sun's pair of bulges then lines up with the Moon's, the two add together, and you get spring tides. (Real coasts also differ: basins whose natural sloshing period is close to 12.4 h have much larger tides, as in the Bay of Fundy.)", + "quote": "When the Sun, Earth and Moon are in a line (at new and full moon), the two sets of bulges add together and you get larger **spring tides**." + }, + "4": { + "answer": "Reasoning from the text: the extra 50 minutes comes only from the Moon moving along its orbit. Without that motion, one turn relative to the Moon would be just one Earth rotation, about 24 hours. With two highs per turn, high tides would come about every 12 hours. The explanation does not say this directly.", + "quote": "That is **two highs and two lows for each turn of the Earth relative to the Moon.**" + } + }, + "stops": [ + "Gradient is used without being defined: \"It is the *gradient* of gravity, and taking a gradient adds another power of distance.\"", + "The approximation is used without saying where it comes from: \"use the approximation (1 − x)⁻² ≈ 1 + 2x for small x\"", + "The sideways leftover is hard to picture: \"the pull there is aimed at the Moon, which is not quite parallel to the pull at the centre\"", + "The side result is stated without working: \"On the sides, the pull tilts inward by an angle of about R/d. That gives an inward leftover of about GMR/d³, half as large.\"", + "\"Up there\" is an unclear reference (does it mean the near and far points?): \"Up there it does almost nothing.\"", + "The 46% figure can't be checked because the Sun's mass is never given: \"The Sun's tidal effect is about 46% of the Moon's.\"", + "Reference frames are mentioned but never explained: \"if you are careful about reference frames, but you don't need it\"", + "This seems to clash with section 5 (the Moon moving along its orbit): \"The Moon is kept fixed on the right, so the Earth appears to turn once every 24 h 50 min\"", + "This is asserted without a reason: \"This part is largest about 45° away from the Moon's direction.\"", + "The second sentence is unclear: \"In some basins the tide circulates around a point where the height hardly changes at all.\"" + ], + "excess": [ + "The Python script: \"If you'd like to check the vector picture yourself, this short Python script computes the leftover pull at points around the Earth\"", + "The centrifugal aside: \"The far-side bulge is often explained as a \"centrifugal effect\" of the Earth–Moon orbit.\"", + "The summary repeats earlier sections: \"The Earth as a whole falls toward the Moon, so the Moon's *average* pull has no effect on the oceans.\"", + "The diurnal-tide detail is off the main question: \"In some basins, such as parts of the Gulf of Mexico, the basin's response strengthens that once-a-day part until it dominates.\"", + "The opening short answer repeats the later sections: \"That difference points toward the Moon on the near side and away from the Moon on the far side. So the ocean is stretched in both directions.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/none/run.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/none/run.json new file mode 100644 index 0000000..cfe593f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/none/run.json @@ -0,0 +1,13 @@ +{ + "result": "I made `explainers/two-tides/explanation.md` and `explainers/two-tides/index.html`: the prose walks through the argument, with a little algebra and a runnable Python check, and the interactive page lets the reader flip from \"the pull\" to \"the pull minus the pull on the centre\" and watch the two bulges form, which a still image shows poorly (not tested in a browser).", + "error": null, + "cost": 0.4116578, + "seconds": 131, + "turns": 3, + "files": [ + "explainers/two-tides/explanation.md", + "explainers/two-tides/index.html" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/explainers/two-tides/explanation.md b/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/explainers/two-tides/explanation.md new file mode 100644 index 0000000..e13b407 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/explainers/two-tides/explanation.md @@ -0,0 +1,77 @@ +# Why are there two high tides a day, not one? + +**Short answer.** The Moon does not just pull the ocean toward itself. It pulls the near side of the Earth harder than the centre, and the centre harder than the far side. That *difference* in pull stretches the Earth's water along the Earth–Moon line. The result is two bulges: one facing the Moon and one facing away. The Earth turns under both bulges each day, so most coastlines get two high tides. + +## The obvious model fails + +The simplest idea is "the Moon's gravity pulls the water toward the Moon." That gives one bulge, on the side that faces the Moon. A place on the Earth would pass under it once per rotation: one high tide a day. + +Most coasts have two high tides a day, about 12 h 25 min apart. So the simple model is missing something. The missing part is that the Moon pulls on the *whole Earth*, not only on the water. + +## Step 1: the whole Earth falls toward the Moon + +The Earth and the Moon orbit their common centre of mass. In practice, the Earth is in free fall toward the Moon. The Moon's gravity accelerates the Earth's centre toward the Moon all the time. + +We live on the Earth and move with it. So what we see is not the Moon's pull itself. We see how the pull at each point **differs from** the pull at the centre. This difference is the **tidal acceleration**. + +> Before you read on: the far side is pulled toward the Moon less than the centre. Relative to the centre, which way does far-side water move? + +## Step 2: compute the pull at three points + +Gravity weakens with distance: acceleration = GM / d². Use the Moon's values: + +- GM of the Moon = 4.90 × 10¹² m³/s² +- Earth–Moon distance d = 384,400 km +- Earth radius R = 6,371 km + +| Point | Distance to Moon | Moon's pull (m/s²) | Minus pull at centre | +|---|---|---|---| +| Near side | d − R = 378,029 km | 3.429 × 10⁻⁵ | **+1.13 × 10⁻⁶** (toward Moon) | +| Centre | d = 384,400 km | 3.316 × 10⁻⁵ | 0 | +| Far side | d + R = 390,771 km | 3.209 × 10⁻⁵ | **−1.07 × 10⁻⁶** (away from Moon) | + +```text +Moon's pull on each point (longer arrow = stronger pull): + + far side centre near side Moon + •──> •───> •────> ● + +Subtract the pull at the centre (the whole Earth already has this motion): + + <• • •> + 1.07e-6 away zero 1.13e-6 toward +``` + +The near-side water moves toward the Moon faster than the Earth does, so it bulges toward the Moon. The far-side water moves toward the Moon *slower* than the Earth does. The Earth falls away from it, so it bulges away from the Moon. The two values are almost equal. Both are close to 2GMR/d³ = 1.10 × 10⁻⁶ m/s². + +At the two sides (90° from the Moon), the Moon's pull points slightly inward, toward the Earth–Moon line. This squeezes the water there by about GMR/d³ = 0.55 × 10⁻⁶ m/s². Water flows from the squeezed sides toward the stretched ends: + +```text + squeeze + ↓ + bulge ← ( Earth ) → bulge Moon ● + ↑ + squeeze +``` + +**Why the far side is not pushed by "centrifugal force from the Earth's spin":** the spin is symmetric about the Earth's axis. It does not know where the Moon is, so it cannot make a bulge that points away from the Moon. The far bulge comes only from the difference in the Moon's pull. + +## Step 3: what the number means + +1.1 × 10⁻⁶ m/s² is about one ten-millionth of the Earth's gravity (9.8 m/s²). It cannot lift water vertically in any useful way. It works because part of it points *along* the surface, and nothing opposes horizontal flow except the slope of the water itself. The ocean flows until the slope of its surface balances the horizontal tidal pull, over a distance of thousands of kilometres. + +For an ideal ocean that covers the whole Earth, the result is a difference of about half a metre between bulge and squeeze (model output, "equilibrium tide"). A larger tidal acceleration would give taller bulges. Because the effect goes as 1/d³, a Moon at half the distance would give 8 times the tidal acceleration. + +## Step 4: the Earth turns under the bulges + +The two bulges stay aligned with the Moon. The Earth turns under them. A point on the coast passes through bulge, squeeze, bulge, squeeze in one turn. That is two high tides and two low tides. + +The cycle is 24 h 50 min, not 24 h. In one day the Moon moves about 13° along its orbit, in the same direction as the Earth turns. The Earth needs about 50 more minutes to face the Moon again. So high tides come every 12 h 25 min, and they are about 50 minutes later each day. + +## The Sun does the same thing + +The Sun produces tidal bulges by the same mechanism. Its pull is much stronger than the Moon's, but tides depend on the *difference* across the Earth, which goes as M/d³. For the Sun, the result is about 46% of the Moon's effect. When the Sun and Moon line up (new and full moon), their bulges add: large **spring tides**. When they are at 90° (quarter moons), they partly cancel: small **neap tides**. + +## What this model leaves out + +This is the idealized model: an ocean that covers the Earth and responds at once. Real tides are also shaped by continents, ocean depth, and resonance of ocean basins. These change the timing and the height a lot (for example, more than 15 m in the Bay of Fundy). In some places, such as parts of the Gulf of Mexico, basin shape and the Moon's tilt relative to the equator make one of the two daily tides dominate, so the coast sees one high tide a day. The two bulges are still the cause; local geography decides how they show up. diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/read.json new file mode 100644 index 0000000..20f24b7 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/read.json @@ -0,0 +1,42 @@ +{ + "answers": { + "1": { + "answer": "The Moon pulls the far side less than it pulls Earth's centre. The whole Earth falls toward the Moon faster than the far-side water does, so relative to the Earth that water is left behind and bulges away from the Moon.", + "quote": "The far-side water moves toward the Moon *slower* than the Earth does. The Earth falls away from it, so it bulges away from the Moon." + }, + "2": { + "answer": "The Moon moves about 13° along its orbit each day, in the same direction the Earth turns. So the Earth needs about 50 extra minutes to face the Moon again. A full cycle takes 24 h 50 min, and two bulges per cycle gives about 12 h 25 min between high tides.", + "quote": "In one day the Moon moves about 13° along its orbit, in the same direction as the Earth turns. The Earth needs about 50 more minutes to face the Moon again." + }, + "3": { + "answer": "At new and full moon (spring tides). The Sun and Moon line up, so their bulges add together. At quarter moons they are at 90° and partly cancel, giving small neap tides.", + "quote": "When the Sun and Moon line up (new and full moon), their bulges add: large **spring tides**." + }, + "4": { + "answer": "Reasoning from the text: the extra 50 minutes comes only from the Moon's orbital motion. Without it, the cycle would be 24 h, so high tides would come every 12 h. This is not stated directly.", + "quote": "The Earth needs about 50 more minutes to face the Moon again. So high tides come every 12 h 25 min" + } + }, + "stops": [ + "\"In practice, the Earth is in free fall toward the Moon.\" It isn't explained how orbiting a common centre of mass counts as free fall, or why that means we only feel differences.", + "\"acceleration = GM / d²\": G and M are never defined; GM is just given as a number.", + "\"Both are close to 2GMR/d³\": the formula appears without saying where the 2 and the R/d³ come from.", + "\"the Moon's pull points slightly inward, toward the Earth–Moon line\": no reason is given for why it points inward at the sides.", + "\"It works because part of it points *along* the surface\": it's unclear which part points along the surface, and where.", + "\"model output, \"equilibrium tide\"\": the term is named but not explained, and it's unclear what model produced the half metre.", + "\"the spin is symmetric about the Earth's axis. It does not know where the Moon is\": this rebuts a misconception the reader may never have held, and it uses phrasing that is a bit abstract.", + "\"resonance of ocean basins\": not defined.", + "\"the Moon's tilt relative to the equator\": appears without any explanation of how it leads to one tide a day." + ], + "excess": [ + "\"| Near side | d − R = 378,029 km | 3.429 × 10⁻⁵ |\": the full numeric table is more detail than the 'why' needs.", + "\"Because the effect goes as 1/d³, a Moon at half the distance would give 8 times the tidal acceleration.\": off the main question.", + "\"for example, more than 15 m in the Bay of Fundy\": a side example." + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/run.json new file mode 100644 index 0000000..d6485d4 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made one file, `explainers/two-tides/explanation.md`: prose with two small inline text diagrams, because the explanation is a short chain of steps (pull at three points, minus the pull at the centre, Earth turning under two bulges) that a worked numeric table and an arrow sketch cover without needing an interactive page.", + "error": null, + "cost": 0.1563002, + "seconds": 42, + "turns": 2, + "files": [ + "explainers/two-tides/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/explanation.md b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/explanation.md new file mode 100644 index 0000000..8115d92 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/explanation.md @@ -0,0 +1,74 @@ +# Why are there two high tides a day, not one? + +Most coasts get two high tides a day. One comes when the Moon is high in the sky. The other comes when the Moon is on the opposite side of the Earth. If the Moon simply pulls water toward itself, the second high tide makes no sense. + +**The Moon does not lift the ocean toward itself. It stretches the ocean along the Earth–Moon line, and a stretch has two ends. The Earth turns through both ends, so a place gets two high tides.** + +## The pull at three points + +The Moon's pull is the acceleration that the Moon's gravity gives at a point. Take three points on the Earth–Moon line: the **near side** (the surface point that faces the Moon), Earth's **centre**, and the **far side** (the surface point opposite the Moon). The unit is µm/s², one millionth of a m/s². + +| Point | Distance to Moon | Moon's pull (toward the Moon) | +|---|---|---| +| near side | 378,029 km | 34.32 µm/s² | +| centre | 384,400 km | 33.19 µm/s² | +| far side | 390,771 km | 32.12 µm/s² | + +All three pulls point toward the Moon. So why does the far-side water not move toward the Moon? + +## Only the part that differs from the centre's pull acts + + +The Earth and the Moon orbit their shared centre of mass. An orbit is free fall: the whole Earth falls toward the Moon, and Earth's centre accelerates at 33.19 µm/s². An acceleration that every part shares does not change the shape of a falling body. + +So subtract the centre's pull. The remainder is the **tidal acceleration**: the Moon's pull at a point minus the Moon's pull at Earth's centre. Here + means toward the Moon. + +- Near side: 34.32 − 33.19 = +1.13. The near-side water falls toward the Moon faster than the centre, so it moves ahead of the centre, toward the Moon. +- Far side: 32.12 − 33.19 = −1.07. The far-side water falls toward the Moon slower than the centre, so it is left behind, away from the Moon. + + +Both results point away from Earth's centre. The Moon does not push the far side away. It pulls the far side less than it pulls the centre, and the centre's fall sets the reference. The near value is about 5% larger because the pull changes faster closer to the Moon. + +Now take a **side point**, 90° from the near side. Its pull has almost the same size as the centre's, 33.18 µm/s², so along the Earth–Moon line the two cancel. But the pull points at the Moon, from 6371 km off the line, so it tilts by R/d = 6371 / 384,400 = 0.0166 rad (0.95°) toward Earth's centre. That tilt leaves 33.19 × 0.0166 = 0.55 µm/s², pointing toward Earth's centre. So the tidal acceleration stretches the Earth along the Earth–Moon line and squeezes it across that line. + +## How a tiny acceleration moves an ocean + +1.1 µm/s² is about one ten-millionth of g (9.81 m/s²). Up or down, the tidal acceleration changes the weight of the water by 1 part in 10 million, which does nothing. But Earth's gravity has no part along a level surface, so the tidal acceleration is the only sideways push. It is largest 45° from the near side, at 0.82 µm/s², and points toward the nearer end of the line. Water flows until the sea surface slopes just enough that gravity along the slope balances this push. The water gathers into two **bulges**: one under the Moon, one opposite it. A **high tide** is the time when a place passes through a bulge. + +```text + side point: 0.55 µm/s² down, low water + | + ↙ v ↘ + . - ~~~~~~~~~~~ - . + far-side bulge <--( Earth )--> near-side bulge -----> Moon + (-1.07 µm/s²) ' - ~~~~~~~~~~~ - ' (+1.13 µm/s²) + ↖ ^ ↗ + | + side point + + ↙ ↘ ↖ ↗ : sideways push along the surface, 0.82 µm/s² at 45°, toward the nearer bulge +``` + +## Why subtract: two tests + + +The simplest idea, that the Moon's pull itself raises the tide, fails two tests. + + +First, imagine that something held the Earth still, so that it did not fall toward the Moon. Then nothing subtracts the centre's 33.19. The far side's 32.12 µm/s² points toward the Moon, which on the far side is down into the Earth. That Earth has one bulge, not two. The far bulge exists because the Earth falls. + +Second, the Sun. Its pull on the Earth is 179 times the Moon's, so the pull alone predicts solar tides 179 times larger. But a pull GM/r² changes by about 2R/d of itself across one Earth radius R, at distance d. So the tidal acceleration is about the pull times 2R/d. For the Moon: 33.19 × 2 × 6371 / 384,400 = 1.10, between 1.13 and 1.07. The Sun is 389 times farther than the Moon, so its tidal acceleration is about 179 / 389 = 0.46 times the Moon's. Observed solar tides are about half the Moon's. + +## Two high tides per turn + + +The bulges stay on the Earth–Moon line, and the Earth turns under them. The Moon also moves about 12° a day along its orbit, in the same direction that the Earth turns. The Earth turns about 15° an hour, so it needs about 50 minutes more than 24 hours to face the Moon again: a **lunar day** of 24 h 50 min. A place passes through both bulges in each lunar day, so high tides come every half lunar day: 12 h 25 min. A high tide at 8 in the morning gives the next at about 8:25 in the evening. + +## The answer + +The Moon pulls the near side more than Earth's centre, and the centre more than the far side. The Earth falls with its centre, so these differences stretch the ocean into two bulges, and the Earth turns through both each lunar day. No centrifugal force is needed. The pulls depend only on distance, so an Earth that fell straight at the Moon, with no orbit, would get the same +1.13 and −1.07 and the same two bulges. + +## Scope + +- **Height:** in an ideal ocean, the bulges rise about 0.36 m above the mean (an estimate). Look up "equilibrium tide". +- **Real oceans:** continents and basin shapes make tides lag the Moon, reach metres on some coasts, and come once a day on others. Look up "dynamic theory of tides". diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/model.md b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/model.md new file mode 100644 index 0000000..f6ab5a9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/model.md @@ -0,0 +1,144 @@ +# Semantic model: two high tides a day + +> Source of truth. Every rendering derives from this file. + +## Central question + +Why does a place on the coast get two high tides a day, when there is only one Moon pulling on one side of the Earth? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (Newton's law of gravity, inverse-square law, acceleration, vectors, free fall), basic programming. New to tides. Knows the Moon causes tides and that the Earth spins once a day. Does not know the idea of a tidal acceleration, or why the far side of the Earth has a bulge. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Earth | A ball of rock of radius 6371 km, covered mostly by ocean. | fact | +| Moon | The body that causes most of the tide, 384,400 km from Earth's centre on average. | fact | +| Moon's pull | The acceleration the Moon's gravity gives an object at a point; it points at the Moon and falls with the square of distance. | established fact (Newton) | +| near side | The point on Earth's surface that faces the Moon, 6371 km closer than the centre. | definition | +| far side | The point on Earth's surface opposite the Moon, 6371 km farther than the centre. | definition | +| side point | A surface point 90° from the near side; 384,453 km from the Moon. | definition | +| tidal acceleration | The Moon's pull at a point minus the Moon's pull at Earth's centre (a vector subtraction). | definition | +| bulge | Water gathered on the Earth–Moon line, because there the tidal acceleration points outward and along the surface it points toward the line. There are two: under the Moon and opposite the Moon. | model output (ideal ocean) | +| high tide | The time when a place passes through a bulge and its water level is highest. | observation | +| lunar day | 24 h 50 min: the time for the Moon to come back to the same place in the sky. | fact | + +## Relationships + +| From | Relationship (verb) | To | Notes | +|---|---|---|---| +| Moon's pull | accelerates | the whole Earth | Earth falls freely toward the Moon (as part of its orbit around the shared centre of mass). | +| Earth's free fall | cancels | the Moon's pull at the centre (33.19 µm/s²) | Earth's centre accelerates at 33.19 µm/s²; relative to the centre, each point feels only its own pull minus 33.19. | +| tidal acceleration | points away from Earth's centre at | near side and far side | Stretches Earth along the Earth–Moon line. | +| tidal acceleration | points toward Earth's centre at | the sides (90° from the Moon) | Squeezes in the other direction. Half the size. | +| tidal acceleration | moves water toward | the two bulges | Its sideways part along the surface moves the water. | +| Earth's rotation | carries a place through | both bulges each lunar day | Two high tides. | + +## Dependencies (learning order) + +1. The Moon's pull is weaker farther from the Moon (inverse square). +2. The whole Earth falls toward the Moon, so the pull it shares with every part does nothing to its shape. +3. Tidal acceleration = pull at a point − pull at the centre. +4. The tidal acceleration points away from the centre on both the near side and the far side → two bulges. +5. The Earth turns under the two bulges → two high tides per lunar day. + +## Causal chain + +```text +pull falls with distance → near side pulled more than centre, centre more than far side +→ Earth falls as a whole, so only the tidal acceleration acts +→ near side pulled ahead of centre; far side left behind centre +→ water stretched into two bulges on the Earth–Moon line +→ Earth turns under both bulges → two high tides each 24 h 50 min +``` + +## Temporal sequence + +1. A place passes under the Moon: high tide (near-side bulge). +2. About 6 h 12 min later the place is 90° away: low tide. +3. About 12 h 25 min after the first, the place is on the far side: second high tide (far-side bulge). +4. About 24 h 50 min after the first, the place is under the Moon again. + +## Quantities + +Constants: GM of Moon = 4.905 × 10¹² m³/s²; Earth–Moon distance d = 384,400 km; Earth radius R = 6371 km; GM of Sun = 1.327 × 10²⁰ m³/s²; Earth–Sun distance 1.496 × 10⁸ km. + +Moon's pull, a = GM/r², in µm/s² (1 µm/s² = 10⁻⁶ m/s²): + +| Point | Distance to Moon (km) | Moon's pull (µm/s²) | Pull minus centre's pull (µm/s²) | +|---|---|---|---| +| near side | 378,029 | 34.32 | +1.13 (toward Moon) | +| centre | 384,400 | 33.19 | 0 | +| far side | 390,771 | 32.12 | 1.07 away from Moon (−1.07) | +| side point | 384,453 | 33.18, tilted 0.95° toward the centre | 0.55 toward Earth's centre | + +- Side point: its pull has almost the same size as the centre's, but it points at the Moon, so it is tilted by R/d = 0.0166 rad toward Earth's centre. The vector difference is 33.19 × 0.0166 = 0.55 µm/s², pointing down. +- On the Earth–Moon line, GM/(d−R)² − GM/d² ≈ (GM/d²)(2R/d) = 2GMR/d³ = 1.10 µm/s². The exact values 1.13 and 1.07 differ from this by about 3%. The factor 1/d³ means the tidal acceleration falls with the cube of distance. +- Sideways part (along the surface): (3/2)(GMR/d³) sin 2θ, θ the angle from the near side. 0 at 0° and 90°; largest at 45°, 0.82 µm/s², pointing toward the nearer bulge. This part moves water. +- Meaning: 1.1 µm/s² is about one ten-millionth (1.1 × 10⁻⁷) of g = 9.81 m/s². Up or down, it changes weight by a negligible amount. Sideways, nothing opposes it except the slope of the sea, so over a whole ocean it gathers water into the bulges. Low value: zero at the centre. High value: near and far sides (up), 45° points (sideways). +- Sun: pull on Earth 5930 µm/s², about 179 times the Moon's pull. Sun's tidal acceleration ≈ 0.51 µm/s², about 0.46 of the Moon's, because the Sun is 389 times farther and the tidal acceleration falls as 1/d³. Check: tidal acceleration ≈ pull × 2R/d, so Sun/Moon ≈ 179 / 389 = 0.46. +- Why the vertical part does nothing and the sideways part acts: vertically the tidal acceleration adds to g = 9.81 m/s² and changes weight by 1 part in 10 million. Along a level surface gravity has no component, so the tidal acceleration is the only sideways push; water flows until the slope of the sea balances it. +- Near/far asymmetry: 1.13 vs 1.07, about 5% apart, because the pull changes faster closer to the Moon. Too small to matter here. +- Lunar day: the Earth turns about 15° an hour; the Moon moves about 12° per day along its orbit, in the same direction Earth turns, so Earth needs about 50 min more than 24 h to face the Moon again: 24 h 50 min. Two high tides per lunar day → one each 12 h 25 min. + +## Alternative states + +| State / scenario | What changes | Why | +|---|---|---| +| Earth held fixed (not falling toward the Moon) | One bulge only, under the Moon. The far side pull (32.12 µm/s²) points toward the Moon, that is down into the Earth. | Nothing subtracts the centre's pull; every point is pulled Moonward. | +| Uniform pull (same at every point) | No tide at all. | Every part falls together; 33.19 − 33.19 = 0 at every point. | +| Sun instead of Moon | Smaller tide (0.46×) despite 179× stronger pull. | Tide follows the tidal acceleration, which falls as 1/d³. | +| Earth falls straight at the Moon (no orbit) | Same two bulges: +1.13 and −1.07. | The pulls at the three points depend only on distance, not on orbital motion. | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| Subtract the centre's pull | Use the Moon's pull itself as the cause of the tide | Predicts one bulge (all three points pulled Moonward: 34.32, 33.19, 32.12 µm/s²), and predicts the Sun's tide 179× the Moon's. Observed: two bulges, Sun's tide about half the Moon's. | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| The far side is stretched away from the Moon (assumption: Earth falls freely toward the Moon) | Centre pulled 33.19, far side 32.12 µm/s²: the far side is left behind by 1.07 µm/s². | Earth held fixed: far side pulled 32.12 µm/s² toward the Moon and into the Earth; no far bulge. | +| Two bulges give two high tides a day (assumption: Earth turns under the Moon) | Earth turns once per 24 h 50 min relative to the Moon → high tide every 12 h 25 min. | — (the rotation is an observation; no alternative needed) | +| Mechanism: subtraction | 34.32 − 33.19 = +1.13; 32.12 − 33.19 = −1.07; side 0.55 inward. | — | + +## Terms + +- **Moon's pull**: the acceleration toward the Moon at a point, from Newton's law. Example 33.19 µm/s² at Earth's centre. +- **tidal acceleration**: Moon's pull at a point minus Moon's pull at the centre. Zero at the centre; +1.13 µm/s² at the near side. +- **near side / far side**: the surface points nearest to / farthest from the Moon. +- **bulge**: raised ocean on the Earth–Moon line; two of them. +- **high tide**: the time a place passes through a bulge. +- **side point**: a surface point 90° from the near side; tidal acceleration 0.55 µm/s² toward the centre. +- **lunar day**: 24 h 50 min; the time for the Moon to come back to the same place in the sky. + +## Epistemic status + +- **Observed / established:** Most coasts have two high tides per 24 h 50 min. Newton's law of gravity. +- **Mathematical consequence:** The tidal acceleration points outward on both near and far sides, with nearly equal size (1.13 vs 1.07). +- **Assumed:** Equilibrium tide: ocean covers the Earth and responds at once to the tidal acceleration. Real oceans do not; see Scope. +- **Estimated:** Equilibrium bulge height about 0.36 m above the mean at its peak (range about 0.5 m). + +## Confusion points + +- "The Moon pushes the far side away" → No. The Moon pulls the far side, but less than it pulls the centre. The far side is left behind. +- "The far bulge comes from centrifugal force of the Earth–Moon spin" → An unnecessary and misleading story. The tidal acceleration alone gives both bulges; it would exist even if Earth fell straight at the Moon with no orbit. +- "Stronger pull means bigger tide" → No; the Sun pulls 179× harder but raises a tide 0.46× the Moon's. The tidal acceleration matters, not the pull. +- "High tides come every 12 h" → Every 12 h 25 min, because the Moon moves on in its orbit. + +## Scope + +- **Out of scope:** Why real tides are often metres high, lag the Moon, or happen once a day at some places (for example the Gulf of Mexico) — the ocean basins respond as resonating bodies; continents block the bulges; the Moon's tilt makes the two bulges unequal at a given latitude. Look up "dynamic theory of tides" and "amphidromic points". +- **Out of scope:** How high the bulges are — in an ideal ocean, about 0.36 m above the mean at the peak (an estimate); real coasts amplify this. Look up "equilibrium tide". +- **Out of scope:** Spring and neap tides — the Sun's bulges add to or partly cancel the Moon's. +- **Out of scope:** The solid Earth also stretches, by a few tens of centimetres; the ocean tide is measured relative to the land. Look up "Earth tide". + +## Representation decision + +- **Stage:** 1 (prose) +- **Reason:** The argument is a short chain: three numbers, one subtraction, one rotation. A table of the three points carries the comparison; a small text figure carries the geometry. Nothing needs exploration with a parameter. +- **Levels:** L1 the answer / L2 the subtraction at three points / L3 why subtract (free fall; Sun case) / L4 not needed. diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/model.yaml b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/model.yaml new file mode 100644 index 0000000..1d3e1c3 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/model.yaml @@ -0,0 +1,87 @@ +# Machine-readable half of model.md. `explainer check two-tides` holds every rendering to it. + +values: + earth_radius_km: 6371 + moon_distance_km: 384400 + near_distance_km: 378029 + far_distance_km: 390771 + pull_near: 34.32 # µm/s² + pull_centre: 33.19 + pull_far: 32.12 + pull_side: 33.18 + tidal_near: 1.13 + tidal_far: 1.07 + tidal_far_signed: -1.07 + tidal_side: 0.55 + near_far_diff_pct: 5 + earth_turn_deg_per_h: 15 + side_tilt_deg: 0.95 + bulge_height_m: 0.36 + example_high_tide_h: 8 + example_next_high_tide_min: 25 + tidal_sideways_max: 0.82 + sun_pull_ratio: 179 + sun_tide_ratio: 0.46 + sun_distance_ratio: 389 + lunar_day: "24 h 50 min" + tide_interval: "12 h 25 min" + +allow: [2, 3, 45, 90, 24, 50, 12, 9.81, 10, 7, 0.0166, 1.1, 6] + +require: + - path: pull_centre + in: [prose] + - path: tidal_near + in: [prose] + - path: tidal_far + in: [prose] + - path: tide_interval + in: [prose] + +claims: + - id: mech-subtract + kind: mechanism + statement: "The tidal acceleration at a point is the Moon's pull there minus the Moon's pull at Earth's centre, because the whole Earth falls toward the Moon with the centre's acceleration." + example: "Near side 34.32 − 33.19 = +1.13 µm/s² (toward the Moon); far side 32.12 − 33.19 = −1.07 µm/s² (away from the Moon); side point 0.55 µm/s² toward Earth's centre." + ask: "On the far side of the Earth, which way does the tidal acceleration point, and why?" + - id: why-subtract + kind: why + statement: "Subtract the centre's pull; the Moon's pull itself does not predict the tides." + alternative: "Take the Moon's pull itself as the cause of the tide." + counterexample: "All points are pulled toward the Moon (34.32, 33.19, 32.12 µm/s²), so this predicts one bulge. It also predicts a solar tide 179 times the Moon's; the observed solar tide is 0.46 times the Moon's." + ask: "The Sun pulls on the Earth about 179 times harder than the Moon does. Why are its tides smaller?" + - id: guar-far-bulge + kind: guarantee + statement: "The far side has a bulge because the Earth falls freely toward the Moon." + example: "Centre pulled 33.19, far side 32.12 µm/s²: the far side is left behind by 1.07 µm/s²." + counterexample: "If the Earth were held fixed, the far side's 32.12 µm/s² would point toward the Moon, down into the Earth: no far bulge, one bulge only." + - id: mech-rotation + kind: mechanism + statement: "The Earth turns under the two bulges, so a place passes through a bulge twice per lunar day." + example: "Lunar day 24 h 50 min, so a high tide every 12 h 25 min." + ask: "Why are high tides about 12 h 25 min apart, not 12 h?" + - id: misc-push + kind: misconception + statement: "The Moon pushes the far-side water away." + correction: "The Moon pulls the far side less than it pulls the centre, so the far-side water is left behind as the Earth falls toward the Moon." + +terms: + - term: tidal acceleration + means: "the Moon's pull at a point minus the Moon's pull at Earth's centre" + avoid: [tide-raising force, differential acceleration, tidal force] + - term: Moon's pull + means: "the acceleration the Moon's gravity gives at a point" + avoid: [average pull] + +budget: {prose: 950, reason: "the subtraction at four points, why the sideways part acts, the why with the Sun counterexample, the held-Earth break case, the rotation timing, and a text figure"} + +quiz: + - q: "Why is there a high tide on the side of the Earth facing away from the Moon?" + expect: "The Moon pulls the far side less than it pulls Earth's centre. The whole Earth falls toward the Moon with the centre's acceleration, so the far-side water is left behind: its tidal acceleration points away from the Moon." + misconception: "The Moon pushes the far side away, or centrifugal force throws water out." + - q: "If the Earth were held fixed in space and did not fall toward the Moon, how many bulges would there be?" + expect: "One, under the Moon, because every point would be pulled toward the Moon and nothing subtracts the centre's pull." + misconception: "Still two." + - q: "A place has a high tide at 08:00. When is the next high tide, roughly, and why not exactly 12 hours later?" + expect: "About 20:25, because the Moon moves along its orbit, so the Earth needs 24 h 50 min to face it again, and there are two bulges in that time." + misconception: "20:00 exactly." diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/narrative.md b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/narrative.md new file mode 100644 index 0000000..46e8fd9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/narrative.md @@ -0,0 +1,52 @@ +# Narrative: two-tides + +## 1. Reader before and after + +- **Before:** A technical reader who knows Newton's inverse-square gravity, acceleration as a vector, free fall, and that the Earth turns once a day. Knows "the Moon causes tides". Has never asked why the side away from the Moon also has a high tide. +- **After:** "The Moon pulls the near side more than Earth's centre, and the centre more than the far side; because the whole Earth falls toward the Moon, only these differences act, and they stretch the ocean into two bulges that the Earth turns through each 24 h 50 min." + +## 2. Question and motive + +- **Question:** The Moon is on one side of the Earth. Why is the water high on both sides? Result in words: the Moon does not lift the water toward itself; it stretches the Earth's ocean along the Earth–Moon line, and a stretch has two ends. +- **Why care:** The obvious picture (the Moon pulls water toward it) predicts one high tide a day. Most coasts have two. The obvious picture is wrong in a way that teaches something about gravity. +- **Why this approach:** Ask what the pull is at three points, near side, centre, far side, with real numbers. Then ask which of those pulls can change the Earth's shape. + +## 3. Introduction ledger + +| Reference | Means | Grounded by | Beat | +|---|---|---|---| +| Moon's pull | acceleration from the Moon's gravity at a point | 33.19 µm/s² at Earth's centre | 2 | +| µm/s² | 10⁻⁶ m/s² | g is 9.81 m/s² = 9,810,000 µm/s² | 2 | +| near side / far side | surface points nearest / farthest from the Moon | 378,029 km / 390,771 km from the Moon | 2 | +| tidal acceleration | Moon's pull at a point minus the pull at the centre | +1.13 at the near side | 4 | +| side point | surface point 90° from the near side | 0.55 µm/s² toward the centre | 5 | +| bulge | water gathered on the Earth–Moon line | the two ends of the stretch | 6 | +| lunar day | 24 h 50 min, time for the Moon to come back to the same place in the sky | Moon moves about 12° a day | 8 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | Why two high tides? | — | question, then the answer in bold | Most coasts get two high tides a day, one when the Moon is overhead and one when it is on the other side of the Earth. Answer: the Moon stretches the ocean along the Earth–Moon line; a stretch has two ends; the Earth turns through both. | the result | +| 2 | instance | How strong is the pull at different points? | So compute it | table: near side 378,029 km, centre 384,400 km, far side 390,771 km | near side 34.32, centre 33.19, far side 32.12 µm/s², all toward the Moon | all three point toward the Moon, unequal | +| 3 | objection | Then why is the far water not pulled toward the Moon? | But | — | The Earth and Moon orbit their shared centre of mass; an orbit is free fall, so Earth's centre accelerates toward the Moon at 33.19. A shared acceleration changes nothing inside a falling body (astronauts float in an orbiting station). | only the part that differs from the centre's pull acts | +| 4 | name | What is left? | Therefore subtract | difference column; sign: + toward the Moon | tidal acceleration = pull − 33.19: near +1.13 (ahead of the centre, toward the Moon), far −1.07 (left behind, away from the Moon). Both point away from Earth's centre. | both ends pulled outward relative to the centre | +| 5 | mechanism | What about the sides? | And | side row | A side point is 90° from the near side. Its pull points at the Moon, so it tilts 0.95° toward Earth's centre; subtracting the centre's pull leaves 0.55 inward. | stretch along the line, squeeze across | +| 6 | mechanism | 1.1 µm/s² is about 10⁻⁷ g. How does it move water? | But it is tiny | ASCII figure | Up/down: it competes with g, so it changes weight by 1 part in 10 million; no effect. Along the surface: gravity has no part along a level surface, so the tidal acceleration (up to 0.82 at 45° from the near side) is the only sideways push; water flows until the sea slopes enough to balance it. Water gathers at two bulges. | two bulges | +| 7 | test | Does the subtraction matter? | So check it | — | If something held the Earth still, nothing would subtract 33.19; the far side's 32.12 points toward the Moon, into the Earth: one bulge. Sun: pull 179× the Moon's but 389× farther; the difference across Earth is about pull × 2R/d, so 179/389 = 0.46× tide. | why subtract | +| 8 | mechanism | When do high tides come? | Therefore | timeline | Earth turns under bulges. The Moon moves about 12° a day in the same direction, so Earth takes 24 h 50 min (a lunar day) to face it again; high tides 12 h 25 min apart. | timing | +| 9 | close | — | So | — | Two high tides because the Moon pulls the near side more and the far side less than Earth's centre, and the falling Earth feels only these differences. The far bulge needs no centrifugal force (a common textbook story). Scope: real heights, once-a-day coasts, spring/neap. | the answer | + +## 5. Concrete to symbol + +Pull = GM/r² first with numbers; then the approximation 2GMR/d³ only to explain the Sun's cube law. + +## 6. Links between representations + +The table and the text use the same four point names and the same numbers. + +## 7. Close the loop + +- **Answer:** Two high tides because the tidal acceleration stretches the ocean along the Earth–Moon line, toward the Moon on the near side and away on the far side. +- **General argument:** Any body falling in a non-uniform gravity field stretches along the line to the source. +- **Payoff:** Predict the next high tide (12 h 25 min later), why the Sun's tide is smaller. Scope: real tide heights, once-a-day tides, spring/neap. diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/decisions.yaml new file mode 100644 index 0000000..7836770 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/decisions.yaml @@ -0,0 +1,173 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T150203-probe#1: + decision: adopted +2026-10-04T150203-probe#10: + decision: adopted +2026-10-04T150203-probe#11: + decision: declined +2026-10-04T150203-probe#12: + decision: declined +2026-10-04T150203-probe#13: + decision: adopted +2026-10-04T150203-probe#14: + decision: adopted +2026-10-04T150203-probe#15: + decision: adopted +2026-10-04T150203-probe#2: + decision: adopted +2026-10-04T150203-probe#3: + decision: adopted +2026-10-04T150203-probe#4: + decision: adopted +2026-10-04T150203-probe#5: + decision: declined +2026-10-04T150203-probe#6: + decision: adopted +2026-10-04T150203-probe#7: + decision: adopted +2026-10-04T150203-probe#8: + decision: adopted +2026-10-04T150203-probe#9: + decision: adopted +2026-10-04T150440-coldread-narrative#1: + decision: adopted +2026-10-04T150440-coldread-narrative#10: + decision: adopted +2026-10-04T150440-coldread-narrative#11: + decision: adopted +2026-10-04T150440-coldread-narrative#12: + decision: adopted +2026-10-04T150440-coldread-narrative#13: + decision: adopted +2026-10-04T150440-coldread-narrative#14: + decision: adopted +2026-10-04T150440-coldread-narrative#15: + decision: declined +2026-10-04T150440-coldread-narrative#16: + decision: adopted +2026-10-04T150440-coldread-narrative#17: + decision: adopted +2026-10-04T150440-coldread-narrative#18: + decision: adopted +2026-10-04T150440-coldread-narrative#19: + decision: adopted +2026-10-04T150440-coldread-narrative#2: + decision: adopted +2026-10-04T150440-coldread-narrative#20: + decision: adopted +2026-10-04T150440-coldread-narrative#21: + decision: adopted +2026-10-04T150440-coldread-narrative#22: + decision: adopted +2026-10-04T150440-coldread-narrative#23: + decision: adopted +2026-10-04T150440-coldread-narrative#24: + decision: adopted +2026-10-04T150440-coldread-narrative#25: + decision: adopted +2026-10-04T150440-coldread-narrative#26: + decision: adopted +2026-10-04T150440-coldread-narrative#27: + decision: adopted +2026-10-04T150440-coldread-narrative#28: + decision: adopted +2026-10-04T150440-coldread-narrative#3: + decision: adopted +2026-10-04T150440-coldread-narrative#4: + decision: adopted +2026-10-04T150440-coldread-narrative#5: + decision: adopted +2026-10-04T150440-coldread-narrative#6: + decision: adopted +2026-10-04T150440-coldread-narrative#7: + decision: adopted +2026-10-04T150440-coldread-narrative#8: + decision: adopted +2026-10-04T150440-coldread-narrative#9: + decision: adopted +2026-10-04T150700-coldread-prose#1: + decision: adopted +2026-10-04T150700-coldread-prose#2: + decision: adopted +2026-10-04T150700-coldread-prose#3: + decision: adopted +2026-10-04T150700-coldread-prose#4: + decision: adopted +2026-10-04T150700-coldread-prose#5: + decision: declined +2026-10-04T150700-coldread-prose#6: + decision: adopted +2026-10-04T150700-coldread-prose#7: + decision: adopted +2026-10-04T150700-coldread-prose#8: + decision: adopted +2026-10-04T150821-quiz-prose#1: + decision: adopted +2026-10-04T150821-quiz-prose#10: + decision: declined +2026-10-04T150821-quiz-prose#11: + decision: adopted +2026-10-04T150821-quiz-prose#12: + decision: declined +2026-10-04T150821-quiz-prose#13: + decision: declined +2026-10-04T150821-quiz-prose#14: + decision: declined +2026-10-04T150821-quiz-prose#15: + decision: declined +2026-10-04T150821-quiz-prose#16: + decision: declined +2026-10-04T150821-quiz-prose#17: + decision: declined +2026-10-04T150821-quiz-prose#18: + decision: declined +2026-10-04T150821-quiz-prose#19: + decision: declined +2026-10-04T150821-quiz-prose#2: + decision: adopted +2026-10-04T150821-quiz-prose#20: + decision: adopted +2026-10-04T150821-quiz-prose#3: + decision: adopted +2026-10-04T150821-quiz-prose#4: + decision: adopted +2026-10-04T150821-quiz-prose#5: + decision: adopted +2026-10-04T150821-quiz-prose#6: + decision: declined +2026-10-04T150821-quiz-prose#7: + decision: adopted +2026-10-04T150821-quiz-prose#8: + decision: declined +2026-10-04T150821-quiz-prose#9: + decision: declined +2026-10-04T150959-quiz-prose#1: + decision: declined +2026-10-04T150959-quiz-prose#10: + decision: declined +2026-10-04T150959-quiz-prose#11: + decision: declined +2026-10-04T150959-quiz-prose#12: + decision: declined +2026-10-04T150959-quiz-prose#13: + decision: adopted +2026-10-04T150959-quiz-prose#14: + decision: adopted +2026-10-04T150959-quiz-prose#15: + decision: declined +2026-10-04T150959-quiz-prose#2: + decision: adopted +2026-10-04T150959-quiz-prose#3: + decision: declined +2026-10-04T150959-quiz-prose#4: + decision: declined +2026-10-04T150959-quiz-prose#5: + decision: declined +2026-10-04T150959-quiz-prose#6: + decision: declined +2026-10-04T150959-quiz-prose#7: + decision: adopted +2026-10-04T150959-quiz-prose#8: + decision: declined +2026-10-04T150959-quiz-prose#9: + decision: adopted diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150203-probe.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150203-probe.json new file mode 100644 index 0000000..3269c81 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150203-probe.json @@ -0,0 +1,224 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "two-tides", + "at": "2026-10-04T15:02:03", + "model": "default", + "prompt_sha": "4e0747130ed6", + "reply": { + "gaps": [ + { + "rule": 4, + "severity": "main", + "about": "The step where the 'sideways part' of the tidal acceleration moves water into the bulges has no numbers and no worked example.", + "why_it_matters": "The model says the 1.1 µm/s² vertical part 'cannot lift water against gravity directly' and that the sideways part does the work. But it never gives that sideways part's size or where on Earth it peaks. Without that, the reader is told the vertical push is too weak and is never shown what actually piles the water up.", + "suggested_addition": "Horizontal tidal acceleration = (3/2)(GM/d³)R·sin2θ, where θ is the angle from the sub-Moon point. It is 0 at θ = 0° and 90°, and largest at θ = 45°: 1.5 × 0.55 = 0.82 µm/s², pointing toward the nearer bulge. A table at θ = 0°, 45°, 90°, 135°, 180° with values 0, +0.82, 0, −0.82, 0 µm/s² would show water being pushed from the sides toward both bulges." + }, + { + "rule": 4, + "severity": "main", + "about": "The inward squeeze at the sides (0.55 µm/s², 'half the size') is stated, but never calculated or explained.", + "why_it_matters": "The low tide and the 'two bulges, not one' shape depend on the sides being pulled inward. But the three-point table only covers points on the Earth–Moon line. The reader sees no subtraction for the side points, so the squeeze looks asserted rather than derived.", + "suggested_addition": "At a side point, the distance to the Moon is ≈ √(384,400² + 6371²) ≈ 384,453 km, so the pull is ≈ 33.18 µm/s², almost the same as at the centre. But it points at the Moon, which is tilted by R/d = 0.01657 rad from the centre's pull direction. Subtracting the vectors leaves ≈ 33.19 × 0.01657 = 0.55 µm/s², pointing toward Earth's centre. Add a fourth row, 'side', to the table: pull 33.18, difference 0.55 inward." + }, + { + "rule": 1, + "severity": "edge", + "about": "The approximation 2GMR/d³ (and the 1/d³ law) is given without saying why it has this form instead of the direct difference.", + "why_it_matters": "The Sun-versus-Moon argument depends on 'differences fall as 1/d³'. The reader can do the direct subtraction but is never shown where the cube comes from, or why it gives 1.10 when the exact values are 1.13 and 1.07.", + "suggested_addition": "GM/(d−R)² − GM/d² ≈ GM/d² × 2R/d = 2GMR/d³ = 33.19 × 2 × 0.01657 = 1.10 µm/s². The exact near and far values (1.13, 1.07) differ by about ±3%, because of higher terms of order 3R/d ≈ 0.05. Then the Sun: d is 389× larger and GM is 2.71×10⁷× larger, so 2.71×10⁷ / 389³ ≈ 0.46." + }, + { + "rule": 2, + "severity": "main", + "about": "The argument names the wrong object: the Relationships row says free fall 'cancels the average Moon's pull', but the operation is subtracting the pull at the centre.", + "why_it_matters": "The pull at the centre (33.19) is not the average of the near and far values ((34.32 + 32.12)/2 = 33.22). Giving the same idea two names here also leaves implicit the key step: the water is measured relative to an Earth whose centre accelerates at exactly 33.19 µm/s².", + "suggested_addition": "Change the row to: 'Earth's free fall cancels the Moon's pull at the centre (33.19 µm/s²)'. Add one sentence: Earth's centre and the oceans all accelerate toward the Moon, and the centre's acceleration is 33.19 µm/s². Relative to the centre, each point only feels its own pull minus 33.19." + }, + { + "rule": 2, + "severity": "edge", + "about": "The 'uniform pull gives no tide' claim and the 'tidally locked' break case have no numbers.", + "why_it_matters": "Each guarantee should have a numeric case where it holds and one where it breaks. These two rows are bare statements.", + "suggested_addition": "Uniform pull: 33.19 at every point, so 33.19 − 33.19 = 0 everywhere, so no bulge. Tidally locked: the Moon's rotation period equals its orbital period (27.3 days), so relative to Earth the 'lunar day' is infinite and the number of high tides per day is 0. Contrast Earth: 1 rotation per 24 h 50 min relative to the Moon gives 2 per 24 h 50 min." + }, + { + "rule": 2, + "severity": "edge", + "about": "The confusion point 'the bulge would exist even if Earth fell straight at the Moon with no orbit' has no worked case.", + "why_it_matters": "This is the model's main rebuttal of the centrifugal story, and the reader is asked to accept it without numbers.", + "suggested_addition": "With no orbit (straight fall from rest at d = 384,400 km), the centre still accelerates at 33.19, the near side at 34.32 and the far side at 32.12. The differences are still +1.13 and −1.07, the same as in orbit. So both bulges appear without any rotation of the Earth–Moon pair." + }, + { + "rule": 4, + "severity": "edge", + "about": "The lunar day of 24 h 50 min is asserted without a worked calculation.", + "why_it_matters": "The 12 h 25 min spacing is one of the model's observable predictions. The reader is told only that 'the Moon moves along its orbit', with no numbers.", + "suggested_addition": "The Moon goes once around relative to the Sun every 29.53 days, about 12.2° per day, in the same direction Earth spins. Earth must turn an extra 12.2°, which at 15° per hour takes about 0.81 h ≈ 49–50 min. So the lunar day is ≈ 24 h 50 min, and half of it is 12 h 25 min." + }, + { + "rule": 3, + "severity": "edge", + "about": "One concept has many names: 'tidal acceleration', 'pull minus centre's pull', 'difference', 'tide-raising force', 'differential acceleration' (Audience section), 'the differences'. The centre's pull is also called 'the average Moon's pull'.", + "why_it_matters": "A new reader may think 'tide-raising force' or 'the difference' is a different quantity from 'tidal acceleration'.", + "suggested_addition": "Use only 'tidal acceleration' after it is defined, and only 'the Moon's pull at the centre (33.19 µm/s²)' for the subtracted term." + }, + { + "rule": 3, + "severity": "edge", + "about": "Terms used before they are defined, or never defined: 'equilibrium tide' (in the Entities table, defined only in Epistemic status), 'lunar day' (used in Relationships, missing from Entities), 'sides (90° from the Moon)', 'tidally locked', 'spring tides'.", + "why_it_matters": "These are new to the stated audience.", + "suggested_addition": "Add Entities rows: 'side point: a surface point 90° from the near side, ≈ 384,453 km from the Moon'; 'lunar day: 24 h 50 min'; 'equilibrium tide: the ocean shape that would balance the tidal acceleration instantly'. Define 'tidally locked' (the same face always points at the other body, period 27.3 days for the Moon) or remove it." + }, + { + "rule": 3, + "severity": "edge", + "about": "The definition of 'bulge' says the tidal acceleration 'pushed' water, which contradicts the confusion point 'the Moon pushes the far side away → No'.", + "why_it_matters": "It plants the very misconception the model is trying to correct.", + "suggested_addition": "Bulge: water gathered on the Earth–Moon line because the tidal acceleration there points outward (+1.13 / −1.07 µm/s² relative to the centre), and along the surface it points toward the line (up to 0.82 µm/s²)." + }, + { + "rule": 5, + "severity": "edge", + "about": "The bulge height (0.36 m) and the tidal acceleration (1.1 µm/s²) are never linked, and the near/far asymmetry (1.13 vs 1.07) is never given a meaning.", + "why_it_matters": "The reader sees a tiny acceleration and a height of 0.36 m but not how one leads to the other, or whether the near-side bulge is noticeably higher.", + "suggested_addition": "Equilibrium: the ocean surface tilts until Earth's gravity along the slope balances the horizontal tidal acceleration, a slope of about 0.82e-6 / 9.81 ≈ 8×10⁻⁸. Over about 6,000 km from bulge to side, that gives a rise and fall of order 0.5 m (range 0.54 m, peak 0.36 m above the mean). The near-side tidal acceleration is about 5% larger than the far side's, so the near bulge is about 5% taller, roughly 2 cm. That is too small to notice." + }, + { + "rule": 5, + "severity": "edge", + "about": "The Sun/Moon tidal ratio of 0.46 is not turned into something the reader can observe.", + "why_it_matters": "A ratio is more meaningful when it predicts something measurable.", + "suggested_addition": "Aligned (spring tide): 1 + 0.46 = 1.46× the Moon-only range. At right angles (neap tide): 1 − 0.46 = 0.54×. So spring ranges are about 2.7× neap ranges in the equilibrium model." + } + ], + "excess": [ + "The 'Sun and Moon in line (spring tides)' row in Alternative states: Scope declares spring tides out of scope.", + "Scope's 'Tidal friction and the Moon moving away' line is harmless but adds nothing to the central question.", + "The tidally locked Moon example, unless 'tidally locked' is defined; as it stands it adds a new idea just to show a break case." + ], + "next_questions": [ + { + "q": "If the tidal acceleration is only 1 ten-millionth of g, how does it move the ocean at all, and how high does each bulge get?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why are the two high tides in a day often different heights, and why do some places get only one?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Why are real tides metres high, and why does high tide come hours after the Moon is overhead?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Does the solid Earth also stretch into two bulges, and if so, why do we see the ocean rise relative to the land?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Isn't the far bulge really caused by centrifugal force from the Earth–Moon orbit, as I was taught?", + "answered_by_model": true, + "declared_out_of_scope": false + } + ] + }, + "usage": { + "cost": 0.17102900000000001, + "seconds": 65 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 4", + "text": "The step where the 'sideways part' of the tidal acceleration moves water into the bulges has no numbers and no worked example. → Horizontal tidal acceleration = (3/2)(GM/d³)R·sin2θ, where θ is the angle from the sub-Moon point. It is 0 at θ = 0° and 90°, and largest at θ = 45°: 1.5 × 0.55 = 0.82 µm/s², pointing toward the nearer bulge. A table at θ = 0°, 45°, 90°, 135°, 180° with values 0, +0.82, 0, −0.82, 0 µm/s² would show water being pushed from the sides toward both bulges." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 4", + "text": "The inward squeeze at the sides (0.55 µm/s², 'half the size') is stated, but never calculated or explained. → At a side point, the distance to the Moon is ≈ √(384,400² + 6371²) ≈ 384,453 km, so the pull is ≈ 33.18 µm/s², almost the same as at the centre. But it points at the Moon, which is tilted by R/d = 0.01657 rad from the centre's pull direction. Subtracting the vectors leaves ≈ 33.19 × 0.01657 = 0.55 µm/s², pointing toward Earth's centre. Add a fourth row, 'side', to the table: pull 33.18, difference 0.55 inward." + }, + { + "n": 3, + "severity": "edge", + "kind": "rule 1", + "text": "The approximation 2GMR/d³ (and the 1/d³ law) is given without saying why it has this form instead of the direct difference. → GM/(d−R)² − GM/d² ≈ GM/d² × 2R/d = 2GMR/d³ = 33.19 × 2 × 0.01657 = 1.10 µm/s². The exact near and far values (1.13, 1.07) differ by about ±3%, because of higher terms of order 3R/d ≈ 0.05. Then the Sun: d is 389× larger and GM is 2.71×10⁷× larger, so 2.71×10⁷ / 389³ ≈ 0.46." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 2", + "text": "The argument names the wrong object: the Relationships row says free fall 'cancels the average Moon's pull', but the operation is subtracting the pull at the centre. → Change the row to: 'Earth's free fall cancels the Moon's pull at the centre (33.19 µm/s²)'. Add one sentence: Earth's centre and the oceans all accelerate toward the Moon, and the centre's acceleration is 33.19 µm/s². Relative to the centre, each point only feels its own pull minus 33.19." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 2", + "text": "The 'uniform pull gives no tide' claim and the 'tidally locked' break case have no numbers. → Uniform pull: 33.19 at every point, so 33.19 − 33.19 = 0 everywhere, so no bulge. Tidally locked: the Moon's rotation period equals its orbital period (27.3 days), so relative to Earth the 'lunar day' is infinite and the number of high tides per day is 0. Contrast Earth: 1 rotation per 24 h 50 min relative to the Moon gives 2 per 24 h 50 min." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 2", + "text": "The confusion point 'the bulge would exist even if Earth fell straight at the Moon with no orbit' has no worked case. → With no orbit (straight fall from rest at d = 384,400 km), the centre still accelerates at 33.19, the near side at 34.32 and the far side at 32.12. The differences are still +1.13 and −1.07, the same as in orbit. So both bulges appear without any rotation of the Earth–Moon pair." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 4", + "text": "The lunar day of 24 h 50 min is asserted without a worked calculation. → The Moon goes once around relative to the Sun every 29.53 days, about 12.2° per day, in the same direction Earth spins. Earth must turn an extra 12.2°, which at 15° per hour takes about 0.81 h ≈ 49–50 min. So the lunar day is ≈ 24 h 50 min, and half of it is 12 h 25 min." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 3", + "text": "One concept has many names: 'tidal acceleration', 'pull minus centre's pull', 'difference', 'tide-raising force', 'differential acceleration' (Audience section), 'the differences'. The centre's pull is also called 'the average Moon's pull'. → Use only 'tidal acceleration' after it is defined, and only 'the Moon's pull at the centre (33.19 µm/s²)' for the subtracted term." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 3", + "text": "Terms used before they are defined, or never defined: 'equilibrium tide' (in the Entities table, defined only in Epistemic status), 'lunar day' (used in Relationships, missing from Entities), 'sides (90° from the Moon)', 'tidally locked', 'spring tides'. → Add Entities rows: 'side point: a surface point 90° from the near side, ≈ 384,453 km from the Moon'; 'lunar day: 24 h 50 min'; 'equilibrium tide: the ocean shape that would balance the tidal acceleration instantly'. Define 'tidally locked' (the same face always points at the other body, period 27.3 days for the Moon) or remove it." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 3", + "text": "The definition of 'bulge' says the tidal acceleration 'pushed' water, which contradicts the confusion point 'the Moon pushes the far side away → No'. → Bulge: water gathered on the Earth–Moon line because the tidal acceleration there points outward (+1.13 / −1.07 µm/s² relative to the centre), and along the surface it points toward the line (up to 0.82 µm/s²)." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 5", + "text": "The bulge height (0.36 m) and the tidal acceleration (1.1 µm/s²) are never linked, and the near/far asymmetry (1.13 vs 1.07) is never given a meaning. → Equilibrium: the ocean surface tilts until Earth's gravity along the slope balances the horizontal tidal acceleration, a slope of about 0.82e-6 / 9.81 ≈ 8×10⁻⁸. Over about 6,000 km from bulge to side, that gives a rise and fall of order 0.5 m (range 0.54 m, peak 0.36 m above the mean). The near-side tidal acceleration is about 5% larger than the far side's, so the near bulge is about 5% taller, roughly 2 cm. That is too small to notice." + }, + { + "n": 12, + "severity": "edge", + "kind": "rule 5", + "text": "The Sun/Moon tidal ratio of 0.46 is not turned into something the reader can observe. → Aligned (spring tide): 1 + 0.46 = 1.46× the Moon-only range. At right angles (neap tide): 1 − 0.46 = 0.54×. So spring ranges are about 2.7× neap ranges in the equilibrium model." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "The 'Sun and Moon in line (spring tides)' row in Alternative states: Scope declares spring tides out of scope." + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "Scope's 'Tidal friction and the Moon moving away' line is harmless but adds nothing to the central question." + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "The tidally locked Moon example, unless 'tidally locked' is defined; as it stands it adds a new idea just to show a break case." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150440-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150440-coldread-narrative.json new file mode 100644 index 0000000..8223fa9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150440-coldread-narrative.json @@ -0,0 +1,339 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "two-tides", + "at": "2026-10-04T15:04:40", + "model": "default", + "prompt_sha": "6fd08b11fa54", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: the answer in bold. Said: \"The Moon stretches the ocean; a stretch has two ends; the Earth turns through both.\"", + "unresolved": [], + "unsaid": [], + "leap": [ + "The question is never asked in words. The beat gives the answer (\"a stretch has two ends\") before the reader hears that there are two high tides a day, or that the far side has one. The reader has to rebuild the puzzle from the answer." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 2 (instance)", + "text": "Shown: table of three points. Said: \"Pull falls with distance squared: 34.32, 33.19, 32.12 µm/s².\"", + "unresolved": [], + "unsaid": [ + "The table's point names (near side, centre, far side) and their distances from the Moon are not said. The words list three numbers without saying which point each belongs to, or that all three point toward the Moon." + ], + "leap": [], + "excess": [ + "\"Pull falls with distance squared\": this reader knows the inverse-square law." + ], + "purpose_unclear": false + }, + { + "at": "Beat 3 (objection)", + "text": "Said: \"The whole Earth falls toward the Moon at 33.19. A shared acceleration changes nothing inside a falling body (as in an orbiting space station).\"", + "unresolved": [], + "unsaid": [], + "leap": [ + "\"The whole Earth falls toward the Moon\": the reader thinks of the Moon orbiting the Earth, not the Earth falling toward the Moon. The beat does not say why the Earth is in free fall toward the Moon. The space-station comparison helps once the reader accepts that." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 4 (name)", + "text": "Shown: difference column. Said: \"tidal acceleration: +1.13 near, −1.07 far\"", + "unresolved": [ + "the + and − signs in \"+1.13\" and \"−1.07\": nothing says which direction is positive" + ], + "unsaid": [ + "The column is centre-subtracted (34.32−33.19 and 32.12−33.19), but the words do not define tidal acceleration as 'pull minus pull at the centre'.", + "The key conclusion, that both ends are pulled outward relative to the centre, is not said. It is only in the plan's 'Reader now knows' column." + ], + "leap": [ + "The step from −1.07 to 'the far side is pushed away from the Moon' is left for the reader to work out." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: side row. Said: \"pull tilted toward centre, 0.55 inward\"", + "unresolved": [ + "which point \"side\" means (90° from the near side) is not said" + ], + "unsaid": [ + "The conclusion 'stretch along the line, squeeze across' is not said." + ], + "leap": [ + "Why the pull at a side point tilts toward the centre (the lines to the Moon converge) is not given." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Shown: ASCII figure. Said: \"The sideways part, up to 0.82 at 45°, is unopposed; water flows to two bulges.\"", + "unresolved": [ + "\"sideways part\": sideways relative to what (along the surface?)", + "\"45°\": measured from what" + ], + "unsaid": [ + "The beat's own problem, that these accelerations are tiny (about 10⁻⁷ g), is never said, so the reader does not know what this beat is answering." + ], + "leap": [ + "\"is unopposed\": no reason is given for why the horizontal component is unopposed when the vertical one is swamped by g. This is the step that explains how a tiny force moves the ocean." + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 7 (test)", + "text": "Said: \"Held Earth: one bulge. Sun: 179× pull but 0.46× tide, because 1/d³.\"", + "unresolved": [ + "\"Held Earth\": a hypothetical Earth held fixed so it does not fall toward the Moon; never explained", + "\"1/d³\": where a cube law comes from is never said (2GMR/d³ is in the plan only)" + ], + "unsaid": [], + "leap": [ + "Why a held Earth would have one bulge is not argued.", + "How a difference of pulls turns into 1/d³ is not shown." + ], + "excess": [ + "The Sun comparison is useful but is a side test, off the main question of two bulges." + ], + "purpose_unclear": false + }, + { + "at": "Beat 8 (mechanism)", + "text": "Shown: timeline. Said: \"Earth turns under bulges; 24 h 50 min lunar day; 12 h 25 min.\"", + "unresolved": [ + "\"lunar day\"" + ], + "unsaid": [], + "leap": [ + "The reader knows the Earth turns in 24 h. Nothing says why the period here is 24 h 50 min (the Moon moves about 12° a day)." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 9 (close)", + "text": "Said: \"Restate; misconception about centrifugal force; scope.\"", + "unresolved": [ + "\"misconception about centrifugal force\": centrifugal force never came up before, so the reader does not know what misconception is meant" + ], + "unsaid": [], + "leap": [ + "The 'Said' is only a stage direction. No actual restatement, general argument or scope is given to the reader." + ], + "excess": [], + "purpose_unclear": true + } + ], + "blocking": [ + "Beat 4, \"+1.13 near, −1.07 far\": the sign convention and the meaning 'both ends pulled outward from the centre' are not said, and this is the core of the two-bulge answer.", + "Beat 6, \"The sideways part, up to 0.82 at 45°, is unopposed\": no reason is given for why the horizontal part acts while the vertical part does not. Without it the reader cannot see how a force of about 10⁻⁷ g moves water.", + "Beat 9, \"Restate; misconception about centrifugal force; scope.\": there is no closing content, so the answer and the general argument are never actually said." + ], + "cuts": [ + "Beat 9: either drop \"misconception about centrifugal force\" or set it up earlier. As it stands it brings in a topic the reader has never seen.", + "Beat 2: drop \"Pull falls with distance squared\" (already known) and use the space to name which number belongs to which point.", + "Beat 7: the Sun comparison (\"179× pull but 0.46× tide, because 1/d³\") could be shortened or moved after the close, because its 1/d³ step is never explained." + ], + "question": { + "text": "The Moon is on one side of the Earth, so why is there a high tide on the far side too, giving two high tides a day?", + "known_at": "Never asked outright in 'Said'. It can be inferred from Beat 1's answer (\"a stretch has two ends\").", + "result_stated": "Yes, in Beat 1: \"The Moon stretches the ocean; a stretch has two ends; the Earth turns through both.\"" + }, + "motive": "Missing from the script. The reason to care (the naive picture predicts one tide but coasts have two) and the reason for the approach (compare the pull at three points, then ask which differences change the Earth's shape) appear only in the plan's 'Question and motive' section. Neither is in Said or Shown.", + "close": "Not delivered. Beat 9 says only \"Restate; misconception about centrifugal force; scope.\" The general argument ('any body falling in a non-uniform field stretches along the line to the source') appears only in plan section 7.", + "recap": "The whole Earth falls toward the Moon, so only the differences in pull matter: the near side is pulled a bit more and the far side a bit less, which stretches the ocean into two bulges.", + "strongest": "Beats 2–3: real numbers at three points, then the free-fall argument (\"A shared acceleration changes nothing inside a falling body (as in an orbiting space station)\"). This takes away the naive picture.", + "weakest": "Beat 9: the close is a placeholder with no content, and it brings in an unexplained centrifugal misconception. A close second is Beat 6's unexplained \"unopposed\".", + "suggestions": [ + "Open with the puzzle in words: 'The Moon pulls water toward it, so why is there also a high tide on the side facing away?'", + "In Beat 4, say it plainly: 'positive means toward the Moon; so the near water is pulled ahead of the Earth and the far water is left behind, and both move away from the centre.'", + "In Beat 6, explain 'unopposed': straight up or down, 1 µm/s² is nothing against g = 9.81 m/s², but along the surface nothing resists it, so the water slides.", + "Define 'held Earth' as 'if the Earth were held still instead of falling', and give a one-line reason for 1/d³ (a difference of 1/d² terms across the Earth's width).", + "Say why the lunar day is 24 h 50 min: the Moon moves about 12° along its orbit each day.", + "Write the close out in full: the answer, then the general rule that anything falling in a non-uniform field is stretched along the line to the source." + ] + }, + "usage": { + "cost": 0.1251516, + "seconds": 46 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4, \"+1.13 near, −1.07 far\": the sign convention and the meaning 'both ends pulled outward from the centre' are not said, and this is the core of the two-bulge answer." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 6, \"The sideways part, up to 0.82 at 45°, is unopposed\": no reason is given for why the horizontal part acts while the vertical part does not. Without it the reader cannot see how a force of about 10⁻⁷ g moves water." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 9, \"Restate; misconception about centrifugal force; scope.\": there is no closing content, so the answer and the general argument are never actually said." + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "Beat 9: either drop \"misconception about centrifugal force\" or set it up earlier. As it stands it brings in a topic the reader has never seen." + }, + { + "n": 5, + "severity": "excess", + "kind": "cut", + "text": "Beat 2: drop \"Pull falls with distance squared\" (already known) and use the space to name which number belongs to which point." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 7: the Sun comparison (\"179× pull but 0.46× tide, because 1/d³\") could be shortened or moved after the close, because its 1/d³ step is never explained." + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): The question is never asked in words. The beat gives the answer (\"a stretch has two ends\") before the reader hears that there are two high tides a day, or that the far side has one. The reader has to rebuild the puzzle from the answer." + }, + { + "n": 8, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (instance): The table's point names (near side, centre, far side) and their distances from the Moon are not said. The words list three numbers without saying which point each belongs to, or that all three point toward the Moon." + }, + { + "n": 9, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (objection): \"The whole Earth falls toward the Moon\": the reader thinks of the Moon orbiting the Earth, not the Earth falling toward the Moon. The beat does not say why the Earth is in free fall toward the Moon. The space-station comparison helps once the reader accepts that." + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (name): the + and − signs in \"+1.13\" and \"−1.07\": nothing says which direction is positive" + }, + { + "n": 11, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name): The column is centre-subtracted (34.32−33.19 and 32.12−33.19), but the words do not define tidal acceleration as 'pull minus pull at the centre'." + }, + { + "n": 12, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (name): The key conclusion, that both ends are pulled outward relative to the centre, is not said. It is only in the plan's 'Reader now knows' column." + }, + { + "n": 13, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (name): The step from −1.07 to 'the far side is pushed away from the Moon' is left for the reader to work out." + }, + { + "n": 14, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 5 (mechanism): which point \"side\" means (90° from the near side) is not said" + }, + { + "n": 15, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): The conclusion 'stretch along the line, squeeze across' is not said." + }, + { + "n": 16, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): Why the pull at a side point tilts toward the centre (the lines to the Moon converge) is not given." + }, + { + "n": 17, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): \"sideways part\": sideways relative to what (along the surface?)" + }, + { + "n": 18, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 6 (mechanism): \"45°\": measured from what" + }, + { + "n": 19, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): The beat's own problem, that these accelerations are tiny (about 10⁻⁷ g), is never said, so the reader does not know what this beat is answering." + }, + { + "n": 20, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): \"is unopposed\": no reason is given for why the horizontal component is unopposed when the vertical one is swamped by g. This is the step that explains how a tiny force moves the ocean." + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): \"Held Earth\": a hypothetical Earth held fixed so it does not fall toward the Moon; never explained" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (test): \"1/d³\": where a cube law comes from is never said (2GMR/d³ is in the plan only)" + }, + { + "n": 23, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test): Why a held Earth would have one bulge is not argued." + }, + { + "n": 24, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (test): How a difference of pulls turns into 1/d³ is not shown." + }, + { + "n": 25, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (mechanism): \"lunar day\"" + }, + { + "n": 26, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (mechanism): The reader knows the Earth turns in 24 h. Nothing says why the period here is 24 h 50 min (the Moon moves about 12° a day)." + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (close): \"misconception about centrifugal force\": centrifugal force never came up before, so the reader does not know what misconception is meant" + }, + { + "n": 28, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (close): The 'Said' is only a stage direction. No actual restatement, general argument or scope is given to the reader." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150700-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150700-coldread-prose.json new file mode 100644 index 0000000..31ffd7c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150700-coldread-prose.json @@ -0,0 +1,270 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "two-tides", + "at": "2026-10-04T15:07:00", + "model": "default", + "prompt_sha": "fb797624d4da", + "reply": { + "paragraphs": [ + { + "at": "Title + line 3 (opening)", + "text": "Most coasts get two high tides a day... If the Moon simply pulls water toward itself, the second high tide makes no sense.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 5 (bold summary)", + "text": "The Moon does not lift the ocean toward itself. It stretches the ocean along the Earth–Moon line...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 9 (The pull at three points)", + "text": "The Moon's pull is the acceleration that the Moon's gravity gives at a point...", + "unresolved": [], + "leap": [], + "excess": [ + "It points at the Moon and gets weaker with the square of the distance." + ], + "purpose_unclear": false + }, + { + "at": "table, lines 11-15", + "text": "| Point | Distance to Moon | Moon's pull | Tidal acceleration |", + "unresolved": [ + "Tidal acceleration (the column header and its values +1.13, 0, −1.07 µm/s² come before the term or its sign convention is defined; both are only given at line 24)" + ], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 17", + "text": "All three pulls point toward the Moon. So why does the far-side water not move toward the Moon?", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 22 (Only the part that differs...)", + "text": "The Earth and the Moon orbit their shared centre of mass. An orbit is free fall...", + "unresolved": [], + "leap": [], + "excess": [ + "Astronauts in an orbiting station float for the same reason." + ], + "purpose_unclear": false + }, + { + "at": "line 24", + "text": "So subtract the centre's pull. The remainder is the tidal acceleration...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 26 (near side bullet)", + "text": "Near side: 34.32 − 33.19 = +1.13...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 27 (far side bullet)", + "text": "Far side: 32.12 − 33.19 = −1.07...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 30", + "text": "Both results point away from Earth's centre. The Moon does not push the far side away...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 32 (side point)", + "text": "Now take a side point, 90° from the near side...", + "unresolved": [], + "leap": [ + "\"Subtract the centre's pull, and 0.55 µm/s² is left, pointing down\": it doesn't say why the along-line part cancels (33.18 vs 33.19), leaving only the 0.95° tilt component." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 36 (How a tiny acceleration moves an ocean)", + "text": "1.1 µm/s² is about one ten-millionth of g... The water gathers into two bulges", + "unresolved": [], + "leap": [ + "\"It is largest 45° from the near side, at 0.82 µm/s²\": stated without a reason or derivation, since only the 0°, 90° and 180° points were worked out." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "diagram, lines 38-47", + "text": "far side ←( ↘ Earth ↙ )→ near side ... (↘ ↙ : the sideways push, 0.82 at 45°, toward the nearer bulge)", + "unresolved": [ + "\"0.55 down\" and \"0.82 at 45°\" have no units in the diagram" + ], + "leap": [ + "The arrows ↘ ↙ both point inward toward the centre. The legend and line 36 say the push points 'toward the nearer bulge', which is outward toward the ends of the line. The reader has to stop and reconcile the picture with the text." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 52 (Why subtract: two tests)", + "text": "The simplest idea is that the Moon's pull itself raises the tide. It fails two tests.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": true + }, + { + "at": "line 55 (first test)", + "text": "First, imagine that something held the Earth still... The far bulge exists because the Earth falls.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 57 (Sun test)", + "text": "Second, the Sun. Its pull on the Earth is 179 times the Moon's...", + "unresolved": [], + "leap": [ + "\"the tidal acceleration is about the pull times 2R/d\": the factor is stated, not derived. The reader can check it against the table (33.19 × 2·6371/384400 ≈ 1.10), but the text doesn't do this." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 62 (Two high tides per turn)", + "text": "The bulges stay on the Earth–Moon line, and the Earth turns under them...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 66 (The answer)", + "text": "The Moon pulls the near side more than Earth's centre... An Earth that fell straight at the Moon, with no orbit, would have the same two bulges.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 70 (Scope: Height)", + "text": "in an ideal ocean, the bulges rise about 0.36 m above the mean (an estimate)...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 71 (Scope: Real oceans)", + "text": "continents block the bulges and basins slosh with their own timing...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "line 72 (Scope: Spring and neap)", + "text": "the Sun's bulges add to the Moon's or partly cancel them.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "Line 22: drop \"Astronauts in an orbiting station float for the same reason.\" The sentence before it already makes the point.", + "Line 9: drop \"It points at the Moon and gets weaker with the square of the distance.\" This reader already knows inverse-square gravity." + ], + "question": { + "text": "Why is there a high tide on the side of the Earth away from the Moon, giving two high tides a day instead of one?", + "known_at": "Title and line 3: \"If the Moon simply pulls water toward itself, the second high tide makes no sense.\"", + "result_stated": "Yes, at line 5: \"It stretches the ocean along the Earth–Moon line, and a stretch has two ends. The Earth turns through both ends, so a place gets two high tides.\"" + }, + "motive": "Reason to care about the question (line 3): \"If the Moon simply pulls water toward itself, the second high tide makes no sense.\" Reason for the approach of subtracting the centre's pull (line 22): \"An orbit is free fall... An acceleration that every part shares does not change the shape of a falling body.\" It is backed up at line 52 by \"It fails two tests\" (the held-still Earth and the Sun ratio).", + "close": "Yes, at line 66: \"The Moon pulls the near side more than Earth's centre, and the centre more than the far side. The falling Earth feels only these differences, which stretch the ocean into two bulges. The Earth turns through both every 24 h 50 min.\"", + "recap": "The whole Earth is falling toward the Moon, so only differences in the Moon's pull count: the near side is pulled more than the centre and the far side less, which stretches the ocean into two bulges, and the Earth turns through both.", + "strongest": "Line 55: the held-still-Earth test. \"That Earth has one bulge, not two. The far bulge exists because the Earth falls.\" It isolates the cause of the far bulge cleanly.", + "weakest": "The diagram at lines 38-47. The ↘ ↙ arrows point inward, but the legend says the push is 'toward the nearer bulge', and the values have no units. A smaller weak spot is the table at lines 11-15, whose 'Tidal acceleration' column appears before the term is defined at line 24.", + "suggestions": [ + "Redraw the sideways-push arrows so they point outward toward the near and far bulges, and add units (µm/s²) inside the diagram.", + "Either leave the 'Tidal acceleration' column out of the first table, or add one line saying it is defined in the next section.", + "Give a one-line reason for the 45° maximum, or for the 2R/d factor, e.g. from differentiating GM/d²." + ] + }, + "usage": { + "cost": 0.12958219999999998, + "seconds": 43 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "Line 22: drop \"Astronauts in an orbiting station float for the same reason.\" The sentence before it already makes the point." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "Line 9: drop \"It points at the Moon and gets weaker with the square of the distance.\" This reader already knows inverse-square gravity." + }, + { + "n": 3, + "severity": "edge", + "kind": "unresolved", + "text": "table, lines 11-15: Tidal acceleration (the column header and its values +1.13, 0, −1.07 µm/s² come before the term or its sign convention is defined; both are only given at line 24)" + }, + { + "n": 4, + "severity": "edge", + "kind": "leap", + "text": "line 32 (side point): \"Subtract the centre's pull, and 0.55 µm/s² is left, pointing down\": it doesn't say why the along-line part cancels (33.18 vs 33.19), leaving only the 0.95° tilt component." + }, + { + "n": 5, + "severity": "edge", + "kind": "leap", + "text": "line 36 (How a tiny acceleration moves an ocean): \"It is largest 45° from the near side, at 0.82 µm/s²\": stated without a reason or derivation, since only the 0°, 90° and 180° points were worked out." + }, + { + "n": 6, + "severity": "edge", + "kind": "unresolved", + "text": "diagram, lines 38-47: \"0.55 down\" and \"0.82 at 45°\" have no units in the diagram" + }, + { + "n": 7, + "severity": "edge", + "kind": "leap", + "text": "diagram, lines 38-47: The arrows ↘ ↙ both point inward toward the centre. The legend and line 36 say the push points 'toward the nearer bulge', which is outward toward the ends of the line. The reader has to stop and reconcile the picture with the text." + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "line 57 (Sun test): \"the tidal acceleration is about the pull times 2R/d\": the factor is stated, not derived. The reader can check it against the table (33.19 × 2·6371/384400 ≈ 1.10), but the text doesn't do this." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150821-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150821-quiz-prose.json new file mode 100644 index 0000000..73e8b89 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150821-quiz-prose.json @@ -0,0 +1,205 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "two-tides", + "at": "2026-10-04T15:08:08", + "model": "default", + "prompt_sha": "94be2bc7b92f", + "reply": { + "answers": { + "1": "The Moon, the Earth (its centre plus three surface points: the near side, the far side and a side point 90° away), the ocean water, the Moon's pull (the acceleration its gravity gives at a point), the tidal acceleration, the two bulges, the Earth's rotation and the lunar day. The Sun is also used as a comparison.", + "2": "The Moon's pull gets weaker with distance: 34.32 µm/s² at the near side, 33.19 at the centre and 32.12 at the far side. Earth and Moon orbit their shared centre of mass, so the whole Earth is in free fall toward the Moon. Only the difference from the centre's pull (the tidal acceleration) changes the Earth's shape. That difference points outward at the near and far sides and inward at the side points. Along the surface it pushes water toward the two ends of the Earth–Moon line, which forms two bulges. The Earth turns under these bulges.", + "3": "Because the Earth falls freely toward the Moon, only the difference between the Moon's pull at a point and its pull at the centre acts on the water. That difference stretches the ocean into two bulges along the Earth–Moon line, and the turning Earth carries each place through both bulges.", + "4": "The key variable is distance d to the tide-raising body. The tidal acceleration is roughly the pull times 2R/d. The pull itself falls as 1/d², so the tide falls faster than the pull, roughly as 1/d³ (the explanation only implies this). A farther body gives smaller tides. That is why the Sun's pull is 179 times the Moon's but its tide is only about 0.46 times the Moon's. A larger Earth radius R would give larger tides.", + "5": "Assumption: an ideal ocean in equilibrium with no continents (the 0.36 m bulge height is called an estimate), and the Earth treated as falling freely. Observation: solar tides are about half the size of the Moon's. Also, high tides come about every 12 h 25 min.", + "6": "The Moon's pull is a little stronger on the near side of the Earth than at the centre, and a little weaker on the far side. The whole Earth is in free fall toward the Moon, so only the differences from the centre's pull matter. Those differences point outward along the Earth–Moon line and inward across it. Along the surface they push water toward the two ends of the line, which makes two bulges. The Earth turns under both bulges once per lunar day of 24 h 50 min, so high tides come every 12 h 25 min.", + "7": "The hardest parts were the side-point tilt (0.95°, 0.0166 rad) and the 2R/d formula. Both are stated, not derived. It would help to say that the tilt angle is R/d, and to show where the factor 2 comes from. It was also unclear how a sideways push of 0.82 µm/s² at 45° turns into a sea slope.", + "8": "The Moon pulls the far side less than it pulls the Earth's centre (32.12 vs 33.19 µm/s²). The Earth as a whole falls toward the Moon, so the far-side water is left behind, away from the Moon. Its tidal acceleration is −1.07 µm/s², which points outward from the Earth's centre.", + "9": "One. Nothing would subtract the centre's pull, so on the far side the Moon's pull would point down into the Earth. Only the near-side bulge would remain.", + "10": "At about 20:25. The Moon moves about 12° a day along its orbit in the same direction the Earth turns. The Earth therefore needs 24 h 50 min to face the Moon again, and half of that is 12 h 25 min.", + "11": "It points away from the Moon, which on the far side is outward from the Earth's centre. The far side is pulled less (32.12) than the centre (33.19), so the difference is −1.07 µm/s².", + "12": "Tides depend on the difference in pull across the Earth, roughly the pull times 2R/d, not on the pull itself. The Sun is 389 times farther away than the Moon, so its tide is about 179/389 ≈ 0.46 times the Moon's.", + "13": "There are two bulges, and a place passes through both in each lunar day. The lunar day is 24 h 50 min, not 24 h, because the Moon moves about 12° a day in the direction the Earth turns. Half of 24 h 50 min is 12 h 25 min." + }, + "audit": { + "terms": [ + "'tide' means two things in the Sun paragraph (line 60). In 'its tide is about 0.46 times the Moon's' it means tidal acceleration. In 'Observed solar tides are about half' it means water height. The explanation treats the two as the same without saying so.", + "'bulge' (line 36 onward) and 'high tide' (lines 3, 65) both name the same feature: one is the water shape, the other is what a coastal place sees. The link is implied rather than stated.", + "'stretches the ocean' (line 5), 'tidal acceleration' (line 24) and 'these differences' (line 69) all name the same quantity." + ], + "unexplained": [ + "Line 13 vs line 15: near side +1.13 vs far side −1.07. Why the two ends are unequal is never said. The reader may wonder whether the near bulge is larger.", + "Line 32, '0.95° (0.0166 rad)': where the angle comes from is not stated (it is R/d).", + "Line 36, '0.82 µm/s² at 45°': given with no derivation and no comparison to the 1.1 values along the line.", + "Line 65, '12° a day' leading to '50 minutes': the conversion is not shown.", + "Line 73, '0.36 m above the mean': not linked to the accelerations, and no hint of how it relates to observed metre-scale tides beyond the Scope note." + ], + "missing_why": [ + "Line 60: 'tidal acceleration is about the pull times 2R/d'. Neither the factor 2 nor the R/d form is justified.", + "Line 36: why the sideways push is largest at 45°.", + "Line 36: 'Water flows until the sea slopes just enough to balance this push'. How a slope balances a sideways force is not explained.", + "Line 32: why the side-point tilt equals the inward component (pull × angle)." + ], + "excess": [ + "Line 9: defining µm/s² as one millionth of a m/s² is something this reader already knows.", + "Line 22: 'An orbit is free fall' restates something the reader already knows (free fall), though it does link that idea to the orbit.", + "Line 69: the centrifugal-force aside adds a claim ('An Earth that fell straight... same two bulges') that the questions do not use.", + "Lines 74–75: the 'Real oceans' and spring/neap bullets are detours the argument and questions do not use. They are short and clearly labelled." + ] + }, + "gaps": [ + "No derivation of the 2R/d tidal formula, so the Sun comparison has to be taken on trust.", + "The near/far asymmetry (1.13 vs 1.07) is not explained.", + "The 45° maximum and the slope-balance mechanism are asserted, not shown.", + "How 12° a day becomes about 50 minutes is not shown." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 1, + "12": 2, + "13": 2 + }, + "reasons": { + "10": "Kept at 2, borderline. It explains why the gap is not exactly 12 h (the Moon moves about 12° a day, so the lunar day is 24 h 50 min) and gives 20:25. It halves the lunar day without saying this is because there are two bulges.", + "11": "It gives the direction (away from the Moon, outward) and the subtraction 32.12 − 33.19 = −1.07. It does not say why the centre's pull is subtracted: the whole Earth falls toward the Moon with the centre's acceleration. The expected answer requires this reason." + }, + "pass": false + }, + "usage": { + "cost": 0.1745564, + "seconds": 42 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "'tide' means two things in the Sun paragraph (line 60). In 'its tide is about 0.46 times the Moon's' it means tidal acceleration. In 'Observed solar tides are about half' it means water height. The explanation treats the two as the same without saying so." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'bulge' (line 36 onward) and 'high tide' (lines 3, 65) both name the same feature: one is the water shape, the other is what a coastal place sees. The link is implied rather than stated." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'stretches the ocean' (line 5), 'tidal acceleration' (line 24) and 'these differences' (line 69) all name the same quantity." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "Line 13 vs line 15: near side +1.13 vs far side −1.07. Why the two ends are unequal is never said. The reader may wonder whether the near bulge is larger." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "Line 32, '0.95° (0.0166 rad)': where the angle comes from is not stated (it is R/d)." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "Line 36, '0.82 µm/s² at 45°': given with no derivation and no comparison to the 1.1 values along the line." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "Line 65, '12° a day' leading to '50 minutes': the conversion is not shown." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "Line 73, '0.36 m above the mean': not linked to the accelerations, and no hint of how it relates to observed metre-scale tides beyond the Scope note." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "Line 60: 'tidal acceleration is about the pull times 2R/d'. Neither the factor 2 nor the R/d form is justified." + }, + { + "n": 10, + "severity": "gap", + "kind": "missing_why", + "text": "Line 36: why the sideways push is largest at 45°." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Line 36: 'Water flows until the sea slopes just enough to balance this push'. How a slope balances a sideways force is not explained." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "Line 32: why the side-point tilt equals the inward component (pull × angle)." + }, + { + "n": 13, + "severity": "excess", + "kind": "excess", + "text": "Line 9: defining µm/s² as one millionth of a m/s² is something this reader already knows." + }, + { + "n": 14, + "severity": "excess", + "kind": "excess", + "text": "Line 22: 'An orbit is free fall' restates something the reader already knows (free fall), though it does link that idea to the orbit." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "Line 69: the centrifugal-force aside adds a claim ('An Earth that fell straight... same two bulges') that the questions do not use." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "Lines 74–75: the 'Real oceans' and spring/neap bullets are detours the argument and questions do not use. They are short and clearly labelled." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "No derivation of the 2R/d tidal formula, so the Sun comparison has to be taken on trust." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "The near/far asymmetry (1.13 vs 1.07) is not explained." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "The 45° maximum and the slope-balance mechanism are asserted, not shown." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "How 12° a day becomes about 50 minutes is not shown." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150959-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150959-quiz-prose.json new file mode 100644 index 0000000..8022c21 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/explainers/two-tides/review/runs/2026-10-04T150959-quiz-prose.json @@ -0,0 +1,167 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "two-tides", + "at": "2026-10-04T15:09:53", + "model": "default", + "prompt_sha": "94be2bc7b92f", + "reply": { + "answers": { + "1": "The Moon, the Earth (its near side, centre, far side and side points), the ocean water, Earth's gravity g, the Sun, and the Earth's rotation, which brings in the lunar day.", + "2": "The Moon's gravity pulls every point of the Earth toward the Moon. The pull is weaker farther away: 34.32, 33.19 and 32.12 µm/s² at the near side, centre and far side. The Earth and Moon orbit their shared centre of mass, so the Earth is in free fall toward the Moon at the centre's acceleration. Only the difference from the centre's pull (the tidal acceleration) changes the ocean's shape. It points outward at the near and far sides and inward at the side points. Its sideways part, along the surface, is not resisted by Earth's gravity, so water flows into two bulges on the Earth–Moon line. The Earth turns under the bulges, and the Moon moves along its orbit, so each place passes through both bulges once per lunar day.", + "3": "The Moon pulls the near side more than the centre and the centre more than the far side, while the whole Earth falls with its centre. What is left is a stretch along the Earth–Moon line with two ends, and the turning Earth carries each place through both ends.", + "4": "The key variable is the distance d to the tide-raising body. The tidal acceleration is about the pull × 2R/d, and the pull falls as 1/d². So the tidal effect falls faster than the pull. A closer Moon would give much bigger bulges and a farther one much smaller bulges. That is why the Sun, which pulls 179 times harder but is 389 times farther, gives only about 0.46 of the Moon's tide. If the Earth stopped falling (held fixed), the subtraction would vanish and only one bulge would remain.", + "5": "Assumption or simplification: an ideal ocean in equilibrium with no continents; the 0.36 m bulge height is called an estimate. Observation: solar tides are seen to be about half the height of lunar tides. Most coasts get two high tides a day, about 12 h 25 min apart.", + "6": "The Moon's pull weakens with distance, so it is stronger at the Earth's near side than at its centre, and stronger at the centre than at the far side. The Earth is in free fall toward the Moon, so a shared acceleration does nothing. Only the leftover, the tidal acceleration, acts: about +1.13 µm/s² at the near side, −1.07 at the far side (both pointing away from the centre) and inward at the sides. This is far too small to lift water against g, but it is the only sideways push along a level surface, so water flows into two bulges on the Earth–Moon line. The Earth turns through both bulges once per lunar day of 24 h 50 min (longer than 24 h because the Moon moves along its orbit), so high tides come about every 12 h 25 min.", + "7": "Two parts were hardest. First, the side-point tilt calculation and the claim that the sideways push peaks at 0.82 µm/s² at 45°: no reason is given for 45° or for that value. A short line on how the stretch along the line and the squeeze across it combine would help. Second, the closing remark that 'no centrifugal force is needed', with an Earth falling straight at the Moon. Centrifugal force was never introduced, so this comes out of nowhere.", + "8": "Because the Earth as a whole is falling toward the Moon at the centre's acceleration (33.19 µm/s²). The far side is pulled less (32.12), so it falls more slowly than the centre and is left behind, away from the Moon. Relative to the centre that is an outward tidal acceleration of −1.07 µm/s², and water gathers there into a second bulge.", + "9": "One. With no fall, nothing subtracts the centre's pull. The far side's pull points toward the Moon, which on the far side is down into the Earth, so only the near side would bulge.", + "10": "At about 20:25. The bulges stay on the Earth–Moon line, and the Moon moves about 12° a day in the same direction the Earth turns. So the Earth needs about 50 minutes more than 24 h to face the Moon again (a lunar day of 24 h 50 min), and half of that is 12 h 25 min.", + "11": "Away from the Moon, which on the far side is outward from Earth's centre. The far side's pull (32.12) is smaller than the centre's (33.19), so far side minus centre is −1.07 µm/s²: it lags behind the falling centre. The Moon does not push it; it just pulls it less.", + "12": "Tides come from the difference in pull across the Earth, about the pull × 2R/d, not from the pull itself. The Sun is 389 times farther away than the Moon, so its tidal acceleration is about 179/389 ≈ 0.46 of the Moon's. That matches observed solar tides of about half the lunar height.", + "13": "The Moon advances about 12° a day along its orbit in the direction of Earth's spin. At 15°/h the Earth needs about 50 extra minutes to face the Moon again, so a lunar day is 24 h 50 min. Two bulges per lunar day gives about 12 h 25 min between high tides." + }, + "audit": { + "terms": [ + "The same horizontal component is called 'the tidal acceleration is the only sideways push' (section 'How a tiny acceleration moves an ocean') and 'sideways push along the surface' (diagram legend). This is fine, but 'push' sits awkwardly next to 'The Moon does not push the far side away' (section 'Only the part that differs').", + "Two words for the same direction: 'pointing down' (side point paragraph) and 'toward Earth's centre' (same paragraph), with 'down into the Earth' in the first test. Minor." + ], + "unexplained": [ + "'It is largest 45° from the near side, at 0.82 µm/s²' (section 'How a tiny acceleration moves an ocean'): no reason is given for why the peak is at 45° or how 0.82 is obtained.", + "'the bulges rise about 0.36 m above the mean (an estimate)' (Scope): there is no hint of how this follows from the 1.1 µm/s² slope balance.", + "'the two bulges differ by about 5%' (section 'Only the part that differs'): it says nothing about whether this matters for observed tide heights.", + "'The Moon also moves about 12° a day' (section 'Two high tides per turn'): presented as a given. It is fine to take it as data, but no source or reasoning is offered." + ], + "missing_why": [ + "'the tidal acceleration is about the pull times 2R/d' (section 'Why subtract: two tests'): nothing explains why it has this form, i.e. where the factor 2 comes from in the inverse-square fall-off. The numeric check shows it works but not why.", + "'Water flows until the sea surface slopes just enough that gravity along the slope balances this push': the step from this balance to two bulges of a given height is not shown.", + "'No centrifugal force is needed: an Earth that fell straight at the Moon, with no orbit, would have the same two bulges' (The answer): stated with no argument, and it answers a question the text never raised." + ], + "excess": [ + "'The answer' section mostly repeats the bold summary at the top and the subtraction section. Its only new content is the centrifugal remark, which is itself unexplained.", + "The Scope bullet on real oceans (dynamic theory, once-a-day tides) is not used by the argument. It is short and acceptable as a pointer." + ] + }, + "gaps": [ + "Why the sideways push peaks at 45° and why its value there is 0.82 µm/s².", + "Where the 2R/d factor comes from (the derivative of an inverse-square pull).", + "The centrifugal-force remark has no setup, so a reader cannot judge it.", + "How 0.36 m follows from the acceleration balance is not shown." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": {}, + "pass": true + }, + "usage": { + "cost": 0.17281239999999998, + "seconds": 39 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "The same horizontal component is called 'the tidal acceleration is the only sideways push' (section 'How a tiny acceleration moves an ocean') and 'sideways push along the surface' (diagram legend). This is fine, but 'push' sits awkwardly next to 'The Moon does not push the far side away' (section 'Only the part that differs')." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "Two words for the same direction: 'pointing down' (side point paragraph) and 'toward Earth's centre' (same paragraph), with 'down into the Earth' in the first test. Minor." + }, + { + "n": 3, + "severity": "gap", + "kind": "unexplained", + "text": "'It is largest 45° from the near side, at 0.82 µm/s²' (section 'How a tiny acceleration moves an ocean'): no reason is given for why the peak is at 45° or how 0.82 is obtained." + }, + { + "n": 4, + "severity": "gap", + "kind": "unexplained", + "text": "'the bulges rise about 0.36 m above the mean (an estimate)' (Scope): there is no hint of how this follows from the 1.1 µm/s² slope balance." + }, + { + "n": 5, + "severity": "gap", + "kind": "unexplained", + "text": "'the two bulges differ by about 5%' (section 'Only the part that differs'): it says nothing about whether this matters for observed tide heights." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'The Moon also moves about 12° a day' (section 'Two high tides per turn'): presented as a given. It is fine to take it as data, but no source or reasoning is offered." + }, + { + "n": 7, + "severity": "gap", + "kind": "missing_why", + "text": "'the tidal acceleration is about the pull times 2R/d' (section 'Why subtract: two tests'): nothing explains why it has this form, i.e. where the factor 2 comes from in the inverse-square fall-off. The numeric check shows it works but not why." + }, + { + "n": 8, + "severity": "gap", + "kind": "missing_why", + "text": "'Water flows until the sea surface slopes just enough that gravity along the slope balances this push': the step from this balance to two bulges of a given height is not shown." + }, + { + "n": 9, + "severity": "gap", + "kind": "missing_why", + "text": "'No centrifugal force is needed: an Earth that fell straight at the Moon, with no orbit, would have the same two bulges' (The answer): stated with no argument, and it answers a question the text never raised." + }, + { + "n": 10, + "severity": "excess", + "kind": "excess", + "text": "'The answer' section mostly repeats the bold summary at the top and the subtraction section. Its only new content is the centrifugal remark, which is itself unexplained." + }, + { + "n": 11, + "severity": "excess", + "kind": "excess", + "text": "The Scope bullet on real oceans (dynamic theory, once-a-day tides) is not used by the argument. It is short and acceptable as a pointer." + }, + { + "n": 12, + "severity": "gap", + "kind": "gap", + "text": "Why the sideways push peaks at 45° and why its value there is 0.82 µm/s²." + }, + { + "n": 13, + "severity": "gap", + "kind": "gap", + "text": "Where the 2R/d factor comes from (the derivative of an inverse-square pull)." + }, + { + "n": 14, + "severity": "gap", + "kind": "gap", + "text": "The centrifugal-force remark has no setup, so a reader cannot judge it." + }, + { + "n": 15, + "severity": "gap", + "kind": "gap", + "text": "How 0.36 m follows from the acceleration balance is not shown." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/read.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/read.json new file mode 100644 index 0000000..db8fe07 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/read.json @@ -0,0 +1,45 @@ +{ + "answers": { + "1": { + "answer": "The whole Earth is in free fall toward the Moon, and its centre accelerates at 33.19 µm/s². The far side is pulled less than that (32.12 µm/s²), so its water falls more slowly than the centre and gets left behind, away from the Moon. The Moon does not push it away. It just pulls the far side less than the centre, and the centre's fall is the reference point.", + "quote": "Far side: 32.12 − 33.19 = −1.07. The far-side water falls toward the Moon slower than the centre, so it is left behind, away from the Moon." + }, + "2": { + "answer": "The Moon moves about 12° a day along its orbit, in the same direction the Earth turns. So the Earth needs about 50 extra minutes to face the Moon again, which makes a lunar day of 24 h 50 min. A place passes through both bulges each lunar day, so high tides come every half lunar day: 12 h 25 min.", + "quote": "The Earth turns about 15° an hour, so it needs about 50 minutes more than 24 hours to face the Moon again: a **lunar day** of 24 h 50 min." + }, + "3": { + "answer": "not covered. The explanation only says solar tides are about half the Moon's. It never says when tides are largest or how the Sun's and Moon's tides combine.", + "quote": "" + }, + "4": { + "answer": "About every 12 hours. The extra 50 minutes comes only from the Moon moving along its orbit. Without that, the lunar day would be 24 h, and two bulges per turn gives a high tide every 12 h.", + "quote": "The Moon also moves about 12° a day along its orbit, in the same direction that the Earth turns. The Earth turns about 15° an hour, so it needs about 50 minutes more than 24 hours" + } + }, + "stops": [ + "\"An acceleration that every part shares does not change the shape of a falling body.\": this is the key step, but it is stated without a reason or an everyday example.", + "\"The Earth and the Moon orbit their shared centre of mass. An orbit is free fall\": it is not obvious why orbiting means the Earth is falling toward the Moon.", + "\"it tilts by R/d = 6371 / 384,400 = 0.0166 rad (0.95°) toward Earth's centre\": R and d are not defined until later, and the tilt geometry is hard to follow.", + "\"It is largest 45° from the near side, at 0.82 µm/s²\": the 0.82 figure is given with no derivation.", + "\"a pull GM/r² changes by about 2R/d of itself across one Earth radius R\": GM/r² and the 2R/d rule come with no explanation.", + "\"The Sun is 389 times farther than the Moon, so its tidal acceleration is about 179 / 389 = 0.46\": dividing by 389 seems to drop part of the 2R/d factor, so it is unclear why this works.", + "\"No centrifugal force is needed.\": centrifugal force was never mentioned before, so this comes out of nowhere.", + "\"an Earth that fell straight at the Moon, with no orbit, would get the same +1.13 and −1.07\": this is hard to square with the earlier statement that the Earth orbits.", + "\"Water flows until the sea surface slopes just enough that gravity along the slope balances this push.\": the step from a sideways push to a sloped surface is quick." + ], + "excess": [ + "\"The unit is µm/s², one millionth of a m/s².\": basic unit material this reader already knows.", + "\"The near value is about 5% larger because the pull changes faster closer to the Moon.\": an aside that the main question does not need.", + "\"For the Moon: 33.19 × 2 × 6371 / 384,400 = 1.10, between 1.13 and 1.07.\": a check on numbers that were already given.", + "\"Height: in an ideal ocean, the bulges rise about 0.36 m above the mean (an estimate).\": off the question.", + "\"Real oceans: continents and basin shapes make tides lag the Moon\": off the question.", + "\"The Moon pulls the near side more than Earth's centre, and the centre more than the far side.\": repeats the earlier sections." + ], + "grounded": { + "1": true, + "2": true, + "3": false, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/published/run.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/run.json new file mode 100644 index 0000000..954e8b9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/two-tides/explanation.md`, a prose explanation (plus its `model.md`, `model.yaml` and `narrative.md`). Prose fits because the argument is a short chain — the Moon's pull at three points, one subtraction, then the Earth's rotation — and a table and a small text figure carry it, so a diagram or interactive page would add nothing; the probe, cold reads, final `explainer check` and blind test all passed, though I made small wording fixes after the blind test and didn't re-run it.", + "error": null, + "cost": 2.064919, + "seconds": 705, + "turns": 101, + "files": [ + "explainers/two-tides/explanation.md" + ], + "review_cost": 0.7731, + "reviews": 5 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/explanation.md b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/explanation.md new file mode 100644 index 0000000..8da97e9 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/explanation.md @@ -0,0 +1,60 @@ +# Why are there two high tides a day, not one? + + + +**The Moon raises two bulges of ocean on opposite sides of Earth: one facing the Moon and one facing away. Earth turns under both bulges, so a point on the coast passes through two high tides in each turn.** + +## Predict first + +The Moon pulls water toward it. If that were the whole story, how many bulges would you expect, and where? Decide before you read on. + +## The Moon pulls every part of Earth by a different amount + +The Moon's gravity gives an acceleration, here called the **pull**, of GM/r². The pull gets weaker with distance r. Earth is 12,742 km wide, so its near side is closer to the Moon than its far side. With the Moon at 384,400 km and Earth's radius 6,371 km: + +| Point on Earth | Distance to Moon | Pull (m/s²) | Pull minus centre's pull (m/s²) | +|---|---|---|---| +| Near side | 378,029 km | 3.433e-5 | 1.13e-6 toward the Moon | +| Centre | 384,400 km | 3.320e-5 | 0 | +| Far side | 390,771 km | 3.213e-5 | 1.07e-6 away from the Moon | + +All three pulls point toward the Moon. So why does the far water move away from the Moon? + +## Only the difference in pull moves water + + +Earth is not held in place. The whole Earth falls toward the Moon with the pull at its centre, 3.320e-5 m/s². (Earth and Moon orbit each other, so this fall never ends in a collision.) The common part of the pull moves rock and water together, so it changes nothing on the surface. +What moves water relative to Earth is the **tidal acceleration**: the pull at a point minus the pull at Earth's centre. That is the last column of the table. + + +- On the near side, the water is pulled more than Earth's centre. It falls toward the Moon a little faster than Earth does: a bulge toward the Moon. +- On the far side, the water is pulled less than Earth's centre. Earth falls toward the Moon a little faster than this water. The water falls behind: a bulge away from the Moon. + + +Nothing pushes the far water. It is pulled toward the Moon, but less than the rest of Earth. + +```text + far bulge near bulge + ___ ___ + (relative) <- / \ EARTH / \ -> ( MOON ) + \___/ centre \___/ + pulled less than centre pulled more than centre +``` + +The tidal acceleration is about 1.1e-6 m/s², about one ten-millionth of g (9.81 m/s²). In an ideal ocean, it raises the water by about 0.5 m from low to high. + +## Evidence that the difference is what counts: the Sun + +The Sun's pull on Earth is 179 times the Moon's. If the raw pull made tides, solar tides would dwarf lunar tides. They do not: solar tides are 0.46 of lunar tides. A difference across a fixed width falls off as 1/d³, not 1/d², and that punishes the distant Sun. + +## Why two highs, and why not exactly every 12 hours + + +The two bulges stay lined up with the Moon. Earth turns under them. A point on the coast passes the near bulge, then the far bulge half a turn later. + +The Moon also moves along its orbit, about 12° per day, in the same direction as Earth turns. Earth must turn a little extra to bring the point back under the Moon. So the cycle is a **lunar day** of 24 h 50 min, and high tides come about every 12 h 25 min. High tide comes about 50 minutes later each day. + +## Limits + +- **Assumption:** this is the idealized "equilibrium tide": one ocean covers Earth and adjusts at once. Real oceans have continents and basins that resonate. Some coasts, such as parts of the Gulf of Mexico, get only one high tide a day. +- **Not covered:** spring and neap tides, where the Sun's smaller pair of bulges adds to or partly cancels the Moon's. diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/model.md b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/model.md new file mode 100644 index 0000000..3180289 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/model.md @@ -0,0 +1,100 @@ +# Semantic model: two high tides a day + +> Source of truth. Every rendering derives from this file. + +## Central question + +Why does a point on the coast get two high tides a day, not one, when the Moon is on only one side of Earth? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (Newton's law of gravity, inverse-square, acceleration), basic programming. New to tides. Knows that the Moon causes tides. Does not know that only the *difference* in pull matters, or why there is a bulge on the side away from the Moon. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| Moon's pull | Gravitational acceleration toward the Moon, GM/r², at a given point. | fact | +| Earth's centre | The point whose acceleration describes how Earth as a whole falls toward the Moon. | fact | +| Tidal acceleration | Moon's pull at a point minus Moon's pull at Earth's centre. | definition | +| Near bulge | Ocean bulge on the side facing the Moon. | fact (idealized) | +| Far bulge | Ocean bulge on the side away from the Moon. | fact (idealized) | +| Lunar day | Time for a point on Earth to return under the Moon: 24 h 50 min. | fact | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| Moon's pull | falls off with distance (1/r²) | near side > centre > far side | | +| Earth's centre | sets the pull that the whole Earth follows | tidal acceleration | uniform part gives no tide | +| Tidal acceleration | points away from Earth's centre on both sides | near bulge, far bulge | | +| Earth's rotation | carries a point through both bulges | two high tides per lunar day | | + +## Causal chain + +```text +pull falls with distance → near side pulled more than centre, far side less +→ relative to Earth's centre, near water moves toward Moon, far water lags behind (away) +→ two bulges on opposite sides → Earth rotates under both → two high tides per 24 h 50 min +``` + +## Quantities + +Constants: G = 6.674e-11, Moon mass 7.35e22 kg, distance 384,400 km, Earth radius 6,371 km. + +| Point | Distance to Moon | Moon's pull (m/s²) | Minus centre (m/s²) | +|---|---|---|---| +| Near side | 378,029 km | 3.433e-5 | +1.13e-6 (toward Moon) | +| Centre | 384,400 km | 3.320e-5 | 0 | +| Far side | 390,771 km | 3.213e-5 | −1.07e-6 (away from Moon, relative) | + +Approximation: tidal acceleration ≈ 2GMR/d³ = 1.10e-6 m/s², about one ten-millionth of g (9.81). + +Sun check: Sun's pull on Earth 5.93e-3 m/s², 179× the Moon's; Sun's tidal acceleration 5.05e-7 m/s², 0.46 of the Moon's. Tide follows 1/d³, the difference, not 1/d², the pull. + +Timing: Moon moves about 12.2° per day along its orbit, so a lunar day is 24 h 50 min, and high tides come every 12 h 25 min. + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks | +|---|---|---| +| Subtract the pull at Earth's centre | Use the raw pull: water flows toward the Moon | Raw pull is toward the Moon everywhere (3.2e-5 to 3.4e-5), predicting one bulge and one high tide a day. Observation: two. Also predicts Sun tides 179× the Moon's; observed is about 0.46×. | + +## Concrete cases + +| Claim | Holds here | Breaks here | +|---|---|---| +| Uniform pull makes no tide | Whole Earth accelerates at 3.320e-5 toward Moon; that part moves water and rock together. | If pull were the same at all points (no 1/r² fall-off), the difference would be 0: no bulges. | +| Two bulges → two highs | Rotation of 360° relative to the Moon in 24 h 50 min passes two bulges 180° apart: highs 12 h 25 min apart. | Real basins: some coasts (parts of the Gulf of Mexico) get one high tide a day, because basin shape and Moon's tilt change the response. | + +## Terms + +- **Pull**: the Moon's gravitational acceleration at one point. +- **Tidal acceleration**: pull at a point minus pull at Earth's centre. Small value (1e-6 m/s²) yet enough to raise water about half a metre in the idealized ocean. +- **Bulge**: region where water is higher than average. +- **Lunar day**: 24 h 50 min. + +## Epistemic status + +- **Established:** two bulges in the equilibrium model; lunar day 24 h 50 min; Sun/Moon ratio 0.46. +- **Mathematical consequence:** near/far tidal accelerations from Newton's law. +- **Assumed:** idealized ocean covering Earth, responding instantly (equilibrium tide). +- **Estimate:** idealized lunar tide range about 0.5 m. + +## Confusion points + +- "The Moon pulls water toward it, so one bulge." → Earth is pulled too; only the difference shapes the water. +- "Far bulge is from centrifugal force of Earth spinning." → Earth's daily spin is the same everywhere around the equator; it shapes Earth's equator, not the tides. The cause is the pull difference. +- "Far side is pushed away." → Nothing pushes. The far side is pulled toward the Moon, but less than Earth's centre, so it falls behind. + +## Scope + +- **Out of scope:** spring and neap tides (Sun and Moon aligned or at right angles) — pointer: add the Sun's bulges, 0.46 of the Moon's. +- **Out of scope:** real tide heights and timing (resonance, continents, Bay of Fundy) — pointer: dynamic tide theory, harmonic tide prediction. +- **Out of scope:** unequal pairs of highs (diurnal inequality) from the Moon's tilt. + +## Representation decision + +- **Stage:** 1 +- **Reason:** A short causal chain with one table of numbers; the geometry fits in one small text figure. +- **Levels:** L1 two bulges, Earth turns under them / L2 pull differs across Earth / L3 subtract the centre's pull, Sun check. diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/model.yaml b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/model.yaml new file mode 100644 index 0000000..b6b7104 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/explainers/two-tides/model.yaml @@ -0,0 +1,63 @@ +# Machine-readable half of model.md. `explainer check two-tides` holds every rendering to it. + +values: + moon_distance_km: 384400 + earth_radius_km: 6371 + earth_diameter_km: 12742 + near_distance_km: 378029 + far_distance_km: 390771 + moon_deg_per_day: 12 + daily_delay_min: 50 + interval_hours: 12 + interval_extra_min: 25 + pull_near: 3.433e-5 + pull_centre: 3.320e-5 + pull_far: 3.213e-5 + tidal_near: 1.13e-6 + tidal_far: 1.07e-6 + g: 9.81 + sun_pull_ratio: 179 + sun_tidal_ratio: 0.46 + lunar_day: "24 h 50 min" + high_tide_interval: "12 h 25 min" + idealized_range_m: 0.5 + +allow: [1, 2, 3, 180, 360, 24] + +require: + - path: pull_near + in: [prose] + - path: pull_far + in: [prose] + - path: lunar_day + in: [prose] + - path: sun_tidal_ratio + in: [prose] + +claims: + - id: why-subtract + kind: why + statement: "Only the pull at a point minus the pull at Earth's centre moves water relative to Earth." + alternative: "Use the raw pull: water flows toward the Moon." + counterexample: "Raw pull points toward the Moon everywhere (3.213e-5 to 3.433e-5 m/s²): one bulge, one high tide a day, and Sun tides 179x the Moon's. Observed: two highs, Sun 0.46x." + ask: "Why is there a bulge on the side of Earth away from the Moon?" + - id: mech-bulges + kind: mechanism + statement: "The near side is pulled more than the centre and the far side less, so relative to the centre both sides move outward." + example: "Near +1.13e-6 m/s² toward the Moon; far 1.07e-6 m/s² away from the Moon, relative to the centre." + - id: mech-rotation + kind: mechanism + statement: "Earth rotates under the two bulges, so a point passes two highs per lunar day." + example: "Lunar day 24 h 50 min; highs every 12 h 25 min." + ask: "How long between two high tides, and why not 12 h exactly?" + - id: misc-push + kind: misconception + statement: "Nothing pushes the far side away; it is pulled less than Earth's centre and falls behind." + correction: "The far side is pulled toward the Moon by 3.213e-5 m/s², less than the centre's 3.320e-5 m/s², so it lags behind Earth's fall." + +accept_unjustified: {} + +terms: + - term: tidal acceleration + means: "pull at a point minus pull at Earth's centre" + avoid: [tidal force pushes, pushed away] diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/read.json new file mode 100644 index 0000000..366b0f0 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/read.json @@ -0,0 +1,41 @@ +{ + "answers": { + "1": { + "answer": "Nothing pushes the far-side water away. It is pulled toward the Moon, just less strongly than Earth's centre is. Earth falls toward the Moon a little faster than that water does, so the water falls behind and makes a bulge on the far side.", + "quote": "On the far side, the water is pulled less than Earth's centre. Earth falls toward the Moon a little faster than this water. The water falls behind: a bulge away from the Moon." + }, + "2": { + "answer": "The Moon moves along its orbit about 12° per day, in the same direction Earth turns. Earth has to turn a little extra to bring a point back under the Moon. That makes a lunar day of 24 h 50 min, so high tides come about every 12 h 25 min.", + "quote": "Earth must turn a little extra to bring the point back under the Moon. So the cycle is a **lunar day** of 24 h 50 min, and high tides come about every 12 h 25 min." + }, + "3": { + "answer": "not covered. The explanation says spring and neap tides, where the Sun's bulges add to or partly cancel the Moon's, are left out. It never says when tides are largest.", + "quote": "" + }, + "4": { + "answer": "About every 12 hours. The extra 25 minutes comes only from the Moon moving along its orbit. Without that motion, Earth would need no extra turn, so one turn would take 24 h and contain two highs.", + "quote": "The Moon also moves along its orbit, about 12° per day, in the same direction as Earth turns. Earth must turn a little extra to bring the point back under the Moon." + } + }, + "stops": [ + "\"GM/r²\": G and M are never defined.", + "\"A difference across a fixed width falls off as 1/d³, not 1/d²\": it says the difference falls off as 1/d³ but never shows why.", + "\"(Earth and Moon orbit each other, so this fall never ends in a collision.)\": it doesn't say why a fall that never ends still counts as falling.", + "\"it raises the water by about 0.5 m from low to high\": it doesn't show how a tiny acceleration becomes 0.5 m.", + "\"solar tides are 0.46 of lunar tides\": no source or working is given for this ratio.", + "\"The two bulges stay lined up with the Moon\": it doesn't say why the bulges don't turn with Earth.", + "\"Real oceans have continents and basins that resonate\": 'resonate' is never explained, and neither is how it gives some coasts only one tide." + ], + "excess": [ + "\"\"", + "\"Earth is 12,742 km wide\": this repeats the radius given in the next sentence.", + "\"Nothing pushes the far water. It is pulled toward the Moon, but less than the rest of Earth.\": this restates the far-side bullet just above it.", + "\"High tide comes about 50 minutes later each day.\": this repeats the 24 h 50 min already given." + ], + "grounded": { + "1": true, + "2": true, + "3": false, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/run.json new file mode 100644 index 0000000..3953c0b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/two-tides/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/two-tides/explanation.md`, a prose explanation of under 600 words. Prose fits because the answer is one short chain of cause and effect: the Moon's pull weakens with distance, so only the difference from the pull at Earth's centre moves water, which gives two bulges. A table of worked numbers and one small text sketch cover the geometry, and Mermaid draws force arrows poorly. The supporting `model.md`/`model.yaml` are alongside it, and `explainer check two-tides` passes.", + "error": null, + "cost": 0.5251108, + "seconds": 114, + "turns": 27, + "files": [ + "explainers/two-tides/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/explainers/vaccine-memory/explanation.md b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/explainers/vaccine-memory/explanation.md new file mode 100644 index 0000000..07fed97 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/explainers/vaccine-memory/explanation.md @@ -0,0 +1,179 @@ +# How a vaccine prepares the immune system for a disease it has never met + +## The puzzle + +Your immune system can make antibodies that bind tightly to a virus that evolved last year. Nothing in your genome was written with that virus in mind. So how can the body "know" in advance what to look for? And how can a vaccine, which is not the disease, leave you protected against the disease? + +The short answer is that the immune system doesn't learn the way a student learns a fact. It works more like a search algorithm: + +1. **Generate** a huge random library of detectors before any threat appears. +2. **Select** the few detectors that happen to fit when a threat shows up. +3. **Amplify and refine** those detectors. +4. **Keep copies** of the best ones for next time. + +A vaccine runs steps 2 to 4 on purpose, using a harmless stand-in for the pathogen. When the real pathogen arrives, the body is already at step 4. + +The rest of this document goes through each step. + +--- + +## Step 0: What is actually being recognised + +A virus or bacterium is covered in molecules, mostly proteins. The immune system doesn't recognise "the measles virus" as a whole. It recognises small patches on these molecules, each a few amino acids across. A molecule the immune system can recognise is called an **antigen**, and the specific patch that a detector binds to is called an **epitope**. + +Recognition is physical. Shape and chemistry have to match closely enough that the two molecules stick together. Think of it as a lock and key, except that the fit is graded rather than all-or-nothing. A good fit sticks strongly (**high affinity**). A poor fit sticks weakly and comes off again. + +So the question "how does the body recognise a new virus?" turns into "how does the body come to have a molecule shaped to fit a patch on that virus?" + +--- + +## Step 1: Generate — a random library, built in advance + +The detectors are receptor proteins on the surface of two kinds of white blood cell, **B cells** and **T cells**. Each individual B or T cell carries one kind of receptor, many copies of it, and that receptor is different from almost every other cell's. + +You have only about 20,000 genes, yet you can make billions of different receptors. Here's how. The receptor genes are stored as *parts lists*, not finished genes. As each young B or T cell develops, it cuts and pastes its own DNA. It picks one segment at random from each of several pools (called V, D and J segments) and joins them. At the joins it adds or deletes a few random DNA letters. The receptor is built from two protein chains, each assembled this way, so the combinations multiply. + +In programming terms it's roughly: + +```python +def make_receptor(): + heavy = random.choice(V_heavy) + junk() + random.choice(D) + junk() + random.choice(J_heavy) + light = random.choice(V_light) + junk() + random.choice(J_light) + return (heavy, light) +``` + +The theoretical number of distinct receptors is astronomically large (estimates are often quoted above 10^13). The number actually present in a person at any time is smaller but still huge: billions of different B and T cell receptors. + +Two consequences matter here: + +- **The library exists before any infection.** Somewhere in your blood and lymph nodes there is probably already a handful of cells whose receptor happens to fit a patch on a virus you have never encountered. Nobody designed it. With enough random shapes, some will fit. +- **Each match is rare.** For any particular epitope, only a tiny fraction of cells fit it, perhaps a few in a million or fewer. A handful of cells can't fight an infection. That is the problem the next steps solve. + +There is also a safety filter. While they develop, cells whose receptors bind strongly to the body's *own* molecules are mostly deleted or switched off. If this filter fails, the result is autoimmune disease. + +--- + +## Step 2: Select — the rare matching cells are found and activated + +When an antigen enters the body, whether from an infection or a vaccine, it gets carried to the lymph nodes. Lymph nodes are meeting points where antigen and immune cells concentrate. Billions of B and T cells pass through them, in effect "trying" the antigen against their receptors. + +A cell whose receptor binds is **selected**. But binding alone is not enough to start a full response. The immune system needs a second signal that says "this is dangerous", not just "this is present": + +- The **innate immune system** is the older, built-in part. It doesn't need a library. It has fixed sensors for broad signs of infection, such as bits of bacterial cell wall, viral genetic material in the wrong place, or damaged tissue. When those sensors fire, they put out alarm signals. +- **Helper T cells**, which have recognised the same pathogen, give B cells permission to proceed. + +This two-key system helps prevent accidental attacks on harmless things. It is also why many vaccines include an **adjuvant**, an ingredient whose job is to trip the innate alarm. A purified protein on its own can look harmless and be ignored. The adjuvant supplies the "this is dangerous" signal. + +--- + +## Step 3: Amplify and refine + +### Amplify: clonal expansion + +A selected cell divides repeatedly, roughly once every 6 to 12 hours, for days. One cell becomes thousands, all carrying the same receptor (a **clone**). This is called **clonal selection**: the antigen doesn't create the receptor, it chooses which pre-existing cells get copied. + +Some of the expanded cells go to work right away: + +- **Plasma cells** are B cells turned into antibody factories. An **antibody** is the secreted, free-floating version of the B cell's receptor. Antibodies coat the pathogen. This blocks it from entering cells and marks it for destruction. +- **Killer (cytotoxic) T cells** find and destroy the body's own cells that have been infected. They recognise fragments of viral protein that infected cells display on their surface. + +### Refine: affinity maturation + +The first matches are usually mediocre. They bind, but not tightly. Inside lymph nodes, in structures called **germinal centres**, B cells run what amounts to an evolutionary algorithm: + +1. **Mutate.** The dividing B cells deliberately introduce random point mutations into their receptor genes, at a rate roughly a million times higher than the normal mutation rate. This is called **somatic hypermutation**. +2. **Compete.** The mutants compete for a limited supply of antigen and for help from T cells. Cells that bind more tightly get more of both. +3. **Survive or die.** Winners get survival signals and divide again. Losers die off. +4. **Repeat** over many rounds, for weeks. + +```python +population = selected_clones +for round in range(many): + population = [mutate(cell) for cell in population for _ in range(2)] + population = keep_best(population, by=binding_strength_to(antigen)) +``` + +The result is antibodies that bind the target much more tightly than the starting ones, often by factors of 10 to 1000 or more. B cells can also **class-switch**, changing the antibody's "tail" while keeping the binding end. This lets them make types suited to the blood (IgG) or to mucous surfaces (IgA), for example. + +--- + +## Step 4: Keep copies — memory + +When the antigen is cleared, most of the expanded army dies off. That's deliberate, since keeping it would waste resources. A minority survive as **memory cells**: + +- **Memory B cells** carry the refined, high-affinity receptors. +- **Memory T cells** (both helper and killer types). +- **Long-lived plasma cells** settle in the bone marrow and keep secreting a baseline level of antibody, sometimes for decades. + +Memory cells differ from the original naive cells in three ways that matter: + +| | Naive (before) | Memory (after) | +|---|---|---| +| How many cells fit this antigen | a handful | hundreds to thousands of times more | +| Quality of fit | mediocre | refined by affinity maturation | +| Activation | needs strong signals, slow | responds faster, to less stimulus | + +This is the whole trick. The **first** encounter with an antigen produces a **primary response**. It is slow to start (useful antibody levels take around 1–2 weeks) and modest in size. A **second** encounter produces a **secondary response**. It starts within days, rises much higher, and uses better antibodies. + +``` +antibody +level + │ ╱‾‾‾‾‾╲ + │ ╱ ╲___ secondary + │ ╱ ‾‾‾─── + │ ╱ + │ primary ╱ + │ ╱‾‾╲ │ + │ _____╱ ╲_______ ____________ ╱ + └───┬──────────────────────────────┬────────────────▶ time + first exposure second exposure + (lag ~1-2 weeks) (lag ~days, higher, tighter binding) +``` + +--- + +## So what does a vaccine do? + +Everything above happens during any infection. The danger of a natural infection is that the primary response is **slow**. During those 1–2 weeks the pathogen is multiplying and can kill you, cripple you, or spread to others before the immune system catches up. + +A vaccine gives the immune system its *first encounter* under safe conditions. It shows the body the right antigens, with enough of an alarm signal to trigger a real response, but without a pathogen that can cause the disease. The primary response, with its slow lag, happens while nothing is at stake. When the real pathogen arrives, it meets a **secondary** response: antibodies already circulating, plus memory cells that expand within days. Often the infection is stopped before it causes symptoms, or kept mild. + +So "preparing the immune system for a disease it has never met" means this: the vaccine doesn't install new knowledge. It **selects, enlarges and sharpens** the part of an already-random library that fits this pathogen, and keeps it on standby. + +### Ways to show the antigen + +Different vaccine types are different ways of presenting the same antigens: + +| Type | What is injected | Trade-off | +|---|---|---| +| **Live attenuated** (e.g. measles, MMR) | A weakened strain that still replicates a little | Very realistic, strong long-lasting immunity; not suitable for people with weakened immune systems | +| **Inactivated** (e.g. many flu shots, polio IPV) | Whole pathogen, killed | Can't cause infection; usually needs adjuvant and boosters | +| **Subunit / protein** (e.g. hepatitis B, HPV) | Just one or a few purified proteins, sometimes assembled into virus-like particles | Very safe and precise; relies on adjuvant | +| **Toxoid** (e.g. tetanus, diphtheria) | Inactivated version of the toxin the bacterium makes | Protects against the toxin, which is what actually causes the disease | +| **mRNA / viral vector** (e.g. several COVID-19 vaccines) | Genetic instructions; your own cells briefly make the antigen protein | Fast to design and update; the instructions are temporary and break down within days | + +In every case, the immune system ends up seeing the target's shapes in a context that sets off the alarm, and the four steps run. + +--- + +## Why some details work the way they do + +**Why boosters?** Each additional exposure runs another round of selection and affinity maturation. The memory pool grows and the antibodies get better. Some antigens, especially non-replicating ones, give a weaker first response, so a second or third dose is needed to build a solid memory. + +**Why does protection sometimes fade?** Antibody levels in the blood fall over time. If protection depends on having antibodies *already present* when the pathogen arrives (as with infections of the nose and throat that move fast), it can wane even though memory cells remain. Memory cells usually still make illness less severe, because they shorten the lag. + +**Why do we need new flu vaccines each year?** Memory is specific to the epitopes it was selected on. Influenza mutates its surface proteins quickly (**antigenic drift**), so last year's memory cells fit this year's virus less well. Measles barely changes its key surface protein, which is one reason a childhood measles vaccine can protect for life. + +**Why doesn't it work for everything?** Some pathogens hide their vulnerable patches, change them constantly (HIV), or live inside cells where antibodies can't reach. For those, finding an antigen that produces protective memory is the hard part. + +**Herd immunity.** If enough people have memory, a pathogen can't find enough hosts to keep spreading. That protects people who can't be vaccinated or who respond poorly. + +--- + +## Summary + +- Your body makes billions of B and T cell receptors at random, **before** meeting any pathogen. A few happen to fit almost anything. +- When an antigen appears along with a danger signal, the rare matching cells are **selected** and copied by the thousands. +- In germinal centres, B cells **mutate and compete**, evolving antibodies that bind much more tightly. +- After the threat is gone, **memory cells** and long-lived antibody producers remain: more of them, better at binding, and quicker to respond. +- A vaccine triggers this process with a harmless form of the antigen. The slow first response happens safely, and the real disease meets the fast second response. diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/read.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/read.json new file mode 100644 index 0000000..37b20f6 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "It has to contain or make the pathogen's antigens (the molecules with the patches the immune system recognises), together with enough of an alarm signal (often from an adjuvant) to start a real response. It doesn't make you ill because there is no pathogen in it that can cause the disease. It might be a killed whole pathogen, a purified protein, a toxoid, or mRNA that has your own cells briefly make the protein. A live attenuated vaccine is a weakened strain that only replicates a little.", + "quote": "It shows the body the right antigens, with enough of an alarm signal to trigger a real response, but without a pathogen that can cause the disease." + }, + "2": { + "answer": "After the first encounter, memory cells are left behind. There are hundreds to thousands of times more of them than the original matching cells, their binding has been sharpened by affinity maturation, and they react faster to less stimulus. Some antibodies are also already circulating. So the response starts within days instead of 1–2 weeks.", + "quote": "When the real pathogen arrives, it meets a **secondary** response: antibodies already circulating, plus memory cells that expand within days." + }, + "3": { + "answer": "Memory is specific. The vaccine selects and multiplies only the cells whose receptors fit the epitopes of that one pathogen. An unrelated virus has different surface shapes, so these memory cells don't fit it, and its first encounter is still a slow primary response.", + "quote": "Memory is specific to the epitopes it was selected on." + }, + "4": { + "answer": "Protection gets weaker because the memory cells and antibodies were selected and refined to fit the old shape. They fit the changed patch less well. This is antigenic drift in flu, and it is why flu vaccines are updated every year. If the patch changes constantly (as in HIV), it is hard to get protective memory at all.", + "quote": "Influenza mutates its surface proteins quickly (**antigenic drift**), so last year's memory cells fit this year's virus less well." + } + }, + "stops": [ + "V, D and J segments are named but never explained beyond 'pools': \"(called V, D and J segments)\"", + "junk() appears in the code without a definition: \"random.choice(V_heavy) + junk() + random.choice(D)\"", + "heavy and light chains appear in the code, but the text only says \"two protein chains\": \"The receptor is built from two protein chains\"", + "It is unclear how helper T cells 'recognised' the pathogen or how they give permission: \"Helper T cells, which have recognised the same pathogen, give B cells permission to proceed.\"", + "'Naive' cells are used in the table without being defined: \"Memory cells differ from the original naive cells\"", + "Killer T cells spotting displayed fragments is mentioned without saying how: \"They recognise fragments of viral protein that infected cells display on their surface.\"", + "It is not clear why non-replicating antigens give a weaker response: \"Some antigens, especially non-replicating ones, give a weaker first response\"", + "It is not explained why memory cells don't fully protect against fast infections: \"as with infections of the nose and throat that move fast\"" + ], + "excess": [ + "The class-switch detail is not needed for the question: \"B cells can also **class-switch**, changing the antibody's \"tail\" while keeping the binding end.\"", + "Herd immunity is off the main question: \"If enough people have memory, a pathogen can't find enough hosts to keep spreading.\"", + "The autoimmune safety filter is a side point: \"There is also a safety filter. While they develop, cells whose receptors bind strongly to the body's *own* molecules\"", + "The summary repeats earlier points: \"A vaccine triggers this process with a harmless form of the antigen. The slow first response happens safely\"", + "The second pseudocode block adds little to the numbered list: \"population = [mutate(cell) for cell in population for _ in range(2)]\"", + "The plan line is just a signpost: \"The rest of this document goes through each step.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/run.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/run.json new file mode 100644 index 0000000..e6ea796 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/none/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/vaccine-memory/explanation.md`, a prose explainer with short Python sketches, tables and an ASCII graph of antibody levels, because the mechanism is a step-by-step process (random generation, then selection, then refinement through mutation and competition, then memory) that a reader who programs can follow best as a story with algorithm-like code.", + "error": null, + "cost": 0.16533019999999998, + "seconds": 62, + "turns": 2, + "files": [ + "explainers/vaccine-memory/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/explanation.md b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/explanation.md new file mode 100644 index 0000000..0017182 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/explanation.md @@ -0,0 +1,102 @@ +# How a vaccine prepares the immune system for a disease it has never met + +**Short answer.** The immune system does not design a defense after it meets a pathogen. Before any infection, it already holds a huge library of randomly shaped receptors. A vaccine shows the body harmless copies of a pathogen's shapes. The few cells that happen to match those shapes multiply, and a large group of them stays behind as *memory*. When the real pathogen arrives, the response starts from thousands of matching cells instead of a few, and it wins the race. + +The rest of this file explains each part of that sentence: the library, the selection, the multiplication, the memory, and the race. + +## Why this matters: infection is a race + +A virus copies itself exponentially. As an illustrative model, suppose an infection starts with 100 virus particles that double every 8 hours: + +| Time | Doublings | Virus particles | +|---|---|---| +| Day 0 | 0 | 100 | +| Day 3 | 9 | ~51,000 | +| Day 7 | 21 | ~210,000,000 | + +A first-time immune response takes about 7–14 days to produce large amounts of antibody. By then the virus has had a week of unopposed growth, and the person is sick. A response from memory takes about 2–4 days, and some antibody is already present on day 0. The vaccine's job is to move the defense from the slow column to the fast one. + +## Terms + +- **Antigen**: a molecular shape on a pathogen, for example a region of a surface protein, that a receptor can bind. +- **Lymphocyte**: a white blood cell that carries one receptor shape, in many copies on its surface. There are two main kinds: + - **B cells** make **antibodies**: free-floating copies of their receptor that stick to a pathogen and block it or mark it for destruction. + - **T cells** kill infected cells and help B cells activate. + +## Step 1: the library exists before the disease + +Each new lymphocyte builds its receptor gene by cutting and joining randomly chosen gene segments. The joins also add random letters. The result is that each lymphocyte carries a receptor shape that no one chose. Across the body's hundreds of billions of lymphocytes, the library contains millions of different shapes. + +This is the key to the question. The immune system can "know" a disease it has never met because it never needed to know it. It made a random library in advance, and some shapes in the library fit almost any antigen. + +A random library would also contain receptors that bind the body's own proteins. During development, cells that bind self strongly are mostly removed. This is why the library attacks foreign shapes and mostly ignores the body. + +The cost of a random library: any single antigen matches very few cells. Estimates are about 1 in 100,000 to 1 in 1,000,000 lymphocytes (estimate; varies by antigen and person). A few rare matching cells cannot stop a virus alone. That is why the first response is slow. + +## Step 2: the vaccine selects the matching cells + +A vaccine delivers the antigen without the disease. Common forms: + +- an inactivated (killed) pathogen; +- a weakened (attenuated) pathogen that copies itself poorly; +- a purified pathogen protein (subunit vaccine); +- mRNA or a harmless carrier virus that makes the body's own cells produce one pathogen protein for a short time. + +Antigen alone is not enough. A purified protein injected by itself often gives a weak response, because lymphocytes also need a **danger signal**. Fast, non-specific immune cells give this signal when they detect damage or microbial patterns. Vaccines supply it with an **adjuvant** (an added ingredient that triggers the signal) or through the vaccine's own microbial parts. Without the danger signal, the body treats the antigen as harmless and ignores it. + +With antigen and danger signal together, the rare matching lymphocytes bind the antigen and activate. Non-matching cells do nothing. This is **clonal selection**: the antigen picks its own responders from the library. + +## Step 3: the selected cells multiply, then most die + +An activated lymphocyte divides about every 8 hours (estimate). Its descendants all carry the same receptor. This is **clonal expansion**. + +Worked example (illustrative numbers): + +1. Start: 100 naive cells match the antigen. +2. Expand for 5 days: 15 doublings. 100 × 2¹⁵ ≈ **3.3 million** cells. +3. The antigen is cleared. About 90–95% of the expanded cells die (estimate). Keep 5%: **~164,000** cells. + +Those 164,000 cells are the **memory cells**. The pool is now about 1,600 times larger than the original 100. + +Now compare a second encounter. To reach 3.3 million cells again from 164,000 takes log₂(20) ≈ 4.3 doublings, about **1.4 days**. The first time it took 5 days. Memory cells also activate more easily than naive cells, so the real gap is larger than this arithmetic shows. + +## Step 4: B cells improve their fit + +During the response, activated B cells gather in lymph nodes and mutate their receptor genes at a high rate. Mutants that bind the antigen more tightly receive more survival signals and divide more. Mutants that bind worse die. This is **affinity maturation**: a small evolution process inside the lymph node, with the antigen as the selection test. + +The result is that memory B cells and their antibodies bind much more tightly than the original random match did. A second vaccine dose (a **booster**) runs this process again: it expands the memory pool further and selects even better binders. + +Some B cells also become **long-lived plasma cells** in the bone marrow. They secrete antibody continuously, in some cases for years. This antibody is already in the blood on day 0 of an infection, so it can block the pathogen before any cell has to activate. + +## Summary of the chain + +```text +random receptor library (before any exposure) + │ vaccine: antigen + danger signal + ▼ +rare matching cells activate (selection) + ▼ +matching cells multiply ~8 h/division (expansion) + ▼ +B cells mutate, best binders kept (affinity maturation) + ▼ +most cells die, 5–10% remain (memory pool ~1000× naive pool) + │ real infection + ▼ +large, pre-tuned pool + antibody already present → response in 2–4 days, not 7–14 +``` + +## Where it breaks + +Memory is specific to shapes. It protects only while the pathogen keeps those shapes. + +- **Holds:** measles keeps the same surface shapes. Two doses of measles vaccine give long, strong protection. +- **Breaks:** influenza changes its surface protein a little each year (**antigenic drift**). Last year's antibodies bind this year's virus poorly, so the race starts closer to the slow column. That is why the flu vaccine is updated every year. + +Protection also weakens when antibody levels fall over time. Memory cells often still remain, so a later infection may not be prevented but is usually milder, because the fast response still arrives before the virus reaches its peak. + +**Predict before you read on:** a pathogen keeps most of its surface but changes one region. Which antibodies still help? *Answer: those that bind the unchanged regions. That is why vaccine designers target protein regions that the pathogen cannot change without breaking its own function.* + +## Left out + +How cells display antigen to T cells (MHC molecules), herd immunity, vaccine side effects, and specific schedules. Each is a separate topic. diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/model.md b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/model.md new file mode 100644 index 0000000..fe681e6 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/model.md @@ -0,0 +1,53 @@ +# Model: vaccine memory + +## Central question +How does a vaccine prepare the immune system for a disease it has never met? + +## Answer in one line +The immune system does not design a defense after it sees a pathogen. It already holds a huge random library of receptor shapes. A vaccine shows harmless pathogen shapes, and the few cells that match them multiply and leave a large, long-lived memory pool. + +## Reader +Technical, school science, new to immunology. Knows cells divide, proteins have shapes, exponential growth. + +## Entities +- Antigen: a molecular shape on a pathogen that a receptor can bind. +- Lymphocyte: white blood cell with one receptor shape. B cells (make antibodies) and T cells (kill infected cells, help B cells). +- Antibody: soluble copy of a B-cell receptor; binds antigen and blocks or marks the pathogen. +- Memory cell: lymphocyte left after a response; more numerous and faster to activate than the naive precursor. +- Long-lived plasma cell: B-cell descendant in bone marrow; secretes antibody for years. +- Adjuvant / danger signal: innate-immune activation that tells lymphocytes the antigen matters. + +## Causal chain +1. Random gene recombination makes each new lymphocyte carry a random receptor. Self-reactive ones are mostly removed. +2. Vaccine delivers antigen + danger signal. +3. Matching rare cells (~1 per 10^5–10^6, estimate) bind antigen and are activated (clonal selection). +4. Activated cells divide every ~8 h (clonal expansion). +5. B cells mutate receptors and the better binders are selected (affinity maturation). +6. Antigen is gone; ~90–95% of the expanded cells die (contraction); 5–10% stay as memory. +7. Real infection: memory pool is ~1000x larger and pre-tuned; preexisting antibody acts at once. + +## Worked numbers (illustrative) +- 100 naive matching cells, 15 doublings (5 days) → ~3.3 million; keep 5% → ~164,000 memory cells. +- Re-expansion to 3.3 million from 164,000 needs ~4.3 doublings ≈ 1.5 days, vs 5 days. +- Virus race: 100 particles doubling every 8 h → ~51,000 at day 3, ~2×10^8 at day 7. + +## Why each step +- Why a library and not design: receptors are made before exposure; no mechanism builds a receptor from the antigen. +- Why a danger signal: antigen alone is often ignored (tolerance); purified protein without adjuvant gives weak responses. +- Why harmless: inactivated, attenuated, protein subunit, mRNA, or viral vector give the shape without the disease. +- Why boosters: a second exposure expands memory further and adds affinity maturation. + +## Failure case +Memory binds specific shapes. Influenza changes its surface protein (antigenic drift); old antibodies bind poorly; new vaccine each year. Antibody levels also wane over time. + +## Confusion points +- "The vaccine teaches the body": no, it selects from what exists. +- Random library would attack self: developmental deletion (tolerance) removes most self-reactive cells. +- Memory does not always prevent infection; often it prevents severe disease. + +## Epistemic labels +- Precursor frequencies, % surviving contraction, division times: estimates, vary by antigen and person. +- Virus race numbers: illustrative model. + +## Scope (left out) +Antigen presentation (MHC) detail, herd immunity, side effects, specific vaccine schedules. diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/model.yaml b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/model.yaml new file mode 100644 index 0000000..0319068 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/explainers/vaccine-memory/model.yaml @@ -0,0 +1,21 @@ +slug: vaccine-memory +precursor_frequency: "1 in 100,000 to 1 in 1,000,000" # estimate +division_time_hours: 8 # estimate, activated lymphocyte +example: + naive_matching_cells: 100 + expansion_days: 5 + doublings: 15 + peak_cells: 3276800 # 100 * 2^15 + surviving_fraction: 0.05 # estimate, 5-10% range + memory_cells: 163840 + fold_increase_over_naive: 1638 + re_expansion_doublings: 4.32 # log2(20) + re_expansion_days: 1.4 +virus_race: # illustrative + start_particles: 100 + doubling_hours: 8 + day3: 51200 + day7: 209715200 +primary_response_days: "7-14" +memory_response_days: "2-4" +failure_case: influenza antigenic drift diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/read.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/read.json new file mode 100644 index 0000000..42e14b1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "It must contain or produce the antigen, meaning the pathogen's molecular shape. Examples are a killed pathogen, a weakened pathogen, a purified protein, or mRNA or a carrier virus that makes one protein. It also needs a danger signal, from an adjuvant or the vaccine's own microbial parts. It does not make you ill because it delivers only the shape, not the working disease: the pathogen is killed, copies itself poorly, or is reduced to a single protein. The explanation states this but does not spell out why a weakened pathogen that still copies itself causes no illness.", + "quote": "A vaccine delivers the antigen without the disease." + }, + "2": { + "answer": "Vaccination leaves a memory pool much larger than the few original matching cells (about 164,000 instead of 100). This pool needs only about 4.3 doublings, roughly 1.4 days, to reach full size, compared with 5 days the first time. Memory cells also activate more easily, their antibodies bind more tightly after affinity maturation, and some antibody is already in the blood on day 0.", + "quote": "To reach 3.3 million cells again from 164,000 takes log₂(20) ≈ 4.3 doublings, about **1.4 days**. The first time it took 5 days. Memory cells also activate more easily than naive cells" + }, + "3": { + "answer": "This is reasoned from the text, which does not discuss unrelated viruses directly. The vaccine selects and expands only the cells whose receptors match its antigen's shapes. An unrelated virus has different shapes, so the expanded memory cells and antibodies do not bind it. Its matching cells are still rare naive cells, so the response stays slow.", + "quote": "Memory is specific to shapes. It protects only while the pathogen keeps those shapes." + }, + "4": { + "answer": "Protection drops. Old antibodies and memory cells were selected to fit the old shape, so they bind the new shape poorly. The response falls back toward the slow first-time timeline. Antibodies that bind unchanged regions still help, which is why the flu vaccine is updated every year.", + "quote": "Last year's antibodies bind this year's virus poorly, so the race starts closer to the slow column. That is why the flu vaccine is updated every year." + } + }, + "stops": [ + "'naive cells' is used but never defined: \"Start: 100 naive cells match the antigen.\"", + "The summary contradicts the worked example: \"most cells die, 5–10% remain (memory pool ~1000× naive pool)\" versus \"Keep 5%\" and \"about 1,600 times larger\"", + "Unclear meaning: \"The joins also add random letters.\"", + "No reason given why a pathogen that still copies itself causes no illness: \"a weakened (attenuated) pathogen that copies itself poorly;\"", + "The danger signal is introduced without saying what it physically is: \"Fast, non-specific immune cells give this signal when they detect damage or microbial patterns.\"", + "Metaphor that refers back to the table: \"the race starts closer to the slow column\"", + "The role of T cells in the memory story is never explained: \"T cells kill infected cells and help B cells activate.\"", + "Plasma cells appear abruptly: \"Some B cells also become **long-lived plasma cells** in the bone marrow.\"", + "Unclear how millions of shapes can fit almost any antigen: \"some shapes in the library fit almost any antigen.\"" + ], + "excess": [ + "\"The rest of this file explains each part of that sentence: the library, the selection, the multiplication, the memory, and the race.\"", + "The summary chain repeats Steps 1–4: \"random receptor library (before any exposure)\"", + "\"How cells display antigen to T cells (MHC molecules), herd immunity, vaccine side effects, and specific schedules.\"", + "Self-tolerance is off the question: \"During development, cells that bind self strongly are mostly removed.\"", + "\"Two doses of measles vaccine give long, strong protection.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/run.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/run.json new file mode 100644 index 0000000..47211e1 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/principles/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/vaccine-memory/explanation.md`, plus `model.md` and `model.yaml` as the source facts behind it. I chose prose because the subject is one causal chain: random receptor library → selection → expansion → memory → a faster race against the virus. A short text diagram inside the file covers the overview, so an interactive page or separate diagram would not have added anything. I could not run `explainer check` because I have no shell here, so the numbers have not been automatically checked against `model.yaml`.", + "error": null, + "cost": 0.2302082, + "seconds": 68, + "turns": 4, + "files": [ + "explainers/vaccine-memory/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/explanation.md b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/explanation.md new file mode 100644 index 0000000..92ce5c5 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/explanation.md @@ -0,0 +1,75 @@ +# How does a vaccine prepare the immune system for a disease it has never met? + + + +**A vaccine does not teach the body a new shape. The body already has a few cells that bind the virus or bacterium, made at random long before. The vaccine makes those few cells multiply once, without the disease, and many of their descendants stay. So when the real infection arrives, the body controls it in about 3 days instead of about 7.** + +Numbers marked *illustrative* are round numbers that show the mechanism; real values vary by disease and person. + +## Infection is a race + +A **pathogen** is a virus or bacterium that multiplies in the body and causes disease. Illustrative: an infection starts with 100 particles, the pathogen grows 10× per day, and symptoms start above 100 million particles. + +Immune cells also multiply. The body **controls** the infection when its immune cells remove the pathogen faster than the pathogen grows. So ask: what does a vaccine change in this race? + +## The matching cells exist before the pathogen does + + +A **lymphocyte** is a white blood cell that carries many copies of one kind of **receptor**. A receptor is a protein that binds one family of similar molecular shapes. The shape that a receptor binds on a pathogen is an **antigen**; for example, the spike protein on the surface of a coronavirus. + +Each new lymphocyte builds its receptor by joining pieces of genes in a random combination and adding random edits at the joins. So one person carries more than 100 million different receptors (estimate). One gene per receptor is impossible: humans have about 20,000 genes, and no gene list could anticipate a virus that evolved last year. + +Because each receptor binds a family of shapes, and a pathogen shows many different antigens, a few lymphocytes bind almost any new pathogen by chance. + + +This is the central idea. The body does not design a receptor after it sees the pathogen. The pathogen selects from a random library that already exists. + +## The first meeting is slow + + +When an antigen arrives, only the lymphocytes whose receptors bind it activate and divide. This is **clonal selection**. A **clone** is one lymphocyte and all its descendants; they share the same receptor. + +For one antigen, about 1 in 100,000 to 1 in 1,000,000 lymphocytes bind it (estimate). Illustrative: 100 cells. Call this number the **precursor count**: the lymphocytes able to respond at the start. Each activated cell divides about every 8 hours. + + +The clone's cells become **effector cells**, which fight now. Some release **antibodies**: free copies of their receptor that stick to the pathogen and block it. Others kill infected cells. One effector cell can release thousands of antibodies per second, so the response does not need as many cells as the pathogen has particles. Illustrative: 1,000,000 effector cells control the pathogen. For simplicity, both cases below use this one target. + + +How long does the first response take? Let n be the number of divisions. Then 100 × 2^n = 1,000,000, so 2^n = 10,000 and n ≈ 13.3. At 8 hours each, that is 4.4 days. The rare matching cells also need about 2.5 days to meet the antigen and activate. Total: about 7 days. This first response is the **primary response**. In 7 days the pathogen grows from 100 to 100 × 10^7 = 1 billion particles. That is above the symptom level of 100 million, so the person is sick. + +## What stays behind + + +After the pathogen is gone, about 99% of the effector cells die. About 1% stay for years or decades as **memory cells**. Illustrative: 1% of 1,000,000 is 10,000 memory cells, which is 100× the original precursor count of 100. + +At the next meeting with the same antigen, 10,000 × 2^n = 1,000,000 gives 2^n = 100, so n ≈ 6.6 divisions, or 2.2 days. Memory cells already wait in the tissues and need less stimulation, so they activate in under a day. Total: about 3 days. This faster response, which starts from memory cells, is the **secondary response**. + +| | First meeting | Same antigen again | +|---|---|---| +| Precursor count | 100 | 10,000 | +| Divisions to reach 1,000,000 | 13.3 | 6.6 | +| Days to control | about 7 | about 3 | +| Pathogen peak (from 100, ×10 per day) | 1 billion | 100,000 | +| Above 100 million (symptoms)? | yes | no, 1,000× below | + +The rule behind the table: each doubling of the precursor count saves one division, which is 8 hours. + +## What the vaccine does + + +A **vaccine** contains the antigen, or instructions for the body's cells to make it. It contains no pathogen that can multiply to disease. So the vaccine runs the slow first response while nothing is growing, and leaves the memory cells behind. When the real pathogen comes, the race starts with the secondary response, not the primary one. + + +Antigen alone is often not enough. A lymphocyte activates only when it binds its antigen *and* receives a **danger signal** from other immune cells that sense a threat. This rule stops attacks on harmless shapes such as food proteins. A purified protein alone gives no danger signal and often only a weak response. So many vaccines contain an **adjuvant**: an ingredient that triggers the danger signal. + +## When memory does not help + +Memory cells bind only the antigen that made them. Measles keeps the same antigens: two doses of measles vaccine prevent about 97% of cases, for decades. Influenza changes its surface antigens from year to year. The old memory receptors bind poorly, so the response falls back toward the slow first one. That is why flu vaccines prevent only about 40–60% of cases in a typical season and are updated every year. The cause is a changed antigen, not memory cells that wear off. + +A vaccine does not stop the pathogen from entering. It shortens the race. + +## Summary + +A vaccine prepares the body for a disease it has never met because the body does not need to have met it: the matching lymphocytes already exist by chance. The vaccine runs the slow primary response safely, so the real infection meets a secondary response that starts with 100× more cells. + +**Not covered here:** how receptors improve their fit over weeks (affinity maturation), how T cells help B cells, how antibody levels fall over months and why some vaccines need boosters, the types of vaccine (live, inactivated, protein, mRNA), and herd immunity. A standard textbook such as *Janeway's Immunobiology* covers each. diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/model.md b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/model.md new file mode 100644 index 0000000..ef96a12 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/model.md @@ -0,0 +1,163 @@ +# Semantic model: how a vaccine prepares the immune system + +> Source of truth. Every rendering derives from this file. + +## Central question + +How does a vaccine prepare the immune system for a disease it has never met? + +## Audience and prior knowledge + +A technical reader: school mathematics and science (cells, DNA and genes, proteins have shapes, exponential growth), basic programming (lookup, cache, random generation). New to immunology. Does not know: antigen, lymphocyte, receptor, clone, antibody, memory cell, adjuvant, primary/secondary response. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| pathogen | A virus or bacterium that multiplies in the body and causes the disease. | fact | +| antigen | A molecular shape on a pathogen (usually part of a surface protein) that a receptor can bind. | fact | +| lymphocyte | A white blood cell (B cell or T cell) that carries one kind of receptor, many copies of it. | fact | +| receptor | The protein on a lymphocyte surface that binds one antigen shape; each lymphocyte has a different, randomly generated receptor. | fact | +| naive lymphocyte | A lymphocyte that has not yet bound its antigen. | fact | +| clone | All lymphocytes descended from one lymphocyte; they share its receptor. | fact | +| clonal selection | Antigen activates only the lymphocytes whose receptors bind it; those multiply. | fact | +| effector cell | An activated lymphocyte that fights now: plasma cells release antibodies; killer T cells kill infected cells. | fact | +| antibody | A free-floating copy of a B cell receptor; it sticks to the antigen and blocks or tags the pathogen. | fact | +| memory cell | A long-lived lymphocyte left after a response, from an expanded clone, that responds fast to the same antigen. | fact | +| danger signal | A second signal from innate (built-in, non-specific) immune cells that says "this is a threat"; a lymphocyte needs it to activate. | fact (simplified) | +| adjuvant | An ingredient in a vaccine that triggers the danger signal. | fact | +| vaccine | A preparation that shows the antigen (and a danger signal) to the immune system without the pathogen's ability to cause disease. | fact | +| primary response | The response to an antigen met for the first time; it starts from naive lymphocytes. | fact | +| secondary response | The response to an antigen met again; it starts from memory cells and existing antibodies. | fact | + +## Relationships + +| From | Relationship | To | Notes | +|---|---|---|---| +| gene-segment shuffling | generates, before any infection | receptor library | random; not guided by any pathogen | +| antigen + danger signal | activates | matching naive lymphocyte | clonal selection | +| activated lymphocyte | divides into | clone of effector cells | and some memory cells | +| memory cells | outnumber | naive precursors | illustrative 100× (estimate range 10–1,000×) | +| vaccine | runs | primary response | without disease | +| pathogen with same antigen | meets | secondary response | fast, large | + +## Dependencies (learning order) + +1. The pathogen grows exponentially; the defence is a race. +2. The receptor library exists before any infection (random generation). +3. Clonal selection: antigen picks the matching lymphocytes; they multiply. +4. The primary response is slow because the matching clone starts small. +5. After the response, memory cells remain, more of them than before. +6. A vaccine runs step 3–5 without the disease. +7. Memory protects only while the antigen stays the same. + +## Causal chain + +```text +random receptor library (before birth onward) + → vaccine shows antigen + danger signal + → matching rare lymphocytes activate and multiply (≈1 week) + → most effector cells die; memory cells + long-lived plasma cells remain + → real pathogen arrives → memory cells + antibodies respond in ≈3 days + → pathogen controlled before it grows enough to cause disease +``` + +## Temporal sequence + +1. Lymphocytes form continuously in bone marrow and thymus, each with a random receptor. +2. Day 0: vaccine. Days 0–3: antigen reaches lymph nodes, rare matching cells find it and activate. +3. Days 3–7: clone divides about every 8 hours. +4. Days 7–14: response peaks. +5. Weeks after: about 99% of effector cells die; memory cells remain for years to decades. +6. Later infection: secondary response in about 2–4 days. + +## Quantities + +All "illustrative" numbers are a round-number model to show the race, labelled as such. + +- Receptor library: more than 100 million different receptors in one person (estimate; true diversity likely higher). Built by randomly joining gene segments and editing the joins. +- Human genes: about 20,000. So one gene per receptor is impossible. +- Coverage: one receptor binds a family of similar shapes, and a pathogen carries many different antigens; so a random library of >10^8 receptors almost always contains some that fit. +- Self-binding: most new lymphocytes whose receptors bind the body's own molecules are removed before release (established; detail out of scope). +- Naive precursors for one antigen: roughly 1 in 100,000 to 1 in 1,000,000 lymphocytes (estimate). Illustrative precursor count: 100 cells. +- Division time of an activated lymphocyte: about 8 hours (3 per day). +- Why fewer cells can beat more virus: the response does not need to outnumber the pathogen. One plasma cell releases thousands of antibodies per second; one killer T cell kills many infected cells. Illustrative: 1,000,000 effector cells clear the pathogen faster than it grows ("control"). +- Primary: 100 × 2^n = 10^6 → n ≈ 13.3 divisions ≈ 4.4 days, + about 2.5 days to find antigen and activate ≈ 7 days. +- Contraction: about 99% of the 10^6 effector cells die; about 1% remain as memory cells (illustrative 10,000 = 100× the naive count; real ratio estimates 10–1,000×). +- Memory: 10,000 → 2^n = 100 → n ≈ 6.6 divisions ≈ 2.2 days, + about 0.5–1 day to activate (memory cells already sit in tissues and lymph nodes and need less stimulation) ≈ 3 days. +- Illustrative pathogen: starts at 100 particles, grows 10× per day; symptoms appear above about 10^8 particles. Primary: 7 days → 100 × 10^7 = 10^9 (sick; symptoms from day 6). Secondary: 3 days → 100 × 10^3 = 10^5, 1,000× below the symptom level. +- Meaning of the precursor count: low (100) → about 7 days to control; high (10,000) → about 3 days. Every 2× more precursors saves one division (8 h). +- Effectiveness: the percentage drop in disease among vaccinated people compared with unvaccinated people (e.g. 97%: 90 of 100 unvaccinated get measles after exposure, about 3 of 100 vaccinated do). +- Measles: two vaccine doses give about 97% protection, for decades (established). +- Influenza: vaccine effectiveness about 40–60% in a typical season, and a new vaccine each year (established range). + +## Alternative states + +| State | What changes | Why | +|---|---|---| +| no vaccine, first infection | primary response, ≈7 days | pathogen grows ≈10^7×; disease likely | +| vaccinated, same antigen | secondary response, ≈3 days, antibodies already present | pathogen grows ≈10^3× at most; often no disease | +| vaccinated, antigen changed (flu drift) | memory receptors bind poorly | response falls back toward primary | +| antigen without adjuvant | weak or no activation | missing danger signal | + +## Why this form + +| Operation | Simplest alternative | What the alternative breaks (with numbers) | +|---|---|---| +| random receptor library | one gene per pathogen receptor | ~20,000 genes vs >100 million receptors needed; genes cannot list unknown future pathogens | +| clonal selection (multiply only the matching clone) | make all lymphocytes multiply | 100 million kinds × growth: the body cannot hold it; and most would attack nothing or self | +| need for danger signal / adjuvant | activate on any binding | would attack harmless food proteins and self; pure protein vaccines often give weak responses | +| vaccine runs the first response | wait for the real infection | real primary response takes ~7 days while pathogen grows 10^7× | + +## Concrete cases + +| Claim | Holds here (numbers) | Breaks here, without its assumption (numbers) | +|---|---|---| +| memory makes the response fast | 10,000 vs 100 starting cells: 6.6 vs 13.3 divisions; ≈3 vs ≈7 days | if the antigen changes (flu drift), memory receptors no longer bind well; effectiveness 40–60% and a yearly new vaccine | +| memory lasts | measles: 2 doses ≈97% protection for decades | influenza: shape change, not memory loss, needs yearly update | +| clonal expansion math | 100 × 2^13.3 ≈ 10^6 | — | + +## Terms + +- pathogen: the thing that causes the disease (virus, bacterium). +- antigen: the molecular shape that a receptor binds (e.g. the SARS-CoV-2 spike protein). +- lymphocyte: B cell or T cell; each carries one kind of receptor. +- receptor: shape-binding protein on a lymphocyte; random; one kind per lymphocyte. +- naive lymphocyte: has not met its antigen. +- clone: a lymphocyte and its descendants, same receptor. +- clonal selection: antigen activates only matching lymphocytes, which multiply. +- effector cell: activated lymphocyte fighting now (plasma cell, killer T cell). +- antibody: a released copy of a B cell receptor. +- memory cell: long-lived lymphocyte that remains after a response. +- danger signal / adjuvant: the "this is a threat" second signal / the vaccine ingredient that provides it. +- primary / secondary response. +- precursor count: number of lymphocytes able to respond at the start (100 naive, illustrative; 10,000 memory). Low → slow response; high → fast. + +## Epistemic status + +- **Observed / established:** clonal selection; V(D)J recombination; memory cells; primary ≈1–2 weeks vs secondary ≈days; measles and flu effectiveness ranges. +- **Mathematical consequence:** division counts from 2^n. +- **Estimated:** library size, precursor frequency, memory-to-naive ratio. +- **Illustrative model:** 100 cells, 10^6 needed, pathogen 10×/day. +- **Simplified:** the danger signal (in reality several signals and T-cell help). + +## Confusion points + +- "The vaccine teaches the body the shape / the body builds an antibody to fit it." → No instruction: the matching receptor already exists by chance; the vaccine selects and multiplies it. +- "A vaccine is a weak infection that the body fights off, and immunity is the antibodies." → Antibodies fade; the lasting part is the larger pool of memory cells (plus long-lived plasma cells). +- "Vaccinated means you cannot be infected." → The pathogen can still enter; memory shortens the race so it is controlled sooner. +- "Flu vaccine fails because memory wears off." → Mainly the antigen changes. + +## Scope + +- **Out of scope:** antibody decay over months; why some vaccines need boosters. +- **Out of scope:** affinity maturation (B-cell receptors improve their fit over weeks) — germinal centres, immunology textbook (Janeway's Immunobiology). +- **Out of scope:** T-cell help, MHC presentation — same textbook. +- **Out of scope:** vaccine types (live attenuated, inactivated, subunit, mRNA) — they differ in how the antigen is delivered, not in the memory mechanism. +- **Out of scope:** herd immunity, boosters' exact schedule, safety. + +## Representation decision + +- **Stage:** 1 +- **Reason:** One causal chain with one before/after comparison; a table holds the two races. No parameter needs exploring beyond the one comparison. +- **Levels:** L1 answer / L2 the steps / L3 why random library, why memory is fast / L4 limits. diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/model.yaml b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/model.yaml new file mode 100644 index 0000000..3ab38ef --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/model.yaml @@ -0,0 +1,111 @@ +# Machine-readable half of model.md. `explainer check vaccine-memory` holds every rendering to it. + +values: + library_size: "100 million" # different receptors in one person (estimate, lower bound) + human_genes: 20000 + naive_precursors: 100 # illustrative precursor count, first meeting + memory_precursors: 10000 # illustrative, 100x naive + memory_ratio: 100 + division_hours: 8 + cells_needed: 1000000 # illustrative effector cells for control + primary_divisions: 13.3 + primary_division_days: 4.4 + activation_days: 2.5 + memory_divisions: 6.6 + memory_division_days: 2.2 + primary_days: 7 + memory_days: 3 + pathogen_start: 100 # particles + pathogen_growth_per_day: 10 + symptom_level: "100 million" # particles, illustrative + primary_peak: "1 billion" + memory_peak: 100000 + precursor_freq_low: 100000 # 1 in 100,000 + precursor_freq_high: 1000000 + contraction_pct: 99 + measles_effectiveness_pct: 97 + flu_effectiveness_low: 40 + flu_effectiveness_high: 60 + +allow: [1, 2, 3, 6, 10, 1000] + +require: + - path: primary_days + in: [prose] + - path: memory_days + in: [prose] + - path: naive_precursors + in: [prose] + - path: memory_precursors + in: [prose] + - path: symptom_level + in: [prose] + +claims: + - id: library-before + kind: why + about: [receptor library] + statement: "The body makes a random library of receptors before any infection, so a matching lymphocyte already exists when a new antigen arrives." + alternative: "One gene for each receptor, written to fit known pathogens." + counterexample: "About 20,000 human genes cannot list more than 100 million receptors, and genes cannot anticipate a pathogen that has not evolved yet." + ask: "Before a person meets a brand-new virus, does their body already have lymphocytes that can bind it? Why?" + - id: clonal-selection + kind: mechanism + statement: "An antigen activates only the few lymphocytes whose receptors bind it; those divide every 8 hours into a clone." + example: "100 matching cells × 2^13.3 ≈ 1,000,000 effector cells after about 4.4 days of division." + - id: primary-slow + kind: mechanism + statement: "The first response is slow because the matching clone starts at about 100 cells; it needs about 7 days to control the pathogen." + example: "100 → 1,000,000 needs 13.3 divisions (4.4 days) plus about 2.5 days to find the antigen and activate: about 7 days. The pathogen grows from 100 to 1 billion particles, above the symptom level of 100 million." + - id: memory-fast + kind: guarantee + statement: "After a response, memory cells remain, 100× more than the naive count; the next response against the same antigen takes about 3 days." + example: "10,000 → 1,000,000 needs 6.6 divisions (2.2 days) plus under a day to activate: about 3 days. The pathogen peaks at 100,000 particles, 1,000× below the symptom level." + counterexample: "Influenza changes its surface antigens each year; old memory receptors bind poorly, so the response falls back toward the slow first response, and flu vaccine effectiveness is about 40–60%." + ask: "Why is the second response to the same antigen faster than the first?" + - id: vaccine-role + kind: mechanism + statement: "A vaccine shows the antigen with a danger signal but without a pathogen that can multiply to disease, so the slow first response runs at no cost and leaves memory cells." + example: "Measles vaccine, two doses: about 97% effectiveness for decades." + ask: "What does a vaccine actually leave behind in the body that protects a person years later?" + - id: why-adjuvant + kind: why + about: [adjuvant] + statement: "A lymphocyte activates only with a danger signal as well as its antigen; a vaccine adjuvant supplies that signal." + alternative: "Activate on any binding." + counterexample: "Lymphocytes would attack harmless food proteins; and a purified protein alone often gives only a weak response." + - id: selection-not-instruction + kind: misconception + statement: "The vaccine teaches the body the germ's shape, and the body builds a receptor to fit it." + correction: "The vaccine does not teach the body a shape; it selects and multiplies lymphocytes whose random receptors already fit." + example: "Of about 100 million receptors, only the about 100 matching cells divide." + - id: fewer-beats-more + kind: mechanism + statement: "The response wins with fewer cells than pathogen particles because each effector cell removes many particles." + example: "One plasma cell releases thousands of antibodies per second." + +accept_unjustified: {} + +terms: + - term: memory cell + means: "a long-lived lymphocyte left after a response; it is not antibody" + avoid: [memory antibodies, antibody memory] + - term: receptor + means: "the binding protein on a lymphocyte; an antibody is a released copy of a B cell receptor" + avoid: [antibody receptor] + - term: precursor count + means: "the number of lymphocytes able to respond at the start: 100 naive, 10,000 memory" + avoid: [head start count, starting cells] + +budget: {prose: 1000, reason: "the answer needs a random library, a race with two sets of numbers, and a case where memory fails"} + +quiz: + - q: "A person gets a vaccine against a virus whose antigen does not change. Two years later the virus enters their body. Roughly how much faster is their immune response than in an unvaccinated person, and why?" + expect: "About 3 days instead of about 7, because memory cells start from about 100× more matching cells (fewer divisions needed) and activate faster; so the virus stays below the symptom level." + misconception: "The antibodies made two years ago are still there and kill the virus instantly; or the vaccine prevents the virus from entering." + - q: "Why do people need a new flu vaccine each year, while a measles vaccine protects for decades?" + expect: "The flu virus changes its surface antigens, so old memory cells' receptors no longer fit well; measles antigens stay the same." + misconception: "Flu vaccine memory wears off quickly." + - q: "How can the immune system have a matching lymphocyte for a virus that appeared only last year?" + expect: "The receptors are made at random, more than 100 million kinds, before any infection; by chance a few fit almost any shape. The virus selects them." + misconception: "The immune system designs a new receptor to fit the virus after seeing it." diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/narrative.md b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/narrative.md new file mode 100644 index 0000000..6c3e59f --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/narrative.md @@ -0,0 +1,56 @@ +# Narrative: vaccine-memory + +## 1. Reader before and after + +- **Before:** A technical reader: school science (cells, DNA, genes, proteins with shapes, exponential growth) and basic programming. Knows "immune system" and "antibody" only as everyday words. Does not know lymphocyte, receptor, antigen, clone, memory cell, adjuvant. +- **After:** "My body already holds, by chance, a few cells that fit almost any germ; a vaccine makes those few multiply once, safely, and leaves many more of them behind, so the real infection is stopped in about 3 days instead of about 7." + +## 2. Question and motive + +- **Question:** How can a vaccine prepare the body for a disease it has never met? Result in words: the vaccine does not teach a shape. The body already has a matching cell by chance. The vaccine multiplies it once, safely, and the extra cells stay. +- **Why care:** It sounds impossible: the body cannot know the shape of a virus that evolved last year. +- **Why this approach:** Treat infection as a race between two exponential growths. Then ask what changes the result of the race. + +## 3. Introduction ledger + +| Reference | Means | Grounded by | Beat | +|---|---|---|---| +| pathogen | virus or bacterium that multiplies and causes disease | 100 particles, ×10 per day | 2 | +| antigen | molecular shape on the pathogen that a receptor binds | the spike protein of SARS-CoV-2 | 3 | +| lymphocyte | white blood cell (B or T cell) carrying one kind of receptor | — one cell, one receptor kind | 3 | +| receptor | binding protein on a lymphocyte, made at random | >100 million kinds in one person | 3 | +| clone | a lymphocyte and its descendants, same receptor | 100 cells → 1,000,000 | 4 | +| clonal selection | antigen makes only matching lymphocytes divide | 100 of 100 million kinds divide | 4 | +| effector cell | activated lymphocyte that fights now | plasma cell, killer T cell | 4 | +| antibody | released copy of a B cell receptor | thousands per second from one plasma cell | 4 | +| precursor count | lymphocytes able to respond at the start | 100 naive vs 10,000 memory | 5 | +| memory cell | long-lived lymphocyte left after a response | 10,000 = 1% of 1,000,000 | 6 | +| danger signal / adjuvant | second signal "this is a threat" / vaccine ingredient giving it | food proteins do not trigger a response | 7 | + +## 4. Beats + +| # | Kind | Reader's question | Bridge | Shown | Said | Reader now knows | +|---|---|---|---|---|---|---| +| 1 | hook | How can it prepare for something unknown? | — | answer in bold | "The vaccine does not teach the body a new shape. The body already has a few cells that fit it, made at random. The vaccine makes those cells multiply once, without the disease, and many of them stay. So the next time, the body controls the germ in about 3 days instead of about 7." | the result, in words | +| 2 | approach | What decides if I get sick? | so | pathogen numbers | "A pathogen (a virus or bacterium) multiplies in the body. Illustrative: it starts at 100 particles and grows 10× per day; symptoms start above 100 million particles. Immune cells multiply too. Whichever side gets big enough first wins: infection is a race between the pathogen's growth and the immune cells' growth." | the race and both runners | +| 3 | mechanism | Which immune cells, and how can one fit an unknown germ? | but | library numbers | "A lymphocyte is a white blood cell with one kind of receptor: a protein that binds one family of similar molecular shapes. The shape it binds on a pathogen is an antigen (e.g. the spike protein of a coronavirus). Each new lymphocyte assembles its receptor by joining gene pieces at random, so one person has more than 100 million kinds. The ~20,000 genes could not list them one by one. Because each receptor fits a family of shapes, and a pathogen shows many antigens, a few lymphocytes fit almost any new pathogen by chance." | lymphocyte, receptor, antigen; the library exists in advance | +| 4 | mechanism | What happens when the antigen arrives? | so | clone arithmetic | "Only the lymphocytes whose receptors bind the antigen activate (clonal selection). For one antigen, about 1 in 100,000 to 1 in 1,000,000 lymphocytes fit: illustrative, 100 cells. Each divides every 8 hours. A clone is a cell and all its descendants, with the same receptor. The clone's cells become effector cells: plasma cells release antibodies (free copies of the receptor), killer T cells kill infected cells. Each effector cell removes many particles, so about 1,000,000 effector cells (illustrative) clear the pathogen faster than it grows." | clonal selection, clone, effector cell, antibody, target 1,000,000 | +| 5 | mechanism | Why is the first time slow? | but | 100 × 2^n = 1,000,000; 7 days; 1 billion particles | "Here n is the number of divisions: 100 × 2^n = 1,000,000 gives n ≈ 13.3, which is 4.4 days at 8 hours each. Finding the antigen and activating take about 2.5 days more: about 7 days. By then the pathogen is at 100 × 10^7 = 1 billion particles, above the 100-million symptom level: the person is sick." | primary response | +| 6 | mechanism | What is left after? | therefore | 99% die, 1% stay: 10,000 | "After the pathogen is gone, about 99% of the effector cells die. About 1% stay for years as memory cells: 10,000, which is 100× the original 100. Next time, 10,000 × 2^n = 1,000,000 gives n ≈ 6.6 divisions, 2.2 days; memory cells also activate in under a day: about 3 days. The pathogen reaches only 100,000 particles, 1,000× below the symptom level." | memory cell, secondary response | +| 7 | name | So what does the vaccine do? | therefore | vaccine = antigen + adjuvant, no disease | "A vaccine contains the antigen, or instructions to make it, but no pathogen that can multiply to disease. It also needs a danger signal: lymphocytes activate only if other immune cells also signal 'threat'; this stops attacks on harmless food proteins. The vaccine ingredient that gives this signal is an adjuvant. So the vaccine runs the slow first response (beats 4–6) safely, and leaves the memory cells." | the answer | +| 8 | objection | When does it fail? | but | flu vs measles | "Memory works only for the same antigen. Influenza changes its surface antigens each year, so old receptors fit poorly and the response falls back toward the slow first one; flu vaccines prevent about 40–60% of cases and are updated yearly. Measles antigens stay the same: two doses prevent about 97% of cases for decades." | limit | +| 9 | close | — | so | recap | answer in the opening words; each 2× more precursors saves one 8-hour division | after-sentence | + +## 5. Concrete to symbol + +100 × 2^n = 1,000,000 → n ≈ 13.3. Said: "here n is the number of divisions". 10,000 × 2^n = 1,000,000 → n ≈ 6.6. + +## 6. Links between representations + +One table: first meeting vs vaccinated, same rows (precursor count, divisions, days, pathogen peak, sick?). + +## 7. Close the loop + +- **Answer:** A vaccine prepares the body by running the slow first response safely, so the cells that fit are already many when the disease arrives. +- **General argument:** Memory cuts the precursor gap; each 2× more precursors saves one 8-hour division. +- **Payoff:** Predict when a vaccine will fail (antigen change) and why boosters or yearly updates exist. Scope pointers. diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/decisions.yaml b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/decisions.yaml new file mode 100644 index 0000000..32a2751 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/decisions.yaml @@ -0,0 +1,193 @@ +# What the author did with each review finding (explainer decide). Read by findings --stats. +2026-10-04T145746-probe#1: + decision: adopted +2026-10-04T145746-probe#10: + decision: adopted +2026-10-04T145746-probe#11: + decision: adopted +2026-10-04T145746-probe#12: + decision: adopted +2026-10-04T145746-probe#13: + decision: adopted +2026-10-04T145746-probe#14: + decision: adopted +2026-10-04T145746-probe#15: + decision: declined +2026-10-04T145746-probe#16: + decision: declined +2026-10-04T145746-probe#2: + decision: adopted +2026-10-04T145746-probe#3: + decision: adopted +2026-10-04T145746-probe#4: + decision: adopted +2026-10-04T145746-probe#5: + decision: declined +2026-10-04T145746-probe#6: + decision: adopted +2026-10-04T145746-probe#7: + decision: adopted +2026-10-04T145746-probe#8: + decision: declined +2026-10-04T145746-probe#9: + decision: adopted +2026-10-04T150002-coldread-narrative#1: + decision: adopted +2026-10-04T150002-coldread-narrative#10: + decision: declined +2026-10-04T150002-coldread-narrative#11: + decision: declined +2026-10-04T150002-coldread-narrative#12: + decision: declined +2026-10-04T150002-coldread-narrative#13: + decision: adopted +2026-10-04T150002-coldread-narrative#14: + decision: adopted +2026-10-04T150002-coldread-narrative#15: + decision: declined +2026-10-04T150002-coldread-narrative#16: + decision: declined +2026-10-04T150002-coldread-narrative#17: + decision: adopted +2026-10-04T150002-coldread-narrative#18: + decision: adopted +2026-10-04T150002-coldread-narrative#19: + decision: declined +2026-10-04T150002-coldread-narrative#2: + decision: adopted +2026-10-04T150002-coldread-narrative#20: + decision: declined +2026-10-04T150002-coldread-narrative#21: + decision: declined +2026-10-04T150002-coldread-narrative#22: + decision: declined +2026-10-04T150002-coldread-narrative#23: + decision: declined +2026-10-04T150002-coldread-narrative#24: + decision: adopted +2026-10-04T150002-coldread-narrative#25: + decision: adopted +2026-10-04T150002-coldread-narrative#26: + decision: adopted +2026-10-04T150002-coldread-narrative#27: + decision: declined +2026-10-04T150002-coldread-narrative#28: + decision: declined +2026-10-04T150002-coldread-narrative#29: + decision: declined +2026-10-04T150002-coldread-narrative#3: + decision: adopted +2026-10-04T150002-coldread-narrative#30: + decision: adopted +2026-10-04T150002-coldread-narrative#31: + decision: adopted +2026-10-04T150002-coldread-narrative#32: + decision: adopted +2026-10-04T150002-coldread-narrative#33: + decision: adopted +2026-10-04T150002-coldread-narrative#34: + decision: adopted +2026-10-04T150002-coldread-narrative#35: + decision: adopted +2026-10-04T150002-coldread-narrative#36: + decision: adopted +2026-10-04T150002-coldread-narrative#37: + decision: adopted +2026-10-04T150002-coldread-narrative#38: + decision: declined +2026-10-04T150002-coldread-narrative#39: + decision: adopted +2026-10-04T150002-coldread-narrative#4: + decision: adopted +2026-10-04T150002-coldread-narrative#40: + decision: declined +2026-10-04T150002-coldread-narrative#41: + decision: declined +2026-10-04T150002-coldread-narrative#42: + decision: adopted +2026-10-04T150002-coldread-narrative#43: + decision: adopted +2026-10-04T150002-coldread-narrative#44: + decision: declined +2026-10-04T150002-coldread-narrative#45: + decision: declined +2026-10-04T150002-coldread-narrative#46: + decision: adopted +2026-10-04T150002-coldread-narrative#47: + decision: adopted +2026-10-04T150002-coldread-narrative#48: + decision: adopted +2026-10-04T150002-coldread-narrative#5: + decision: adopted +2026-10-04T150002-coldread-narrative#6: + decision: adopted +2026-10-04T150002-coldread-narrative#7: + decision: adopted +2026-10-04T150002-coldread-narrative#8: + decision: adopted +2026-10-04T150002-coldread-narrative#9: + decision: declined +2026-10-04T150207-coldread-prose#1: + decision: adopted +2026-10-04T150207-coldread-prose#10: + decision: declined +2026-10-04T150207-coldread-prose#2: + decision: adopted +2026-10-04T150207-coldread-prose#3: + decision: adopted +2026-10-04T150207-coldread-prose#4: + decision: adopted +2026-10-04T150207-coldread-prose#5: + decision: adopted +2026-10-04T150207-coldread-prose#6: + decision: adopted +2026-10-04T150207-coldread-prose#7: + decision: adopted +2026-10-04T150207-coldread-prose#8: + decision: declined +2026-10-04T150207-coldread-prose#9: + decision: adopted +2026-10-04T150328-quiz-prose#1: + decision: adopted +2026-10-04T150328-quiz-prose#10: + decision: declined +2026-10-04T150328-quiz-prose#11: + decision: declined +2026-10-04T150328-quiz-prose#12: + decision: declined +2026-10-04T150328-quiz-prose#13: + decision: declined +2026-10-04T150328-quiz-prose#14: + decision: adopted +2026-10-04T150328-quiz-prose#15: + decision: declined +2026-10-04T150328-quiz-prose#16: + decision: adopted +2026-10-04T150328-quiz-prose#17: + decision: declined +2026-10-04T150328-quiz-prose#18: + decision: declined +2026-10-04T150328-quiz-prose#19: + decision: declined +2026-10-04T150328-quiz-prose#2: + decision: adopted +2026-10-04T150328-quiz-prose#20: + decision: declined +2026-10-04T150328-quiz-prose#21: + decision: adopted +2026-10-04T150328-quiz-prose#22: + decision: declined +2026-10-04T150328-quiz-prose#3: + decision: adopted +2026-10-04T150328-quiz-prose#4: + decision: adopted +2026-10-04T150328-quiz-prose#5: + decision: declined +2026-10-04T150328-quiz-prose#6: + decision: declined +2026-10-04T150328-quiz-prose#7: + decision: declined +2026-10-04T150328-quiz-prose#8: + decision: adopted +2026-10-04T150328-quiz-prose#9: + decision: declined diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T145746-probe.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T145746-probe.json new file mode 100644 index 0000000..43d8b4b --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T145746-probe.json @@ -0,0 +1,231 @@ +{ + "tool": "probe", + "rendering": null, + "slug": "vaccine-memory", + "at": "2026-10-04T14:57:46", + "model": "default", + "prompt_sha": "74ac0a8c8132", + "reply": { + "gaps": [ + { + "rule": 2, + "severity": "main", + "about": "The race argument: the clone grows more slowly than the pathogen", + "why_it_matters": "In the file's own numbers the clone triples its doublings each day (2^3 = 8× per day) while the pathogen grows 10× per day. With these numbers the clone never catches up. The file never says why the defence wins, so the core 'race' story doesn't hold together as written.", + "suggested_addition": "Say that one effector cell or antibody removes many pathogen units, so the relevant comparison is killing capacity, not cell count. Illustrative example: if each of the 10^6 effector cells clears 100 virus particles per day, they clear 10^8 per day. A pathogen load of 10^7 then shrinks. Also give a stopping threshold: 'control' means clearance per day exceeds growth per day." + }, + { + "rule": 5, + "severity": "main", + "about": "Pathogen growth of 10^7× vs 10^3× has no disease threshold or starting dose", + "why_it_matters": "The reader can't tell what 10^3× means for symptoms. Without a starting dose and a threshold, 'often no disease' is just asserted.", + "suggested_addition": "Illustrative: the infection starts with 100 particles and symptoms begin around 10^8. Primary response: 100 × 10^7 = 10^9, which is above the threshold, so the person gets sick. Secondary response: 100 × 10^3 = 10^5, which is 1,000× below the threshold, so infection happens with no disease." + }, + { + "rule": 2, + "severity": "main", + "about": "The claim that 'the matching receptor already exists by chance' has no coverage argument", + "why_it_matters": "A learner will ask how 10^8 random receptors can cover every possible antigen shape. The missing step is that each receptor binds many similar shapes (cross-reactivity) and each pathogen carries many antigen shapes. There is also no case where the claim fails.", + "suggested_addition": "Example: if one receptor binds about 1 in 10^5 random shapes, then a library of 10^8 receptors has about 1,000 binders for any shape. A pathogen with 10 separate antigen spots is almost certain to be hit. Breaking case: a small library, such as 10^4 receptors (low diversity in the very old or very young), gives about 0.1 expected binders per shape, so responses can be missing." + }, + { + "rule": 2, + "severity": "main", + "about": "The contraction numbers contradict the memory count", + "why_it_matters": "If 90–95% of 10^6 effector cells die, 50,000–100,000 cells are left, not 10,000. The ratio would then be 500–1,000× the naive count, which is outside the stated 10–100× estimate.", + "suggested_addition": "Either say that only about 1% become memory (10^6 → 10^4), or rework the example: 5% of 10^6 = 50,000. Also show the low end of the estimate: at 10× the naive count (1,000 cells), 2^n = 1,000 gives n ≈ 10 divisions ≈ 3.3 days, plus 0.75 day, ≈ 4 days." + }, + { + "rule": 1, + "severity": "edge", + "about": "'Keep all effector cells forever' fails on the file's own numbers", + "why_it_matters": "10^6 cells × 1,000 infections = 10^9 cells, which is about 0.1% of roughly 10^12 lymphocytes. So the stated 'space runs out' argument doesn't work at these numbers.", + "suggested_addition": "Use real expansion sizes: one response can make 10^7–10^9 cells, up to about 10% of all killer T cells. Add that effector cells are short-lived and cause inflammation. Example: 100 infections × 10% each would exceed the whole pool." + }, + { + "rule": 1, + "severity": "edge", + "about": "Why memory cells activate faster (0.5–1 day vs 2.5 days)", + "why_it_matters": "The ≈3-day result depends on this unexplained shortcut as well as on the head start.", + "suggested_addition": "Say that memory cells already sit in tissues and lymph nodes and need less stimulation to activate. Show the time cost of not having this: 2.2 days of division + 2.5 days of activation = 4.7 days instead of about 3." + }, + { + "rule": 5, + "severity": "edge", + "about": "Measles '97% protection' vs influenza '40–60% effectiveness' are undefined and use two different names", + "why_it_matters": "The reader doesn't know what the percentage measures, so they can't compare the two vaccines.", + "suggested_addition": "Define it once as the reduction in risk of disease, vaccinated vs unvaccinated. Example: if 90 of 100 unvaccinated exposed people get measles, 97% means about 2.7 of 100 vaccinated people do. For flu at 50%: if 10 of 100 unvaccinated people get flu, 5 of 100 vaccinated people do. Use one term throughout." + }, + { + "rule": 5, + "severity": "edge", + "about": "Pre-existing antibodies in the secondary response are not quantified", + "why_it_matters": "The 'Alternative states' table says antibodies are already present, but the model gives no number for their effect and never says how fast they fade.", + "suggested_addition": "Example: antibody levels halve about every 3 weeks (illustrative), so after 1 year about 2^-17 of the peak remains. Long-lived plasma cells keep a low baseline that neutralises part of the starting dose, e.g. 100 → 10 particles." + }, + { + "rule": 3, + "severity": "edge", + "about": "Terms used without definition: long-lived plasma cell, heavy/light chain, V/D/J, affinity maturation, precursor", + "why_it_matters": "These appear in the causal chain, Quantities and Relationships before they are defined (or are never defined). 'Naive precursors' appears in Relationships, but 'precursor count' is defined only in the Terms section.", + "suggested_addition": "Add entity rows: long-lived plasma cell (keeps releasing antibodies for years); heavy and light chain (the two parts of a receptor, each assembled from gene segments); precursor (a lymphocyte able to respond at the start: 100 naive, 10,000 memory)." + }, + { + "rule": 3, + "severity": "edge", + "about": "Concepts with more than one name", + "why_it_matters": "Several names for one concept make readers think there are several concepts.", + "suggested_addition": "Use one name for each: 'danger signal' (not also 'second signal' or 'signals'); 'precursor count' (not also 'starting cells' or 'head-start'); 'receptor fit' (not also 'affinity'); 'protection' (not also 'effectiveness')." + }, + { + "rule": 4, + "severity": "edge", + "about": "The clonal selection step has no worked example with numbers", + "why_it_matters": "The step that answers the central question gets no arithmetic.", + "suggested_addition": "Example: a lymph node scans about 10^7 lymphocytes. At a frequency of 1 in 10^5, about 100 bind the vaccine antigen. Only those 100 divide; the other 99.999% stay idle." + }, + { + "rule": 2, + "severity": "edge", + "about": "Self-reactivity of a random library", + "why_it_matters": "A random library must also make receptors that bind the body's own proteins. Only the danger signal is offered as protection, and the removal of self-binding cells during development is never mentioned.", + "suggested_addition": "Add that most newly made lymphocytes that bind self proteins are removed before release. For T cells, roughly 90%+ of developing cells die in the thymus (estimate)." + } + ], + "excess": [ + "Detailed gene-segment arithmetic (40 V × 23 D × 6 J, light chains ≈320, pairs ≈1.8 million): the comparison of ~20,000 genes vs >10^8 receptors is enough", + "The affinity maturation relationship and the 'B-cell receptors improve fit' step: the Scope section declares this out of scope and it is not needed for the central answer", + "Killer T cells as an effector type: the model's arithmetic follows only one generic clone", + "The Representation decision section, which is plan metadata, not reader content" + ], + "next_questions": [ + { + "q": "How can random receptors already match a pathogen nobody has seen, and why don't they attack my own body?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "If the clone grows 8× per day and the pathogen 10× per day, how does the immune system ever win?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "Why does measles immunity last decades while some vaccines (tetanus, pertussis) need boosters?", + "answered_by_model": false, + "declared_out_of_scope": true + }, + { + "q": "Why does the flu antigen change while measles doesn't?", + "answered_by_model": false, + "declared_out_of_scope": false + }, + { + "q": "How does an mRNA vaccine show the antigen without the pathogen?", + "answered_by_model": false, + "declared_out_of_scope": true + } + ] + }, + "usage": { + "cost": 0.1667154, + "seconds": 62 + }, + "findings": [ + { + "n": 1, + "severity": "main", + "kind": "rule 2", + "text": "The race argument: the clone grows more slowly than the pathogen → Say that one effector cell or antibody removes many pathogen units, so the relevant comparison is killing capacity, not cell count. Illustrative example: if each of the 10^6 effector cells clears 100 virus particles per day, they clear 10^8 per day. A pathogen load of 10^7 then shrinks. Also give a stopping threshold: 'control' means clearance per day exceeds growth per day." + }, + { + "n": 2, + "severity": "main", + "kind": "rule 5", + "text": "Pathogen growth of 10^7× vs 10^3× has no disease threshold or starting dose → Illustrative: the infection starts with 100 particles and symptoms begin around 10^8. Primary response: 100 × 10^7 = 10^9, which is above the threshold, so the person gets sick. Secondary response: 100 × 10^3 = 10^5, which is 1,000× below the threshold, so infection happens with no disease." + }, + { + "n": 3, + "severity": "main", + "kind": "rule 2", + "text": "The claim that 'the matching receptor already exists by chance' has no coverage argument → Example: if one receptor binds about 1 in 10^5 random shapes, then a library of 10^8 receptors has about 1,000 binders for any shape. A pathogen with 10 separate antigen spots is almost certain to be hit. Breaking case: a small library, such as 10^4 receptors (low diversity in the very old or very young), gives about 0.1 expected binders per shape, so responses can be missing." + }, + { + "n": 4, + "severity": "main", + "kind": "rule 2", + "text": "The contraction numbers contradict the memory count → Either say that only about 1% become memory (10^6 → 10^4), or rework the example: 5% of 10^6 = 50,000. Also show the low end of the estimate: at 10× the naive count (1,000 cells), 2^n = 1,000 gives n ≈ 10 divisions ≈ 3.3 days, plus 0.75 day, ≈ 4 days." + }, + { + "n": 5, + "severity": "edge", + "kind": "rule 1", + "text": "'Keep all effector cells forever' fails on the file's own numbers → Use real expansion sizes: one response can make 10^7–10^9 cells, up to about 10% of all killer T cells. Add that effector cells are short-lived and cause inflammation. Example: 100 infections × 10% each would exceed the whole pool." + }, + { + "n": 6, + "severity": "edge", + "kind": "rule 1", + "text": "Why memory cells activate faster (0.5–1 day vs 2.5 days) → Say that memory cells already sit in tissues and lymph nodes and need less stimulation to activate. Show the time cost of not having this: 2.2 days of division + 2.5 days of activation = 4.7 days instead of about 3." + }, + { + "n": 7, + "severity": "edge", + "kind": "rule 5", + "text": "Measles '97% protection' vs influenza '40–60% effectiveness' are undefined and use two different names → Define it once as the reduction in risk of disease, vaccinated vs unvaccinated. Example: if 90 of 100 unvaccinated exposed people get measles, 97% means about 2.7 of 100 vaccinated people do. For flu at 50%: if 10 of 100 unvaccinated people get flu, 5 of 100 vaccinated people do. Use one term throughout." + }, + { + "n": 8, + "severity": "edge", + "kind": "rule 5", + "text": "Pre-existing antibodies in the secondary response are not quantified → Example: antibody levels halve about every 3 weeks (illustrative), so after 1 year about 2^-17 of the peak remains. Long-lived plasma cells keep a low baseline that neutralises part of the starting dose, e.g. 100 → 10 particles." + }, + { + "n": 9, + "severity": "edge", + "kind": "rule 3", + "text": "Terms used without definition: long-lived plasma cell, heavy/light chain, V/D/J, affinity maturation, precursor → Add entity rows: long-lived plasma cell (keeps releasing antibodies for years); heavy and light chain (the two parts of a receptor, each assembled from gene segments); precursor (a lymphocyte able to respond at the start: 100 naive, 10,000 memory)." + }, + { + "n": 10, + "severity": "edge", + "kind": "rule 3", + "text": "Concepts with more than one name → Use one name for each: 'danger signal' (not also 'second signal' or 'signals'); 'precursor count' (not also 'starting cells' or 'head-start'); 'receptor fit' (not also 'affinity'); 'protection' (not also 'effectiveness')." + }, + { + "n": 11, + "severity": "edge", + "kind": "rule 4", + "text": "The clonal selection step has no worked example with numbers → Example: a lymph node scans about 10^7 lymphocytes. At a frequency of 1 in 10^5, about 100 bind the vaccine antigen. Only those 100 divide; the other 99.999% stay idle." + }, + { + "n": 12, + "severity": "edge", + "kind": "rule 2", + "text": "Self-reactivity of a random library → Add that most newly made lymphocytes that bind self proteins are removed before release. For T cells, roughly 90%+ of developing cells die in the thymus (estimate)." + }, + { + "n": 13, + "severity": "excess", + "kind": "model entry", + "text": "Detailed gene-segment arithmetic (40 V × 23 D × 6 J, light chains ≈320, pairs ≈1.8 million): the comparison of ~20,000 genes vs >10^8 receptors is enough" + }, + { + "n": 14, + "severity": "excess", + "kind": "model entry", + "text": "The affinity maturation relationship and the 'B-cell receptors improve fit' step: the Scope section declares this out of scope and it is not needed for the central answer" + }, + { + "n": 15, + "severity": "excess", + "kind": "model entry", + "text": "Killer T cells as an effector type: the model's arithmetic follows only one generic clone" + }, + { + "n": 16, + "severity": "excess", + "kind": "model entry", + "text": "The Representation decision section, which is plan metadata, not reader content" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150002-coldread-narrative.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150002-coldread-narrative.json new file mode 100644 index 0000000..5be48ad --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150002-coldread-narrative.json @@ -0,0 +1,486 @@ +{ + "tool": "coldread", + "rendering": "narrative", + "slug": "vaccine-memory", + "at": "2026-10-04T15:00:02", + "model": "default", + "prompt_sha": "7327675ff3e8", + "reply": { + "beats": [ + { + "at": "Beat 1 (hook)", + "text": "Shown: answer in 3 sentences. Said: selection, not instruction; memory cells; 7 → 3 days", + "unresolved": [ + "selection, not instruction", + "memory cells", + "7 → 3 days" + ], + "unsaid": [ + "'answer in 3 sentences' is shown, but the Said column gives only three fragments. The reader never hears the answer as sentences." + ], + "leap": [ + "It is not said what '7 → 3 days' measures (days until what?), or what is being selected." + ], + "excess": [], + "purpose_unclear": true + }, + { + "at": "Beat 2 (approach)", + "text": "Shown: pathogen numbers. Said: infection is a race; 100 particles ×10/day; symptoms above 100 million", + "unresolved": [ + "pathogen", + "particles" + ], + "unsaid": [ + "The 'pathogen numbers' are shown, but the words never say what the pathogen is (a virus or bacterium)." + ], + "leap": [ + "'infection is a race': only one runner is named. The reader is not told what the germ is racing against.", + "No reason is given for treating infection as a race." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 3 (mechanism)", + "text": "Shown: library numbers. Said: random receptors, >100 million kinds; 20,000 genes can't list them; loose fit + many antigens; self-binders removed", + "unresolved": [ + "receptors", + "antigens", + "self-binders", + "library" + ], + "unsaid": [ + "The 'library numbers' are shown, but the words never call it a library or say what it holds. The cell that carries the receptor (lymphocyte) is never named or described." + ], + "leap": [ + "'20,000 genes can't list them': it is not said how random receptors are made from a limited number of genes.", + "'loose fit + many antigens': it is not said why this means some cell will fit." + ], + "excess": [ + "'self-binders removed' is not needed for the main argument at this point." + ], + "purpose_unclear": false + }, + { + "at": "Beat 4 (mechanism)", + "text": "Shown: clone arithmetic. Said: only matching ~100 cells divide every 8 h; effector cells; why fewer can beat more", + "unresolved": [ + "clone", + "effector cells", + "the antigen" + ], + "unsaid": [ + "The 'clone arithmetic' is shown, but the words do not say what a clone is or give the 1,000,000 target." + ], + "leap": [ + "Where '~100 cells' comes from, after '>100 million kinds'.", + "'why fewer can beat more': the reason is named but not given." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 5 (mechanism)", + "text": "Shown: 13.3 divisions; 7 days; 1 billion particles. Said: the clone starts tiny", + "unresolved": [], + "unsaid": [ + "'13.3 divisions' appears with no equation and no target count in the words.", + "'1 billion particles' is shown, but the words do not compare it with the 100 million symptom threshold (so: sick)." + ], + "leap": [ + "13.3 divisions × 8 h is about 4.4 days, not '7 days'. The difference is not explained." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 6 (mechanism)", + "text": "Shown: 99% die, 1% stay: 10,000. Said: memory cells; 6.6 divisions; 3 days; 100,000 particles", + "unresolved": [], + "unsaid": [ + "'99% die, 1% stay' is shown, but the words do not say it.", + "The words never say that 100,000 particles is below the symptom threshold (so: not sick)." + ], + "leap": [ + "Why 1% stay.", + "6.6 × 8 h is about 2.2 days, not '3 days'." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 7 (name)", + "text": "Shown: vaccine = antigen + adjuvant, no disease. Said: runs beat 4–6 safely; why adjuvant", + "unresolved": [ + "adjuvant", + "beat 4–6" + ], + "unsaid": [ + "The 'adjuvant' is shown, but the words say only 'why adjuvant' and never explain it (no danger signal is given)." + ], + "leap": [ + "It is not said why antigen alone causes no disease." + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Beat 8 (objection)", + "text": "Shown: flu vs measles. Said: antigen change; effectiveness defined", + "unresolved": [ + "effectiveness" + ], + "unsaid": [ + "The flu vs measles comparison is shown, but no numbers or definition of effectiveness are given in the words." + ], + "leap": [ + "It is not said why a change in antigen breaks memory (no new matching cells)." + ], + "excess": [ + "The 'effectiveness defined' detail is not needed for the main argument." + ], + "purpose_unclear": false + }, + { + "at": "Beat 9 (close)", + "text": "Shown: recap. Said: answer in the opening words; cache analogy and where it breaks", + "unresolved": [ + "cache analogy" + ], + "unsaid": [], + "leap": [ + "A new analogy appears at the very end, and its content and where it breaks are not given." + ], + "excess": [ + "'answer in the opening words' repeats Beat 1." + ], + "purpose_unclear": false + } + ], + "blocking": [ + "Beat 2: 'infection is a race': the opposing runner (the immune cells) is never named.", + "Beat 3: 'receptors' and 'antigens' are used undefined, and the lymphocyte is never introduced.", + "Beat 4: 'clone' and '~100 cells' are unexplained.", + "Beat 5: '13.3 divisions' has no stated target or equation, and it does not match '7 days'.", + "Beat 7: 'adjuvant' is never explained." + ], + "cuts": [ + "Beat 3: 'self-binders removed'", + "Beat 9: 'cache analogy and where it breaks'", + "Beat 8: 'effectiveness defined'", + "Beat 3: '20,000 genes can't list them' (or explain it)" + ], + "question": { + "text": "How can a vaccine prepare the body for a germ it has never met?", + "known_at": "Never stated in Said/Shown. Beat 1 only implies it through the answer.", + "result_stated": "Only in compressed form, Beat 1: 'selection, not instruction; memory cells; 7 → 3 days'" + }, + "motive": "Missing from the beats. 'sounds impossible' appears only in the plan. A reason for the approach is also missing: Beat 2 asserts 'infection is a race' with no reason.", + "close": "Beat 9: 'answer in the opening words'. The answer is not quoted, and the general argument (each doubling of starting cells saves one 8 h division) is not stated in the beats.", + "recap": "Your body already has a few cells that fit a germ by chance; a vaccine multiplies them safely once, so next time there are many more and the race is won in about 3 days instead of about 7.", + "strongest": "Beats 5–6: the same arithmetic, 100 vs 10,000 starting cells, gives 13.3 vs 6.6 divisions and 1 billion vs 100,000 particles.", + "weakest": "Beat 3: four undefined terms ('receptors', 'antigens', 'self-binders', 'library') arrive at once, and the cell carrying the receptor is never named.", + "suggestions": [ + "Name the second runner in Beat 2: immune cells that must multiply to fight back.", + "Define lymphocyte, receptor and antigen in plain words in Beat 3.", + "Say the 1,000,000 target and the 100×2^n equation aloud, and explain why 4.4 days becomes 7 (start-up delay).", + "Explain the adjuvant as the 'this is a threat' signal.", + "State the question and why it seems impossible at the start of Beat 1." + ] + }, + "usage": { + "cost": 0.11253679999999999, + "seconds": 36 + }, + "findings": [ + { + "n": 1, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 2: 'infection is a race': the opposing runner (the immune cells) is never named." + }, + { + "n": 2, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 3: 'receptors' and 'antigens' are used undefined, and the lymphocyte is never introduced." + }, + { + "n": 3, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 4: 'clone' and '~100 cells' are unexplained." + }, + { + "n": 4, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 5: '13.3 divisions' has no stated target or equation, and it does not match '7 days'." + }, + { + "n": 5, + "severity": "blocking", + "kind": "blocking", + "text": "Beat 7: 'adjuvant' is never explained." + }, + { + "n": 6, + "severity": "excess", + "kind": "cut", + "text": "Beat 3: 'self-binders removed'" + }, + { + "n": 7, + "severity": "excess", + "kind": "cut", + "text": "Beat 9: 'cache analogy and where it breaks'" + }, + { + "n": 8, + "severity": "excess", + "kind": "cut", + "text": "Beat 8: 'effectiveness defined'" + }, + { + "n": 9, + "severity": "excess", + "kind": "cut", + "text": "Beat 3: '20,000 genes can't list them' (or explain it)" + }, + { + "n": 10, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): selection, not instruction" + }, + { + "n": 11, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): memory cells" + }, + { + "n": 12, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 1 (hook): 7 → 3 days" + }, + { + "n": 13, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 1 (hook): 'answer in 3 sentences' is shown, but the Said column gives only three fragments. The reader never hears the answer as sentences." + }, + { + "n": 14, + "severity": "edge", + "kind": "leap", + "text": "Beat 1 (hook): It is not said what '7 → 3 days' measures (days until what?), or what is being selected." + }, + { + "n": 15, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (approach): pathogen" + }, + { + "n": 16, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 2 (approach): particles" + }, + { + "n": 17, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 2 (approach): The 'pathogen numbers' are shown, but the words never say what the pathogen is (a virus or bacterium)." + }, + { + "n": 18, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (approach): 'infection is a race': only one runner is named. The reader is not told what the germ is racing against." + }, + { + "n": 19, + "severity": "edge", + "kind": "leap", + "text": "Beat 2 (approach): No reason is given for treating infection as a race." + }, + { + "n": 20, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): receptors" + }, + { + "n": 21, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): antigens" + }, + { + "n": 22, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): self-binders" + }, + { + "n": 23, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 3 (mechanism): library" + }, + { + "n": 24, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 3 (mechanism): The 'library numbers' are shown, but the words never call it a library or say what it holds. The cell that carries the receptor (lymphocyte) is never named or described." + }, + { + "n": 25, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): '20,000 genes can't list them': it is not said how random receptors are made from a limited number of genes." + }, + { + "n": 26, + "severity": "edge", + "kind": "leap", + "text": "Beat 3 (mechanism): 'loose fit + many antigens': it is not said why this means some cell will fit." + }, + { + "n": 27, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): clone" + }, + { + "n": 28, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): effector cells" + }, + { + "n": 29, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 4 (mechanism): the antigen" + }, + { + "n": 30, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 4 (mechanism): The 'clone arithmetic' is shown, but the words do not say what a clone is or give the 1,000,000 target." + }, + { + "n": 31, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): Where '~100 cells' comes from, after '>100 million kinds'." + }, + { + "n": 32, + "severity": "edge", + "kind": "leap", + "text": "Beat 4 (mechanism): 'why fewer can beat more': the reason is named but not given." + }, + { + "n": 33, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): '13.3 divisions' appears with no equation and no target count in the words." + }, + { + "n": 34, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 5 (mechanism): '1 billion particles' is shown, but the words do not compare it with the 100 million symptom threshold (so: sick)." + }, + { + "n": 35, + "severity": "edge", + "kind": "leap", + "text": "Beat 5 (mechanism): 13.3 divisions × 8 h is about 4.4 days, not '7 days'. The difference is not explained." + }, + { + "n": 36, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): '99% die, 1% stay' is shown, but the words do not say it." + }, + { + "n": 37, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 6 (mechanism): The words never say that 100,000 particles is below the symptom threshold (so: not sick)." + }, + { + "n": 38, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): Why 1% stay." + }, + { + "n": 39, + "severity": "edge", + "kind": "leap", + "text": "Beat 6 (mechanism): 6.6 × 8 h is about 2.2 days, not '3 days'." + }, + { + "n": 40, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (name): adjuvant" + }, + { + "n": 41, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 7 (name): beat 4–6" + }, + { + "n": 42, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 7 (name): The 'adjuvant' is shown, but the words say only 'why adjuvant' and never explain it (no danger signal is given)." + }, + { + "n": 43, + "severity": "edge", + "kind": "leap", + "text": "Beat 7 (name): It is not said why antigen alone causes no disease." + }, + { + "n": 44, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 8 (objection): effectiveness" + }, + { + "n": 45, + "severity": "edge", + "kind": "unsaid", + "text": "Beat 8 (objection): The flu vs measles comparison is shown, but no numbers or definition of effectiveness are given in the words." + }, + { + "n": 46, + "severity": "edge", + "kind": "leap", + "text": "Beat 8 (objection): It is not said why a change in antigen breaks memory (no new matching cells)." + }, + { + "n": 47, + "severity": "edge", + "kind": "unresolved", + "text": "Beat 9 (close): cache analogy" + }, + { + "n": 48, + "severity": "edge", + "kind": "leap", + "text": "Beat 9 (close): A new analogy appears at the very end, and its content and where it breaks are not given." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150207-coldread-prose.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150207-coldread-prose.json new file mode 100644 index 0000000..97d3374 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150207-coldread-prose.json @@ -0,0 +1,294 @@ +{ + "tool": "coldread", + "rendering": "prose", + "slug": "vaccine-memory", + "at": "2026-10-04T15:02:07", + "model": "default", + "prompt_sha": "6b72086ee875", + "reply": { + "paragraphs": [ + { + "at": "Title + bold opening (L1, L5)", + "text": "A vaccine does not teach the body a new shape... controls it in about 3 days instead of about 7.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L7", + "text": "Numbers marked illustrative are round numbers...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Infection is a race, L11", + "text": "A pathogen is a virus or bacterium... symptoms start above 100 million particles.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L13", + "text": "Immune cells also multiply. Whichever side gets big enough first wins...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The matching cells exist before the germ does, L18", + "text": "A lymphocyte is a white blood cell... spike protein on the surface of a coronavirus.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L20", + "text": "Each new lymphocyte builds its receptor by joining pieces of genes...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L22", + "text": "Because each receptor fits a family of shapes... mostly removed while they develop.", + "unresolved": [], + "leap": [], + "excess": [ + "Lymphocytes whose receptors bind the body's own molecules are mostly removed while they develop." + ], + "purpose_unclear": false + }, + { + "at": "L25", + "text": "This is the central idea... The germ selects from a random library that already exists.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "The first meeting is slow, L30", + "text": "When an antigen arrives, only the lymphocytes whose receptors bind it activate and divide...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L32", + "text": "For one antigen, about 1 in 100,000 to 1 in 1,000,000 lymphocytes fit... divides about every 8 hours.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L35", + "text": "The clone's cells become effector cells... 1,000,000 effector cells clear the pathogen faster than it grows.", + "unresolved": [ + "Plasma cells", + "B-cell receptor", + "Killer T cells" + ], + "leap": [ + "Why 1,000,000 effector cells is the number that wins: the text gives only 'thousands of antibodies per second', not how that compares with the pathogen's growth" + ], + "excess": [ + "Killer T cells kill infected cells." + ], + "purpose_unclear": false + }, + { + "at": "L38", + "text": "How long does the first response take? Let n be the number of divisions...", + "unresolved": [ + "lymph node" + ], + "leap": [ + "Why activation takes about 2.5 days is only partly explained ('meet the antigen in a lymph node')" + ], + "excess": [], + "purpose_unclear": false + }, + { + "at": "What stays behind, L43", + "text": "After the pathogen is gone, about 99% of the effector cells die. About 1% stay...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L45", + "text": "At the next meeting with the same antigen, 10,000 × 2^n = 1,000,000... Some antibodies may also still be in the blood.", + "unresolved": [], + "leap": [], + "excess": [ + "Some antibodies may also still be in the blood." + ], + "purpose_unclear": false + }, + { + "at": "Table L47-53", + "text": "First meeting vs same antigen again", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L55", + "text": "The rule behind the table: each doubling of the precursor count saves one division...", + "unresolved": [], + "leap": [], + "excess": [ + "A 100× larger count saves about 6.6 divisions. (repeats the table's 13.3 vs 6.6)" + ], + "purpose_unclear": false + }, + { + "at": "What the vaccine does, L60", + "text": "A vaccine contains the antigen, or instructions... the race starts with the secondary response", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L63", + "text": "Antigen alone is often not enough... an adjuvant: an ingredient that triggers the danger signal.", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "When memory does not help, L67", + "text": "Memory cells fit only the antigen that made them. Measles... Influenza...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "L69", + "text": "Also note what a vaccine does not do: it does not stop the pathogen from entering...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Summary L73", + "text": "The body made the matching cells at random, before any infection...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + }, + { + "at": "Not covered L75", + "text": "Not covered here: how B-cell receptors improve their fit...", + "unresolved": [], + "leap": [], + "excess": [], + "purpose_unclear": false + } + ], + "blocking": [], + "cuts": [ + "L35: 'Killer T cells kill infected cells.' and the unexplained 'Plasma cells'/'B-cell' names. Saying 'some clone cells release antibodies, free copies of their receptor' keeps the point.", + "L45: 'Some antibodies may also still be in the blood.' This side point is never used.", + "L22: the sentence about removing self-binding lymphocytes. The argument does not need it here (L63 covers the safety point).", + "L55: 'A 100× larger count saves about 6.6 divisions.' The table already shows this." + ], + "question": { + "text": "How can a vaccine make the body ready for a germ it has never met, when it can't know the germ's shape ahead of time? The title asks this, and the reader knows it from L1. L13 then reframes it as a race: 'what does a vaccine change in this race?'", + "known_at": "L1 (title), reframed at L13", + "result_stated": "Yes, in words at L5: 'The vaccine makes those few cells multiply once, without the disease, and many of their descendants stay. So when the real germ arrives, the body controls it in about 3 days instead of about 7.'" + }, + "motive": "Reason for the approach: L13 'Immune cells also multiply. Whichever side gets big enough first wins. So ask: what does a vaccine change in this race?' This sets up the counting and doubling argument. Reason to care: mostly implied. The puzzle in the title (preparing for something never met) and L20 'no gene list could anticipate a virus that evolved last year' supply it. No separate stakes are given.", + "close": "Yes. L73: 'The body made the matching cells at random, before any infection. The vaccine selects the few that fit, multiplies them safely, and leaves about 100× more behind. So the real infection is controlled in about 3 days instead of about 7.'", + "recap": "Your body already has a few cells that fit any germ by chance, and a vaccine safely multiplies them ahead of time, so the real infection starts with 100× more fighters and is beaten in about 3 days instead of 7, before symptoms appear.", + "strongest": "L38–L53: the worked doubling arithmetic (100×2^n vs 10,000×2^n) and the table. They make 7 vs 3 days and 1 billion vs 100,000 particles concrete, and every figure checks out.", + "weakest": "L35: 'Plasma cells', 'B-cell receptor' and 'Killer T cells' appear without definition. The claim that 1,000,000 effector cells is enough is asserted with little reason.", + "suggestions": [ + "In L35, say that lymphocytes come in two kinds (B cells, whose receptors can be released as antibodies, and T cells), or leave out the names.", + "Give a one-line reason why about 1% of effector cells survive as memory cells, or say plainly that this is an observed fact.", + "Briefly say what a lymph node is (a small hub where immune cells and antigen meet)." + ] + }, + "usage": { + "cost": 0.121891, + "seconds": 38 + }, + "findings": [ + { + "n": 1, + "severity": "excess", + "kind": "cut", + "text": "L35: 'Killer T cells kill infected cells.' and the unexplained 'Plasma cells'/'B-cell' names. Saying 'some clone cells release antibodies, free copies of their receptor' keeps the point." + }, + { + "n": 2, + "severity": "excess", + "kind": "cut", + "text": "L45: 'Some antibodies may also still be in the blood.' This side point is never used." + }, + { + "n": 3, + "severity": "excess", + "kind": "cut", + "text": "L22: the sentence about removing self-binding lymphocytes. The argument does not need it here (L63 covers the safety point)." + }, + { + "n": 4, + "severity": "excess", + "kind": "cut", + "text": "L55: 'A 100× larger count saves about 6.6 divisions.' The table already shows this." + }, + { + "n": 5, + "severity": "edge", + "kind": "unresolved", + "text": "L35: Plasma cells" + }, + { + "n": 6, + "severity": "edge", + "kind": "unresolved", + "text": "L35: B-cell receptor" + }, + { + "n": 7, + "severity": "edge", + "kind": "unresolved", + "text": "L35: Killer T cells" + }, + { + "n": 8, + "severity": "edge", + "kind": "leap", + "text": "L35: Why 1,000,000 effector cells is the number that wins: the text gives only 'thousands of antibodies per second', not how that compares with the pathogen's growth" + }, + { + "n": 9, + "severity": "edge", + "kind": "unresolved", + "text": "L38: lymph node" + }, + { + "n": 10, + "severity": "edge", + "kind": "leap", + "text": "L38: Why activation takes about 2.5 days is only partly explained ('meet the antigen in a lymph node')" + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150328-quiz-prose.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150328-quiz-prose.json new file mode 100644 index 0000000..7c97599 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/explainers/vaccine-memory/review/runs/2026-10-04T150328-quiz-prose.json @@ -0,0 +1,230 @@ +{ + "tool": "quiz", + "rendering": "prose", + "slug": "vaccine-memory", + "at": "2026-10-04T15:03:12", + "model": "default", + "prompt_sha": "a0a72909c349", + "reply": { + "answers": { + "1": "The main objects are the pathogen (a virus or bacterium that multiplies) and its antigens (the shapes a receptor binds). On the immune side there are lymphocytes, each carrying many copies of one kind of receptor, and clones (one lymphocyte plus its descendants). Clones produce effector cells, the antibodies some effector cells release, and memory cells. Finally there are the vaccine, the adjuvant and the danger signal.", + "2": "Each lymphocyte's receptor is built at random from gene pieces, so the body has a large library of different receptors. When a pathogen's antigen appears, only the lymphocytes whose receptors bind it activate and divide into a clone. To activate they also need a danger signal. The clone's cells become effector cells, which kill infected cells or release antibodies (free copies of the receptor) that block the pathogen. After the pathogen is cleared, about 99% of effector cells die and about 1% remain as memory cells for that same antigen. A vaccine supplies the antigen without a pathogen that can multiply, plus often an adjuvant to trigger the danger signal, so the first expansion happens safely and memory cells are left behind.", + "3": "A vaccine's antigen selects the few lymphocytes that already fit it and makes them multiply, leaving about 100× more memory cells than the original number. With that larger starting count, a real infection needs fewer doublings to reach an effective force, so it is controlled in about 3 days instead of 7, before the pathogen reaches the symptom level.", + "4": "The key variable is the precursor count (cells able to respond at the start). Raising it cuts the divisions needed to reach about 1,000,000 effector cells: each doubling saves one division, about 8 hours. Going from 100 to 10,000 cuts 13.3 divisions to 6.6, and the pathogen peak falls from 1 billion to 100,000, below the symptom level. Lowering it, or having an antigen that has changed so memory fits poorly, pushes the response back toward 7 days and illness, because the pathogen grows 10× per day while the response grows only by doubling.", + "5": "Assumption: the illustrative numbers. Infection starts with 100 particles growing 10× per day, symptoms start above 100 million particles, 1,000,000 effector cells is enough to win, and a fixed 8-hour division time holds. Established fact: two doses of measles vaccine prevent about 97% of cases for decades. Also: humans have about 20,000 genes.", + "6": "Each lymphocyte builds a random receptor by joining gene pieces, so the body already holds over 100 million receptor types, and a few of them fit almost any pathogen. An antigen selects those few cells, which divide every ~8 hours. Starting from about 100 cells, the first response needs ~7 days, by which time the pathogen has caused symptoms. Afterward about 1% of the expanded clone survives as memory cells (~100× the original count), so a second meeting needs half the divisions and takes ~3 days, keeping the pathogen below symptom level. A vaccine runs that first expansion with antigen but no growing pathogen (plus often an adjuvant for the needed danger signal). This works only while the antigen stays the same: measles yes, flu no.", + "7": "Hardest: the time totals. The 2.5 days for a first activation and 'under a day' for memory cells are stated without a reason, and the 3-day total only works if you add 2.2 + under a day yourself. It is also unclear how '1 in 100,000 to 1,000,000' turns into 100 cells, because the total number of lymphocytes is never given. Also, 'secondary/primary response' appears only in the vaccine section and never in the table. Stating the total lymphocyte count, giving a short reason for each activation delay, and using one name for the first and second responses would help.", + "8": "According to the explanation, about 3 days to control instead of about 7, so roughly 2.3× faster (about 4 days sooner). Memory cells last 'years or decades', so after two years about 10,000 matching cells remain instead of 100. That means 6.6 divisions instead of 13.3, and memory cells activate in under a day instead of ~2.5 days. The pathogen peaks around 100,000 particles instead of 1 billion, so there are no symptoms.", + "9": "Memory cells fit only the antigen that made them. Measles keeps the same antigens, so old memory still fits and gives the fast response for decades. Influenza changes its surface antigens each year, so old memory receptors fit poorly and the response falls back toward the slow first one. The vaccine is therefore updated yearly. The explanation says the cause is the changed antigen, not memory wearing off.", + "10": "Receptors are not stored one per gene, and they are not designed after the virus appears. Each new lymphocyte joins gene pieces at random and adds random edits, giving more than 100 million receptor types. Each receptor binds a family of similar shapes, and a virus shows many antigens. So by chance a few existing lymphocytes fit even a brand-new virus; the virus selects them.", + "11": "Yes, a few: about 1 in 100,000 to 1,000,000 lymphocytes per antigen (illustrative: 100 cells). The reason is that the receptor library is made at random beforehand, is very large, and each receptor fits a family of shapes.", + "12": "There are more starting cells: about 10,000 memory cells versus 100, so fewer doublings (6.6 vs 13.3) are needed to reach ~1,000,000 effector cells. Also, memory cells already wait in the tissues and need less stimulation, so they activate in under a day instead of ~2.5 days.", + "13": "Memory cells: the roughly 1% of the expanded clone's effector cells that survive for years or decades. For the same antigen they form a starting pool about 100× larger than the original precursor count (illustrative: 10,000 vs 100). Antibodies are not the lasting protection here; their decline is listed as not covered." + }, + "audit": { + "terms": [ + "'germ' (lines 5, 13, 24) and 'pathogen' (line 11 onward) name the same thing; 'germ' is never defined as equal to 'pathogen'.", + "'secondary response' / 'primary one' (line 58) mean the same as 'first meeting' / 'next meeting' / 'Same antigen again' (lines 26, 44, table), but 'primary/secondary' are used without being introduced.", + "'controls it' (lines 5, 49), 'clear the pathogen' (line 34) and 'wins' (line 13) all describe the same end point of the race, but it is never said they are one threshold (reaching 1,000,000 effector cells).", + "'fit' (lines 5, 22, 31), 'bind' (lines 18, 29) and 'match' (heading, line 15) are used interchangeably for receptor-antigen binding.", + "'shape' (lines 5, 18) and 'antigen' (line 18) overlap: line 5 says the vaccine does not teach 'a new shape', and only later is it said that the shape on a pathogen is the antigen." + ], + "unexplained": [ + "'about 1 in 100,000 to 1 in 1,000,000 lymphocytes fit' becomes 'Illustrative: 100 cells' (line 31) without the total lymphocyte count, so the reader cannot check the 100 or see what a higher or lower fraction would mean.", + "'about 2.5 days to meet the antigen and activate' (line 37) and 'under a day' for memory cells (line 44): no basis is given, yet these delays make up a large share of both totals.", + "'about 99% of the effector cells die. About 1% stay' (line 42): it is not stated that 10,000 = 1% of 1,000,000, or what a higher or lower survival share would change (it sets the 100× gain).", + "'about 97% of cases' vs 'about 40–60%' (line 65): it is not said whether these are per person or per population, or how they relate to the 3-day vs 7-day model.", + "'1,000,000 effector cells clear the pathogen faster than it grows' (line 34): the threshold is asserted; it is unclear how it depends on pathogen size, given the pathogen is at 1 billion by then in the first response." + ], + "missing_why": [ + "Why only about 1% of effector cells become long-lived memory cells while the rest die (line 42).", + "Why memory cells 'need less stimulation' and activate faster (line 44).", + "Why the first activation takes about 2.5 days: 'rare matching cells also need ... to meet the antigen' (line 37) hints at search time but does not explain it.", + "Why one fixed threshold of 1,000,000 effector cells applies even though the pathogen count at that time differs by 10,000× between the two scenarios (lines 34, 50)." + ], + "excess": [ + "Line 3 HTML comment 'Stage 1, rendered from model.md in STE-80' is production metadata, not content.", + "Line 71 Summary repeats the bolded opening (line 5) almost word for word, adding only '100× more'." + ] + }, + "gaps": [ + "The total number of lymphocytes is never given, so the step from '1 in 100,000–1,000,000' to 100 precursor cells cannot be checked.", + "There is no reason for the activation delays (2.5 days vs under a day), even though they are a large part of the 7-vs-3-day result.", + "Antibody levels and how long they last are excluded, so the reader cannot tell whether protection years later involves anything besides memory cells.", + "No mechanism is given for why ~1% of effector cells survive as memory.", + "'Primary/secondary response' appear once without definition.", + "It is unclear whether a partly changed antigen (flu) gives a partial speed-up or none; 'falls back toward' is not quantified." + ] + }, + "grade": { + "scores": { + "1": 2, + "2": 2, + "3": 2, + "4": 2, + "5": 2, + "6": 2, + "7": 2, + "8": 2, + "9": 2, + "10": 2, + "11": 2, + "12": 2, + "13": 2 + }, + "reasons": { + "1": "Full marks. Correct and specific, though it leaves out naive lymphocyte, clonal selection and primary/secondary response as named entities.", + "2": "Full marks. The full mechanism is present: random library, clonal selection, danger signal, effector/antibody, 99% contraction, vaccine with adjuvant.", + "3": "Full marks. Selection plus 100× memory, fewer doublings, about 3 vs 7 days, below symptom level.", + "4": "Full marks. Precursor count is correctly identified as the key variable, with numbers (13.3 vs 6.6 divisions, 8 h per doubling) and the antigen-change limit.", + "5": "Full marks. Illustrative assumptions are separated correctly from established facts (measles 97%, about 20,000 genes).", + "6": "Full marks. The summary covers the whole causal chain and its limit.", + "7": "Full marks. Gives specific, plausible points of difficulty with concrete suggested fixes.", + "8": "Full marks. Gives about 3 vs 7 days, 100× more starting cells (6.6 vs 13.3 divisions), faster activation, and the pathogen staying below symptom level. Does not fall into either 'wrong if' trap.", + "9": "Full marks. The changed surface antigen is named as the cause, and the answer explicitly rejects memory wearing off.", + "10": "Full marks. Random generation of more than 100 million receptors before infection, binding to shape families, and selection by the virus; it explicitly denies that receptors are designed after infection.", + "11": "Full marks. Yes, because a random library exists beforehand; precursor frequency is given.", + "12": "Full marks. 100× more starting memory cells with fewer divisions and faster activation. The 3-day total is not stated outright but follows from the components given.", + "13": "Full marks. Memory cells, about 100× the precursor count, are correctly named as the lasting protection. It does not mention the danger-signal and no-disease framing, but it answers what is left behind correctly." + }, + "pass": true + }, + "usage": { + "cost": 0.20738799999999996, + "seconds": 56 + }, + "findings": [ + { + "n": 1, + "severity": "gap", + "kind": "terms", + "text": "'germ' (lines 5, 13, 24) and 'pathogen' (line 11 onward) name the same thing; 'germ' is never defined as equal to 'pathogen'." + }, + { + "n": 2, + "severity": "gap", + "kind": "terms", + "text": "'secondary response' / 'primary one' (line 58) mean the same as 'first meeting' / 'next meeting' / 'Same antigen again' (lines 26, 44, table), but 'primary/secondary' are used without being introduced." + }, + { + "n": 3, + "severity": "gap", + "kind": "terms", + "text": "'controls it' (lines 5, 49), 'clear the pathogen' (line 34) and 'wins' (line 13) all describe the same end point of the race, but it is never said they are one threshold (reaching 1,000,000 effector cells)." + }, + { + "n": 4, + "severity": "gap", + "kind": "terms", + "text": "'fit' (lines 5, 22, 31), 'bind' (lines 18, 29) and 'match' (heading, line 15) are used interchangeably for receptor-antigen binding." + }, + { + "n": 5, + "severity": "gap", + "kind": "terms", + "text": "'shape' (lines 5, 18) and 'antigen' (line 18) overlap: line 5 says the vaccine does not teach 'a new shape', and only later is it said that the shape on a pathogen is the antigen." + }, + { + "n": 6, + "severity": "gap", + "kind": "unexplained", + "text": "'about 1 in 100,000 to 1 in 1,000,000 lymphocytes fit' becomes 'Illustrative: 100 cells' (line 31) without the total lymphocyte count, so the reader cannot check the 100 or see what a higher or lower fraction would mean." + }, + { + "n": 7, + "severity": "gap", + "kind": "unexplained", + "text": "'about 2.5 days to meet the antigen and activate' (line 37) and 'under a day' for memory cells (line 44): no basis is given, yet these delays make up a large share of both totals." + }, + { + "n": 8, + "severity": "gap", + "kind": "unexplained", + "text": "'about 99% of the effector cells die. About 1% stay' (line 42): it is not stated that 10,000 = 1% of 1,000,000, or what a higher or lower survival share would change (it sets the 100× gain)." + }, + { + "n": 9, + "severity": "gap", + "kind": "unexplained", + "text": "'about 97% of cases' vs 'about 40–60%' (line 65): it is not said whether these are per person or per population, or how they relate to the 3-day vs 7-day model." + }, + { + "n": 10, + "severity": "gap", + "kind": "unexplained", + "text": "'1,000,000 effector cells clear the pathogen faster than it grows' (line 34): the threshold is asserted; it is unclear how it depends on pathogen size, given the pathogen is at 1 billion by then in the first response." + }, + { + "n": 11, + "severity": "gap", + "kind": "missing_why", + "text": "Why only about 1% of effector cells become long-lived memory cells while the rest die (line 42)." + }, + { + "n": 12, + "severity": "gap", + "kind": "missing_why", + "text": "Why memory cells 'need less stimulation' and activate faster (line 44)." + }, + { + "n": 13, + "severity": "gap", + "kind": "missing_why", + "text": "Why the first activation takes about 2.5 days: 'rare matching cells also need ... to meet the antigen' (line 37) hints at search time but does not explain it." + }, + { + "n": 14, + "severity": "gap", + "kind": "missing_why", + "text": "Why one fixed threshold of 1,000,000 effector cells applies even though the pathogen count at that time differs by 10,000× between the two scenarios (lines 34, 50)." + }, + { + "n": 15, + "severity": "excess", + "kind": "excess", + "text": "Line 3 HTML comment 'Stage 1, rendered from model.md in STE-80' is production metadata, not content." + }, + { + "n": 16, + "severity": "excess", + "kind": "excess", + "text": "Line 71 Summary repeats the bolded opening (line 5) almost word for word, adding only '100× more'." + }, + { + "n": 17, + "severity": "gap", + "kind": "gap", + "text": "The total number of lymphocytes is never given, so the step from '1 in 100,000–1,000,000' to 100 precursor cells cannot be checked." + }, + { + "n": 18, + "severity": "gap", + "kind": "gap", + "text": "There is no reason for the activation delays (2.5 days vs under a day), even though they are a large part of the 7-vs-3-day result." + }, + { + "n": 19, + "severity": "gap", + "kind": "gap", + "text": "Antibody levels and how long they last are excluded, so the reader cannot tell whether protection years later involves anything besides memory cells." + }, + { + "n": 20, + "severity": "gap", + "kind": "gap", + "text": "No mechanism is given for why ~1% of effector cells survive as memory." + }, + { + "n": 21, + "severity": "gap", + "kind": "gap", + "text": "'Primary/secondary response' appear once without definition." + }, + { + "n": 22, + "severity": "gap", + "kind": "gap", + "text": "It is unclear whether a partly changed antigen (flu) gives a partial speed-up or none; 'falls back toward' is not quantified." + } + ] +} diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/read.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/read.json new file mode 100644 index 0000000..a8b0bbc --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "A vaccine has to contain the antigen, or instructions that make the body's cells produce it. It does not make you ill because it contains no pathogen that can multiply to disease levels. The slow first response happens while nothing is growing.", + "quote": "A **vaccine** contains the antigen, or instructions for the body's cells to make it. It contains no pathogen that can multiply to disease." + }, + "2": { + "answer": "About 1% of the effector cells stay on as memory cells. In the illustrative numbers that is 10,000 cells instead of 100, so only about 6.6 divisions are needed instead of 13.3. Memory cells are also already waiting in the tissues and need less stimulation, so they activate in under a day. Control comes in about 3 days instead of 7.", + "quote": "Memory cells already wait in the tissues and need less stimulation, so they activate in under a day. Total: about 3 days." + }, + "3": { + "answer": "Memory cells bind only the antigen that made them. An unrelated virus shows different antigens, so the expanded clones don't bind it. The response would start again from the small, random precursor count, which means a slow primary response. This is my inference from the text, which never discusses unrelated viruses directly.", + "quote": "Memory cells bind only the antigen that made them." + }, + "4": { + "answer": "Protection falls. The old memory receptors bind the changed antigen poorly, so the response slides back toward the slow first response. The cause is the changed antigen, not memory cells wearing off. This is the influenza case, which is why flu vaccines are updated every year.", + "quote": "The old memory receptors bind poorly, so the response falls back toward the slow first one." + } + }, + "stops": [ + "Never says how symptoms or damage relate to particle count, or why 100 million is the threshold: \"symptoms start above 100 million particles\"", + "\"Controls\" is defined as removing the pathogen faster than it grows, but the 1,000,000-cell target is just asserted: \"Illustrative: 1,000,000 effector cells control the pathogen.\"", + "Gives no reason the first activation takes 2.5 days: \"The rare matching cells also need about 2.5 days to meet the antigen and activate.\"", + "\"Instructions for the body's cells to make it\" is not explained (mRNA is only named in Not covered): \"or instructions for the body's cells to make it\"", + "Unclear which cells give the danger signal and how they sense a threat: \"receives a **danger signal** from other immune cells that sense a threat\"", + "Unclear how a receptor can bind a whole \"family\" of shapes, and how similar is similar enough: \"A receptor is a protein that binds one family of similar molecular shapes.\"", + "The 1,000× gap in the table needs a moment to check: \"no, 1,000× below\"", + "Says a vaccine doesn't stop entry, but the earlier figure says it prevents 97% of cases, which seems to conflict: \"A vaccine does not stop the pathogen from entering. It shortens the race.\"", + "Doesn't say why only some effector cells become memory cells: \"About 1% stay for years or decades as **memory cells**.\"" + ], + "excess": [ + "Explains the receptor-generation mechanism in more detail than the questions need: \"adding random edits at the joins\"", + "The summary repeats points made earlier: \"The vaccine runs the slow primary response safely, so the real infection meets a secondary response that starts with 100× more cells.\"", + "The Not covered list and textbook pointer are off the question: \"A standard textbook such as *Janeway's Immunobiology* covers each.\"", + "A leftover authoring comment appears in the page: \"\"", + "The rule of thumb restates what the table already shows: \"each doubling of the precursor count saves one division, which is 8 hours.\"" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/run.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/run.json new file mode 100644 index 0000000..3ed63ce --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/published/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/vaccine-memory/explanation.md` as prose with one comparison table: the subject is a single causal chain (a random library of cells, the vaccine picking and multiplying the matching ones, the memory cells left behind) plus one before-and-after race, and prose with numbers carries that better than a diagram or page would. The supporting files `model.md`, `model.yaml` and `narrative.md` are in the same folder; the probe, cold reads and blind test (passed, full marks) all ran with `--run`, `explainer check` passes after the post-test edits, and the blind test was not re-run on those small final edits.", + "error": null, + "cost": 1.5137304, + "seconds": 543, + "turns": 65, + "files": [ + "explainers/vaccine-memory/explanation.md" + ], + "review_cost": 0.6085, + "reviews": 4 +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/explanation.md b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/explanation.md new file mode 100644 index 0000000..6d65505 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/explanation.md @@ -0,0 +1,46 @@ +# How does a vaccine prepare the immune system for a disease it has never met? + + + +**The immune system does not design a defence after it sees a disease. Before any infection, it already holds billions of cells with random receptor shapes. A vaccine shows it a harmless piece of the pathogen, the rare matching cells multiply, and some stay as memory cells. When the real pathogen arrives, the response starts from a large, ready population instead of a few cells.** + +## The parts + +- An **antigen** is a molecular shape on a pathogen, for example a protein on the surface of a virus. +- A **lymphocyte** is a white blood cell (a B cell or a T cell). Each lymphocyte carries one **receptor** type, a protein that binds one shape. +- An **antibody** is a free-floating copy of a B cell receptor. It binds the antigen and blocks the pathogen or marks it for destruction. + +## How it works + + +1. **Random receptors, made in advance.** Each new lymphocyte shuffles gene segments at random to build its receptor. The number of possible receptors is more than 10^11. The set of receptors in your body is your **repertoire**. It already contains a few cells that fit almost any shape, including shapes from viruses that did not exist when you were born. +2. **The antigen selects the match.** For a given antigen, only about 1 in 100,000 to 1 in 1,000,000 lymphocytes has a receptor that binds it. The vaccine brings the antigen to the lymph nodes, where those rare cells meet it. +3. **Clonal expansion.** A lymphocyte that binds the antigen, and also gets an alarm signal, starts to divide. Over about a week, it makes a clone of thousands of identical cells. B cells in the clone make antibodies. +4. **Memory cells remain.** After the response, most clone cells die. Some become **memory cells**: long-lived lymphocytes with the matching receptor. There are now about 10–100 times more matching cells than before. Many have a better fit, because the clone mutates its receptors during expansion and the best binders are kept. + +Analogy: the repertoire is a large ring of random keys. The vaccine does not cut a new key. It finds the keys that fit this lock and copies them. (Unlike keys, the copies can also improve their fit.) + +A vaccine supplies the antigen without the disease: an inactivated pathogen, one purified protein, or mRNA that tells your cells to make that one protein briefly. + +## Why it works: a race + + + +Infection is a race between pathogen growth and immune expansion. Assume, for illustration, a pathogen that multiplies 10 times per day. + +| | First meeting (no vaccine) | Second meeting (after vaccine) | +|---|---|---| +| Matching cells at the start | a few rare naive cells | 10–100 times more memory cells | +| Useful antibody after | 7–14 days | 2–4 days | +| Antibody level | baseline | 10–100 times higher | +| Pathogen growth before response (10×/day) | 7 days: 10,000,000-fold | 3 days: 1,000-fold | + +The vaccine moves the slow first response to a safe time. The real infection then meets the fast second response. The pathogen can still enter the body, but the immune system stops it while it is small, so you get no illness or a mild one. + +All day counts and frequencies are typical estimates; they vary by antigen, vaccine, and person. + +## Limits + +- **Changed antigen.** Memory receptors fit the old shape. A variant with a changed antigen (as with influenza each year) binds less well, so protection drops and the vaccine needs an update. +- **Decline over time.** Antibody levels fall, and for some vaccines memory also weakens. Boosters restore them (tetanus: about every 10 years). +- **Weak immune system.** A person who makes few lymphocytes (for example, during chemotherapy) cannot expand a clone well, so the vaccine protects less. diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/model.md b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/model.md new file mode 100644 index 0000000..c28d752 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/model.md @@ -0,0 +1,98 @@ +# Semantic model: how a vaccine prepares the immune system + +> Source of truth. Every rendering derives from this file. + +## Central question + +How does a vaccine prepare the immune system for a disease it has never met? + +## Audience and prior knowledge + +A technical reader (school science, basic programming) new to immunology. Knows cells divide, proteins have shapes, exponential growth. Does not know lymphocyte, antigen, receptor diversity, clonal selection, memory cell. + +## Entities + +| Entity | Definition (one sentence) | Status | +|---|---|---| +| antigen | A molecular shape on a pathogen (for example, a virus surface protein) that a receptor can bind. | fact | +| lymphocyte | A white blood cell (B cell or T cell) that carries one receptor type on its surface. | fact | +| receptor | The binding protein of one lymphocyte; its shape is made by random gene shuffling before any infection. | fact | +| repertoire | The full set of different receptors across all lymphocytes in one body. | fact | +| clonal expansion | Rapid division of a lymphocyte whose receptor binds an antigen, making a clone of identical cells. | fact | +| memory cell | A long-lived lymphocyte left after a response; it keeps the matching receptor. | fact | +| antibody | A soluble copy of a B cell receptor; it binds the antigen and blocks or marks the pathogen. | fact | +| vaccine | A harmless source of the antigen, plus an alarm signal (adjuvant or the vaccine's own components). | fact | + +## Relationships + +| From | Relationship | To | +|---|---|---| +| random gene shuffling | makes | repertoire | +| antigen | selects | lymphocyte with a matching receptor | +| selected lymphocyte | undergoes | clonal expansion | +| clonal expansion | leaves | memory cells | +| memory cells | shorten | the second response | + +## Causal chain + +```text +random receptors (before exposure) → vaccine antigen binds the rare match → clonal expansion (~1 week) +→ memory cells remain → real pathogen → large, fast response in days → pathogen stopped early +``` + +## Quantities (estimates, order of magnitude) + +- Matching naive lymphocytes: about 1 in 100,000 to 1 in 1,000,000 lymphocytes. +- Repertoire size: more than 10^11 possible receptors (gene shuffling); about 10^8 or more distinct receptors present at once. +- First response: useful antibody after about 7–14 days. +- Second response: useful antibody after about 2–4 days, at 10–100 times higher levels. +- Memory cells: present in about 10–100 times more cells than the naive match; last years to decades. +- Illustrative race: a pathogen that multiplies 10× per day grows 10^7-fold in 7 days, but only 10^3-fold in 3 days. + +## Alternative states + +| Scenario | What changes | Why | +|---|---|---| +| No vaccine, first infection | Response needs ~7–14 days | Few matching cells; expansion must start from them | +| Vaccinated, infection | Response in ~2–4 days | Memory cells start from a larger, better-fitting clone | +| Variant with a changed antigen | Weaker protection | Memory receptors fit the old shape less well | +| Years later, no booster | Weaker protection for some vaccines | Antibody levels and memory decline | + +## Why this form + +| Operation | Simplest alternative | What it breaks | +|---|---|---| +| Make receptors at random, in advance | Design a receptor after seeing the pathogen | The body has no mechanism to design a protein from a shape; it can only select from what exists | +| Use a harmless antigen | Use the real pathogen | The first response takes ~7 days, while a 10×/day pathogen grows 10^7-fold | + +## Concrete cases + +| Claim | Holds | Breaks | +|---|---|---| +| Memory gives a faster response | Vaccinated: response at day ~3, pathogen grew ~10^3-fold | Variant with a changed antigen: memory fits less well; response is slower | + +## Terms + +antigen, lymphocyte, receptor, repertoire, clonal expansion, memory cell, antibody, vaccine — as above. "Learn" is used only with the meaning "select and amplify", never "design". + +## Epistemic status + +- **Established:** clonal selection, random receptor generation, memory, faster secondary response. +- **Estimated:** all frequencies and days are typical orders of magnitude; they vary by antigen and person. +- **Assumed (illustration):** the 10×/day pathogen growth rate. + +## Confusion points + +- The immune system "studies" the pathogen and designs a counter → no; matching receptors already exist by chance; the vaccine selects them. +- A vaccine contains a weak dose of the disease → many contain only one protein, or instructions (mRNA) to make one. +- Antibodies are the memory → antibodies decline; memory cells are the durable record. + +## Scope + +- **Out of scope:** T cell killing, affinity maturation detail, self-tolerance (why receptors that bind your own body are removed), herd immunity — see an immunology text (for example, Janeway's *Immunobiology*). + +## Representation decision + +- **Stage:** 1 +- **Reason:** One sequential causal chain; one comparison table holds the two responses. +- **Levels:** L1 the answer / L2 the steps / L3 why it works (the race) / L4 limits. diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/model.yaml b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/model.yaml new file mode 100644 index 0000000..2ed23e8 --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/explainers/vaccine-memory/model.yaml @@ -0,0 +1,52 @@ +# Machine-readable half of model.md. `explainer check vaccine-memory` holds every rendering to it. + +values: + match_frequency_low_inverse: 1000000 + match_frequency_high_inverse: 100000 + repertoire_possible_exp: 11 + first_response_days_min: 7 + first_response_days_max: 14 + second_response_days_min: 2 + second_response_days_max: 4 + fold_increase_min: 10 + fold_increase_max: 100 + growth_per_day: 10 + growth_7_days: 10000000 + growth_3_days: 1000 + +allow: [1, 2, 3, 4, 5, 6, 7] + +require: + - path: first_response_days_min + in: [prose] + - path: second_response_days_max + in: [prose] + - path: growth_7_days + in: [prose] + +claims: + - id: selection-not-design + kind: mechanism + statement: "Receptors are made at random before exposure; the vaccine antigen selects the rare matching lymphocyte, which then divides." + example: "About 1 in 100,000 to 1 in 1,000,000 lymphocytes binds a given antigen." + ask: "Does the immune system build a new receptor after it sees the vaccine?" + - id: why-harmless-antigen + kind: why + statement: "The vaccine gives the slow first response without the danger of the real pathogen." + alternative: "Meet the real pathogen first." + counterexample: "A pathogen that grows 10× per day grows 10,000,000-fold during a 7-day first response." + - id: memory-faster + kind: guarantee + statement: "Memory cells make the second response faster and larger." + example: "Second response in 2–4 days at 10–100 times the antibody level; pathogen grows only about 1,000-fold in 3 days." + counterexample: "A variant with a changed antigen fits memory receptors less well, so protection drops." + ask: "Why does a vaccinated person get less sick even though the pathogen still enters the body?" + +accept_unjustified: {} + +terms: + - term: select + means: "the antigen picks an existing matching cell; nothing is designed" + avoid: [designs an antibody, builds a new receptor to fit] + +quiz: [] diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/read.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/read.json new file mode 100644 index 0000000..3c17ecf --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/read.json @@ -0,0 +1,44 @@ +{ + "answers": { + "1": { + "answer": "It must contain or produce the pathogen's antigen: an inactivated pathogen, one purified protein, or mRNA that makes your cells produce that one protein for a short time. It doesn't make you ill because it supplies only a harmless piece or shape of the pathogen, not the disease itself. The explanation never says why an inactivated pathogen or a single protein can't cause disease (for example, that it can't multiply).", + "quote": "A vaccine supplies the antigen without the disease: an inactivated pathogen, one purified protein, or mRNA that tells your cells to make that one protein briefly." + }, + "2": { + "answer": "The first response starts from only a few rare matching cells, which need about a week to multiply into a large clone. After vaccination there are already 10–100 times more matching memory cells, and many fit better. So useful antibody appears in 2–4 days instead of 7–14, while the pathogen is still small.", + "quote": "When the real pathogen arrives, the response starts from a large, ready population instead of a few cells." + }, + "3": { + "answer": "Not stated directly, so this is inferred. Each receptor binds only one shape, and the vaccine expands only the cells that match its antigen. An unrelated virus has different antigens, so the memory cells don't fit it. Its matching cells are still the rare naive ones, and the slow first response happens.", + "quote": "Each lymphocyte carries one **receptor** type, a protein that binds one shape." + }, + "4": { + "answer": "Protection drops. The memory receptors were selected to fit the old shape, so they bind the changed antigen less well. The vaccine then needs an update, as with influenza each year.", + "quote": "Memory receptors fit the old shape. A variant with a changed antigen (as with influenza each year) binds less well, so protection drops and the vaccine needs an update." + } + }, + "stops": [ + "'and also gets an alarm signal' — the alarm signal is never explained (where it comes from, or why it is needed)", + "'a few rare naive cells' — 'naive' is used in the table but never defined", + "'shuffles gene segments at random' — gene segments are not explained", + "'(a B cell or a T cell)' — only B cells' role (making antibodies) is described; what T cells do is never said", + "'marks it for destruction' — destruction by what is not said", + "'Antibody level | baseline' — unclear what baseline means for a first meeting, or what it is measured against", + "'an inactivated pathogen' — why an inactivated pathogen is harmless is not explained", + "'the clone mutates its receptors during expansion and the best binders are kept' — the explanation does not say how the best binders are selected", + "'brings the antigen to the lymph nodes' — lymph nodes are not introduced, and how the vaccine gets the antigen there is not said" + ], + "excess": [ + "'' — internal note, not content", + "'' — internal markup", + "'Boosters restore them (tetanus: about every 10 years).' — off the main question", + "'A person who makes few lymphocytes (for example, during chemotherapy) cannot expand a clone well' — a side case", + "'All day counts and frequencies are typical estimates; they vary by antigen, vaccine, and person.' — a caveat this reader did not need" + ], + "grounded": { + "1": true, + "2": true, + "3": true, + "4": true + } +} \ No newline at end of file diff --git a/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/run.json b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/run.json new file mode 100644 index 0000000..1fe0b8c --- /dev/null +++ b/evals/artifact/results/2026-10-04-dev/work/vaccine-memory/quick/run.json @@ -0,0 +1,12 @@ +{ + "result": "I made `explainers/vaccine-memory/explanation.md` as prose, with a supporting `model.md` and `model.yaml`; `explainer check` passes. I chose prose because the subject is one chain of steps, and a single table can hold the comparison between a first and a second response.", + "error": null, + "cost": 0.4694588000000001, + "seconds": 94, + "turns": 25, + "files": [ + "explainers/vaccine-memory/explanation.md" + ], + "review_cost": 0, + "reviews": 0 +} \ No newline at end of file diff --git a/explainer_kit/agent.py b/explainer_kit/agent.py index a3f578a..33aae70 100644 --- a/explainer_kit/agent.py +++ b/explainer_kit/agent.py @@ -15,10 +15,13 @@ def ask(system: str, prompt: str, model: str | None = None, read: list[Path] | None = None, - timeout: int = 900) -> str: + timeout: int = 900, usage: dict | None = None) -> str: + """With `usage`, the call's cost (USD) and seconds are added to that dict.""" # The prompt goes right after -p: --tools, --allowedTools, and --add-dir take several values each and would # swallow a prompt placed after them. cmd = ["claude", "-p", prompt, "--setting-sources", "project", "--strict-mcp-config", "--system-prompt", system] + if usage is not None: + cmd += ["--output-format", "json"] if read: cmd += ["--tools", "Read", "--allowedTools", "Read"] for folder in sorted({str(Path(p).resolve()) for p in read}): @@ -32,7 +35,41 @@ def ask(system: str, prompt: str, model: str | None = None, read: list[Path] | N stdin=subprocess.DEVNULL) if out.returncode != 0: raise RuntimeError(f"claude -p failed: {out.stderr.strip()[:300]}") - return out.stdout.strip() + if usage is None: + return out.stdout.strip() + data = json.loads(out.stdout) + usage["cost"] = usage.get("cost", 0) + (data.get("total_cost_usd") or 0) + usage["seconds"] = usage.get("seconds", 0) + round((data.get("duration_ms") or 0) / 1000) + return (data.get("result") or "").strip() + + +def work(prompt: str, cwd: Path, tools: list[str], allowed: list[str], append_system: str | None = None, + plugin_dir: Path | None = None, env: dict[str, str] | None = None, model: str | None = None, + timeout: int = 1800) -> dict: + """One fresh agent that does a task in `cwd` with the given tools, under Claude Code's own system prompt + (plus `append_system`). Still no settings files, no MCP servers, and no stdin; `plugin_dir` loads one + plugin. Returns the reply, cost, seconds, and turns (from --output-format json).""" + cmd = ["claude", "-p", prompt, "--setting-sources", "project", "--strict-mcp-config", "--output-format", "json", + "--tools", ",".join(tools), "--allowedTools", *allowed] + if append_system: + cmd += ["--append-system-prompt", append_system] + if plugin_dir: + cmd += ["--plugin-dir", str(plugin_dir)] + if model: + cmd += ["--model", model] + try: + out = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout, + stdin=subprocess.DEVNULL, env=env) + except subprocess.TimeoutExpired: + return {"result": "", "error": f"timeout after {timeout} s", "cost": None, "seconds": timeout, "turns": None} + try: + data = json.loads(out.stdout) + except json.JSONDecodeError: + return {"result": out.stdout[-2000:], "error": out.stderr.strip()[:500] or "no JSON reply", "cost": None, + "seconds": None, "turns": None} + return {"result": data.get("result", ""), "error": data.get("is_error") and data.get("subtype", "error") or None, + "cost": data.get("total_cost_usd"), "seconds": round((data.get("duration_ms") or 0) / 1000), + "turns": data.get("num_turns")} def parse_json(text: str) -> dict: diff --git a/explainer_kit/artifact_eval.py b/explainer_kit/artifact_eval.py new file mode 100644 index 0000000..3c60f2f --- /dev/null +++ b/explainer_kit/artifact_eval.py @@ -0,0 +1,451 @@ +"""Artifact eval: on subjects it was never tuned on, does the toolkit help an agent build an explainer that a +reader understands with less effort? + +For each subject in evals/artifact/questions.yaml and each condition, three fresh `claude -p` agents run: + +1. The builder gets the question and the reader, and writes an explanation (prose, diagram, or page) in an + empty project folder. The conditions differ only in the guidance it gets: none, the principles appended + to its system prompt, or the explain skill (plugin) at the quick or published tier. +2. The reader sees only the finished renderings (no model.md) and answers the subject's fixed quiz. Each + answer must quote the passage it rests on; an answer whose quote is not in the renderings scores 0, so a + reader that answers from its own knowledge earns nothing. It also lists where it would stop or reread, + and what it did not need. +3. The grader sees, per subject, every condition's answers and visible text under shuffled labels. It scores + each answer against the expected one and checks the text for the misconception and for factual errors. + +Every stage saves its output and is skipped when the output exists, so a failed run resumes. +""" + +from __future__ import annotations + +import json +import os +import random +import re +import shutil +import tempfile +from concurrent.futures import ThreadPoolExecutor +from datetime import date +from pathlib import Path + +import yaml + +from explainer_kit import agent +from explainer_kit.evals import system_prompt, words +from explainer_kit.model import visible_html, visible_markdown +from explainer_kit.paths import CHECKOUT, display, project_root + +RENDERINGS = {"explanation.md": "prose", "diagram.md": "diagram", "index.html": "html"} +PLUGIN = CHECKOUT / "plugin" + +TASK = """Explain this subject to a reader, as a file the reader will read on their own. + +Question: {q} +The reader: {reader} + +Write the explanation as one or more of: explainers/{id}/explanation.md (prose), explainers/{id}/diagram.md (Mermaid +diagrams with their text), explainers/{id}/index.html (an interactive page). Choose the form that serves this reader +best; make more than one only if each adds something. Do not make a video. Do not search the web. +{extra} +When done, reply with one line: the files you made, and why that form.""" + +SKILL_TIER = { + "quick": "Use the explain skill (explainer:explain) at the Quick artifact tier, with the slug {id}.", + "published": "Use the explain skill (explainer:explain) at the Published tier, with the slug {id}: run the probe, " + "the cold read, and the blind test with --run, and fix what they find before you finish.", +} +TIMEOUT = {"quick": 2400, "published": 5400} + +READER_SYSTEM = "You are a careful reader taking a test on an explanation. Reply with JSON only." +GRADER_SYSTEM = "You are a strict technical editor and a careful scientist. You grade explanations. Reply with JSON only." + + +def questions_file() -> Path: + return project_root() / "evals" / "artifact" / "questions.yaml" + + +def load(split: str = "dev", only: list[str] | None = None) -> tuple[str, list[dict]]: + spec = yaml.safe_load(questions_file().read_text()) + subjects = [s for s in spec["subjects"] if (split == "all" or s["split"] == split) and (not only or s["id"] in only)] + return spec["reader"], subjects + + +def condition(name: str, source: str) -> dict: + """quick and published use the plugin; any other source is a system-prompt source (evals.system_prompt): + none, principles, git:, or a file.""" + if source in SKILL_TIER: + return {"kind": "skill", "tier": source} + return {"kind": "prompt", "append": None if source == "none" else system_prompt(source)} + + +# ---------------------------------------------------------------- 1. build + +_SHIM: Path | None = None + + +def shim() -> Path: + """A bin folder whose `explainer` runs this checkout, so the plugin uses the code under test, not a release. + It lives in a private temp folder (mkdtemp: mode 0700, unpredictable name), made once per process: a fixed + name in a shared /tmp would let another user plant the executable. It holds a local path, which does not + belong in the results.""" + global _SHIM + if _SHIM is not None: + return _SHIM + bin_dir = Path(tempfile.mkdtemp(prefix="explainer-eval-bin-")) + exe = bin_dir / "explainer" + exe.write_text(f'#!/bin/sh\nexec uv run --quiet --project "{CHECKOUT}" explainer "$@"\n') + exe.chmod(0o700) + _SHIM = bin_dir + return bin_dir + + +def inside_project(path: Path) -> Path | None: + """The first folder at or above `path` that would give an agent instructions (CLAUDE.md or .claude/).""" + for d in (path, *path.parents): + if (d / "CLAUDE.md").exists() or (d / ".claude").is_dir(): + return d + return None + + +def build(subject: dict, name: str, cond: dict, reader: str, out: Path, model: str | None) -> dict: + """Builds in a temporary folder outside every project, then copies the result into out/work//. + Inside a project, Claude Code would load that project's CLAUDE.md (in this repo: the principles) whatever + the condition.""" + dest = out / "work" / subject["id"] / name + done = dest / "run.json" + if done.exists(): + return json.loads(done.read_text()) + ws = Path(tempfile.mkdtemp(prefix=f"explainer-eval-{subject['id']}-{name}-")).resolve() + if found := inside_project(ws): + raise SystemExit(f"the build folder {ws} is inside {found}, which has CLAUDE.md or .claude/; set TMPDIR " + f"to a folder outside every project") + (ws / "explainers").mkdir(parents=True) + tools = ["Read", "Write", "Edit", "Glob", "Grep"] + allowed = list(tools) + env = {**os.environ, "EXPLAINER_ROOT": str(ws)} + extra, kwargs = "", {} + if cond["kind"] == "skill": + extra = SKILL_TIER[cond["tier"]].format(id=subject["id"]) + tools += ["Bash", "Skill"] + allowed += ["Bash(explainer:*)", "Skill"] + env["PATH"] = f"{shim()}{os.pathsep}{env['PATH']}" + kwargs = {"plugin_dir": PLUGIN, "timeout": TIMEOUT[cond["tier"]]} + else: + kwargs = {"append_system": cond["append"], "timeout": 1200} + prompt = TASK.format(q=subject["q"], reader=reader, id=subject["id"], extra=extra) + run = agent.work(prompt, ws, tools, allowed, env=env, model=model, **kwargs) + run["files"] = [display_rel(p, ws) for p in renderings(ws, subject["id"])] + # The reviews a published build runs are separate claude -p calls; their cost is in each saved run. + reviews = [json.loads(f.read_text()).get("usage", {}) for f in ws.glob("explainers/*/review/runs/*.json")] + run["review_cost"] = round(sum(u.get("cost", 0) for u in reviews), 4) + run["reviews"] = len(reviews) + if dest.exists(): + shutil.rmtree(dest) + shutil.copytree(ws, dest) + shutil.rmtree(ws, ignore_errors=True) + done.write_text(json.dumps(run, indent=2, ensure_ascii=False)) + return run + + +def display_rel(path: Path, root: Path) -> str: + return str(path.relative_to(root)) + + +def renderings(ws: Path, sid: str) -> list[Path]: + """The renderings the builder made: in explainers//, else anywhere in its folder.""" + folder = ws / "explainers" / sid + found = [folder / f for f in RENDERINGS if (folder / f).exists()] + if not found: + found = sorted(p for f in RENDERINGS for p in ws.rglob(f)) + return found + + +# ---------------------------------------------------------------- 2. read + +def visible(path: Path) -> str: + """What a reader sees, for counting words: Markdown without code, styling, and comments; a page's text.""" + text = path.read_text() + return visible_html(text) if path.suffix == ".html" else visible_markdown(text) + + +def shown_text(path: Path) -> str: + """What the grader reads: Markdown as written (formulas, code, and Mermaid included), minus comments the + reader never sees; a page's visible text.""" + text = path.read_text() + return visible_html(text) if path.suffix == ".html" else re.sub(r"", "", text, flags=re.S) + + +def reader_prompt(files: list[Path], reader: str, quiz: list[dict]) -> str: + return "\n".join([ + "You are reading an explanation for the first time, as this reader: " + reader, + "", + "Read ONLY these files. Do not open any other file, and do not search the web:", + *[f"- {f}" for f in files], + "", + "An .html file is an interactive page: read its visible text, and its script to learn what the controls show.", + "", + "Answer each question from the explanation alone, as this reader would after reading it once. For each, give " + "your answer and the exact passage (a quote of at most 40 words, copied character for character from the " + "files) that your answer rests on. If the explanation does not answer the question, say \"not covered\" " + "and leave the quote empty. Do not fill gaps with your own knowledge; a gap is useful evidence. Some " + "questions ask about a case the explanation does not mention: reason from what it does say, and quote the " + "passage your reasoning uses.", + "", + "Questions:", + *[f"{i}. {item['q']}" for i, item in enumerate(quiz, 1)], + "", + "Then list, each with a short quote, at most 10 of each:", + "- stops: places where this reader would stop, reread, or lose the thread (an undefined term, a step " + "without its reason, an ambiguous reference).", + "- excess: passages this reader did not need (already known, said before, or off the question).", + "", + 'Reply with JSON only: {"answers": {"1": {"answer": "...", "quote": "..."}, ...}, "stops": ["..."], ' + '"excess": ["..."]}', + ]) + + +def read(subject: dict, name: str, reader: str, out: Path, model: str | None) -> dict: + ws = out / "work" / subject["id"] / name + saved = ws / "read.json" + if saved.exists(): + return json.loads(saved.read_text()) + files = renderings(ws, subject["id"]) + if not files: + result = {"answers": {}, "stops": [], "excess": [], "missing": True} + else: + # The reader gets copies of the renderings alone, so it cannot open model.md or the review runs. + shown = out / "shown" / subject["id"] / name + if shown.exists(): + shutil.rmtree(shown) + shown.mkdir(parents=True) + copies = [] + for f in files: + shutil.copy(f, shown / f.name) + copies.append(shown / f.name) + prompt = reader_prompt(copies, reader, subject["quiz"]) + for _ in range(2): # a smaller reader model sometimes breaks the JSON shape; ask once more + result = reader_reply(agent.ask(READER_SYSTEM, prompt, model=model, read=[shown]), len(subject["quiz"])) + if len(result["answers"]) == len(subject["quiz"]): + break + shutil.rmtree(shown) # copies of files kept in work/ + result["grounded"] = {k: grounded(v.get("quote", ""), files) for k, v in result.get("answers", {}).items()} + saved.write_text(json.dumps(result, indent=2, ensure_ascii=False)) + return result + + +def reader_reply(text: str, n: int) -> dict: + """The reader's JSON, with answers under "answers" even when the reply put them at the top level.""" + try: + reply = agent.parse_json(text) + except ValueError: + reply = {} + answers = reply.get("answers") + if not isinstance(answers, dict): + answers = {k: v for k, v in reply.items() if k.isdigit()} + answers = {k: v for k, v in answers.items() if isinstance(v, dict) and k.isdigit() and 1 <= int(k) <= n} + return {"answers": answers, "stops": reply.get("stops", []), "excess": reply.get("excess", [])} + + +def norm(text: str) -> list[str]: + text = text.lower().replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"') + return re.findall(r"[a-z0-9]+", text) + + +def grounded(quote: str, files: list[Path]) -> bool: + """True when the quote, or a run of 6 of its words in order, is in a rendering (raw or as rendered).""" + q = norm(quote) + if not q: + return False + hay = " " + " ".join(w for f in files for t in (f.read_text(), visible(f)) for w in norm(t)) + " " + n = min(6, len(q)) + return any(" " + " ".join(q[i:i + n]) + " " in hay for i in range(len(q) - n + 1)) + + +# ---------------------------------------------------------------- 3. grade + +TEXT_LIMIT = 60_000 # characters per explanation shown to the grader (line breaks kept: tables, Mermaid) + + +def grade_prompt(subject: dict, reader: str, entries: dict[str, dict]) -> str: + quiz = "\n".join(f" {i}. {item['q']}\n Expected: {item['expect']}" for i, item in enumerate(subject["quiz"], 1)) + blocks = [] + for label, e in entries.items(): + answers = "\n".join(f" {k}. {a.get('answer', '')}" for k, a in sorted(e["answers"].items())) + text = e["text"][:TEXT_LIMIT] + blocks.append(f"=== Explanation {label} ===\n{text}\n\n--- Reader's answers from {label} ---\n{answers}") + return "\n".join([ + f"Subject: {subject['q']}", + f"The reader: {reader}", + "", + "A reader read each explanation once and answered this quiz from it:", + quiz, + f"The explanation must not teach this misconception: {subject['misconception']}", + "", + *blocks, + "", + "Grade each explanation on its own. For each:", + "- quiz: one score per question: 0 wrong or not covered, 1 vague or partly right, 2 correct and specific " + "(the answer may use other words or another example than the expected one).", + "- misconception: 'taught' (the text states or implies it), 'corrected' (it names and refutes it), or 'absent'.", + "- errors: every factual error in the explanation's text, quoted.", + f"Then rank the explanations (labels {', '.join(entries)}) from the one that lets this reader build an " + "accurate mental model with the least effort to the one that costs the most. Effort comes from what is " + "missing (a step, a reason, a term left undefined) and from what is extra (what the reader knows, " + "repeats, detours). Say in one sentence why the first is first.", + "", + 'Reply with JSON only: {"grades": {"