@creader/benchmark

The first structural benchmark for narrative AI systems.

Not "does AI write well?" but "can AI understand story structure?" — three-act detection, character arc tracing, false-positive resistance on literary masterworks, and regression detection across versions.

What It Measures

Evaluator	What It Tests	Method
guardian	Can your system distinguish literary devices from real inconsistencies — including violations of author-declared world constraints (rules / prohibitions)?	Precision/recall/FPR against annotated false-positive traps
analysis	Can it detect literary qualities, character arcs, and causal chains?	Quality detection + arc mapping + causal coverage
style-prose	Does it avoid flagging well-written prose as flawed?	False-positive resistance across 5 style detectors
chapter-suspense	Can it classify chapter types and measure tension?	Suspense detection + thread coverage + cliffhanger false-positive resistance
plot-structure	Can it find three-act structure, inciting incidents, and foreshadowing?	Act detection + incident positioning + foreshadowing tracking

Fixtures

Two gold-standard literary works, fully annotated with expected qualities, arcs, and causal chains:

• Hamlet (English) — 11 chapters covering the complete 5-act dramatic trajectory
• Dream of the Red Chamber / Hong Lou Meng (Chinese) — 20 analytical chapters covering Cao Xueqin's first 80 chapters

Each fixture includes:

• Chapter content (analytical prose, not raw text)
• Knowledge base entries (characters, locations)
• Expected literary qualities with evidence
• Expected character arcs with beats
• Expected causal chains
• False-positive traps (things that should NOT be flagged)

Guardian v3 sync (2026-07)

The product's Guardian is a 5-layer system (L1 consistency / L2 style & prose / L3 analysis / L4 chapter & suspense / L5 plot structure). The local evaluators here cover the pattern-matching tiers; detectors that only exist as LLM calls ship as annotated gold fixtures instead. Full coverage map + open gaps: docs/guardian-v3-coverage-gap.md.

Ported local detectors (scored by the default run):

• l2.pacing-rhythm + l2.gesture-gloss — added to prose-detectors.ts.
• l4.opening-quality — the chapter classifier already computed openingErrors; the chapter evaluator now scores it as the noFalseOpeningErrors false-positive-resistance dimension.

LLM-detector gold fixtures (tagged llm-detector, skipped by the default bench guardian run — local analyzers cannot score LLM-only detectors, and pretending otherwise would report a meaningless zero-recall floor; they feed the LLM-detector evaluation path, which needs an API budget):

• world-constraints-{en,zh}.json — l1.constraint-violation: violations of author-declared rule/prohibition entries, plus honored-rule / dream / spoken-intent / in-constraint-exception / metaphor traps.
• timeline-violations-{en,zh}.json — l1.timeline-violation: chapters contradicting the canonical event order fixed by KB event entries, plus flashback / character-misremembering / foreshadowing traps. Tagged proposed-detector: the shipped product detector is a marker-frequency heuristic; no shipped code implements these semantics yet (see gap doc).
• entity-contradictions-{en,zh}.json — l1.entity-contradiction: prose contradicting canonical entity facts, plus perception-under-light / in-world-change / hearsay / alias / metaphor traps. Tagged proposed-detector: the shipped detector reasons over relations[], a different input universe (see gap doc).
• proofread-{en,zh}.json — l2.proofread: seeded mechanical errors, plus dialect-in-dialogue / classical-quotation / in-world-spelling / intentional-fragment traps.

Quick Start

# Install
pnpm add @creader/benchmark

# Run all evaluators
pnpm run bench

# Run a single evaluator
pnpm run bench -- --evaluator guardian

# Filter by tag
pnpm run bench -- --tags hamlet

# Enable LLM-as-Judge (costs tokens, more accurate)
pnpm run bench -- --llm

# Compare against a saved baseline
pnpm run bench -- --save-baseline v1.0.0
pnpm run bench -- --compare v1.0.0

# Verbose output (per-fixture details)
pnpm run bench -- --verbose

# "Why is this written well?" reverse-engineering
pnpm run bench:why
pnpm run bench:why -- --fixture hamlet

LLM-as-Judge

By default, evaluators use local analyzers (zero cost, fast). Add --llm to enable semantic evaluation via LLM:

• Quality detection: Does the text actually exhibit the annotated literary qualities?
• Arc mapping: Are character arc beats present and in correct order?
• Plot structure: Inciting incident, midpoint, and foreshadowing identification
• Suspense validation: Are cliffhanger warnings genuine structural issues or natural drama?

Configure the LLM provider:

# Default: OpenAI gpt-4o-mini
export OPENAI_API_KEY=sk-...

# Or use Anthropic
export BENCH_PROVIDER=anthropic
export BENCH_MODEL=claude-sonnet-4-20250514
export ANTHROPIC_API_KEY=sk-ant-...

Regression Detection

Save a baseline, then compare future runs:

# After a known-good state
pnpm run bench -- --save-baseline v1.0.0

# After changes — exits with code 1 if regressions detected
pnpm run bench -- --compare v1.0.0

Regressions are any metric that drops by more than 10 points. Use in CI to prevent narrative understanding from degrading across model updates.

Analyze Why

A unique reverse-engineering tool: feeds local analyzer signals into an LLM to produce a structured report explaining why a text works.

pnpm run bench:why -- --fixture hamlet --locale zh

Outputs a structured literary analysis covering:

• Rhythm design (chapter-level tension patterns)
• Structural skeleton (three-act, inciting incident, midpoint)
• Suspense weaving (thread distribution)
• Character arc interplay
• Causal chain construction
• Actionable craft lessons for writers

Adding Fixtures

Create a JSON file in fixtures/{evaluator}/:

{
  "id": "unique-id",
  "name": "Display Name",
  "description": "What this fixture tests",
  "locale": "en",        // "en" or "zh"
  "contentType": "novel",
  "chapters": [
    {
      "id": "ch-1",
      "title": "Chapter Title",
      "content": "Chapter text...",
      "orderIndex": 0
    }
  ],
  "expectedQualities": [
    {
      "dimension": "character",
      "quality": "What quality to detect",
      "evidence": "Substring from the text proving this quality"
    }
  ],
  "expectedArcs": [
    {
      "entity": "Character Name",
      "arcType": "rise-fall",
      "beats": ["Beat 1 description", "Beat 2 description"]
    }
  ],
  "expectedCausalChains": [
    {
      "label": "Chain Name",
      "links": [
        { "event": "Cause", "consequence": "Effect" }
      ]
    }
  ],
  "expectedAbsentFlaws": ["Things that should NOT be flagged"],
  "tags": ["tag1", "tag2"]
}

Project Structure

src/
  runner.ts              CLI entry point
  analyze-why.ts         Literary reverse-engineering
  types.ts               All type definitions
  evaluators/            5 evaluators (guardian, analysis, style, chapter, plot)
  judges/                LLM-as-Judge (quality, arc, plot, suspense)
  analyzers/             Built-in local analyzers
    style/               Prose quality detectors (5 detectors)
    chapter/             Chapter classifier (type, suspense, structure)
    plot/                Plot structure (acts, foreshadowing, incidents)
    langs/               Language modules (en, zh)
  lib/                   Infrastructure (fixture loader, scoring, model factory)

fixtures/                Annotated test data (Hamlet, Dream of the Red Chamber)
baselines/               Versioned score snapshots
reports/                 Generated benchmark reports (gitignored)

License

Apache-2.0

Built by Creader — the world expression platform for novelists.