llm-wiki-example

A small LLM wiki following Karpathy's LLM wiki pattern: an AI agent (Claude Code) builds and maintains a structured markdown knowledge base from raw source documents. The human drops documents in and asks questions; the LLM does all the bookkeeping.

The corpus is 10 documents from RepLiQA — synthetic documents deliberately absent from LLM training data, so every answer the wiki gives must come from the corpus, not from the model's memory. Each document ships with gold Q&A pairs (in eval/), which makes grading the wiki's answers trivial.

Layout

raw/docs/        Layer 1 — immutable source documents (never edited)
wiki/            Layer 2 — LLM-generated pages
  index.md         catalog of all pages
  log.md           append-only operation log
  sources/         one summary per raw document
  concepts/        ideas & themes
  entities/        people, orgs, places
  comparisons/     contrast pages
CLAUDE.md        Layer 3 — the schema: page templates, naming rules, workflows
eval/qa_pairs.md gold Q&A answer key (off-limits to the LLM; for grading only)
outputs/         lint reports
scripts/         corpus downloader

Steps

Requires only stock python3 (stdlib) and Claude Code.

0. Fetch the corpus (already done once; re-run to verify or extend)

python3 scripts/fetch_repliqa.py

Downloads 10 documents into raw/docs/ and their gold Q&A into eval/qa_pairs.md. Idempotent — existing documents are skipped. To grow the corpus, raise NUM_DOCS in the script and re-run.

1. Ingest

Open Claude Code in this directory and prompt:

Ingest all new documents in raw/ following CLAUDE.md.

Claude creates a source-summary per document, spins up concept/entity pages, updates wiki/index.md, and logs the operation in wiki/log.md. You can also ingest incrementally — add a few docs at a time and re-run.

2. Query

Pick a question from eval/qa_pairs.md (don't paste the answer!) and prompt:

Query the wiki:

Claude consults wiki/index.md, reads the relevant pages, and answers with citations. Compare against the gold answer in eval/qa_pairs.md.

3. Lint

Run the lint workflow from CLAUDE.md.

Claude scans the whole wiki for contradictions, orphan pages, red links, stale pages, and schema violations, then writes a report to outputs/lint-<date>.md.

4. Watch it evolve

Commit after each operation — the git history shows the knowledge base compounding:

git add -A && git commit -m "ingest: <docs>"