chitragupta (चित्रगुप्त)

The celestial record-keeper: a local-first knowledge library for humans and AI agents. Hybrid FTS5 + vector retrieval, PageIndex-style vectorless document trees, and web/PDF acquisition — one SQLite file plus one ANN sidecar. No server, no infrastructure.

chitragupta combines three retrieval philosophies that are usually sold as competitors, because each one wins on a different query:

Technique	Wins when	Powered by
FTS5 keyword search (porter-stemmed, query widening)	exact terms, names, rare words — "Vimshottari", "Sade Sati"	litesearch
Vector ANN search (HNSW)	paraphrase, fuzzy recall — "seven year saturn transit over moon"	usearch native index, used directly
Vectorless tree reasoning (à la PageIndex)	"where in this 500-page book is X discussed?", synthesis across a chapter, doc-structure questions	toc() / read() over a heading tree an LLM can navigate

Results from the first two are fused with Reciprocal Rank Fusion, then aggregated up the document tree so agents get sections with citations, not just isolated chunks. Every answer is traceable: book › chapter › section › page.

Why this beats stock RAG

1. Chunks know where they live. Every chunk is linked to a node in a PageIndex-style tree built at ingest time (markdown headings → chapter heuristics → page windows, in that order — every doc gets a navigable tree even without an LLM in the loop). Retrieval evidence rolls up to sections; agents read() a whole section instead of guessing from 400-char fragments.
2. Reasoning-based retrieval stays available. toc() emits the tree (titles + summaries + page ranges) exactly like PageIndex's node structure. An agent that knows it wants "the chapter on dashas" never needs an embedding — it reads the tree and opens the node.
3. ANN without infrastructure. Vectors live in SQLite blobs; the HNSW index is a disposable sidecar file built with usearch's native Python index (no SQLite-extension download, works air-gapped) and can always be rebuilt with reindex(). Numpy brute-force covers the fallback path.
4. Write-back memory (à la ChatIndex): agents note() distilled findings; notes are searched alongside the corpus, so the library gets smarter with use.
5. The web is a source, not an afterthought. Through fossick: URLs, arXiv papers, YouTube transcripts, GitHub repos, JS-heavy or bot-walled pages, and OCR for scanned PDFs (pdf2md).

Install

uv add chitragupta            # core: litesearch + usearch + numpy
uv add chitragupta[web]       # + fossick for URL/arXiv/YouTube/GitHub ingestion

Quick start — learn astrology from a shelf of books

from chitragupta import Library, add_pdf, add_url

lib = Library('astro/library.db')            # encoder='auto': model2vec if available, hash fallback

# ingest books (e.g. from ayushman1024/ASTROLOGY-BOOKS-DATABASE)
add_pdf(lib, 'books/saravali.pdf', title='Kalyana Varma — Saravali')
add_pdf(lib, 'books/brihat_jataka.pdf', ocr='auto')   # scanned? fossick OCR takes over
add_url(lib, 'https://en.wikipedia.org/wiki/Hindu_astrology')

# 1. hybrid search: FTS5 + ANN + RRF, cited to the page
lib.search('effects of saturn in the seventh house', k=5)
# [{'content': ..., 'doc': 'Kalyana Varma — Saravali',
#   'breadcrumb': 'Kalyana Varma — Saravali › Chapter 30 › Saturn', 'page': 214, ...}]

# 2. section-level research: evidence aggregated up the tree
lib.research('how is sade sati timed and interpreted', k=3)
# [{'title': 'Saturn Transits', 'pages': (201, 219), 'score': ...,
#   'snippets': [...], 'read': "read('a1b2c3d4e5f6a7b8#12')"}]

# 3. vectorless navigation (PageIndex-style; no embeddings touched)
lib.toc('Saravali', max_depth=2)   # tree with titles, summaries, page ranges
lib.read('a1b2c3d4e5f6a7b8#12')    # full section text + breadcrumb + children

# 4. agent memory: distilled findings persist and are retrieved with the corpus
lib.note('Sade Sati = Saturn transiting 12th, 1st, 2nd from natal moon; 3 phases of ~2.5y',
         topic='saturn')
lib.search('sade sati phases', include_notes=True)

# 5. evidence packs for filling templated docs
print(lib.evidence(['What are the effects of Jupiter in Cancer?',
                    'Which dashas indicate career rise?']))

CLI (agent-friendly: every command takes --json)

chitragupta add books/                       # whole directory of PDFs
chitragupta add https://arxiv.org/abs/2506.12345
chitragupta search "saturn seventh house" --json
chitragupta research "timing of marriage in vedic astrology"
chitragupta toc saravali --depth 2
chitragupta read 'a1b2c3d4e5f6a7b8#12'
chitragupta note "Jaimini uses chara karakas, not natural karakas" --topic jaimini
chitragupta evidence "What is Sade Sati?" "Saturn dasha effects?" --out evidence.md
chitragupta status

Library path resolution: --lib flag → $CHITRAGUPTA_DB → ./.chitragupta/library.db.

Data structures

library.db (SQLite, WAL)                     library.db.usearch (HNSW sidecar)
├── docs    id · title · source · kind ·     └── chunk_id → f32 vector (cos)
│           pages · meta · added_at              rebuildable: lib.reindex()
├── nodes   id('doc#seq') · doc_id · parent_id · level · seq ·
│           title · page_start · page_end · summary · nchunks
├── chunks  content · embedding(f32 blob) · doc_id · node_id · page
│           └── FTS5 index (porter), synced by triggers   [litesearch store]
├── notes   content · embedding · topic · source           [agent memory]
└── meta    encoder name · dim · schema version

Design choices worth knowing:

• Content-addressed docs — doc_id = sha1(source, title)[:16]; re-adding is a no-op (force=True re-ingests).
• Embeddings are pluggable and recorded. A library remembers its encoder (meta) and refuses a mismatched one. auto prefers model2vec (potion-retrieval-32M, 512-d, CPU-fast static embeddings), falls back to a deterministic char-ngram HashEncoder when downloads are impossible — FTS still carries exact terms, and the hash vectors still catch morphology and typos. Upgrade later with lib.reindex(encoder='model2vec'). fastencode (litesearch ONNX, EmbeddingGemma) is the quality ceiling.
• usearch is used directly for ANN (native Index), not through litesearch's SQLite extension — no binary download at import time, HNSW scaling, and the SQL brute-force path remains as a fallback.
• Trees without an LLM — PageIndex generates its tree with an LLM; chitragupta gets 90% of the value structurally (markdown headings from pdf-oxide's detect_headings, chapter-line heuristics for classic books, page windows as the floor) at zero token cost. Both build_tree and node summaries accept callables, so an LLM can be dropped in where it pays.

Chunking pipeline (adapted from RAGLite)

Ingestion runs a three-stage, cost-model-driven splitter (chitragupta.chunking):

1. Sentences — markdown-aware splitting: headings/lists/blockquotes/tables are atomic lines, prose splits on punctuation with an abbreviation guard. (RAGLite uses a wtpsplit SaT model here; the structural signal covers most of its value for books and docs without a model download.)
2. Chunklets — dynamic programming groups sentences into ~3-"statement" units using RAGLite's exact cost model: chunklets should start on a structural boundary (heading > blockquote > paragraph > list) and a "statement" is a quantile-normalized sentence word count.
3. Semantic chunks — chunklets merge into ≤1600-char chunks by cutting where adjacent chunklet embeddings are least similar, after projecting out the document's "discourse vector" so local topic shifts stand out. RAGLite solves this with binary integer programming; the same objective decomposes, so we solve it exactly with DP — no scipy.

Two more RAGLite ideas are built in:

• Contextual chunk headings — every chunk is embedded together with its heading path (Doc › Part › Chapter, straight from the tree) and the heading is FTS-indexed via the metadata column, while stored content stays clean. Chunks are no longer "out of context" fragments.

• Late chunking — two tiers. With pooled encoders (model2vec/hash), chunk vectors are blended with their same-node neighbors (ctx_blend=0.3) as a cheap approximation. With encoder='late' (or 'late:nomic', 'late:modernbert', 'late:gemma'), chitragupta does true token-level late chunking: a node's chunks are tokenized jointly in long sliding windows with a golden-ratio left preamble, the ONNX model contextualizes every token, and each chunk vector is the mean of its own tokens' contextualized embeddings. This works because transformers.js-style ONNX exports output last_hidden_state (token embeddings) — pooling normally happens client-side, so we pool after chunk-boundary assignment instead. Recommended models (all wrapped via litesearch's FastEncode):

  Spec	Model	Ctx	Pooling	Why
  late:nomic (default)	nomic-ai/nomic-embed-text-v1.5	8192	mean	long ctx + mean pooling = the late-chunking sweet spot; Apache-2.0
  late:modernbert	nomic-ai/modernbert-embed-base	8192	mean	fastest CPU inference (ModernBERT), same prompts
  late:gemma	onnx-community/embeddinggemma-300m-ONNX	2048	mean	strongest retrieval quality; shorter windows

  (RAGLite implements this via llama.cpp GGUF embedders with pooling disabled — its default is lm-kit/bge-m3-gguf@512. BGE-M3 is multilingual but CLS-pooled and degrades past 512 tokens, so for English corpora the mean-pooled long-context models above are a better fit — RAGLite's own config notes the same.)

Searching techniques, end to end

1. Query is widened for FTS5 (litesearch.data.pre: keyword extraction + prefix wildcards + OR-broadening) with graceful fallback to a quoted literal query.
2. The same query is embedded and sent to the usearch HNSW index (cosine, f32), with numpy brute-force over SQLite blobs as fallback.
3. Both ranked lists fuse via adaptively weighted RRF (k=60): quoted phrases and name/number-heavy queries lean the fusion toward FTS, zero FTS recall leans it toward vectors, and items found by both legs float to the top. Pass adaptive=False for classic RRF.
4. search(spans=1) merges adjacent chunk hits within a node into contiguous chunk spans (RAGLite-style) padded with ±1 neighbors — better context for a generating LLM than isolated fragments.
5. search() returns cited chunks; research() groups hits by tree node, sums RRF mass per section, and returns ranked sections with snippets and read() handles — the agent's next action is in the payload.
6. toc()/read() skip steps 1–5 entirely when the agent prefers to reason over structure (long docs, "summarise chapter 3", template filling).

Deliberately not adopted from RAGLite (documented trade-offs): the SaT sentence model and llama.cpp token-level late chunking (heavy downloads for marginal gain on structured text), LLM-decides adaptive retrieval (chitragupta is the tool an agent decides to call; the SKILL.md loop covers it), and the Procrustes query adapter (needs accumulated relevance feedback — a good future step once note() usage provides it).

Using with Claude Code / agent harnesses

chitragupta/SKILL.md ships an agent skill: point your harness at a library and it gets search / research / toc / read / note / evidence as cheap JSON tools. The intended loop for "learn astrology from these books" or "fill this templated report":

research(question) → read(best node) → note(distilled finding) → repeat
                                     ↘ evidence(template questions) → fill doc

Development

uv venv && uv pip install -e '.[test]'
pytest tests/ -q        # fully offline (HashEncoder)