Offline-first RAG system. Your documents, your models, your machine.
LocalRAG ingests your local documents, stores embeddings in a local ChromaDB database, and answers questions using Ollama (or OpenAI / Anthropic) models. No cloud account, no API key, and no data leaving your machine on the default path.
Most RAG examples are demos: fixed-size text slicing, vector-only search, and no way to tell whether retrieval actually got better. LocalRAG is built around the parts that decide whether answers are trustworthy.
- Runs fully offline. The happy path is Ollama plus embedded ChromaDB on your own hardware — private by construction, no mandatory API bills, and it works on a plane.
- Chunks by structure, not by character count. Splitting happens on headings, tables, and code blocks, so retrieved context contains complete facts instead of sentences cut in half.
- Hybrid retrieval by default. Vector search finds meaning, BM25 finds exact tokens like error codes, SKUs, and version strings. Each covers the other's blind spot.
- Prefers fresh answers. When two chunks are equally relevant, the newer one wins, which avoids the common "correct but out-of-date policy" failure.
- Measures quality instead of asserting it. A bundled dataset, RAGAS metrics, a reproducible benchmark runner, and versioned results ship with the project — retrieval changes can be compared, not guessed at.
- No vendor lock-in. LLM and embedding providers sit behind interfaces, so switching between Ollama, OpenAI, and Anthropic is configuration rather than a rewrite.
- Bounded agent mode. The agent endpoint has exactly two tools (
search_documents,answer_directly) instead of an open-ended loop, so its behavior stays explainable. - MCP transport. MCP-capable clients can search, answer, ingest, and list collections through a separate authenticated process.
Requires Python 3.13+, uv, and Ollama.
uv sync --locked
ollama serve
ollama pull nomic-embed-text
ollama pull gemma3:4b
cp .env.example .env
uv run localrag ingest ./docs
uv run localrag query "What are the key topics in these documents?"That's the whole loop — no cloud API keys needed for local Ollama mode. Ollama install details, including running it in Docker, are in docs/ollama.md.
uv run uvicorn localrag.api.main:app --reloadOpen http://127.0.0.1:8000/docs for interactive Swagger documentation of every endpoint, generated from the code itself.
Set API_KEY in .env to require an X-API-Key header on everything except the health and metrics probes.
task docker-upStarts localrag-api, ollama, chromadb, prometheus, and grafana, and pulls the configured models first via a one-shot localrag-setup service.
That service exits 0 when it finishes; that is expected, not a failure.
- API:
http://localhost:8000/docs - MCP:
http://localhost:8002/mcp - Grafana:
http://localhost:3000(admin / admin) - Prometheus:
http://localhost:9090
flowchart TD
userReq[User Request] --> apiLayer[FastAPI Endpoints]
apiLayer --> queryJson["POST /query (JSON)"]
apiLayer --> queryStream["POST /query/stream (SSE)"]
apiLayer --> agentQuery["POST /agent/query"]
queryJson --> ragEngine[RAG Engine]
queryStream --> ragEngine
agentQuery --> agentService[Agent Service]
agentService -->|search_documents| ragEngine
agentService -->|answer_directly| llmProvider[LLM Provider]
ragEngine --> llmProvider
llmProvider --> providers["Ollama | OpenAI | Anthropic"]
ragEngine --> vectorStore[(ChromaDB)]
apiLayer --> metrics["GET /metrics (Prometheus)"]
metrics --> prometheus[Prometheus]
prometheus --> grafana[Grafana]
Documents are parsed, chunked on structural boundaries, embedded, and stored in ChromaDB. A query retrieves candidates with hybrid search, optionally reranks and compresses them, and passes the surviving context to the answering model. docs/architecture.md covers the layers and data flow in full.
uv run localrag --help
uv run localrag ingest ./docs
uv run localrag query "How does chunking work?"
uv run localrag collections list
uv run localrag inspect --collection localrag --sample-count 5inspect is read-only and never calls a model or the network.
Full command reference, JSON schemas, and exit codes: docs/cli.md.
Copy .env.example to .env and edit it, or use structured YAML:
uv run localrag --config config.yaml ingest ./documents
uv run localrag --config config.yaml config-showSettings resolve highest-first from CLI --set, the environment, .env, then YAML — so YAML is a base layer the environment overrides.
Every variable, default, and the full precedence rules are in docs/configuration.md.
uv run localrag eval --offlineResults land in evals/results/.
Runs are seeded and record dataset identity, git SHA, model digests, and a settings snapshot, so numbers stay comparable across machines and over time.
Evaluation and benchmarks are always manual; nothing triggers a model run in CI.
The bundled localrag-core dataset has 23 balanced in-scope and out-of-scope records, with these baseline targets:
| Metric | Target |
|---|---|
| faithfulness | ≥ 0.7 |
| answer_relevancy | ≥ 0.7 |
| context_precision | ≥ 0.6 |
| context_recall | ≥ 0.6 |
Datasets, metrics, reproducibility, benchmark matrices, HTML reports, and the leaderboard all start from docs/evaluation.md.
Kubernetes manifests live in k8s/.
The deployment is intentionally single-replica with a ReadWriteOnce volume, because Chroma and ingest jobs are node-local; do not apply k8s/hpa.yaml until those are externalized.
See docs/deployment.md for the persistence, readiness, and trust boundaries.
MCP setup and tool contracts are documented in docs/mcp.md.
Setup, tests, quality gates, and the trunk-based Git workflow are in CONTRIBUTING.md. AGENTS.md is the equivalent map for coding agents.
| Document | Contents |
|---|---|
| docs/architecture.md | Layers, data flow, extension points |
| docs/configuration.md | Every setting, default, and precedence rule |
| docs/cli.md | CLI reference, schemas, exit codes |
| docs/rag-retrieval.md | Ranking math, chunking, retrieval tuning |
| docs/document-formats.md | Supported file types and how each is parsed |
| docs/ollama.md | Installing and running Ollama |
| docs/ocr.md | Scanned-PDF OCR and Tesseract setup |
| docs/observability.md | Logging, metrics, optional OpenTelemetry |
| docs/data-lifecycle.md | Upload and query-audit retention |
| docs/deployment.md | Compose and Kubernetes contracts |
| docs/mcp.md | MCP tools, transports, and authentication |
| docs/plugin-author-guide.md | Writing a retriever plugin |
| docs/evaluation.md | Start here for evaluation — datasets, metrics, reproducibility, reports, leaderboard |
| docs/agent-navigation.md | Fast codebase orientation for agents |
| docs/adr/README.md | Architecture Decision Records, indexed by status |
| docs/research-pipeline-performance.md | Dated research: stage-by-stage pipeline audit against primary sources |
| docs/issues-and-fixes-reddit-rag.md | Dated research: common RAG failure modes mapped to LocalRAG's fixes |
| ROADMAP.md | Milestones and contributor guidance |
MIT — see LICENSE.