RAG Assistant

A small, fully local Retrieval-Augmented Generation service. Upload documents (PDF / Markdown / text), and ask questions that are answered grounded in those documents, with inline source citations. Everything runs on your machine — no external APIs. Embeddings and the LLM are served by Ollama, vectors are stored in Chroma, and the HTTP layer is FastAPI.

How it works

          ┌──────────── ingest ────────────┐         ┌──────────── query ───────────┐
 PDF/MD/TXT → chunk (512/50) → embed ──────►  Chroma  ◄──── embed(question) ── retrieve top-3
                              (nomic-embed-text)  │                                  │
                                                  └──── chunks ──► LLM (qwen2.5:3b) ─┘
                                                                   → answer + citations

1. Ingest — a document is loaded, split into ~512-character chunks (50-char overlap), embedded with nomic-embed-text, and stored in a Chroma collection named documents (cosine similarity).
2. Query — the question is embedded, the top 3 most similar chunks are retrieved, and qwen2.5:3b answers using only that context, citing sources as [filename, chunk N].

Stack

Component	Choice
API	FastAPI + Uvicorn
Embeddings	nomic-embed-text (via Ollama)
LLM	qwen2.5:3b (via Ollama)
Vector store	Chroma 0.6.3
Package manager	uv
Python	3.12

Project layout

.
├── app/
│   ├── main.py        # FastAPI app: web UI + /health, /ingest, /query, /documents
│   ├── index.html     # self-contained web UI served at /
│   ├── ingest.py      # load → chunk → embed → store in Chroma
│   └── query.py       # embed question → retrieve → LLM answer
├── docs/              # documents to ingest (mounted into the container)
├── Dockerfile         # uv-based image for the API
├── docker-compose.yml # ollama + chromadb + rag-api
├── pyproject.toml     # dependencies (managed by uv)
└── uv.lock            # pinned, reproducible lockfile

API

Method	Path	Description
GET	/	Web UI — upload documents, ask questions, manage indexed docs
GET	/health	Liveness check → {"status": "ok"}
POST	/ingest	Upload a .pdf / .md / .txt file (multipart) and index it. Re-uploading an already-indexed filename returns 409 — delete it first
GET	/documents	List indexed documents → [{"source": ..., "chunks": N}, ...]
DELETE	/documents/{filename}	Remove a document — deletes its chunks and the uploaded file from docs/ (404 if not indexed)
POST	/query	Body {"question": "..."} → {"answer": ..., "sources": [...]}. Chunks below the similarity floor are dropped; if none qualify, the answer says so and sources is empty

A browser UI is served at / (the simplest way to use the app), and interactive API docs are at /docs (Swagger) once the API is running.

Running with Docker (recommended)

# Start the whole stack — that's it.
docker compose up -d --build

On first run the ollama-init service automatically pulls both models before the API starts, so no manual ollama pull is needed. Then open http://localhost:8080 in your browser to upload documents and ask questions.

Prefer the command line? The same endpoints are on host port 8080:

curl -F "file=@docs/test.pdf" http://localhost:8080/ingest
curl -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is this document about?"}'

The first run downloads ~2 GB of models; they live in the ollama_data named volume, so later starts are fast and need no re-pull. Use docker compose down to stop; avoid down -v, which wipes the volumes (models and the vector store).

Running locally (without Docker)

Requires uv and a running Ollama with both models pulled (ollama pull nomic-embed-text && ollama pull qwen2.5:3b).

# Install dependencies into a project venv
uv sync

# Start a local Chroma server (separate terminal)
uv run chroma run --host localhost --port 8000 --path ./chroma_data

# Ingest a document (scripts run from the app/ directory)
cd app && uv run python ingest.py            # ingests ../docs/test.pdf

# Ask a question from the CLI
uv run python query.py "What is RAG?"

# Or run the API
uv run uvicorn main:app --reload             # http://localhost:8000

When running the API locally, set DOCS_DIR to a real path (it defaults to the container path /app/docs), e.g. DOCS_DIR=../docs uv run uvicorn main:app.

Configuration

The scripts read these environment variables (with host-friendly defaults), so the same code works both locally and inside the compose network:

Variable	Default	Purpose
CHROMA_HOST	localhost	Chroma server host
CHROMA_PORT	8000	Chroma server port
OLLAMA_HOST	localhost	Ollama host
OLLAMA_PORT	11434	Ollama port
DOCS_DIR	/app/docs	Where uploaded files are written
SIMILARITY_THRESHOLD	0.4	Minimum cosine similarity for a retrieved chunk to be used as context

Notes

• Chroma is pinned to 0.6.3 to match the Python client. A newer server image breaks the 0.6.3 client (KeyError: '_type').
• The collection uses cosine similarity, so query.py reports a similarity score in [0, 1] (computed as 1 - cosine_distance).
• Chroma is not exposed on the host — only rag-api reaches it, over the internal compose network as chromadb:8000. Keeping the vector store network-internal avoids unnecessary attack surface. (For one-off debugging you can publish it via a docker-compose.override.yml rather than the base file.)