fix(parser): fitz.open(stream=data) loads entire PDF into RAM with no size cap, enabling OOM on large uploads #246

Summary

ingestion/parser.py calls fitz.open(stream=data, filetype='pdf') where data is the raw bytes of the uploaded PDF. PyMuPDF copies the entire byte string into an in-memory stream, and IngestInput.data also holds those bytes simultaneously. A single oversized upload can double peak memory and OOM the worker process with no size guard.

Root Cause

ingestion/parser.py around line 133:

def parse(self, data: bytes) -> tuple[ParsedPage, ...]:
    doc = fitz.open(stream=data, filetype='pdf')   # ← entire bytes loaded into RAM
    # data is also still held by the caller as IngestInput.data
    # peak memory ≈ 2 × len(data) + parsed output

There is no file-size check before calling parse(), no streaming mode, and no limit in the pipeline entry point.

Failure Scenario

1. Tenant uploads a 2 GB PDF via the API.
2. The API layer reads the full body into memory as bytes.
3. PipelineIngestor._run_pipeline() passes those bytes to parser.parse().
4. fitz.open(stream=data) allocates another ~2 GB buffer internally.
5. Peak RSS ≈ 4+ GB → worker OOM-killed → all in-flight requests on that worker fail simultaneously.

Proposed Fix

Two layers of defense:

1. API-level size limit (in backend/app/api/documents.py):

MAX_UPLOAD_BYTES = 100 * 1024 * 1024  # 100 MB
if len(await file.read(MAX_UPLOAD_BYTES + 1)) > MAX_UPLOAD_BYTES:
    raise HTTPException(status_code=413, detail="File too large (max 100 MB)")

2. Parser-level guard (in ingestion/parser.py):

MAX_PDF_BYTES = 100 * 1024 * 1024
def parse(self, data: bytes) -> tuple[ParsedPage, ...]:
    if len(data) > MAX_PDF_BYTES:
        raise ParseError(f"PDF exceeds maximum size ({len(data)} bytes > {MAX_PDF_BYTES})")

Make MAX_PDF_BYTES configurable via Settings so operators can tune it without a code change.