fix(pipeline): re-upload with transient TOC failure overwrites ingest_status='indexed' to 'fallback_only' #250

Summary

When a previously successfully indexed document is re-uploaded and the TOC extraction transiently fails (network flap, LLM quota), the pipeline deduplicates correctly (reuses doc_id) but then calls upsert_document(ingest_status='fallback_only'), permanently corrupting the row's status even though valid section data still exists in the database from the first successful ingest.

Root Cause

ingestion/pipeline.py around line 455–488:

# Hash lookup finds existing doc:
existing = await find_doc_id_by_hash(content_hash)   # finds the indexed doc
doc_id = existing.doc_id                              # reuses ID ✓

# TOC extraction fails transiently → fallback_only=True
toc_result = await extract_toc(...)   # LLM quota hit → fallback_only=True

# Pipeline proceeds to upsert with degraded status:
await upsert_document(
    doc_id=doc_id,
    ingest_status='fallback_only',     # ← OVERWRITES 'indexed' with 'fallback_only'
    fallback_only=True,                # ← OVERWRITES False with True
)

The upsert unconditionally overwrites ingest_status and fallback_only without checking whether the existing row already has a better status.

Failure Scenario

1. Tenant uploads file.pdf → fully indexed (ingest_status='indexed', 15 sections).
2. Two days later, same tenant re-uploads the same file.pdf.
3. LLM refiner hits a quota limit → toc_result.fallback_only=True.
4. upsert_document() writes ingest_status='fallback_only' and fallback_only=True over the row.
5. The 15 sections still exist in Postgres and Qdrant (they weren't deleted on this path), but the routing layer now treats the document as fallback_only=True, bypassing section-level routing forever.
6. Only a fresh re-upload during a working LLM window restores the document.

Proposed Fix

Use a "don't downgrade" strategy in upsert_document:

UPDATE documents
SET ingest_status = CASE
    WHEN ingest_status = 'indexed' AND :new_status = 'fallback_only' THEN 'indexed'
    ELSE :new_status
END,
fallback_only = CASE
    WHEN ingest_status = 'indexed' AND :new_fallback = true THEN false
    ELSE :new_fallback
END
WHERE doc_id = :doc_id AND tenant_id = :tenant_id

Or equivalently, check in the pipeline before calling upsert:

if existing and existing.ingest_status == 'indexed' and toc_result.fallback_only:
    logger.warning("Skipping status downgrade for previously indexed doc %s", doc_id)
    # proceed with re-ingest but preserve status