fix(ingestor): bare except Exception reclassifies all pipeline failures as DependencyUnavailable 503 #244

Summary

PipelineIngestor.ingest_document() wraps the entire pipeline call in a bare except Exception that re-raises every unrecognised exception as DependencyUnavailable. This hides programming bugs, memory errors, and data-corruption exceptions from operators and callers — making every internal crash look like a transient infrastructure outage.

Root Cause

backend/app/adapters/ingestor.py around line 214–220:

try:
    result = await anyio.to_thread.run_sync(...)
except (AssertionError, ValueError, TypeError) as exc:
    raise  # domain errors propagate
except Exception as exc:
    raise DependencyUnavailable(            # ← ALL other exceptions become 503
        f"Ingestion pipeline failed: {exc}"
    ) from exc

MemoryError, RecursionError, KeyError, RuntimeError, and library-specific exceptions from Qdrant or SQLAlchemy are all silently promoted to "dependency unavailable". The document may be half-written (sections in Postgres, no Qdrant points) with no signal to the caller whether retrying is appropriate.

Failure Scenario

1. Pipeline raises MemoryError (large PDF, server OOM).
2. Caller receives 503 Ingestion pipeline failed: ….
3. Caller retries indefinitely — each attempt OOMs the worker again.
4. There is no way to distinguish a broken pipeline from an unavailable database using only the HTTP response.
5. Meanwhile, the document sits in Postgres with ingest_status='pending' and zero sections, permanently invisible in search.

Proposed Fix

except (MemoryError, RecursionError) as exc:
    logger.critical("Non-retriable pipeline failure", exc_info=True)
    raise DependencyUnavailable("Pipeline resource exhaustion") from exc
except Exception as exc:
    logger.error(
        "Unexpected pipeline exception — possible data corruption",
        exc_info=True,
        extra={"doc_id": doc_id, "tenant_id": tenant_id},
    )
    raise  # propagate as 500, not 503 — caller should NOT retry

Separate retriable infrastructure errors (DB down, Qdrant down) from programming bugs and resource exhaustion so callers can make correct retry decisions.