fix(pipeline): embed() called with all chunks in one batch, causing OOM or API rejection on large documents #247

Summary

pipeline.py calls embedder.embed([chunk.text for chunk in chunks]) with all chunks in a single batch and no size limit. A large document (500+ pages) can produce thousands of chunks, exceeding the OpenAI API's per-request token budget or exhausting BgeM3's GPU/CPU memory — causing the entire ingest to fail.

Root Cause

ingestion/pipeline.py around line 473:

vectors = await asyncio.get_event_loop().run_in_executor(
    None,
    embedder.embed,
    [chunk.text for chunk in chunks],   # ← no batch size limit
)

OpenAI's embeddings API has a per-request token limit. BgeM3 processes all texts simultaneously in a single matrix operation. Neither embedder has an internal chunking loop.

Failure Scenario

1. Tenant uploads a 400-page technical manual producing 3,000 chunks.
2. embedder.embed(all_3000_texts) is called.
3. OpenAI rejects the request (payload too large) → ingest fails with APIError.
4. Or: BgeM3 attempts to allocate a 3,000 × 1024 float32 matrix → MemoryError.
5. Document is permanently in ingest_status='pending' with no sections.

Proposed Fix

EMBED_BATCH_SIZE = 256  # configurable via Settings

async def _embed_in_batches(
    embedder: Embedder,
    chunks: list[Chunk],
    batch_size: int,
) -> list[list[float]]:
    vectors = []
    for i in range(0, len(chunks), batch_size):
        batch = chunks[i : i + batch_size]
        batch_vectors = await asyncio.get_event_loop().run_in_executor(
            None,
            embedder.embed,
            [c.text for c in batch],
        )
        vectors.extend(batch_vectors)
    return vectors

Replace the single embedder.embed(...) call with _embed_in_batches(embedder, chunks, EMBED_BATCH_SIZE). Add embed_batch_size: int = 256 to Settings.