Skip to content

fix(chunker): image-only pages silently skipped, leaving sections with zero chunks but visible in TOC #276

Closed

Description

@piyushmakhija28

Summary

chunker.py's _section_page_text() filters out pages where page.text.strip() is falsy. Pages with only images (no extractable text) produce empty page.text, are silently skipped with no warning, and the resulting section has zero chunks — yet it still appears in the document's TOC and routing responses, returning zero search results.

Root Cause

ingestion/chunker.py around line 95:

def _section_page_text(doc: ParsedDocument, entry: TocEntry) -> Iterator[tuple[int, str]]:
    for page in doc.pages:
        if page.page_num < entry.page_start or page.page_num > entry.page_end:
            continue
        if not page.text.strip():   # ← image-only pages silently skipped
            continue
        yield page.page_num, page.text

No log entry is emitted when a page is skipped. A section spanning 5 pages where pages 2–4 are full-page images produces a chunk from page 1 and page 5 only — without the caller knowing that 60% of the section's content was silently ignored.

Failure Scenario

  1. A regulatory document has a section "Compliance Matrix" spanning pages 10–15.
  2. Pages 10–14 are full-page tables rendered as images (no text extractable).
  3. Page 15 has a one-line summary.
  4. chunk_section produces one chunk from page 15.
  5. The section appears in TOC as "Compliance Matrix (pages 10-15)".
  6. A query "What is in the compliance matrix?" returns only the one-line summary from page 15.
  7. 85% of the section's content is permanently invisible, with no indication to the operator.

Proposed Fix

Log a warning when pages are skipped:

if not page.text.strip():
    logger.warning(
        "Page %d in section '%s' has no extractable text (image-only?) — skipping",
        page.page_num,
        entry.title,
        extra={"doc_id": doc.doc_id, "section_id": entry.section_id},
    )
    continue

Expose skipped_pages count in IngestResult so operators can detect image-heavy documents:

@dataclass
class IngestResult:
    ...
    skipped_image_pages: int = 0

Activity

  1. piyushmakhija28 commented on Jun 21, 2026

    @piyushmakhija28
    ContributorAuthor

    Fixed in PR #280 (Sprint R3/R4: fix(sprint-r3): root-cause fixes for all 39 bugs from code review round 3 + Sprint R4 round 4)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcode-reviewFound via automated code reviewpriority: mediumFunctional error with workaround

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions