fix(toc_extractor): unstripped block.text in raw list causes false degenerate-structure detection, losing document sections #248

Summary

_detect_headers() in toc_extractor.py appends block.text (with leading/trailing whitespace) to the raw list without stripping. _is_degenerate_structure() then builds a distinct_titles set from these unstripped values. Two blocks with the same title but different whitespace (e.g. ' Introduction ' and 'Introduction') are counted as distinct titles, falsely triggering the degenerate-structure path and discarding all section structure.

Root Cause

ingestion/toc_extractor.py around line 274:

raw.append((level, block.text, page_num))  # ← block.text NOT stripped

Then in _is_degenerate_structure():

distinct_titles = {entry.title for entry in raw}  # ← whitespace variants treated as distinct
if len(survivors) <= 1 and len(distinct_titles) > 1:
    return True   # ← falsely returns "degenerate"

The fix str(title).strip() IS applied at _derive_page_ranges (TocEntry creation), but that's too late — the degenerate check runs first on raw data.

Failure Scenario

1. A PDF has two heuristic header candidates on the same page: one block with text ' Introduction ' (leading/trailing spaces) and one with 'Introduction' (no spaces).
2. Both clamp to the same page_start, leaving one survivors entry.
3. _is_degenerate_structure: distinct_titles = {' Introduction ', 'Introduction'} → size 2, survivors size 1 → returns True.
4. Extractor falls through to Scenario C, emits fallback_only=True.
5. Document is indexed with no sections, permanently returning zero results for any routing query.

Proposed Fix

Strip titles before appending to raw:

raw.append((level, block.text.strip(), page_num))

And add a regression test:

def test_whitespace_variants_of_same_title_not_counted_as_distinct():
    raw = [(1, '  Introduction  ', 2), (1, 'Introduction', 2)]
    assert not _is_degenerate_structure(raw, total_pages=5)