feat: promote beholder 4.0 main-content data into core schema#227
Merged
Conversation
BREAKING CHANGE: @nitpicker/analyze-main-contents is deleted with no compatibility shim. Beholder 4.0.0 now extracts the same main-content metrics during crawling, promoted into the core page schema instead of a post-hoc analyze plugin.
BREAKING CHANGE: NitpickerConfig no longer accepts a '@nitpicker/analyze-main-contents' override — the plugin is gone (see the analyze-main-contents package removal).
Bump @d-zero/beholder to 4.0.0, which extracts main-content region metrics (word counts, headings, images, tables, buttons, iframes, videos, audios, canvases, scroll heights) during crawling instead of requiring a post-hoc analyze pass. - Add 17 flat aggregate columns to page_meta (denormalised, same pattern as tag_count/jsonld_count) plus 8 new child tables (page_main_content_headings/images/tables/buttons/iframes/videos/ audios/canvases), each individually self-healing via hasTable guards - Add a lightweight ALTER TABLE migration for page_meta's new columns and for info.mainContentSelector, so existing archives pick up the schema without a REQUIRED_FORMAT_VERSION bump - Thread a new mainContentSelector crawl option from Config through CrawlerOptions into Scraper.scrapeStart, letting callers override beholder's automatic main-content-region detection - Clear the new child tables (and denormalised counts) on --retry-failed / --append re-promotion, mirroring the existing page_tags/page_jsonld reset behaviour - Add typed Page getters/methods for the new columns and child tables BREAKING CHANGE: page_meta gains 17 new columns and 8 new adjunct tables are created on next archive open; existing archives are migrated additively (no REQUIRED_FORMAT_VERSION change).
… list views - Add getPageMainContents(accessor, url), returning the detected main-content region's identity, aggregate counts, and all 8 child-entity arrays (headings/images/tables/buttons/iframes/videos/ audios/canvases) in one call - Extend PageListItem/PageDetail with the 17 new scalar fields and ListPagesOptions.sortBy with the sortable ones - Add 12 numeric main-content columns to the viewer_pages read model (NOT NULL DEFAULT 0, matching tag_count/jsonld_count's keyset-safe pattern) and wire them into the cursor sort-spec
Mirrors get_page_tags' shape (archiveId + url, jsonResult wrapper). Also updates get_page_detail's description to mention the new aggregate-only main-content fields it now returns.
Appends 13 columns after tags_providers (selector + 10 counts + 2 scroll heights). The old analyze-main-contents plugin's dynamic pageData columns are gone along with the plugin itself.
- Add 17 new columns to the Pages table; register sort UI for the 12 numeric ones and extend to-page-sort-by's allow-list (it gates both the viewer_pages fast path and the legacy listPages path) - Add a "main content" section to the page detail view: aggregate summary plus full drill-down for all 8 child-entity arrays, backed by a new /api/pages/main-contents route and usePageMainContents hook - Add en/ja i18n strings for the new section
--main-content-selector moves from an analyze/pipeline plugin flag to a top-level crawl-time flag (alongside crawl/pipeline's other crawl options), matching where beholder now applies it. Also drops the @nitpicker/analyze-main-contents dependency and its analyze/pipeline wiring. BREAKING CHANGE: `nitpicker analyze --main-content-selector` and `nitpicker pipeline --main-content-selector` (as an analyze option) no longer exist. Use `nitpicker crawl --main-content-selector` (or `pipeline`'s own crawl-time flag of the same name) instead.
…s fixture - Swap analyze-pipeline.e2e.ts's use of analyze-main-contents (deleted) for analyze-search - Add /main-content/ — a page with one of each element type beholder's main-content extraction recognizes — and a real headless-browser E2E crawling it, verifying the extraction and all 8 child tables against actual DOM detection rather than a hand-built fixture object
…ts references Add the new page_meta columns and 8 page_main_content_* child tables to ARCHITECTURE.md's DB schema section, and remove the deleted analyze-main-contents package from README.md's package table, the CONTRIBUTING.md concurrency example, and the git skill's package tier tables.
Rebasing onto origin/dev's v0.13.0 release caused yarn.lock conflicts (package version bumps colliding with local workspace edits) that were resolved by taking the rebase's current side wholesale, which inadvertently dropped the @d-zero/beholder 4.0.0 upgrade (and its @d-zero/puppeteer-page-scan/puppeteer-scroll transitive bumps) from the lockfile. Restore those entries and let yarn install prune the now-deleted analyze-main-contents package's stale entries.
verify-shards caught the gap: the new E2E test file wasn't listed in any shard, so CI would never actually run it.
This pre-existing test seeds 240 setPage() calls plus a full read-model build in beforeAll — legitimately heavy, and apparently marginal against vitest's 10s default hook timeout on CI hardware (passed consistently locally). Pass an explicit 30s timeout, per vitest's own suggested fix, rather than trimming the fixture's scale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps
@d-zero/beholderto 4.0.0, which now extracts main-content region metrics(word counts, headings, images, tables, buttons, iframes, videos, audios, canvases,
scroll heights) during crawling. This promotes that data into the core crawl-time
schema and removes the old post-hoc
@nitpicker/analyze-main-contentsplugin.@nitpicker/analyze-main-contentsentirely — no compatibility shim,no deprecation warning. Its former
--main-content-selectoroption moves from ananalyze/pipeline flag to a top-level crawl-time flag (
nitpicker crawl --main-content-selector), matching where beholder now applies it.page_meta(denormalised, same pattern as
tag_count/jsonld_count) plus 8 new child tables(
page_main_content_headings/images/tables/buttons/iframes/videos/audios/canvases).Existing archives get a lightweight additive migration — no
REQUIRED_FORMAT_VERSIONbump.@nitpicker/query: newgetPageMainContents(accessor, url)returning the fulldrill-down (identity + aggregates + all 8 child arrays);
PageListItem/PageDetailgain the 17 scalar fields;
viewer_pagesread model gains 12 sortable numericcolumns.
detail section with full child-entity drill-down, backed by a new
/api/pages/main-contentsroute.get_page_main_contentstool.tags_providers.Review process
6 of 8 new insert paths never exercised with real data, viewer read model's new
columns unverified, reset/repromote only checked 2 of 8 child tables, report
cell values unverified, MCP tool only tested the null path, no E2E exercising
beholder's actual DOM detection, viewer frontend untested) — all fixed, including
a new crawler-level E2E (
main-contents.e2e.ts) that crawls a real fixture pagethrough a real headless browser and verifies extraction against actual DOM
heuristics rather than a hand-built fixture object.
(null-vs-empty-object semantics) — fixed.
(
www.d-zero.co.jp-*-0.13.nitpicker): migration runs cleanly,query/viewerread paths return the new fields as
nullwith no runtime errors, and theoriginal archive file is left untouched (migration only touches the ephemeral
extraction).
Test plan
yarn build— all 13 packagesyarn test— 479 files / 3372 tests passingyarn vitest run --config vitest.e2e.config.ts— 29 files / 136 tests passingyarn workspace @nitpicker/viewer test:e2e— 48/48 Playwright tests passingyarn lint— clean🤖 Generated with Claude Code