From d047963f956ac9ed04285b6478f5f9bf98a8a363 Mon Sep 17 00:00:00 2001 From: Yusuke Hirao Date: Sat, 25 Jul 2026 00:32:46 +0900 Subject: [PATCH 1/2] feat(page-cluster)!: report cluster selection reasons via onClusterReason Nitpicker needs to explain why pages ended up in the same template cluster (shared CSS, common header/footer, sidenav presence) instead of returning an opaque clusterKey. The old includeLandmarkPositions design held every member page's landmark data until final clustering completed, which is why it rejected corpora over 20,000 pages outright - fatal for a 170k-page production crawl. ClusterReason is sized by cluster count instead of page count, so it has no such ceiling and works unmodified on the streaming path. BREAKING CHANGE: includeLandmarkPositions, PageClusterKeyResult, and PageLandmarkReport are removed (never used in production). Landmark position extraction and chrome/content classification are now decomposed into independently-callable, stateless public APIs (extractLandmarks plus the newly public isChromeLandmarkInstance) that callers combine with onClusterReason's ClusterReason.landmarks[type].shellTokens themselves, instead of receiving one pre-built per-page report from the library. The CLI's --include-landmark-positions flag is replaced by --cluster-reasons-file , which writes a clusterKey-keyed ClusterReason object once per run instead of per-page JSONL fields. --- packages/@d-zero/page-cluster/README.md | 73 ++-- packages/@d-zero/page-cluster/package.json | 8 + .../src/build-cluster-reason.spec.ts | 121 +++++++ .../page-cluster/src/build-cluster-reason.ts | 159 +++++++++ .../src/build-page-landmark-report.spec.ts | 64 ---- .../src/build-page-landmark-report.ts | 107 ------ packages/@d-zero/page-cluster/src/cli.spec.ts | 83 +++-- packages/@d-zero/page-cluster/src/cli.ts | 109 +++--- .../src/derive-blocking-reason.ts | 35 ++ .../src/extract-landmarks.spec.ts | 4 +- .../src/merge-cross-block-clusters.spec.ts | 40 ++- .../src/merge-cross-block-clusters.ts | 63 +++- .../page-cluster/src/pass0-blocking.spec.ts | 63 ++++ .../page-cluster/src/pass0-blocking.ts | 57 ++- .../src/per-page-landmark-signatures.ts | 9 +- .../src/reassign-orphan-block-keys.ts | 7 +- .../src/resolve-blocking-group-keys.spec.ts | 45 +++ .../src/resolve-blocking-group-keys.ts | 39 ++- ...esolve-page-cluster-keys-streaming.spec.ts | 42 ++- .../src/resolve-page-cluster-keys.spec.ts | 110 +++--- .../src/resolve-page-cluster-keys.ts | 324 ++++++++++-------- .../@d-zero/page-cluster/src/shell-quorum.ts | 8 +- 22 files changed, 1038 insertions(+), 532 deletions(-) create mode 100644 packages/@d-zero/page-cluster/src/build-cluster-reason.spec.ts create mode 100644 packages/@d-zero/page-cluster/src/build-cluster-reason.ts delete mode 100644 packages/@d-zero/page-cluster/src/build-page-landmark-report.spec.ts delete mode 100644 packages/@d-zero/page-cluster/src/build-page-landmark-report.ts create mode 100644 packages/@d-zero/page-cluster/src/derive-blocking-reason.ts diff --git a/packages/@d-zero/page-cluster/README.md b/packages/@d-zero/page-cluster/README.md index 9f2d1198..3069cfe9 100644 --- a/packages/@d-zero/page-cluster/README.md +++ b/packages/@d-zero/page-cluster/README.md @@ -15,7 +15,7 @@ yarn add @d-zero/page-cluster ### CLI ```sh -page-cluster [--content-block-attribute ] [--include-landmark-positions] < pages.jsonl > clusters.jsonl +page-cluster [--content-block-attribute ] [--cluster-reasons-file ] < pages.jsonl > clusters.jsonl ``` **入力**: JSONL 1 行 1 ページ。フィールドは以下。`html` 以外はすべて任意（`paths` / `stylesheetHrefs` がないと粗い分類になる）。 @@ -36,44 +36,38 @@ page-cluster [--content-block-attribute ] [--include-landmark-positions] < { "id": "任意の識別子", "clusterKey": "..." } ``` -`--include-landmark-positions` を指定すると、各行に `landmarks` フィールドが追加される。header / footer / nav / aside / form / search / main のインスタンスごとに、HTML 内の位置（1-based の line/column と文字列オフセットの両方）を返す。header 〜 search の 6 種は追加で、そのページが属する最終クラスタ内での頻度分析（`shellQuorum`）に基づく `isChrome`（サイト/セクション共通の chrome か、ページ固有のコンテンツか）を持つ。`main` は常にコンテンツなので `isChrome` を持たない。 +`--cluster-reasons-file ` を指定すると、処理完了後に別ファイルとして「クラスタ選定理由」を書き出す。ページ単位ではなく**クラスタ単位**（`clusterKey` をキーにしたオブジェクト、1 クラスタにつき 1 エントリ）なので、ファイルサイズはページ数ではなくクラスタ数に比例する — ページ単位のレポートと違い、コーパスサイズの上限はない。各エントリは「なぜこのページ達が同じクラスタになったか」の根拠を構造化データとして返す: ブロッキング理由（共有 stylesheet 集合 or URL パスプレフィックス）、クラスタ内で共有されている DOM 構造トークンのコア、landmark タイプ（header/footer/nav/aside/form/search）ごとのクラスタ内共通性、そして同一ブロッキンググループ内で分岐した兄弟クラスタのキー一覧。 ```json { - "id": "任意の識別子", - "clusterKey": "...", - "landmarks": { - "header": [ - { - "startLine": 1, - "startColumn": 7, - "endLine": 1, - "endColumn": 30, - "startOffset": 6, - "endOffset": 29, - "isChrome": true - } + "[\"path:news\",\"cluster:0\"]": { + "memberCount": 42, + "blocking": [ + { "blockKey": "path:news", "reason": { "kind": "path", "pathKey": "news" } } ], - "footer": [], - "nav": [], - "aside": [], - "form": [], - "search": [], - "main": [ - { - "startLine": 2, - "startColumn": 1, - "endLine": 10, - "endColumn": 8, - "startOffset": 40, - "endOffset": 120 + "structuralCoreTokens": ["body>main>article", "..."], + "landmarks": { + "header": { + "presenceRate": 1, + "chromeRate": 1, + "shellTokens": ["..."], + "memberCountWithInstance": 42 + }, + "aside": { + "presenceRate": 0.3, + "chromeRate": 0, + "shellTokens": [], + "memberCountWithInstance": 13 } - ] + }, + "siblingClusterKeys": ["[\"path:news\",\"cluster:1\"]"] } } ``` -20,000 ページを超えるコーパス（ストリーミング経路）では `--include-landmark-positions` は使えない（エラーで終了する）。ストリーミング経路はリザーバサンプリングと近似割当を使うため、ページ単位の chrome 判定に必要な「そのページが属する最終クラスタの shell トークン」という概念を持たないため。 +上の例は、header はクラスタ全体で共通の chrome（`chromeRate: 1`）である一方、aside は 42 ページ中 13 ページにしか無く chrome とは判定されていない（`chromeRate: 0`）ことを示す。つまり「ヘッダーは共通だが、サイドナビの有無で分かれているページがある」という状況を数値で表している。`siblingClusterKeys` は同じブロッキンググループ内で Stage A/B が結局別クラスタのままにした相手のキーで、それぞれの `ClusterReason` を突き合わせれば「何が違って分かれたか」を呼び出し側で解釈できる。理由は構造化データのみで、文言化（「ヘッダーが共通です」等）は呼び出し側の責務。 + +landmark の位置情報そのもの（HTML 内のどこにあるか）が必要な場合は、ライブラリの `extractLandmarks`（ステートレス・公開 API）をページの HTML に対して自分で呼び、その結果と `ClusterReason.landmarks[type].shellTokens` を突き合わせて `isChromeLandmarkInstance`（同じく公開 API）で chrome 判定すればよい。詳細は [Library](#library) 節を参照。 クローラ出力が JSON 配列の場合は `jq` で line-delimited に変換して食わせる: @@ -84,7 +78,7 @@ jq -c '.[]' crawl-output.json | page-cluster > clusters.jsonl #### オプション - `--content-block-attribute ` — CMS が自由編集コンテンツブロックに付与している属性名（例: `data-bgb`）が分かっている場合に指定する。指定すると比較前にその属性を持つ要素配下を無視するので、同じテンプレートで本文構成だけ違うページを混同しなくなる。唯一の site-specific なオプションで、未指定でも `
` / `role="main"` を起点にした自動深さキャップが常時働く（詳細は `resolve-page-cluster-keys.ts` の JSDoc を参照） -- `--include-landmark-positions` — 出力の各行に上記の `landmarks` フィールドを追加する。20,000 ページ超のコーパスでは使えない。指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため） +- `--cluster-reasons-file ` — 上記の「クラスタ選定理由」を `` に JSON として書き出す。ページ数の上限はない。20,000 ページ以下のコーパスでは、指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため。20,000 ページ超のストリーミング経路では進捗表示・クラスタ理由の両方が動く） - `--help` / `-h` — ヘルプを表示する - `--version` / `-v` — バージョンを表示する @@ -116,12 +110,15 @@ silence したい場合は `2>/dev/null`。ログに残したい場合は `2> pr サブパスエクスポート構成。import パスと提供関数の対応は以下。 -| import パス | 提供関数 | -| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `@d-zero/page-cluster` | `tokenize` — `` 配下を構造トークン列に変換する低レベルプリミティブ | -| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。いずれも `includeLandmarkPositions: true` を渡すと `clusterKey` に加えて位置情報つきの `landmarks`（`PageLandmarkReport`）を返す | -| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す | -| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類 | +| import パス | 提供関数 | +| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `@d-zero/page-cluster` | `tokenize` — `` 配下を構造トークン列に変換する低レベルプリミティブ | +| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。`onClusterReason` コールバックを渡すと、確定したクラスタごとに 1 回だけ `ClusterReason` を通知する | +| `@d-zero/page-cluster/build-cluster-reason` | `ClusterReason` / `LandmarkClusterProfile` 型、`buildClusterReason` — クラスタ選定理由の型定義と組み立て関数（通常は `resolvePageClusterKeys` の `onClusterReason` 経由で使うので直接呼ぶ必要はない） | +| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す | +| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類 | +| `@d-zero/page-cluster/is-chrome-landmark-instance` | `isChromeLandmarkInstance` — 1 つの landmark インスタンスのトークン集合と `ClusterReason.landmarks[type].shellTokens` のようなシェルトークン集合を突き合わせて chrome/content を判定するステートレス関数 | +| `@d-zero/page-cluster/jaccard-similarity` | `jaccardSimilarity` — 2 つのトークン集合の Jaccard 類似度。`ClusterReason` 同士（`structuralCoreTokens` や `shellTokens`）を比較して兄弟クラスタとの差分を調べる用途などに使う | ```ts import { resolvePageClusterKeysFromArray } from '@d-zero/page-cluster/resolve-page-cluster-keys'; @@ -175,7 +172,7 @@ flowchart TD - **chrome discovery** — 全ページのランドマーク署名の度数分布に auto-cut を当て、閾値以上を「グローバル chrome」（サイト共通のヘッダー等）として比較から除外し、閾値未満かつ 2 ページ以上に出現するものを「ローカル chrome」（セクション固有のナビ等）としてトークン再注入する - **Stage A（ブロック内クラスタリング）** — ブロックごとに直線的な処理。`
` の深さキャップ（候補深度を全走査して knee を探す自動選択）→ tokenize → complete-linkage 階層クラスタリング → max-gap auto-cut でカット高を決定 → 最後に包含関係にあるクラスタを吸収する包含割当（割当チェーンを辿り、循環はメンバー最大のクラスタをルートに選んで解決） - **Pass 1b（ストリーミング時のみ）** — 20,000 ページ超では各ブロックをリザーバサンプリング（最大 100 ページ、ブロックキーをシードにした決定的乱数）で代表させ、サンプル外のページは Stage A 完了後に max-Jaccard で最寄りクラスタへ一括割当する。メモリ使用量はコーパス全体ではなくサンプルサイズに比例する -- **Stage B（ブロック越えマージ）** — ブロック分割はあくまで比較コスト削減のためなので、最後に同一テンプレートがブロックを跨いで分かれていないか再統合する。これが唯一の反復処理（次節） +- **Stage B（ブロック越えマージ）** — ブロック分割はあくまで比較コスト削減のためなので、最後に同一テンプレートがブロックを跨いで分かれていないか再統合する。これが唯一の反復処理（次節）。収束後、`onClusterReason` が指定されていれば、確定した最終クラスタごとに `ClusterReason` を 1 回ずつ組み立てて通知する — 追加の全コーパススキャンではなく、Stage A/B が既に計算済みの中間データ（quorum core、landmark インスタンス、ブロッキング根拠）を再利用するだけなので、クラスタ数にしか比例しない ### Stage B: ブロック越え統合の不動点ループ diff --git a/packages/@d-zero/page-cluster/package.json b/packages/@d-zero/page-cluster/package.json index f987ca4b..765d91c6 100644 --- a/packages/@d-zero/page-cluster/package.json +++ b/packages/@d-zero/page-cluster/package.json @@ -24,6 +24,14 @@ "./resolve-page-cluster-keys": { "import": "./dist/resolve-page-cluster-keys.js", "types": "./dist/resolve-page-cluster-keys.d.ts" + }, + "./is-chrome-landmark-instance": { + "import": "./dist/is-chrome-landmark-instance.js", + "types": "./dist/is-chrome-landmark-instance.d.ts" + }, + "./jaccard-similarity": { + "import": "./dist/jaccard-similarity.js", + "types": "./dist/jaccard-similarity.d.ts" } }, "bin": "./dist/cli.js", diff --git a/packages/@d-zero/page-cluster/src/build-cluster-reason.spec.ts b/packages/@d-zero/page-cluster/src/build-cluster-reason.spec.ts new file mode 100644 index 00000000..c3be3a05 --- /dev/null +++ b/packages/@d-zero/page-cluster/src/build-cluster-reason.spec.ts @@ -0,0 +1,121 @@ +import type { BlockingReason } from './derive-blocking-reason.js'; +import type { LandmarkType } from './extract-landmarks.js'; +import type { PerPageLandmarkInstance } from './per-page-landmark-signatures.js'; + +import { describe, expect, test } from 'vitest'; + +import { buildClusterReason } from './build-cluster-reason.js'; + +const zeroPosition = { + startOffset: 0, + endOffset: 0, + startLine: 1, + startColumn: 1, + endLine: 1, + endColumn: 1, +}; + +/** + * + * @param type + * @param tokens + */ +function instance( + type: LandmarkType, + tokens: readonly string[], +): PerPageLandmarkInstance { + return { + type, + tokens: new Set(tokens), + signature: [...tokens].toSorted().join('|'), + position: zeroPosition, + }; +} + +describe('buildClusterReason', () => { + test('passes blocking and siblingClusterKeys through unchanged', () => { + const blocking: readonly { blockKey: string; reason: BlockingReason }[] = [ + { + blockKey: 'css:abc', + reason: { kind: 'css', distinctiveStylesheetHrefs: ['/a.css'] }, + }, + ]; + const reason = buildClusterReason({ + tokenSets: [new Set(['a'])], + landmarkInstances: [[]], + blocking, + siblingClusterKeys: ['sibling-1'], + }); + expect(reason.blocking).toBe(blocking); + expect(reason.siblingClusterKeys).toEqual(['sibling-1']); + expect(reason.memberCount).toBe(1); + }); + + test('structuralCoreTokens keeps only tokens shared by the quorum fraction of members', () => { + const tokenSets = [ + new Set(['body>main>.card', 'body>main>.unique-a']), + new Set(['body>main>.card', 'body>main>.unique-b']), + new Set(['body>main>.card', 'body>main>.unique-c']), + ]; + const reason = buildClusterReason({ + tokenSets, + landmarkInstances: [[], [], []], + blocking: [], + siblingClusterKeys: [], + }); + expect(reason.structuralCoreTokens).toEqual(['body>main>.card']); + }); + + test('landmarks omits types with no instance on any member page', () => { + const reason = buildClusterReason({ + tokenSets: [new Set(['a']), new Set(['a'])], + landmarkInstances: [[], []], + blocking: [], + siblingClusterKeys: [], + }); + expect(reason.landmarks).toEqual({}); + }); + + test('header/footer common across all members yields chromeRate 1 and full presence', () => { + const header = ['header>nav>a']; + const footer = ['footer>p']; + const landmarkInstances = Array.from({ length: 5 }, () => [ + instance('header', header), + instance('footer', footer), + ]); + const reason = buildClusterReason({ + tokenSets: Array.from({ length: 5 }, () => new Set(['body>main>.card'])), + landmarkInstances, + blocking: [], + siblingClusterKeys: [], + }); + expect(reason.landmarks.header?.presenceRate).toBe(1); + expect(reason.landmarks.header?.chromeRate).toBe(1); + expect(reason.landmarks.header?.memberCountWithInstance).toBe(5); + expect(reason.landmarks.footer?.presenceRate).toBe(1); + expect(reason.landmarks.footer?.chromeRate).toBe(1); + }); + + test('a landmark type present on only some members, with per-member-varying tokens, reads as partial presence and non-chrome (sidenav-style split)', () => { + const header = ['header>nav>a']; + const landmarkInstances = [ + [instance('header', header), instance('aside', ['aside>.promo'])], + [instance('header', header), instance('aside', ['aside>.other'])], + [instance('header', header)], + [instance('header', header)], + [instance('header', header)], + ]; + const reason = buildClusterReason({ + tokenSets: Array.from({ length: 5 }, () => new Set(['body>main>.card'])), + landmarkInstances, + blocking: [], + siblingClusterKeys: [], + }); + expect(reason.landmarks.header?.presenceRate).toBe(1); + expect(reason.landmarks.header?.chromeRate).toBe(1); + expect(reason.landmarks.aside?.presenceRate).toBe(0.4); + expect(reason.landmarks.aside?.memberCountWithInstance).toBe(2); + expect(reason.landmarks.aside?.chromeRate).toBe(0); + expect(reason.landmarks.aside?.shellTokens).toEqual([]); + }); +}); diff --git a/packages/@d-zero/page-cluster/src/build-cluster-reason.ts b/packages/@d-zero/page-cluster/src/build-cluster-reason.ts new file mode 100644 index 00000000..d26d4af4 --- /dev/null +++ b/packages/@d-zero/page-cluster/src/build-cluster-reason.ts @@ -0,0 +1,159 @@ +import type { BlockingReason } from './derive-blocking-reason.js'; +import type { LandmarkType } from './extract-landmarks.js'; +import type { PerPageLandmarkInstance } from './per-page-landmark-signatures.js'; + +import { + DEFAULT_CHROME_OVERLAP_THRESHOLD, + isChromeLandmarkInstance, +} from './is-chrome-landmark-instance.js'; +import { computeQuorumCore } from './merge-cross-block-clusters.js'; +import { ALL_LANDMARK_TYPES } from './per-page-landmark-signatures.js'; +import { shellQuorum } from './shell-quorum.js'; + +/** + * One final cluster's common-vs-varying profile for a single landmark type + * (header/footer/nav/aside/form/search), derived by running + * {@link ./shell-quorum.js | shellQuorum} on just that type's instances across + * every member page. Absent from {@link ClusterReason}'s `landmarks` entirely + * when no member page carries an instance of the type. + */ +export type LandmarkClusterProfile = { + /** Fraction (0–1) of the cluster's member pages carrying at least one instance of this type. */ + readonly presenceRate: number; + /** + * Fraction (0–1) of this type's instances, across all member pages, that + * {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance} + * classified as chrome against this cluster's own shell for the type — + * the "header/footer is shared chrome" half of the user-facing story. + */ + readonly chromeRate: number; + /** + * The shell token set `shellQuorum` discovered for this type within this + * cluster. Exposed as raw evidence — comparing two sibling clusters' + * `shellTokens` for the same type (e.g. via + * {@link ./jaccard-similarity.js | jaccardSimilarity}) is how a caller + * answers "do these two clusters share the same footer but differ in + * whether they have a sidebar nav". + */ + readonly shellTokens: readonly string[]; + /** How many member pages contributed at least one instance of this type. */ + readonly memberCountWithInstance: number; +}; + +/** + * Structured, uninterpreted explanation of why a final cluster's member + * pages ended up together, and which sibling clusters (same Pass-0 block, + * different final cluster) they were nonetheless split from. Deliberately + * carries no human-readable text or verdicts ("these are the same template + * except for X") — that judgment belongs to the caller, which has the + * product context (and the localization requirements) `@d-zero/page-cluster` + * itself does not. + */ +export type ClusterReason = { + /** Number of pages this final cluster contains. */ + readonly memberCount: number; + /** + * The distinct Pass-0 blocking keys that fed into this final cluster, and + * the evidence behind each. Usually one entry; more than one means Stage B + * merged pages that started in different blocks — itself a notable part + * of the explanation (e.g. two differently-styled URL sections turned out + * to share the same DOM template). + */ + readonly blocking: readonly { + readonly blockKey: string; + readonly reason: BlockingReason; + }[]; + /** + * The frequency-quorum core of this cluster's DOM structural tokens (see + * {@link ./merge-cross-block-clusters.js | computeQuorumCore}) — the + * "this DOM structure is common" evidence, independent of CSS or + * landmarks. + */ + readonly structuralCoreTokens: readonly string[]; + /** Per-landmark-type commonality. Types absent from every member page are omitted. */ + readonly landmarks: { readonly [K in LandmarkType]?: LandmarkClusterProfile }; + /** + * Final cluster keys that share at least one Pass-0 block with this + * cluster but were not merged into it by Stage A/B — candidates for "why + * did these split" comparison. Does not include clusters that started in + * a different block from this one. + */ + readonly siblingClusterKeys: readonly string[]; +}; + +/** + * Builds one final cluster's {@link ClusterReason} from data Stage A/B + * already computed for clustering itself — no re-tokenization, no re-running + * `shellQuorum`'s corpus-wide discovery pass, just re-deriving per-type shells + * and quorum cores from the same pooled member state + * {@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters} already + * built and would otherwise have discarded. + * @param input + * @param input.tokenSets + * @param input.landmarkInstances + * @param input.blocking + * @param input.siblingClusterKeys + * @param input.chromeThreshold + * @example + * ```ts + * const reason = buildClusterReason({ + * tokenSets: finalGroup.tokenSets, + * landmarkInstances: finalGroup.landmarkInstances, + * blocking: [{ blockKey: 'css:abc123', reason: { kind: 'css', distinctiveStylesheetHrefs: ['/a.css'] } }], + * siblingClusterKeys: ['["css:abc123","cluster:1"]'], + * }); + * ``` + */ +export function buildClusterReason(input: { + readonly tokenSets: readonly ReadonlySet[]; + readonly landmarkInstances: readonly (readonly PerPageLandmarkInstance[])[]; + readonly blocking: readonly { + readonly blockKey: string; + readonly reason: BlockingReason; + }[]; + readonly siblingClusterKeys: readonly string[]; + readonly chromeThreshold?: number; +}): ClusterReason { + const memberCount = input.tokenSets.length; + const chromeThreshold = input.chromeThreshold ?? DEFAULT_CHROME_OVERLAP_THRESHOLD; + + const structuralCoreTokens = [...computeQuorumCore(input.tokenSets)].toSorted(); + + const landmarks: { [K in LandmarkType]?: LandmarkClusterProfile } = {}; + for (const type of ALL_LANDMARK_TYPES) { + const perMemberInstancesOfType = input.landmarkInstances.map((instances) => + instances.filter((instance) => instance.type === type), + ); + const membersWithInstance = perMemberInstancesOfType.filter( + (instances) => instances.length > 0, + ); + if (membersWithInstance.length === 0) continue; + + const shellTokens = shellQuorum(perMemberInstancesOfType); + let chromeInstanceCount = 0; + let totalInstanceCount = 0; + for (const instances of membersWithInstance) { + for (const instance of instances) { + totalInstanceCount++; + if (isChromeLandmarkInstance(instance.tokens, shellTokens, chromeThreshold)) { + chromeInstanceCount++; + } + } + } + + landmarks[type] = { + presenceRate: membersWithInstance.length / memberCount, + chromeRate: totalInstanceCount === 0 ? 0 : chromeInstanceCount / totalInstanceCount, + shellTokens: [...shellTokens].toSorted(), + memberCountWithInstance: membersWithInstance.length, + }; + } + + return { + memberCount, + blocking: input.blocking, + structuralCoreTokens, + landmarks, + siblingClusterKeys: input.siblingClusterKeys, + }; +} diff --git a/packages/@d-zero/page-cluster/src/build-page-landmark-report.spec.ts b/packages/@d-zero/page-cluster/src/build-page-landmark-report.spec.ts deleted file mode 100644 index 33c1659a..00000000 --- a/packages/@d-zero/page-cluster/src/build-page-landmark-report.spec.ts +++ /dev/null @@ -1,64 +0,0 @@ -import { describe, expect, test } from 'vitest'; - -import { buildPageLandmarkReport } from './build-page-landmark-report.js'; -import { extractLandmarks } from './extract-landmarks.js'; -import { tokenize } from './tokenize.js'; - -describe('buildPageLandmarkReport', () => { - test('a header instance whose tokens are all shell tokens is reported as chrome', () => { - const landmarks = extractLandmarks( - '
content
', - ); - const shellTokens = new Set( - tokenize(`${landmarks.header[0]!.html}`).tokens, - ); - const report = buildPageLandmarkReport(landmarks, shellTokens); - expect(report.header).toHaveLength(1); - expect(report.header[0]!.isChrome).toBe(true); - }); - - test('an instance with no overlap with shell tokens is reported as content', () => { - const landmarks = extractLandmarks( - '

x

content
', - ); - const shellTokens = new Set(['body>totally>unrelated']); - const report = buildPageLandmarkReport(landmarks, shellTokens); - expect(report.header[0]!.isChrome).toBe(false); - }); - - test('main instances carry no isChrome field', () => { - const landmarks = extractLandmarks('
content
'); - const report = buildPageLandmarkReport(landmarks, new Set()); - expect(report.main).toHaveLength(1); - expect(Object.keys(report.main[0]!)).not.toContain('isChrome'); - }); - - test('a page with no landmarks yields all-empty arrays', () => { - const landmarks = extractLandmarks('content only'); - const report = buildPageLandmarkReport(landmarks, new Set()); - expect(report).toStrictEqual({ - header: [], - footer: [], - nav: [], - aside: [], - form: [], - search: [], - main: [], - }); - }); - - test('each reported instance carries the instance position, not its html', () => { - const landmarks = extractLandmarks('
H
M
'); - const report = buildPageLandmarkReport(landmarks, new Set()); - const source = landmarks.header[0]!; - expect(report.header[0]).toStrictEqual({ - startOffset: source.startOffset, - endOffset: source.endOffset, - startLine: source.startLine, - startColumn: source.startColumn, - endLine: source.endLine, - endColumn: source.endColumn, - isChrome: false, - }); - }); -}); diff --git a/packages/@d-zero/page-cluster/src/build-page-landmark-report.ts b/packages/@d-zero/page-cluster/src/build-page-landmark-report.ts deleted file mode 100644 index 6f401947..00000000 --- a/packages/@d-zero/page-cluster/src/build-page-landmark-report.ts +++ /dev/null @@ -1,107 +0,0 @@ -import type { - ExtractLandmarksResult, - LandmarkInstance, - LandmarkPosition, -} from './extract-landmarks.js'; -import type { TokenizeOptions } from './types.js'; - -import { isChromeLandmarkInstance } from './is-chrome-landmark-instance.js'; -import { ALL_LANDMARK_TYPES } from './per-page-landmark-signatures.js'; -import { tokenize } from './tokenize.js'; - -/** - * A landmark instance's position plus whether - * {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance} judged - * it shared site/section chrome (`true`) or page-specific content (`false`), - * against the unit's {@link ./shell-quorum.js | shellQuorum} shell tokens. - */ -export type ReportedLandmarkInstance = LandmarkPosition & { readonly isChrome: boolean }; - -/** - * Per-page landmark position report built by - * {@link ./build-page-landmark-report.js | buildPageLandmarkReport}. `main` - * carries no `isChrome` verdict — it never participates in chrome/shell - * discovery (see `extractLandmarks`'s "main handling" note) and is always - * content. - */ -export type PageLandmarkReport = { - header: ReportedLandmarkInstance[]; - footer: ReportedLandmarkInstance[]; - nav: ReportedLandmarkInstance[]; - aside: ReportedLandmarkInstance[]; - form: ReportedLandmarkInstance[]; - search: ReportedLandmarkInstance[]; - main: LandmarkPosition[]; -}; - -/** - * Strips `html` off a {@link LandmarkInstance}, keeping only its position. - * `buildPageLandmarkReport`'s output is meant to be serialized per page - * across a whole corpus (the CLI's JSONL output), so the report - * deliberately excludes each instance's raw HTML to keep that payload from - * scaling with markup size — callers who also need the HTML already have - * `ExtractLandmarksResult` in hand. - * @param instance - */ -function toPosition(instance: LandmarkInstance): LandmarkPosition { - return { - startOffset: instance.startOffset, - endOffset: instance.endOffset, - startLine: instance.startLine, - startColumn: instance.startColumn, - endLine: instance.endLine, - endColumn: instance.endColumn, - }; -} - -/** - * Builds a page's landmark position report: every landmark instance's - * location, with `header`/`footer`/`nav`/`aside`/`form`/`search` instances - * additionally classified as chrome or content against `shellTokens`. - * - * Reads `landmarks` directly — the full, non-deduplicated instance list - * `extractLandmarks` produced — rather than going through - * {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}'s - * per-page-deduplicated `PerPageLandmarkInstance[]`: that dedupe collapses - * same-signature instances to one entry, which would silently drop the - * position of every duplicate instance a position report needs to include. - * @param landmarks - * @param shellTokens The unit-level shell token set from - * {@link ./shell-quorum.js | shellQuorum}, computed once per final cluster - * and shared across every member page's report. - * @param tokenizeOptions - * @example - * ```ts - * const shellTokens = shellQuorum(clusterPerPageInstances); - * const report = buildPageLandmarkReport(extractLandmarks(page.html), shellTokens); - * ``` - */ -export function buildPageLandmarkReport( - landmarks: ExtractLandmarksResult, - shellTokens: ReadonlySet, - tokenizeOptions?: TokenizeOptions, -): PageLandmarkReport { - const report: PageLandmarkReport = { - header: [], - footer: [], - nav: [], - aside: [], - form: [], - search: [], - main: landmarks.main.map(toPosition), - }; - - for (const type of ALL_LANDMARK_TYPES) { - for (const instance of landmarks[type]) { - const tokens = instance.html - ? new Set(tokenize(`${instance.html}`, tokenizeOptions).tokens) - : new Set(); - report[type].push({ - ...toPosition(instance), - isChrome: isChromeLandmarkInstance(tokens, shellTokens), - }); - } - } - - return report; -} diff --git a/packages/@d-zero/page-cluster/src/cli.spec.ts b/packages/@d-zero/page-cluster/src/cli.spec.ts index 78cb49f4..0f63b4e5 100644 --- a/packages/@d-zero/page-cluster/src/cli.spec.ts +++ b/packages/@d-zero/page-cluster/src/cli.spec.ts @@ -1,3 +1,6 @@ +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; import { Readable, Writable } from 'node:stream'; import { describe, expect, test } from 'vitest'; @@ -68,9 +71,15 @@ describe('parseArgs', () => { }); }); - test('--include-landmark-positions', () => { - expect(parseArgs(['--include-landmark-positions'])).toEqual({ - includeLandmarkPositions: true, + test('--cluster-reasons-file takes a value', () => { + expect(parseArgs(['--cluster-reasons-file', 'reasons.json'])).toEqual({ + clusterReasonsFile: 'reasons.json', + }); + }); + + test('--cluster-reasons-file without a value flags an error', () => { + expect(parseArgs(['--cluster-reasons-file'])).toEqual({ + unknownFlag: '--cluster-reasons-file requires a value', }); }); @@ -149,7 +158,7 @@ describe('runCli', () => { expect(stdout.read()).toMatch(/Usage:/); }); - test('--help mentions --include-landmark-positions', async () => { + test('--help mentions --cluster-reasons-file', async () => { const stdout = makeCollector(); const stderr = makeCollector(); await runCli({ @@ -159,7 +168,7 @@ describe('runCli', () => { argv: ['--help'], version: '0.0.0', }); - expect(stdout.read()).toMatch(/--include-landmark-positions/); + expect(stdout.read()).toMatch(/--cluster-reasons-file/); }); test('--version prints version and exits 0', async () => { @@ -310,7 +319,7 @@ describe('runCli', () => { ); }); - test('--include-landmark-positions adds a landmarks field to each output line', async () => { + test('--cluster-reasons-file writes a clusterKey-keyed ClusterReason object, output lines stay landmarks-free', async () => { const input = [ JSON.stringify({ id: 'a', @@ -325,31 +334,61 @@ describe('runCli', () => { html: '
H
two
', }), ].join('\n'); + const stdout = makeCollector(); + const stderr = makeCollector(); + const dir = await mkdtemp(path.join(tmpdir(), 'page-cluster-cli-')); + const reasonsFile = path.join(dir, 'reasons.json'); + try { + const code = await runCli({ + stdin: makeStdin(input), + stdout: stdout.stream, + stderr: stderr.stream, + argv: ['--cluster-reasons-file', reasonsFile], + version: '0.0.0', + }); + expect(code).toBe(0); + + const lines = stdout.read().split('\n').filter(Boolean); + expect(lines).toHaveLength(2); + const parsed = lines.map( + (line) => JSON.parse(line) as { id: string; clusterKey: string }, + ); + expect(parsed[0]).not.toHaveProperty('landmarks'); + expect(parsed[0]!.clusterKey).toBe(parsed[1]!.clusterKey); + + const reasonsByKey = JSON.parse(await readFile(reasonsFile, 'utf8')) as Record< + string, + { memberCount: number; landmarks: { header?: { chromeRate: number } } } + >; + const reason = reasonsByKey[parsed[0]!.clusterKey]; + expect(reason?.memberCount).toBe(2); + expect(reason?.landmarks.header?.chromeRate).toBe(1); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + test('--cluster-reasons-file write failure is reported as a clean exit-1 error, not an unhandled rejection', async () => { const stdout = makeCollector(); const stderr = makeCollector(); const code = await runCli({ - stdin: makeStdin(input), + stdin: makeStdin( + JSON.stringify({ id: 'a', html: '
H
' }), + ), stdout: stdout.stream, stderr: stderr.stream, - argv: ['--include-landmark-positions'], + // A path under a nonexistent directory always fails ENOENT on write. + argv: [ + '--cluster-reasons-file', + path.join(tmpdir(), 'page-cluster-nonexistent-dir', 'reasons.json'), + ], version: '0.0.0', }); - expect(code).toBe(0); - const lines = stdout.read().split('\n').filter(Boolean); - expect(lines).toHaveLength(2); - const parsed = lines.map( - (line) => - JSON.parse(line) as { - id: string; - clusterKey: string; - landmarks: { header: { isChrome: boolean }[]; main: object[] }; - }, - ); - expect(parsed[0]!.landmarks.header[0]!.isChrome).toBe(true); - expect(parsed[0]!.landmarks.main).toHaveLength(1); + expect(code).toBe(1); + expect(stripAnsi(stderr.read())).toMatch(/\[page-cluster\] error: /); }); - test('without --include-landmark-positions, output lines carry no landmarks field', async () => { + test('without --cluster-reasons-file, no reasons file is written and output lines carry no landmarks field', async () => { const stdout = makeCollector(); const stderr = makeCollector(); const code = await runCli({ diff --git a/packages/@d-zero/page-cluster/src/cli.ts b/packages/@d-zero/page-cluster/src/cli.ts index c6d5e1af..99d935c8 100644 --- a/packages/@d-zero/page-cluster/src/cli.ts +++ b/packages/@d-zero/page-cluster/src/cli.ts @@ -5,13 +5,14 @@ // and streams progress to stderr via `@d-zero/dealer`'s `Lanes` — in-place // animated header on a TTY, appended `[page-cluster] …` lines otherwise. +import type { ClusterReason } from './build-cluster-reason.js'; import type { - PageClusterKeyResult, PageClusterSignals, ProgressEvent, ResolvePageClusterKeysOptions, } from './resolve-page-cluster-keys.js'; +import { writeFile } from 'node:fs/promises'; import process from 'node:process'; import { Lanes } from '@d-zero/dealer'; @@ -24,14 +25,14 @@ import { resolvePageClusterKeys } from './resolve-page-cluster-keys.js'; */ type CliArgs = { readonly contentBlockAttribute?: string; - readonly includeLandmarkPositions?: boolean; + readonly clusterReasonsFile?: string; readonly help?: boolean; readonly version?: boolean; readonly unknownFlag?: string; }; const HELP_TEXT = `Usage: - page-cluster [--content-block-attribute ] [--include-landmark-positions] < pages.jsonl > clusters.jsonl + page-cluster [--content-block-attribute ] [--cluster-reasons-file ] < pages.jsonl > clusters.jsonl Input (JSONL, one page per line): { @@ -45,20 +46,27 @@ Input (JSONL, one page per line): Output (JSONL, one line per input page, in input order): { "id": "...", "clusterKey": "..." } - With --include-landmark-positions, each line additionally carries a - \`landmarks\` field: every header/footer/nav/aside/form/search/main - instance's position (1-based line/column plus string offsets), with the - six excisable types (all but main) also carrying an \`isChrome\` verdict - against that page's final cluster: + With --cluster-reasons-file , a separate JSON file is written once + processing completes: an object keyed by clusterKey, one entry per final + cluster (not per page — a ClusterReason is sized by cluster count, so + this file has no page-count limit, unlike a per-page report would). + Each ClusterReason reports the blocking evidence that grouped the + cluster (shared stylesheet set or URL path prefix), the shared DOM- + structural token core, per-landmark-type (header/footer/nav/aside/form/ + search) commonality within the cluster, and the sibling cluster keys it + was split from within the same blocking group — e.g. "header/footer are + common chrome across both clusters, but they differ in whether a + sidebar nav is present": { - "id": "...", "clusterKey": "...", - "landmarks": { - "header": [{ "startLine": 1, "startColumn": 7, "endLine": 1, - "endColumn": 30, "startOffset": 6, "endOffset": 29, - "isChrome": true }], - "footer": [...], "nav": [...], "aside": [...], "form": [...], "search": [...], - "main": [{ "startLine": 2, "startColumn": 1, "endLine": 10, - "endColumn": 8, "startOffset": 40, "endOffset": 120 }] + "[\\"path:news\\",\\"cluster:0\\"]": { + "memberCount": 42, + "blocking": [{ "blockKey": "path:news", "reason": { "kind": "path", "pathKey": "news" } }], + "structuralCoreTokens": ["body>main>article", "..."], + "landmarks": { + "header": { "presenceRate": 1, "chromeRate": 1, "shellTokens": ["..."], "memberCountWithInstance": 42 }, + "aside": { "presenceRate": 0.3, "chromeRate": 0, "shellTokens": [], "memberCountWithInstance": 13 } + }, + "siblingClusterKeys": ["[\\"path:news\\",\\"cluster:1\\"]"] } } @@ -66,14 +74,10 @@ Options: --content-block-attribute CMS-provided attribute marking freeform content blocks that should be stripped before comparison (e.g. \`data-bgb\`). - --include-landmark-positions Add the \`landmarks\` field described - above to every output line. Not - supported for corpora over 20,000 pages - (throws instead of streaming). Disables - progress output on stderr — this option - always routes through the same - non-progress-emitting code path as a - run without progress. + --cluster-reasons-file Write the per-cluster ClusterReason + object described above to after + processing completes. No page-count + limit. --help Print this help and exit. --version Print the package version and exit. @@ -111,7 +115,7 @@ const VERBOSE_HEADER = '[page-cluster]'; export function parseArgs(argv: readonly string[]): CliArgs { const out: { contentBlockAttribute?: string; - includeLandmarkPositions?: boolean; + clusterReasonsFile?: string; help?: boolean; version?: boolean; unknownFlag?: string; @@ -139,8 +143,14 @@ export function parseArgs(argv: readonly string[]): CliArgs { i++; break; } - case '--include-landmark-positions': { - out.includeLandmarkPositions = true; + case '--cluster-reasons-file': { + const next = argv[i + 1]; + if (next === undefined) { + out.unknownFlag = `${arg} requires a value`; + return out; + } + out.clusterReasonsFile = next; + i++; break; } default: { @@ -432,30 +442,26 @@ export async function runCli(options: { renderProgress(lanes, useTty, readingDoneLine(pages.length)); + // Only worth collecting when the caller asked for the file — a + // ClusterReason Map costs bookkeeping proportional to cluster count, + // not page count, but there's no reason to pay even that when unused. + const reasonsByClusterKey = args.clusterReasonsFile + ? new Map() + : undefined; + const resolveOptions: ResolvePageClusterKeysOptions = { contentBlockAttribute: args.contentBlockAttribute, onProgress: (event) => { renderProgress(lanes, useTty, formatProgressLine(event, elapsed())); }, + onClusterReason: reasonsByClusterKey + ? (key, reason) => reasonsByClusterKey.set(key, reason) + : undefined, }; - // `includeLandmarkPositions` always routes resolvePageClusterKeys - // through its non-progress-emitting sync path (see that option's own - // JSDoc), so the onProgress callback above is set but never invoked - // in this branch — no separate "quiet" resolveOptions variant needed. let clusterKeys: string[]; - let landmarksByIndex: PageClusterKeyResult['landmarks'][] | undefined; try { - if (args.includeLandmarkPositions) { - const results = await resolvePageClusterKeys(() => pages, { - ...resolveOptions, - includeLandmarkPositions: true, - }); - clusterKeys = results.map((r) => r.clusterKey); - landmarksByIndex = results.map((r) => r.landmarks); - } else { - clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions); - } + clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions); } catch (error) { renderProgress(lanes, useTty, errorLine((error as Error).message)); return 1; @@ -465,13 +471,22 @@ export async function runCli(options: { renderProgress(lanes, useTty, doneLine(pages.length, clusterCount, elapsed())); for (const [index, key] of clusterKeys.entries()) { - const row: { id: string | number; clusterKey: string; landmarks?: unknown } = { - id: ids[index] ?? index, - clusterKey: key, - }; - if (landmarksByIndex) row.landmarks = landmarksByIndex[index]; + const row = { id: ids[index] ?? index, clusterKey: key }; options.stdout.write(`${JSON.stringify(row)}\n`); } + + if (args.clusterReasonsFile && reasonsByClusterKey) { + try { + await writeFile( + args.clusterReasonsFile, + JSON.stringify(Object.fromEntries(reasonsByClusterKey), null, 2), + ); + } catch (error) { + renderProgress(lanes, useTty, errorLine((error as Error).message)); + return 1; + } + } + return 0; } finally { lanes.close(); diff --git a/packages/@d-zero/page-cluster/src/derive-blocking-reason.ts b/packages/@d-zero/page-cluster/src/derive-blocking-reason.ts new file mode 100644 index 00000000..f08283a7 --- /dev/null +++ b/packages/@d-zero/page-cluster/src/derive-blocking-reason.ts @@ -0,0 +1,35 @@ +/** + * The evidence behind one Pass-0 blocking key — which signal + * {@link ./resolve-blocking-group-keys.js | resolveBlockingGroupKeys} (and, + * for the `orphanMerge` variant, {@link ./reassign-orphan-block-keys.js | + * reassignOrphanBlockKeys}) actually used to decide it, carried verbatim with + * no added interpretation. Every page sharing a `css:` blocking key + * shares the exact same `distinctiveStylesheetHrefs` set by construction (the + * hash is derived from that set), so one `BlockingReason` per distinct + * blocking key is enough — it does not need to vary per page. + */ +export type BlockingReason = + | { + readonly kind: 'css'; + /** + * The sorted, deduplicated, first-party stylesheet hrefs left after + * corpus-wide chrome removal — the exact set + * {@link ./derive-stylesheet-group-key.js | deriveStylesheetGroupKey} + * hashed into this blocking key. + */ + readonly distinctiveStylesheetHrefs: readonly string[]; + } + | { + readonly kind: 'path'; + /** The `derivePathGroupKey` result this blocking key was derived from. */ + readonly pathKey: string; + } + | { + readonly kind: 'orphanMerge'; + /** + * The confined `path:` key this stylesheet-less page was folded into a + * same-section `css:` block under — see + * {@link ./reassign-orphan-block-keys.js | reassignOrphanBlockKeys}. + */ + readonly pathKey: string; + }; diff --git a/packages/@d-zero/page-cluster/src/extract-landmarks.spec.ts b/packages/@d-zero/page-cluster/src/extract-landmarks.spec.ts index d5814a8f..286ebebf 100644 --- a/packages/@d-zero/page-cluster/src/extract-landmarks.spec.ts +++ b/packages/@d-zero/page-cluster/src/extract-landmarks.spec.ts @@ -312,8 +312,8 @@ describe('extractLandmarks (main handling)', () => { test('main is absent from ALL_LANDMARK_TYPES-driven chrome discovery by construction (position-only)', () => { // extractLandmarks itself has no chrome/shell notion — this test only // pins that main instances are reported at all, alongside the other - // six, with no isChrome-like field (that lives in - // build-page-landmark-report.ts, a separate layer downstream). + // six, with no isChrome-like field (that classification is derived + // downstream, e.g. by build-cluster-reason.ts). const html = '
content
'; const result = extractLandmarks(html); expect(result.main).toHaveLength(1); diff --git a/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.spec.ts b/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.spec.ts index d834826e..1ad41e51 100644 --- a/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.spec.ts +++ b/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.spec.ts @@ -66,8 +66,10 @@ const landmarksWith = ( }; describe('mergeCrossBlockClusters', () => { - test('empty input returns empty map', () => { - expect(mergeCrossBlockClusters([], {})).toEqual(new Map()); + test('empty input returns empty maps', () => { + const result = mergeCrossBlockClusters([], {}); + expect(result.rootByKey).toEqual(new Map()); + expect(result.finalGroupsByRoot).toEqual(new Map()); }); test('single unit maps to itself', () => { @@ -77,7 +79,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit], {}); - expect(result.get('k1')).toBe('k1'); + expect(result.rootByKey.get('k1')).toBe('k1'); }); test('two units with identical token sets merge into one', () => { @@ -93,7 +95,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('k1')).toBe(result.get('k2')); + expect(result.rootByKey.get('k1')).toBe(result.rootByKey.get('k2')); }); test('two units with disjoint token sets stay separate', () => { @@ -112,8 +114,8 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('k1')).toBe('k1'); - expect(result.get('k2')).toBe('k2'); + expect(result.rootByKey.get('k1')).toBe('k1'); + expect(result.rootByKey.get('k2')).toBe('k2'); }); test('multi-page units with same shape but different class names merge via shape-Jaccard', () => { @@ -148,7 +150,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks, noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('reports')).toBe(result.get('projects')); + expect(result.rootByKey.get('reports')).toBe(result.rootByKey.get('projects')); }); test('single-page units are excluded from shape-Jaccard comparison', () => { @@ -169,8 +171,8 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('solo-a')).toBe('solo-a'); - expect(result.get('solo-b')).toBe('solo-b'); + expect(result.rootByKey.get('solo-a')).toBe('solo-a'); + expect(result.rootByKey.get('solo-b')).toBe('solo-b'); }); test('result map has an entry for every input unit key', () => { @@ -190,9 +192,9 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2, unit3], {}); - expect(result.has('a')).toBe(true); - expect(result.has('b')).toBe(true); - expect(result.has('c')).toBe(true); + expect(result.rootByKey.has('a')).toBe(true); + expect(result.rootByKey.has('b')).toBe(true); + expect(result.rootByKey.has('c')).toBe(true); }); test('result is deterministic: same input twice produces identical output', () => { @@ -209,8 +211,8 @@ describe('mergeCrossBlockClusters', () => { }; const r1 = mergeCrossBlockClusters([unit1, unit2], {}); const r2 = mergeCrossBlockClusters([unit1, unit2], {}); - expect(r1.get('k1')).toBe(r2.get('k1')); - expect(r1.get('k2')).toBe(r2.get('k2')); + expect(r1.rootByKey.get('k1')).toBe(r2.rootByKey.get('k1')); + expect(r1.rootByKey.get('k2')).toBe(r2.rootByKey.get('k2')); }); test('merged result preserves an existing unit key (no freshly invented key)', () => { @@ -226,7 +228,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - const rootKey = result.get('block-a')!; + const rootKey = result.rootByKey.get('block-a')!; expect(rootKey === 'block-a' || rootKey === 'block-b').toBe(true); }); @@ -247,7 +249,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([landmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('l1')).toBe(result.get('l2')); + expect(result.rootByKey.get('l1')).toBe(result.rootByKey.get('l2')); }); }); @@ -304,7 +306,7 @@ describe('mergeCrossBlockClusters shellQuorum (via mergeCrossBlockClusters exerc // have been at risk. The stronger correctness assertion is that // shellQuorum returns non-empty for these landmarks (exercised // implicitly by not crashing on the L2 shell lookup). - expect(result.get('A')).toBe(result.get('B')); + expect(result.rootByKey.get('A')).toBe(result.rootByKey.get('B')); }); test('landmark instances that vary per page do not falsely act as shell (histogram cuts them off)', () => { @@ -333,7 +335,7 @@ describe('mergeCrossBlockClusters shellQuorum (via mergeCrossBlockClusters exerc // separate. Precondition: shell histogram correctly filters out the // per-page varying headers rather than admitting them all via a // union fallback. - expect(result.get('A')).toBe('A'); - expect(result.get('B')).toBe('B'); + expect(result.rootByKey.get('A')).toBe('A'); + expect(result.rootByKey.get('B')).toBe('B'); }); }); diff --git a/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.ts b/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.ts index c01437ca..1444c38b 100644 --- a/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.ts +++ b/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.ts @@ -117,10 +117,16 @@ const GENERIC_SEGMENTS = new Set([ // --------------------------------------------------------------------------- /** - * + * Frequency-based token core of a group of member pages: a token must be + * present in at least `QUORUM_FRACTION` of members to enter the core, with a + * full-union fallback when no token clears that bar (see + * {@link mergeCrossBlockClusters}'s JSDoc for why quorum beats strict + * intersection or full union). Exported so callers building a `ClusterReason` + * (`build-cluster-reason.ts`) can re-derive a final group's structural core + * from `finalGroupsByRoot` without duplicating this logic. * @param memberDistinctiveTokens */ -function quorumCore( +export function computeQuorumCore( memberDistinctiveTokens: readonly ReadonlySet[], ): ReadonlySet { const n = memberDistinctiveTokens.length; @@ -204,11 +210,38 @@ function l2Contained(xSig: Map, ySig: Map): bool // Main function // --------------------------------------------------------------------------- +/** + * One root key's pooled member state after Stage B converges: every + * `tokenSets`/`landmarkInstances` entry folded in from every unit merged into + * this root (down-sampled to `capMembers` when the caller opts in, same as + * during merging). This is the exact state `mergeCrossBlockClusters` already + * builds internally to run quorum-core/shell comparisons each round — it was + * discarded once `keyToRoot` was returned. Exposing it lets a `ClusterReason` + * (`build-cluster-reason.ts`) be built from data Stage B already computed, + * with no extra pass over the corpus. + */ +export type FinalGroupMembers = { + readonly tokenSets: readonly ReadonlySet[]; + readonly landmarkInstances: readonly (readonly PerPageLandmarkInstance[])[]; +}; + +/** + * `mergeCrossBlockClusters`'s result: the root-key mapping every caller + * needs for `clusterKey` resolution, plus each surviving root's final pooled + * member state for callers that also want to explain *why* (`ClusterReason`). + */ +export type MergeCrossBlockClustersResult = { + /** Every input unit's `key` mapped to its final root key. Units not absorbed into any other unit map to themselves. */ + readonly rootByKey: ReadonlyMap; + /** Every surviving root key's final pooled member state. */ + readonly finalGroupsByRoot: ReadonlyMap; +}; + /** * Merges cross-block clusters (Stage B) via recursive quorum-core comparison. * - * Returns a `Map` from each input unit's `key` to its final root key. Units - * not absorbed into any other unit map to themselves. + * Returns the root-key mapping (see {@link MergeCrossBlockClustersResult}). + * Units not absorbed into any other unit map to themselves. * * Three merge mechanisms run per round, in order: * 1. **Fine stage** — complete-linkage at `CROSS_BLOCK_THRESHOLD` on quorum @@ -247,9 +280,17 @@ export function mergeCrossBlockClusters( */ capMembers?: number; }, -): Map { +): MergeCrossBlockClustersResult { if (units.length <= 1) { - return new Map(units.map((u) => [u.key, u.key])); + return { + rootByKey: new Map(units.map((u) => [u.key, u.key])), + finalGroupsByRoot: new Map( + units.map((u) => [ + u.key, + { tokenSets: u.memberTokenSets, landmarkInstances: u.memberLandmarkInstances }, + ]), + ), + }; } const threshold = options?.similarityThreshold ?? CROSS_BLOCK_THRESHOLD; @@ -334,7 +375,7 @@ export function mergeCrossBlockClusters( // Quorum core per group const cores = new Map>(); for (const key of groupKeys) { - cores.set(key, quorumCore(groupDistinctive.get(key) ?? [])); + cores.set(key, computeQuorumCore(groupDistinctive.get(key) ?? [])); } // --------------------------------------------------------------- @@ -533,5 +574,11 @@ export function mergeCrossBlockClusters( applyMerges(l2Merges); } - return keyToRoot; + const finalGroupsByRoot = new Map( + [...groups.entries()].map(([root, g]) => [ + root, + { tokenSets: g.tokenSets, landmarkInstances: g.landmarkInstances }, + ]), + ); + return { rootByKey: keyToRoot, finalGroupsByRoot }; } diff --git a/packages/@d-zero/page-cluster/src/pass0-blocking.spec.ts b/packages/@d-zero/page-cluster/src/pass0-blocking.spec.ts index 369ffc94..0ecc4b51 100644 --- a/packages/@d-zero/page-cluster/src/pass0-blocking.spec.ts +++ b/packages/@d-zero/page-cluster/src/pass0-blocking.spec.ts @@ -105,6 +105,69 @@ describe('resolveBlockKeys', () => { }); }); +describe('resolveBlockKeys (includeReasons)', () => { + test('css/path reasons pass through resolveBlockingGroupKeys unchanged when no orphan reassignment happens', () => { + const { blockKeys, reasonsByBlockKey } = resolveBlockKeys( + [ + { paths: ['news', '1'], stylesheetHrefs: ['https://example.com/a.css'] }, + { paths: ['news', '2'], stylesheetHrefs: ['https://example.com/a.css'] }, + { paths: ['about'], stylesheetHrefs: ['https://example.com/b.css'] }, + ], + { includeReasons: true }, + ); + expect(reasonsByBlockKey.get(blockKeys[0]!)).toEqual({ + kind: 'css', + distinctiveStylesheetHrefs: ['https://example.com/a.css'], + }); + expect(reasonsByBlockKey.get('path:about')).toEqual({ + kind: 'path', + pathKey: 'about', + }); + }); + + test('an orphan-merged key gets an orphanMerge reason carrying the confined pathKey', () => { + const pages = [ + { + paths: ['news', '1'], + stylesheetHrefs: ['https://example.com/a.css', 'https://example.com/shared.css'], + }, + { + paths: ['news', '2'], + stylesheetHrefs: ['https://example.com/a.css', 'https://example.com/shared.css'], + }, + { + paths: ['about'], + stylesheetHrefs: ['https://example.com/b.css', 'https://example.com/shared.css'], + }, + // Orphan (no stylesheets) in the news section — reassigned to the + // news css: block, which should carry an `orphanMerge` reason + // distinct from the plain `css` reason of a non-reassigned key. + { paths: ['news', '4'], stylesheetHrefs: [] }, + ]; + const { blockKeys, reasonsByBlockKey } = resolveBlockKeys(pages, { + includeReasons: true, + }); + expect(blockKeys[0]).toMatch(/^orphan-merge:/); + expect(reasonsByBlockKey.get(blockKeys[0]!)).toEqual({ + kind: 'orphanMerge', + pathKey: 'news', + }); + }); + + test('reasonsByBlockKey has exactly one entry per distinct block key', () => { + const { blockKeys, reasonsByBlockKey } = resolveBlockKeys( + [ + { paths: ['news', '1'], stylesheetHrefs: ['https://example.com/a.css'] }, + { paths: ['news', '2'], stylesheetHrefs: ['https://example.com/a.css'] }, + { paths: ['about'], stylesheetHrefs: ['https://example.com/b.css'] }, + ], + { includeReasons: true }, + ); + expect(blockKeys).toHaveLength(3); + expect(reasonsByBlockKey.size).toBe(2); + }); +}); + describe('groupIndicesByBlockKey', () => { test('groups indices by key, preserving first-seen order of keys and input order within each', () => { const groups = groupIndicesByBlockKey(['a', 'b', 'a', 'c', 'a', 'b']); diff --git a/packages/@d-zero/page-cluster/src/pass0-blocking.ts b/packages/@d-zero/page-cluster/src/pass0-blocking.ts index c2a43071..cb5474e7 100644 --- a/packages/@d-zero/page-cluster/src/pass0-blocking.ts +++ b/packages/@d-zero/page-cluster/src/pass0-blocking.ts @@ -1,7 +1,11 @@ +import type { BlockingReason } from './derive-blocking-reason.js'; import type { ResolveBlockingGroupKeysOptions } from './resolve-blocking-group-keys.js'; import { filterFirstPartyStylesheetHrefs } from './filter-first-party-stylesheet-hrefs.js'; -import { reassignOrphanBlockKeys } from './reassign-orphan-block-keys.js'; +import { + REASSIGNED_KEY_PREFIX, + reassignOrphanBlockKeys, +} from './reassign-orphan-block-keys.js'; import { resolveBlockingGroupKeys } from './resolve-blocking-group-keys.js'; /** @@ -38,6 +42,13 @@ export type ResolveBlockKeysOptions = ResolveBlockingGroupKeysOptions & { readonly restrictStylesheetsToFirstParty?: boolean; }; +/** Return shape when `includeReasons: true` is passed to `resolveBlockKeys`. */ +export type BlockKeysWithReasons = { + readonly blockKeys: string[]; + /** One entry per distinct final block key produced, keyed by that key. */ + readonly reasonsByBlockKey: ReadonlyMap; +}; + /** * Splits `resolvePageClusterKeys` into a size-flat first pass so the driver * can decide per-block memory strategy before loading any page HTML. Runs the @@ -78,19 +89,24 @@ export type ResolveBlockKeysOptions = ResolveBlockingGroupKeysOptions & { * // ['css:', 'css:', 'path:about'] * ``` */ +export function resolveBlockKeys( + pages: readonly Pass0PageSignals[], + options: ResolveBlockKeysOptions & { includeReasons: true }, +): BlockKeysWithReasons; export function resolveBlockKeys( pages: readonly Pass0PageSignals[], options?: ResolveBlockKeysOptions, -): string[] { +): string[]; +export function resolveBlockKeys( + pages: readonly Pass0PageSignals[], + options?: ResolveBlockKeysOptions & { includeReasons?: boolean }, +): string[] | BlockKeysWithReasons { const restrictStylesheetsToFirstParty = options?.restrictStylesheetsToFirstParty ?? true; const blockingPages = restrictStylesheetsToFirstParty ? filterFirstPartyStylesheetHrefs(pages) : pages; - const rawBlockKeys = resolveBlockingGroupKeys(blockingPages, options); - const reassignOrphans = options?.reassignOrphans ?? true; - if (!reassignOrphans) return rawBlockKeys; // Orphan reassignment always uses a numeric `pathDepth`. When the caller // asked for `'auto'`, fall back to the historical default 1 here — a // future PR that wires the auto-cut depth through can compute it once @@ -98,7 +114,36 @@ export function resolveBlockKeys( // call to keep them consistent. const numericPathDepth = typeof options?.pathDepth === 'number' ? options.pathDepth : undefined; - return reassignOrphanBlockKeys(blockingPages, rawBlockKeys, numericPathDepth); + const reassignOrphans = options?.reassignOrphans ?? true; + + if (!options?.includeReasons) { + const rawBlockKeys = resolveBlockingGroupKeys(blockingPages, options); + if (!reassignOrphans) return rawBlockKeys; + return reassignOrphanBlockKeys(blockingPages, rawBlockKeys, numericPathDepth); + } + + const { keys: rawBlockKeys, reasonsByKey } = resolveBlockingGroupKeys(blockingPages, { + ...options, + includeReasons: true, + }); + if (!reassignOrphans) { + return { blockKeys: rawBlockKeys, reasonsByBlockKey: reasonsByKey }; + } + + const finalBlockKeys = reassignOrphanBlockKeys( + blockingPages, + rawBlockKeys, + numericPathDepth, + ); + const reasonsByBlockKey = new Map(reasonsByKey); + for (const blockKey of finalBlockKeys) { + if (reasonsByBlockKey.has(blockKey)) continue; + reasonsByBlockKey.set(blockKey, { + kind: 'orphanMerge', + pathKey: blockKey.slice(REASSIGNED_KEY_PREFIX.length), + }); + } + return { blockKeys: finalBlockKeys, reasonsByBlockKey }; } /** diff --git a/packages/@d-zero/page-cluster/src/per-page-landmark-signatures.ts b/packages/@d-zero/page-cluster/src/per-page-landmark-signatures.ts index c0be4c95..b089588a 100644 --- a/packages/@d-zero/page-cluster/src/per-page-landmark-signatures.ts +++ b/packages/@d-zero/page-cluster/src/per-page-landmark-signatures.ts @@ -32,11 +32,10 @@ export const ALL_LANDMARK_TYPES: readonly LandmarkType[] = [ * * `position` is the instance's location in the page it came from, computed * once by {@link ./extract-landmarks.js | extractLandmarks} and carried here - * unchanged — a back-reference for callers (e.g. - * {@link ./build-page-landmark-report.js | buildPageLandmarkReport}) that - * need to report where a chrome-classified instance actually sits, not just - * that it exists. It plays no part in `tokens`/`signature` computation or in - * the corpus-frequency logic that consumes this type. + * unchanged — a back-reference for callers that need to report where a + * chrome-classified instance actually sits, not just that it exists. It + * plays no part in `tokens`/`signature` computation or in the corpus- + * frequency logic that consumes this type. */ export type PerPageLandmarkInstance = { readonly type: LandmarkType; diff --git a/packages/@d-zero/page-cluster/src/reassign-orphan-block-keys.ts b/packages/@d-zero/page-cluster/src/reassign-orphan-block-keys.ts index 87302fd1..4e6f7a8a 100644 --- a/packages/@d-zero/page-cluster/src/reassign-orphan-block-keys.ts +++ b/packages/@d-zero/page-cluster/src/reassign-orphan-block-keys.ts @@ -5,9 +5,12 @@ import { derivePathGroupKey } from './derive-path-group-key.js'; /** * Prefix distinguishing a reassigned key from the `css:`/`path:` keys * {@link ./resolve-blocking-group-keys.js | resolveBlockingGroupKeys} itself - * produces, so the two families can never collide. + * produces, so the two families can never collide. Exported so + * {@link ./pass0-blocking.js | resolveBlockKeys} can recover the confined + * path key back out of a reassigned block key when building `BlockingReason`s, + * without duplicating this literal. */ -const REASSIGNED_KEY_PREFIX = 'orphan-merge:'; +export const REASSIGNED_KEY_PREFIX = 'orphan-merge:'; /** * Reads `values[index]`, throwing instead of returning `undefined`. Every diff --git a/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.spec.ts b/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.spec.ts index df3d6782..b2050e57 100644 --- a/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.spec.ts +++ b/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.spec.ts @@ -157,3 +157,48 @@ describe('resolveBlockingGroupKeys', () => { ); }); }); + +describe('resolveBlockingGroupKeys (includeReasons)', () => { + const pages = [ + { + paths: ['dept-a', 'news', '1'], + stylesheetHrefs: ['https://example.com/a.css', 'https://example.com/common.css'], + }, + { + paths: ['dept-a', 'news', '2'], + stylesheetHrefs: ['https://example.com/a.css', 'https://example.com/common.css'], + }, + { paths: ['dept-b', 'about'], stylesheetHrefs: ['https://example.com/common.css'] }, + { paths: ['dept-c', 'contact'], stylesheetHrefs: ['https://example.com/common.css'] }, + ]; + + test('a css: key reason carries the exact distinctive hrefs that were hashed (common.css excluded)', () => { + const { keys, reasonsByKey } = resolveBlockingGroupKeys(pages, { + includeReasons: true, + }); + const reason = reasonsByKey.get(keys[0]!); + expect(reason).toEqual({ + kind: 'css', + distinctiveStylesheetHrefs: ['https://example.com/a.css'], + }); + }); + + test('a path: key reason carries the pathKey it fell back to', () => { + const { reasonsByKey } = resolveBlockingGroupKeys(pages, { includeReasons: true }); + expect(reasonsByKey.get('path:dept-b')).toEqual({ kind: 'path', pathKey: 'dept-b' }); + expect(reasonsByKey.get('path:dept-c')).toEqual({ kind: 'path', pathKey: 'dept-c' }); + }); + + test('reasonsByKey has exactly one entry per distinct key, not one per page', () => { + const { keys, reasonsByKey } = resolveBlockingGroupKeys(pages, { + includeReasons: true, + }); + expect(keys).toHaveLength(4); + expect(reasonsByKey.size).toBe(3); + }); + + test('without includeReasons, the return value is still a plain string[]', () => { + const result = resolveBlockingGroupKeys(pages); + expect(Array.isArray(result)).toBe(true); + }); +}); diff --git a/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.ts b/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.ts index 182f3991..6b2295f3 100644 --- a/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.ts +++ b/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.ts @@ -1,3 +1,5 @@ +import type { BlockingReason } from './derive-blocking-reason.js'; + import { computeDocumentFrequency } from './compute-document-frequency.js'; import { derivePathClusterKeys } from './derive-path-cluster-keys.js'; import { derivePathGroupKey } from './derive-path-group-key.js'; @@ -44,6 +46,13 @@ export type ResolveBlockingGroupKeysOptions = { hrefCommonThreshold?: number; }; +/** Return shape when `includeReasons: true` is passed to `resolveBlockingGroupKeys`. */ +export type BlockingGroupKeysWithReasons = { + readonly keys: string[]; + /** One entry per distinct blocking key produced, keyed by that key. */ + readonly reasonsByKey: ReadonlyMap; +}; + const DEFAULT_MIN_CSS_GROUP_SIZE = 2; /** @@ -129,10 +138,18 @@ const DEFAULT_MIN_CSS_GROUP_SIZE = 2; * // common.css is loaded by all 3 pages and is filtered out as non-discriminative chrome. * ``` */ +export function resolveBlockingGroupKeys( + pages: readonly PageBlockingSignals[], + options: ResolveBlockingGroupKeysOptions & { includeReasons: true }, +): BlockingGroupKeysWithReasons; export function resolveBlockingGroupKeys( pages: readonly PageBlockingSignals[], options?: ResolveBlockingGroupKeysOptions, -): string[] { +): string[]; +export function resolveBlockingGroupKeys( + pages: readonly PageBlockingSignals[], + options?: ResolveBlockingGroupKeysOptions & { includeReasons?: boolean }, +): string[] | BlockingGroupKeysWithReasons { const pathDepthOption = options?.pathDepth; const minCssGroupSize = options?.minCssGroupSize ?? DEFAULT_MIN_CSS_GROUP_SIZE; const hrefCommonThreshold = options?.hrefCommonThreshold; @@ -187,15 +204,29 @@ export function resolveBlockingGroupKeys( } } - return pages.map((page, index) => { + const reasonsByKey = new Map(); + const keys = pages.map((page, index) => { const cssKey = cssKeys[index]; if (cssKey !== undefined && (cssKeyCounts.get(cssKey) ?? 0) >= minCssGroupSize) { - return `css:${cssKey}`; + const key = `css:${cssKey}`; + if (!reasonsByKey.has(key)) { + reasonsByKey.set(key, { + kind: 'css', + distinctiveStylesheetHrefs: [...(distinctiveHrefs[index] ?? [])].toSorted(), + }); + } + return key; } const pathKey = perPagePathKeys === null ? derivePathGroupKey(page.paths, pathDepthOption as number | undefined) : (perPagePathKeys[index] ?? ''); - return `path:${pathKey}`; + const key = `path:${pathKey}`; + if (!reasonsByKey.has(key)) { + reasonsByKey.set(key, { kind: 'path', pathKey }); + } + return key; }); + + return options?.includeReasons ? { keys, reasonsByKey } : keys; } diff --git a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys-streaming.spec.ts b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys-streaming.spec.ts index a42e1fcd..68169c1b 100644 --- a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys-streaming.spec.ts +++ b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys-streaming.spec.ts @@ -1,3 +1,4 @@ +import type { ClusterReason } from './build-cluster-reason.js'; import type { PageClusterSignals, ProgressEvent } from './resolve-page-cluster-keys.js'; import { describe, expect, test } from 'vitest'; @@ -166,31 +167,43 @@ describe('resolvePageClusterKeys onProgress on small corpus', () => { }); }); -describe('resolvePageClusterKeys (includeLandmarkPositions)', () => { - test('small corpus with includeLandmarkPositions matches resolvePageClusterKeysInMemory', async () => { +describe('resolvePageClusterKeys (onClusterReason)', () => { + test('small corpus with onClusterReason produces the same keys and reasons as resolvePageClusterKeysInMemory', async () => { const pages = buildTinyCorpus(); + const streamedReasons = new Map(); const streamed = await resolvePageClusterKeys(() => pages, { - includeLandmarkPositions: true, + onClusterReason: (key, reason) => streamedReasons.set(key, reason), }); + const inMemoryReasons = new Map(); const inMemory = resolvePageClusterKeysInMemory(pages, { - includeLandmarkPositions: true, + onClusterReason: (key, reason) => inMemoryReasons.set(key, reason), }); expect(streamed).toEqual(inMemory); + expect(streamedReasons).toEqual(inMemoryReasons); + expect(streamedReasons.size).toBeGreaterThan(0); }); - test('combined with onProgress, still returns reports but emits no progress events (routed through the sync path)', async () => { + test('combined with onProgress, still fires onClusterReason but emits no progress events (routed through the sync path)', async () => { const pages = buildTinyCorpus(); const events: ProgressEvent[] = []; + const reasons = new Map(); const result = await resolvePageClusterKeys(() => pages, { - includeLandmarkPositions: true, + onClusterReason: (key, reason) => reasons.set(key, reason), onProgress: (event) => events.push(event), }); expect(result).toHaveLength(pages.length); - expect(result[0]).toHaveProperty('landmarks'); + expect(reasons.size).toBeGreaterThan(0); expect(events).toStrictEqual([]); }); - test('a corpus above CORPUS_INLINE_THRESHOLD rejects with a RangeError instead of streaming', async () => { + test('a corpus above CORPUS_INLINE_THRESHOLD completes on the streaming path and fires onClusterReason once per final cluster, without throwing', async () => { + // Regression guard for the fixed design flaw this replaces: the old + // `includeLandmarkPositions` retained every member page's landmark + // data until final clustering completed, which is why it had to + // reject corpora over this threshold outright. `ClusterReason` is + // sized by cluster count, not page count, so it has no such ceiling — + // this test's whole point is proving that by actually crossing the + // threshold instead of asserting a thrown error. const bigCount = CORPUS_INLINE_THRESHOLD + 1; /** * @yields {PageClusterSignals} A minimal page, `bigCount` times. @@ -200,8 +213,13 @@ describe('resolvePageClusterKeys (includeLandmarkPositions)', () => { yield { paths: ['p', String(i)], stylesheetHrefs: [], html: '' }; } } - await expect( - resolvePageClusterKeys(() => generate(), { includeLandmarkPositions: true }), - ).rejects.toThrow(RangeError); - }); + const reasons = new Map(); + const keys = await resolvePageClusterKeys(() => generate(), { + onClusterReason: (key, reason) => reasons.set(key, reason), + }); + expect(keys).toHaveLength(bigCount); + const distinctKeys = new Set(keys); + expect(reasons.size).toBe(distinctKeys.size); + for (const key of distinctKeys) expect(reasons.has(key)).toBe(true); + }, 30_000); }); diff --git a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.spec.ts b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.spec.ts index 566c22db..0d51c06f 100644 --- a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.spec.ts +++ b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.spec.ts @@ -1,7 +1,8 @@ +import type { ClusterReason } from './build-cluster-reason.js'; + import { describe, expect, test } from 'vitest'; import { - assertLandmarkPositionsSupportedForPageCount, resolvePageClusterKeysInMemory, resolvePageClusterKeysInMemory as resolvePageClusterKeys, } from './resolve-page-cluster-keys.js'; @@ -644,8 +645,39 @@ describe('resolvePageClusterKeys (local-landmark pseudo-token injection)', () => }); }); -describe('resolvePageClusterKeysInMemory (includeLandmarkPositions)', () => { - test('omitting includeLandmarkPositions returns a plain string[], unchanged from before', () => { +describe('resolvePageClusterKeysInMemory (onClusterReason)', () => { + test('two clusters split within the same block are reported as siblings of each other', () => { + // Same fixture as "pages within the same block but with dissimilar + // structure get different final keys" above: header/footer are + // identical (and excluded from comparison by default), but the + //
vs
distinguishing element still splits Stage A into + // two clusters within the single `path:dept-a` block. + const cardHtml = + '
H
C
F
'; + const formHtml = + '
H
F
F
'; + const reasons = new Map(); + const result = resolvePageClusterKeysInMemory( + [ + { paths: ['dept-a', 'page1'], stylesheetHrefs: [], html: cardHtml }, + { paths: ['dept-a', 'page2'], stylesheetHrefs: [], html: cardHtml }, + { paths: ['dept-a', 'page3'], stylesheetHrefs: [], html: formHtml }, + ], + { onClusterReason: (key, reason) => reasons.set(key, reason) }, + ); + + // Precondition: the two clusters really did split within one block. + expect(result[2]).not.toBe(result[0]); + expect(reasons.size).toBe(2); + + const cardReason = reasons.get(result[0]!)!; + const formReason = reasons.get(result[2]!)!; + expect(cardReason.siblingClusterKeys).toEqual([result[2]]); + expect(formReason.siblingClusterKeys).toEqual([result[0]]); + expect(cardReason.blocking).toEqual(formReason.blocking); + }); + + test('omitting onClusterReason returns a plain string[], unchanged from before', () => { const html = '
H
content
'; const result = resolvePageClusterKeysInMemory([ { paths: ['a'], stylesheetHrefs: [], html }, @@ -653,47 +685,47 @@ describe('resolvePageClusterKeysInMemory (includeLandmarkPositions)', () => { expect(result).toStrictEqual(['["path:a","cluster:0"]']); }); - test("a header shared across the whole cluster is reported as chrome; each page's own unique nav is not", () => { + test("a header shared across the whole cluster reads as chrome (chromeRate 1); each page's own unique nav does not", () => { const sharedHeader = '
'; const html1 = `${sharedHeader}
one
`; const html2 = `${sharedHeader}
two
`; + const reasons = new Map(); const result = resolvePageClusterKeysInMemory( [ { paths: ['p', '1'], stylesheetHrefs: [], html: html1 }, { paths: ['p', '2'], stylesheetHrefs: [], html: html2 }, ], - { includeLandmarkPositions: true }, + { onClusterReason: (key, reason) => reasons.set(key, reason) }, ); - expect(result).toHaveLength(2); // Both pages land in the same final cluster (identical header + same - // article structure), so shellQuorum is computed over both together. - expect(result[0]!.clusterKey).toBe(result[1]!.clusterKey); - expect(result[0]!.landmarks.header[0]!.isChrome).toBe(true); - expect(result[1]!.landmarks.header[0]!.isChrome).toBe(true); + // article structure), so one ClusterReason covers both. + expect(result[0]).toBe(result[1]); + expect(reasons.size).toBe(1); + const reason = reasons.get(result[0]!)!; + expect(reason.memberCount).toBe(2); + expect(reason.landmarks.header?.chromeRate).toBe(1); // Each page's nav carries a page-unique class not shared by the other // — content, not shell. - expect(result[0]!.landmarks.nav[0]!.isChrome).toBe(false); - expect(result[1]!.landmarks.nav[0]!.isChrome).toBe(false); + expect(reason.landmarks.nav?.chromeRate).toBe(0); }); - test('main instances are reported with position but no isChrome verdict', () => { + test('main never appears in ClusterReason.landmarks — chrome discovery does not classify content', () => { const html = '
H
content
'; - const result = resolvePageClusterKeysInMemory( - [{ paths: ['a'], stylesheetHrefs: [], html }], - { includeLandmarkPositions: true }, - ); - expect(result[0]!.landmarks.main).toHaveLength(1); - expect(Object.keys(result[0]!.landmarks.main[0]!)).not.toContain('isChrome'); + const reasons = new Map(); + resolvePageClusterKeysInMemory([{ paths: ['a'], stylesheetHrefs: [], html }], { + onClusterReason: (key, reason) => reasons.set(key, reason), + }); + expect(Object.keys([...reasons.values()][0]!.landmarks)).not.toContain('main'); }); - test('two distinct clusters each get their own shellQuorum, not a corpus-wide one', () => { + test('two distinct clusters each get their own shellQuorum-derived ClusterReason, not a corpus-wide one', () => { // Cluster A's header/nav tokens never appear anywhere in cluster B's // pages, and vice versa. If shellQuorum were mistakenly computed once // over all 6 pages instead of once per final cluster, each header's // corpus-wide frequency would be 3/6 = 0.5 — below the single-distinct- - // signature fallback clamp (0.8) — and neither would be chrome. Per- - // cluster computation gives each header a 3/3 = 1.0 frequency within - // its own cluster, so both are correctly chrome. + // signature fallback clamp (0.8) — and neither would read as chrome. + // Per-cluster computation gives each header a 3/3 = 1.0 frequency + // within its own cluster, so both correctly read as chrome. const headerA = '
'; const headerB = '
'; const clusterAPages = Array.from({ length: 3 }, (_, i) => ({ @@ -706,34 +738,24 @@ describe('resolvePageClusterKeysInMemory (includeLandmarkPositions)', () => { stylesheetHrefs: [], html: `${headerB}
content ${i}
`, })); + const reasons = new Map(); const result = resolvePageClusterKeysInMemory([...clusterAPages, ...clusterBPages], { - includeLandmarkPositions: true, + onClusterReason: (key, reason) => reasons.set(key, reason), }); // Precondition: the two groups really are in different final clusters // (different blocking path AND different tag skeleton: article vs // section) — otherwise this test wouldn't be exercising the // per-cluster grouping at all. - const clusterKeys = new Set(result.map((r) => r.clusterKey)); - expect(clusterKeys.size).toBe(2); - expect(result[0]!.clusterKey).toBe(result[1]!.clusterKey); - expect(result[0]!.clusterKey).toBe(result[2]!.clusterKey); - expect(result[3]!.clusterKey).toBe(result[4]!.clusterKey); - expect(result[3]!.clusterKey).toBe(result[5]!.clusterKey); - - for (const r of result) { - expect(r.landmarks.header[0]!.isChrome).toBe(true); - } - }); -}); - -describe('assertLandmarkPositionsSupportedForPageCount', () => { - test('does not throw when pageCount is at or below threshold', () => { - expect(() => assertLandmarkPositionsSupportedForPageCount(5, 5)).not.toThrow(); - expect(() => assertLandmarkPositionsSupportedForPageCount(4, 5)).not.toThrow(); - }); + expect(reasons.size).toBe(2); + expect(result[0]).toBe(result[1]); + expect(result[0]).toBe(result[2]); + expect(result[3]).toBe(result[4]); + expect(result[3]).toBe(result[5]); - test('throws a RangeError when pageCount exceeds threshold', () => { - expect(() => assertLandmarkPositionsSupportedForPageCount(6, 5)).toThrow(RangeError); + for (const reason of reasons.values()) { + expect(reason.landmarks.header?.chromeRate).toBe(1); + expect(reason.memberCount).toBe(3); + } }); }); diff --git a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.ts b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.ts index 1b7c1f45..3a5af04e 100644 --- a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.ts +++ b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.ts @@ -1,15 +1,14 @@ +import type { ClusterReason } from './build-cluster-reason.js'; +import type { BlockingReason } from './derive-blocking-reason.js'; import type { ExtractLandmarksResult } from './extract-landmarks.js'; -import type { CrossBlockUnit } from './merge-cross-block-clusters.js'; +import type { CrossBlockUnit, FinalGroupMembers } from './merge-cross-block-clusters.js'; import type { PerPageLandmarkInstance } from './per-page-landmark-signatures.js'; import type { ResolveBlockingGroupKeysOptions } from './resolve-blocking-group-keys.js'; import type { ResolveStructuralClusterKeysOptions } from './resolve-structural-cluster-keys.js'; import type { TokenizeOptions } from './types.js'; import { autoCutThreshold } from './auto-cut-threshold.js'; -import { - type PageLandmarkReport, - buildPageLandmarkReport, -} from './build-page-landmark-report.js'; +import { buildClusterReason } from './build-cluster-reason.js'; import { capContentDepth } from './cap-content-depth.js'; import { detectContentDepthCap, @@ -22,7 +21,6 @@ import { mergeCrossBlockClusters } from './merge-cross-block-clusters.js'; import { groupIndicesByBlockKey, resolveBlockKeys } from './pass0-blocking.js'; import { computePerPageLandmarkInstances } from './per-page-landmark-signatures.js'; import { removeContentBlocks } from './remove-content-blocks.js'; -import { shellQuorum } from './shell-quorum.js'; import { stageAPerBlock } from './stage-a-per-block.js'; import { tokenize } from './tokenize.js'; @@ -287,41 +285,70 @@ export function computeLocalLandmarkTokens( } /** - * Builds every page's {@link PageLandmarkReport} for the `includeLandmarkPositions` - * result path. Groups pages by their *final* cluster key (post–Stage-B), runs - * {@link ./shell-quorum.js | shellQuorum} once per cluster over the pooled - * member `PerPageLandmarkInstance`s (the same "shell tokens" concept Stage B - * itself uses for L2 shell corroboration, just recomputed at the final- - * cluster granularity rather than the pre-merge unit granularity), then - * classifies each member page's own landmark instances against that - * cluster-level shell. - * @param finalKeys - * @param landmarks - * @param perPageInstances + * Builds and emits one {@link ClusterReason} per final cluster via + * `onClusterReason`, from data Stage A/B already computed for clustering + * itself: `crossBlockUnits` (Stage A's pre-merge units, each carrying its + * originating block key inside `JSON.parse(unit.key)[0]`), Stage B's + * `rootByKey`/`finalGroupsByRoot`, the per-block-key `BlockingReason`s Pass 0 + * derived, and the per-block sibling-unit-key lists the driver accumulated + * alongside its Stage A loop. No re-tokenization and no extra corpus pass — + * this only re-groups references the driver already held. + * @param crossBlockUnits + * @param rootByKey + * @param finalGroupsByRoot + * @param reasonsByBlockKey + * @param siblingUnitKeysByBlock + * @param onClusterReason */ -function buildLandmarkReportsByCluster( - finalKeys: readonly string[], - landmarks: readonly ExtractLandmarksResult[], - perPageInstances: readonly (readonly PerPageLandmarkInstance[])[], -): PageLandmarkReport[] { - const indicesByKey = new Map(); - for (const [i, key] of finalKeys.entries()) { - const indices = indicesByKey.get(key); - if (indices) { - indices.push(i); +function emitClusterReasons( + crossBlockUnits: readonly CrossBlockUnit[], + rootByKey: ReadonlyMap, + finalGroupsByRoot: ReadonlyMap, + reasonsByBlockKey: ReadonlyMap, + siblingUnitKeysByBlock: ReadonlyMap, + onClusterReason: (clusterKey: string, reason: ClusterReason) => void, +): void { + const unitKeysByRoot = new Map(); + for (const unit of crossBlockUnits) { + const root = rootByKey.get(unit.key) ?? unit.key; + const list = unitKeysByRoot.get(root); + if (list) { + list.push(unit.key); } else { - indicesByKey.set(key, [i]); + unitKeysByRoot.set(root, [unit.key]); } } - const reports: PageLandmarkReport[] = Array.from({ length: finalKeys.length }); - for (const indices of indicesByKey.values()) { - const shellTokens = shellQuorum(indices.map((i) => perPageInstances[i]!)); - for (const i of indices) { - reports[i] = buildPageLandmarkReport(landmarks[i]!, shellTokens); + for (const [rootKey, unitKeys] of unitKeysByRoot) { + const finalGroup = finalGroupsByRoot.get(rootKey); + if (!finalGroup) continue; + + const seenBlockKeys = new Set(); + const blocking: { blockKey: string; reason: BlockingReason }[] = []; + const siblingRoots = new Set(); + for (const unitKey of unitKeys) { + const blockKey = JSON.parse(unitKey)[0] as string; + if (!seenBlockKeys.has(blockKey)) { + seenBlockKeys.add(blockKey); + const reason = reasonsByBlockKey.get(blockKey); + if (reason) blocking.push({ blockKey, reason }); + } + for (const siblingUnitKey of siblingUnitKeysByBlock.get(blockKey) ?? []) { + const siblingRoot = rootByKey.get(siblingUnitKey) ?? siblingUnitKey; + if (siblingRoot !== rootKey) siblingRoots.add(siblingRoot); + } } + + onClusterReason( + rootKey, + buildClusterReason({ + tokenSets: finalGroup.tokenSets, + landmarkInstances: finalGroup.landmarkInstances, + blocking, + siblingClusterKeys: [...siblingRoots].toSorted(), + }), + ); } - return reports; } /** @@ -408,50 +435,44 @@ export type ResolvePageClusterKeysOptions = TokenizeOptions & * sync helper to running a per-block async loop that emits * `pass1-block-complete` and `stage-b-start`. Omitting `onProgress` * keeps the small-corpus branch on the pre-refactor sync path with - * zero yield overhead. Ignored when `includeLandmarkPositions` is - * `true` — that option always routes the small-corpus branch through - * the sync helper (see `includeLandmarkPositions`'s own JSDoc), so no - * progress events fire in that combination. + * zero yield overhead. Ignored when `onClusterReason` is set — that + * option always routes the small-corpus branch through the sync + * helper (see `onClusterReason`'s own JSDoc), so no progress events + * fire in that combination. */ onProgress?: (event: ProgressEvent) => void; /** - * When `true`, every result entry additionally carries a - * {@link PageLandmarkReport} — each landmark instance's position, plus - * a chrome/content verdict for the six excisable types (see - * {@link ./build-page-landmark-report.js | buildPageLandmarkReport}). - * Changes the return shape from `string[]` to - * `{@link PageClusterKeyResult}[]` (see the overloads on - * `resolvePageClusterKeysInMemory`/`resolvePageClusterKeys`/ - * `resolvePageClusterKeysFromArray`). Defaults to `false`, in which - * case every existing caller's behavior and return type are - * unchanged. + * Optional observability hook invoked once per **final cluster** (not + * per page) with that cluster's {@link ClusterReason} — the blocking + * signal that grouped it, its DOM-structural token core, its + * per-landmark-type commonality, and the sibling cluster keys it was + * split from within the same Pass-0 block. Unlike `onProgress`, this + * fires on every path — small-corpus and streaming alike — because a + * `ClusterReason` is sized by cluster count, not page count, so it + * carries no streaming-path memory risk the way a per-page report + * would. * - * Not supported on the streaming path - * (`pageCount > {@link CORPUS_INLINE_THRESHOLD}`): reservoir sampling - * and Jaccard-based non-sample assignment there have no notion of - * "this page's shell tokens" to classify chrome against, and - * retrofitting one is out of scope. `resolvePageClusterKeys` throws a - * `RangeError` up front if both apply to the same call, rather than - * silently degrading semantics. + * Building the reasons re-uses Stage A/B's own intermediate state (the + * quorum core, the per-unit landmark instances, the blocking + * evidence) — it does not re-tokenize pages or re-run corpus-wide + * discovery. Omitting `onClusterReason` skips that bookkeeping + * entirely, so existing callers pay nothing for this option. * - * On the async factory-based `resolvePageClusterKeys`, this option + * On the async factory-based `resolvePageClusterKeys`, setting this * forces the small-corpus branch through the sync * `resolvePageClusterKeysInMemory` helper regardless of `onProgress` * — see `onProgress`'s own JSDoc. + * @example + * ```ts + * const reasons = new Map(); + * const keys = await resolvePageClusterKeys(pages, { + * onClusterReason: (key, reason) => reasons.set(key, reason), + * }); + * ``` */ - includeLandmarkPositions?: boolean; + onClusterReason?: (clusterKey: string, reason: ClusterReason) => void; }; -/** - * One page's clustering result when `includeLandmarkPositions` is `true`: - * the same `clusterKey` every caller already gets, plus that page's - * {@link PageLandmarkReport}. - */ -export type PageClusterKeyResult = { - readonly clusterKey: string; - readonly landmarks: PageLandmarkReport; -}; - /** * Corpus size at or below which the async factory-based * `resolvePageClusterKeys` reads the entire input into an array and delegates @@ -475,27 +496,6 @@ export type PageClusterKeyResult = { */ export const CORPUS_INLINE_THRESHOLD = 20_000; -/** - * Throws when `includeLandmarkPositions` is combined with a corpus over - * `threshold` pages (the streaming path — see `includeLandmarkPositions`'s - * own JSDoc for why it has no sample-based equivalent there). Split out from - * its call site so tests can exercise the boundary with a small injected - * `threshold` instead of constructing a 20,001-page fixture to cross the - * real {@link CORPUS_INLINE_THRESHOLD}. - * @param pageCount - * @param threshold - */ -export function assertLandmarkPositionsSupportedForPageCount( - pageCount: number, - threshold: number, -): void { - if (pageCount > threshold) { - throw new RangeError( - `resolvePageClusterKeys: includeLandmarkPositions is not supported for corpora larger than ${threshold} pages (the streaming path) — this corpus has ${pageCount}`, - ); - } -} - /** * Reservoir-sample size per block on the streaming path. Blocks larger than * this have Stage A run on a random sample of `BLOCK_SAMPLE_SIZE` pages, @@ -549,18 +549,10 @@ export const BLOCK_SAMPLE_SIZE = 100; * @param pages * @param options */ -export function resolvePageClusterKeysInMemory( - pages: readonly PageClusterSignals[], - options: ResolvePageClusterKeysOptions & { includeLandmarkPositions: true }, -): PageClusterKeyResult[]; -export function resolvePageClusterKeysInMemory( - pages: readonly PageClusterSignals[], - options?: ResolvePageClusterKeysOptions, -): string[]; export function resolvePageClusterKeysInMemory( pages: readonly PageClusterSignals[], options?: ResolvePageClusterKeysOptions, -): string[] | PageClusterKeyResult[] { +): string[] { const excludeLandmarks = options?.excludeLandmarks ?? true; const similarityThreshold = options?.similarityThreshold ?? 0.8; @@ -571,18 +563,15 @@ export function resolvePageClusterKeysInMemory( } // Always computed: landmark fields are needed by Stage B's shell - // corroboration regardless of `excludeLandmarks`, and `remainderHtml` is - // needed whenever `excludeLandmarks` is true. + // corroboration regardless of `excludeLandmarks`. const landmarks: readonly ExtractLandmarksResult[] = pages.map((page) => extractLandmarks(page.html), ); - // Corpus-level chrome discovery. `perPageInstances` is only consumed - // below when `includeLandmarkPositions` is set — computed unconditionally - // anyway since `computeLocalChromeArtifacts` already builds it internally - // for chrome discovery, so exposing it here costs nothing extra. - const { localTokensByPage: localLandmarkTokensByPage, perPageInstances } = - computeLocalChromeArtifacts(landmarks, options); + const { localTokensByPage: localLandmarkTokensByPage } = computeLocalChromeArtifacts( + landmarks, + options, + ); const contentBlockAttribute = options?.contentBlockAttribute; const preparedHtml = pages.map((page, index) => { @@ -601,7 +590,19 @@ export function resolvePageClusterKeysInMemory( ? filterFirstPartyStylesheetHrefs(pages) : pages; - const blockKeys = resolveBlockKeys(blockingPages, options); + // Reasons (blocking evidence) are only worth deriving when a caller + // actually asked for `onClusterReason` — see that option's own JSDoc for + // why this is the only place ClusterReason bookkeeping is opt-in. + const onClusterReason = options?.onClusterReason; + let blockKeys: string[]; + let reasonsByBlockKey: ReadonlyMap | undefined; + if (onClusterReason) { + const result = resolveBlockKeys(blockingPages, { ...options, includeReasons: true }); + blockKeys = result.blockKeys; + reasonsByBlockKey = result.reasonsByBlockKey; + } else { + blockKeys = resolveBlockKeys(blockingPages, options); + } const indicesByBlockKey = groupIndicesByBlockKey(blockKeys); // Validated here, eagerly, because it's otherwise only reached from @@ -612,6 +613,9 @@ export function resolvePageClusterKeysInMemory( const finalKeys: string[] = Array.from({ length: pages.length }); const crossBlockUnits: CrossBlockUnit[] = []; + const siblingUnitKeysByBlock = onClusterReason + ? new Map() + : undefined; for (const [blockKey, indices] of indicesByBlockKey) { const result = stageAPerBlock( @@ -628,22 +632,37 @@ export function resolvePageClusterKeysInMemory( finalKeys[pageIndex] = key; } crossBlockUnits.push(...result.crossBlockUnits); + siblingUnitKeysByBlock?.set( + blockKey, + result.crossBlockUnits.map((u) => u.key), + ); } // Stage B: cross-block merge — always runs regardless of options - const stageBResult = mergeCrossBlockClusters(crossBlockUnits, options); + const { rootByKey, finalGroupsByRoot } = mergeCrossBlockClusters( + crossBlockUnits, + options, + ); for (let i = 0; i < finalKeys.length; i++) { const currentKey = finalKeys[i]!; - const rootKey = stageBResult.get(currentKey); + const rootKey = rootByKey.get(currentKey); if (rootKey !== undefined && rootKey !== currentKey) { finalKeys[i] = rootKey; } } - if (!options?.includeLandmarkPositions) return finalKeys; + if (onClusterReason && reasonsByBlockKey && siblingUnitKeysByBlock) { + emitClusterReasons( + crossBlockUnits, + rootByKey, + finalGroupsByRoot, + reasonsByBlockKey, + siblingUnitKeysByBlock, + onClusterReason, + ); + } - const reports = buildLandmarkReportsByCluster(finalKeys, landmarks, perPageInstances); - return finalKeys.map((clusterKey, i) => ({ clusterKey, landmarks: reports[i]! })); + return finalKeys; } /** @@ -746,10 +765,10 @@ async function resolveSmallCorpusWithProgress( onProgress({ phase: 'stage-b-start', unitCount: crossBlockUnits.length }); - const stageBResult = mergeCrossBlockClusters(crossBlockUnits, options); + const { rootByKey } = mergeCrossBlockClusters(crossBlockUnits, options); for (let i = 0; i < finalKeys.length; i++) { const currentKey = finalKeys[i]!; - const rootKey = stageBResult.get(currentKey); + const rootKey = rootByKey.get(currentKey); if (rootKey !== undefined && rootKey !== currentKey) { finalKeys[i] = rootKey; } @@ -821,18 +840,10 @@ export type PageFactory = () => * }); * ``` */ -export async function resolvePageClusterKeys( - pages: PageFactory, - options: ResolvePageClusterKeysOptions & { includeLandmarkPositions: true }, -): Promise; -export async function resolvePageClusterKeys( - pages: PageFactory, - options?: ResolvePageClusterKeysOptions, -): Promise; export async function resolvePageClusterKeys( pages: PageFactory, options?: ResolvePageClusterKeysOptions, -): Promise { +): Promise { const onProgress = options?.onProgress; // Pass 0: HTML-free — collect blocking signals (paths, stylesheetHrefs, // host) into an array. This is the only per-page state we keep across @@ -872,27 +883,18 @@ export async function resolvePageClusterKeys( // into per-block progress, so delegate to the untouched sync path — // keeping behavior byte-for-byte identical (and yield-overhead-free) // to how library-only consumers experienced this before the CLI - // progress work landed. `includeLandmarkPositions` always routes here - // too (see its own JSDoc): `resolveSmallCorpusWithProgress` has no - // landmark-report support, and duplicating that logic into the + // progress work landed. `onClusterReason` always routes here too (see + // its own JSDoc): `resolveSmallCorpusWithProgress` has no + // cluster-reason support, and duplicating that logic into the // progress-emitting path for a reporting feature that has nothing to // do with progress observability isn't worth the added surface. - if (onProgress === undefined || options?.includeLandmarkPositions) { + if (onProgress === undefined || options?.onClusterReason) { return resolvePageClusterKeysInMemory(fullPages, options); } return resolveSmallCorpusWithProgress(fullPages, onProgress, options); } - // Large corpus: streaming path. includeLandmarkPositions has no sample- - // based equivalent (see its own JSDoc) — fail fast rather than silently - // ignoring the option or returning a semantically-wrong report. - if (options?.includeLandmarkPositions) { - assertLandmarkPositionsSupportedForPageCount( - blockingSignals.length, - CORPUS_INLINE_THRESHOLD, - ); - } - + // Large corpus: streaming path. const excludeLandmarks = options?.excludeLandmarks ?? true; const similarityThreshold = options?.similarityThreshold ?? 0.8; if (!(similarityThreshold >= 0 && similarityThreshold <= 1)) { @@ -907,12 +909,31 @@ export async function resolvePageClusterKeys( const blockingPagesForKeys = restrictStylesheetsToFirstParty ? filterFirstPartyStylesheetHrefs(blockingSignals) : blockingSignals; - const blockKeys = resolveBlockKeys(blockingPagesForKeys, options); + + // Reasons (blocking evidence) cost nothing beyond bookkeeping — a Map + // keyed by distinct block key, not by page — but are only derived when a + // caller actually asked for `onClusterReason`. + const onClusterReason = options?.onClusterReason; + let blockKeys: string[]; + let reasonsByBlockKey: ReadonlyMap | undefined; + if (onClusterReason) { + const result = resolveBlockKeys(blockingPagesForKeys, { + ...options, + includeReasons: true, + }); + blockKeys = result.blockKeys; + reasonsByBlockKey = result.reasonsByBlockKey; + } else { + blockKeys = resolveBlockKeys(blockingPagesForKeys, options); + } const indicesByBlockKey = groupIndicesByBlockKey(blockKeys); const finalKeys: string[] = Array.from({ length: blockingSignals.length }); const crossBlockUnits: CrossBlockUnit[] = []; const contentBlockAttribute = options?.contentBlockAttribute; + const siblingUnitKeysByBlock = onClusterReason + ? new Map() + : undefined; /** * Per-block bucket accumulates a reservoir sample of the block's pages @@ -989,6 +1010,10 @@ export async function resolvePageClusterKeys( finalKeys[idx] = key; } crossBlockUnits.push(...result.crossBlockUnits); + siblingUnitKeysByBlock?.set( + bucket.blockKey, + result.crossBlockUnits.map((u) => u.key), + ); if (bucket.seenCount > bucket.reservoirIndices.length) { // Save assignment artifacts for Pass 1b. @@ -1134,17 +1159,28 @@ export async function resolvePageClusterKeys( if (onProgress) { onProgress({ phase: 'stage-b-start', unitCount: crossBlockUnits.length }); } - const stageBResult = mergeCrossBlockClusters(crossBlockUnits, { + const { rootByKey, finalGroupsByRoot } = mergeCrossBlockClusters(crossBlockUnits, { ...options, capMembers: BLOCK_SAMPLE_SIZE, }); for (let i = 0; i < finalKeys.length; i++) { const currentKey = finalKeys[i]!; - const rootKey = stageBResult.get(currentKey); + const rootKey = rootByKey.get(currentKey); if (rootKey !== undefined && rootKey !== currentKey) { finalKeys[i] = rootKey; } } + + if (onClusterReason && reasonsByBlockKey && siblingUnitKeysByBlock) { + emitClusterReasons( + crossBlockUnits, + rootByKey, + finalGroupsByRoot, + reasonsByBlockKey, + siblingUnitKeysByBlock, + onClusterReason, + ); + } return finalKeys; } @@ -1164,17 +1200,9 @@ export async function resolvePageClusterKeys( * ]); * ``` */ -export function resolvePageClusterKeysFromArray( - pages: readonly PageClusterSignals[], - options: ResolvePageClusterKeysOptions & { includeLandmarkPositions: true }, -): Promise; export function resolvePageClusterKeysFromArray( pages: readonly PageClusterSignals[], options?: ResolvePageClusterKeysOptions, -): Promise; -export function resolvePageClusterKeysFromArray( - pages: readonly PageClusterSignals[], - options?: ResolvePageClusterKeysOptions, -): Promise { +): Promise { return resolvePageClusterKeys(() => pages, options); } diff --git a/packages/@d-zero/page-cluster/src/shell-quorum.ts b/packages/@d-zero/page-cluster/src/shell-quorum.ts index e62b8899..367efcde 100644 --- a/packages/@d-zero/page-cluster/src/shell-quorum.ts +++ b/packages/@d-zero/page-cluster/src/shell-quorum.ts @@ -74,10 +74,10 @@ const SHELL_QUORUM_FALLBACK_FRACTION = 0.8; * * Originally private to {@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters}'s * Stage B L2 corroboration; exported from its own module so - * {@link ./resolve-page-cluster-keys.js | resolvePageClusterKeysInMemory}'s - * `includeLandmarkPositions` reporting path can run it once per final - * cluster to classify individual landmark instances as chrome (see - * {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance}). + * {@link ./build-cluster-reason.js | buildClusterReason} can run it once per + * final cluster, per landmark type, to classify individual landmark + * instances as chrome (see {@link ./is-chrome-landmark-instance.js | + * isChromeLandmarkInstance}). * @param perPageInstances */ export function shellQuorum( From fad3d24d62e5a083c24950521825bb4f774906db Mon Sep 17 00:00:00 2001 From: Yusuke Hirao Date: Sat, 25 Jul 2026 01:22:51 +0900 Subject: [PATCH 2/2] docs(page-cluster): clarify chromeRate's per-page dedup denominator Verifying ClusterReason against a real 1,416-page crawl by independently reconstructing chromeRate via the public extractLandmarks + tokenize + isChromeLandmarkInstance recipe surfaced one apparent mismatch. It traced to the verification script skipping computePerPageLandmarkInstances' pre-existing per-page, cross-type signature dedup (the same rule that already prevents a CMS-duplicated footer, or a
matching both header and nav, from double-counting) - not a bug in the library. Classifying one instance you already have is unaffected either way; only reproducing the aggregate ratio by counting raw, un-deduplicated instances needs to know this. --- .../@d-zero/page-cluster/src/build-cluster-reason.ts | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/packages/@d-zero/page-cluster/src/build-cluster-reason.ts b/packages/@d-zero/page-cluster/src/build-cluster-reason.ts index d26d4af4..05a71c7b 100644 --- a/packages/@d-zero/page-cluster/src/build-cluster-reason.ts +++ b/packages/@d-zero/page-cluster/src/build-cluster-reason.ts @@ -25,6 +25,18 @@ export type LandmarkClusterProfile = { * {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance} * classified as chrome against this cluster's own shell for the type — * the "header/footer is shared chrome" half of the user-facing story. + * + * The denominator is per-page-deduplicated instances (same rule + * {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances} + * always applies: two instances on the same page that tokenize to the + * same signature — a CMS-duplicated footer, or a `
` matching both `header` and `nav` at the identical + * span — count once). A caller reconstructing a single instance's chrome + * verdict via `extractLandmarks` + `tokenize` + + * `isChromeLandmarkInstance` classifies that one instance correctly + * regardless of this rule; only a caller trying to reproduce this exact + * ratio by counting raw, un-deduplicated instances across a page would + * see a different number. */ readonly chromeRate: number; /**