diff --git a/packages/@d-zero/page-cluster/README.md b/packages/@d-zero/page-cluster/README.md index 9f2d1198..3069cfe9 100644 --- a/packages/@d-zero/page-cluster/README.md +++ b/packages/@d-zero/page-cluster/README.md @@ -15,7 +15,7 @@ yarn add @d-zero/page-cluster ### CLI ```sh -page-cluster [--content-block-attribute ] [--include-landmark-positions] < pages.jsonl > clusters.jsonl +page-cluster [--content-block-attribute ] [--cluster-reasons-file ] < pages.jsonl > clusters.jsonl ``` **入力**: JSONL 1 行 1 ページ。フィールドは以下。`html` 以外はすべて任意（`paths` / `stylesheetHrefs` がないと粗い分類になる）。 @@ -36,44 +36,38 @@ page-cluster [--content-block-attribute ] [--include-landmark-positions] < { "id": "任意の識別子", "clusterKey": "..." } ``` -`--include-landmark-positions` を指定すると、各行に `landmarks` フィールドが追加される。header / footer / nav / aside / form / search / main のインスタンスごとに、HTML 内の位置（1-based の line/column と文字列オフセットの両方）を返す。header 〜 search の 6 種は追加で、そのページが属する最終クラスタ内での頻度分析（`shellQuorum`）に基づく `isChrome`（サイト/セクション共通の chrome か、ページ固有のコンテンツか）を持つ。`main` は常にコンテンツなので `isChrome` を持たない。 +`--cluster-reasons-file ` を指定すると、処理完了後に別ファイルとして「クラスタ選定理由」を書き出す。ページ単位ではなく**クラスタ単位**（`clusterKey` をキーにしたオブジェクト、1 クラスタにつき 1 エントリ）なので、ファイルサイズはページ数ではなくクラスタ数に比例する — ページ単位のレポートと違い、コーパスサイズの上限はない。各エントリは「なぜこのページ達が同じクラスタになったか」の根拠を構造化データとして返す: ブロッキング理由（共有 stylesheet 集合 or URL パスプレフィックス）、クラスタ内で共有されている DOM 構造トークンのコア、landmark タイプ（header/footer/nav/aside/form/search）ごとのクラスタ内共通性、そして同一ブロッキンググループ内で分岐した兄弟クラスタのキー一覧。 ```json { - "id": "任意の識別子", - "clusterKey": "...", - "landmarks": { - "header": [ - { - "startLine": 1, - "startColumn": 7, - "endLine": 1, - "endColumn": 30, - "startOffset": 6, - "endOffset": 29, - "isChrome": true - } + "[\"path:news\",\"cluster:0\"]": { + "memberCount": 42, + "blocking": [ + { "blockKey": "path:news", "reason": { "kind": "path", "pathKey": "news" } } ], - "footer": [], - "nav": [], - "aside": [], - "form": [], - "search": [], - "main": [ - { - "startLine": 2, - "startColumn": 1, - "endLine": 10, - "endColumn": 8, - "startOffset": 40, - "endOffset": 120 + "structuralCoreTokens": ["body>main>article", "..."], + "landmarks": { + "header": { + "presenceRate": 1, + "chromeRate": 1, + "shellTokens": ["..."], + "memberCountWithInstance": 42 + }, + "aside": { + "presenceRate": 0.3, + "chromeRate": 0, + "shellTokens": [], + "memberCountWithInstance": 13 } - ] + }, + "siblingClusterKeys": ["[\"path:news\",\"cluster:1\"]"] } } ``` -20,000 ページを超えるコーパス（ストリーミング経路）では `--include-landmark-positions` は使えない（エラーで終了する）。ストリーミング経路はリザーバサンプリングと近似割当を使うため、ページ単位の chrome 判定に必要な「そのページが属する最終クラスタの shell トークン」という概念を持たないため。 +上の例は、header はクラスタ全体で共通の chrome（`chromeRate: 1`）である一方、aside は 42 ページ中 13 ページにしか無く chrome とは判定されていない（`chromeRate: 0`）ことを示す。つまり「ヘッダーは共通だが、サイドナビの有無で分かれているページがある」という状況を数値で表している。`siblingClusterKeys` は同じブロッキンググループ内で Stage A/B が結局別クラスタのままにした相手のキーで、それぞれの `ClusterReason` を突き合わせれば「何が違って分かれたか」を呼び出し側で解釈できる。理由は構造化データのみで、文言化（「ヘッダーが共通です」等）は呼び出し側の責務。 + +landmark の位置情報そのもの（HTML 内のどこにあるか）が必要な場合は、ライブラリの `extractLandmarks`（ステートレス・公開 API）をページの HTML に対して自分で呼び、その結果と `ClusterReason.landmarks[type].shellTokens` を突き合わせて `isChromeLandmarkInstance`（同じく公開 API）で chrome 判定すればよい。詳細は [Library](#library) 節を参照。 クローラ出力が JSON 配列の場合は `jq` で line-delimited に変換して食わせる: @@ -84,7 +78,7 @@ jq -c '.[]' crawl-output.json | page-cluster > clusters.jsonl #### オプション - `--content-block-attribute ` — CMS が自由編集コンテンツブロックに付与している属性名（例: `data-bgb`）が分かっている場合に指定する。指定すると比較前にその属性を持つ要素配下を無視するので、同じテンプレートで本文構成だけ違うページを混同しなくなる。唯一の site-specific なオプションで、未指定でも `
` / `role="main"` を起点にした自動深さキャップが常時働く（詳細は `resolve-page-cluster-keys.ts` の JSDoc を参照） -- `--include-landmark-positions` — 出力の各行に上記の `landmarks` フィールドを追加する。20,000 ページ超のコーパスでは使えない。指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため） +- `--cluster-reasons-file ` — 上記の「クラスタ選定理由」を `` に JSON として書き出す。ページ数の上限はない。20,000 ページ以下のコーパスでは、指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため。20,000 ページ超のストリーミング経路では進捗表示・クラスタ理由の両方が動く） - `--help` / `-h` — ヘルプを表示する - `--version` / `-v` — バージョンを表示する @@ -116,12 +110,15 @@ silence したい場合は `2>/dev/null`。ログに残したい場合は `2> pr サブパスエクスポート構成。import パスと提供関数の対応は以下。 -| import パス | 提供関数 | -| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `@d-zero/page-cluster` | `tokenize` — `` 配下を構造トークン列に変換する低レベルプリミティブ | -| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。いずれも `includeLandmarkPositions: true` を渡すと `clusterKey` に加えて位置情報つきの `landmarks`（`PageLandmarkReport`）を返す | -| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す | -| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類 | +| import パス | 提供関数 | +| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `@d-zero/page-cluster` | `tokenize` — `` 配下を構造トークン列に変換する低レベルプリミティブ | +| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。`onClusterReason` コールバックを渡すと、確定したクラスタごとに 1 回だけ `ClusterReason` を通知する | +| `@d-zero/page-cluster/build-cluster-reason` | `ClusterReason` / `LandmarkClusterProfile` 型、`buildClusterReason` — クラスタ選定理由の型定義と組み立て関数（通常は `resolvePageClusterKeys` の `onClusterReason` 経由で使うので直接呼ぶ必要はない） | +| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す | +| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類 | +| `@d-zero/page-cluster/is-chrome-landmark-instance` | `isChromeLandmarkInstance` — 1 つの landmark インスタンスのトークン集合と `ClusterReason.landmarks[type].shellTokens` のようなシェルトークン集合を突き合わせて chrome/content を判定するステートレス関数 | +| `@d-zero/page-cluster/jaccard-similarity` | `jaccardSimilarity` — 2 つのトークン集合の Jaccard 類似度。`ClusterReason` 同士（`structuralCoreTokens` や `shellTokens`）を比較して兄弟クラスタとの差分を調べる用途などに使う | ```ts import { resolvePageClusterKeysFromArray } from '@d-zero/page-cluster/resolve-page-cluster-keys'; @@ -175,7 +172,7 @@ flowchart TD - **chrome discovery** — 全ページのランドマーク署名の度数分布に auto-cut を当て、閾値以上を「グローバル chrome」（サイト共通のヘッダー等）として比較から除外し、閾値未満かつ 2 ページ以上に出現するものを「ローカル chrome」（セクション固有のナビ等）としてトークン再注入する - **Stage A（ブロック内クラスタリング）** — ブロックごとに直線的な処理。`
` の深さキャップ（候補深度を全走査して knee を探す自動選択）→ tokenize → complete-linkage 階層クラスタリング → max-gap auto-cut でカット高を決定 → 最後に包含関係にあるクラスタを吸収する包含割当（割当チェーンを辿り、循環はメンバー最大のクラスタをルートに選んで解決） - **Pass 1b（ストリーミング時のみ）** — 20,000 ページ超では各ブロックをリザーバサンプリング（最大 100 ページ、ブロックキーをシードにした決定的乱数）で代表させ、サンプル外のページは Stage A 完了後に max-Jaccard で最寄りクラスタへ一括割当する。メモリ使用量はコーパス全体ではなくサンプルサイズに比例する -- **Stage B（ブロック越えマージ）** — ブロック分割はあくまで比較コスト削減のためなので、最後に同一テンプレートがブロックを跨いで分かれていないか再統合する。これが唯一の反復処理（次節） +- **Stage B（ブロック越えマージ）** — ブロック分割はあくまで比較コスト削減のためなので、最後に同一テンプレートがブロックを跨いで分かれていないか再統合する。これが唯一の反復処理（次節）。収束後、`onClusterReason` が指定されていれば、確定した最終クラスタごとに `ClusterReason` を 1 回ずつ組み立てて通知する — 追加の全コーパススキャンではなく、Stage A/B が既に計算済みの中間データ（quorum core、landmark インスタンス、ブロッキング根拠）を再利用するだけなので、クラスタ数にしか比例しない ### Stage B: ブロック越え統合の不動点ループ diff --git a/packages/@d-zero/page-cluster/package.json b/packages/@d-zero/page-cluster/package.json index f987ca4b..765d91c6 100644 --- a/packages/@d-zero/page-cluster/package.json +++ b/packages/@d-zero/page-cluster/package.json @@ -24,6 +24,14 @@ "./resolve-page-cluster-keys": { "import": "./dist/resolve-page-cluster-keys.js", "types": "./dist/resolve-page-cluster-keys.d.ts" + }, + "./is-chrome-landmark-instance": { + "import": "./dist/is-chrome-landmark-instance.js", + "types": "./dist/is-chrome-landmark-instance.d.ts" + }, + "./jaccard-similarity": { + "import": "./dist/jaccard-similarity.js", + "types": "./dist/jaccard-similarity.d.ts" } }, "bin": "./dist/cli.js", diff --git a/packages/@d-zero/page-cluster/src/build-cluster-reason.spec.ts b/packages/@d-zero/page-cluster/src/build-cluster-reason.spec.ts new file mode 100644 index 00000000..c3be3a05 --- /dev/null +++ b/packages/@d-zero/page-cluster/src/build-cluster-reason.spec.ts @@ -0,0 +1,121 @@ +import type { BlockingReason } from './derive-blocking-reason.js'; +import type { LandmarkType } from './extract-landmarks.js'; +import type { PerPageLandmarkInstance } from './per-page-landmark-signatures.js'; + +import { describe, expect, test } from 'vitest'; + +import { buildClusterReason } from './build-cluster-reason.js'; + +const zeroPosition = { + startOffset: 0, + endOffset: 0, + startLine: 1, + startColumn: 1, + endLine: 1, + endColumn: 1, +}; + +/** + * + * @param type + * @param tokens + */ +function instance( + type: LandmarkType, + tokens: readonly string[], +): PerPageLandmarkInstance { + return { + type, + tokens: new Set(tokens), + signature: [...tokens].toSorted().join('|'), + position: zeroPosition, + }; +} + +describe('buildClusterReason', () => { + test('passes blocking and siblingClusterKeys through unchanged', () => { + const blocking: readonly { blockKey: string; reason: BlockingReason }[] = [ + { + blockKey: 'css:abc', + reason: { kind: 'css', distinctiveStylesheetHrefs: ['/a.css'] }, + }, + ]; + const reason = buildClusterReason({ + tokenSets: [new Set(['a'])], + landmarkInstances: [[]], + blocking, + siblingClusterKeys: ['sibling-1'], + }); + expect(reason.blocking).toBe(blocking); + expect(reason.siblingClusterKeys).toEqual(['sibling-1']); + expect(reason.memberCount).toBe(1); + }); + + test('structuralCoreTokens keeps only tokens shared by the quorum fraction of members', () => { + const tokenSets = [ + new Set(['body>main>.card', 'body>main>.unique-a']), + new Set(['body>main>.card', 'body>main>.unique-b']), + new Set(['body>main>.card', 'body>main>.unique-c']), + ]; + const reason = buildClusterReason({ + tokenSets, + landmarkInstances: [[], [], []], + blocking: [], + siblingClusterKeys: [], + }); + expect(reason.structuralCoreTokens).toEqual(['body>main>.card']); + }); + + test('landmarks omits types with no instance on any member page', () => { + const reason = buildClusterReason({ + tokenSets: [new Set(['a']), new Set(['a'])], + landmarkInstances: [[], []], + blocking: [], + siblingClusterKeys: [], + }); + expect(reason.landmarks).toEqual({}); + }); + + test('header/footer common across all members yields chromeRate 1 and full presence', () => { + const header = ['header>nav>a']; + const footer = ['footer>p']; + const landmarkInstances = Array.from({ length: 5 }, () => [ + instance('header', header), + instance('footer', footer), + ]); + const reason = buildClusterReason({ + tokenSets: Array.from({ length: 5 }, () => new Set(['body>main>.card'])), + landmarkInstances, + blocking: [], + siblingClusterKeys: [], + }); + expect(reason.landmarks.header?.presenceRate).toBe(1); + expect(reason.landmarks.header?.chromeRate).toBe(1); + expect(reason.landmarks.header?.memberCountWithInstance).toBe(5); + expect(reason.landmarks.footer?.presenceRate).toBe(1); + expect(reason.landmarks.footer?.chromeRate).toBe(1); + }); + + test('a landmark type present on only some members, with per-member-varying tokens, reads as partial presence and non-chrome (sidenav-style split)', () => { + const header = ['header>nav>a']; + const landmarkInstances = [ + [instance('header', header), instance('aside', ['aside>.promo'])], + [instance('header', header), instance('aside', ['aside>.other'])], + [instance('header', header)], + [instance('header', header)], + [instance('header', header)], + ]; + const reason = buildClusterReason({ + tokenSets: Array.from({ length: 5 }, () => new Set(['body>main>.card'])), + landmarkInstances, + blocking: [], + siblingClusterKeys: [], + }); + expect(reason.landmarks.header?.presenceRate).toBe(1); + expect(reason.landmarks.header?.chromeRate).toBe(1); + expect(reason.landmarks.aside?.presenceRate).toBe(0.4); + expect(reason.landmarks.aside?.memberCountWithInstance).toBe(2); + expect(reason.landmarks.aside?.chromeRate).toBe(0); + expect(reason.landmarks.aside?.shellTokens).toEqual([]); + }); +}); diff --git a/packages/@d-zero/page-cluster/src/build-cluster-reason.ts b/packages/@d-zero/page-cluster/src/build-cluster-reason.ts new file mode 100644 index 00000000..05a71c7b --- /dev/null +++ b/packages/@d-zero/page-cluster/src/build-cluster-reason.ts @@ -0,0 +1,171 @@ +import type { BlockingReason } from './derive-blocking-reason.js'; +import type { LandmarkType } from './extract-landmarks.js'; +import type { PerPageLandmarkInstance } from './per-page-landmark-signatures.js'; + +import { + DEFAULT_CHROME_OVERLAP_THRESHOLD, + isChromeLandmarkInstance, +} from './is-chrome-landmark-instance.js'; +import { computeQuorumCore } from './merge-cross-block-clusters.js'; +import { ALL_LANDMARK_TYPES } from './per-page-landmark-signatures.js'; +import { shellQuorum } from './shell-quorum.js'; + +/** + * One final cluster's common-vs-varying profile for a single landmark type + * (header/footer/nav/aside/form/search), derived by running + * {@link ./shell-quorum.js | shellQuorum} on just that type's instances across + * every member page. Absent from {@link ClusterReason}'s `landmarks` entirely + * when no member page carries an instance of the type. + */ +export type LandmarkClusterProfile = { + /** Fraction (0–1) of the cluster's member pages carrying at least one instance of this type. */ + readonly presenceRate: number; + /** + * Fraction (0–1) of this type's instances, across all member pages, that + * {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance} + * classified as chrome against this cluster's own shell for the type — + * the "header/footer is shared chrome" half of the user-facing story. + * + * The denominator is per-page-deduplicated instances (same rule + * {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances} + * always applies: two instances on the same page that tokenize to the + * same signature — a CMS-duplicated footer, or a `
` matching both `header` and `nav` at the identical + * span — count once). A caller reconstructing a single instance's chrome + * verdict via `extractLandmarks` + `tokenize` + + * `isChromeLandmarkInstance` classifies that one instance correctly + * regardless of this rule; only a caller trying to reproduce this exact + * ratio by counting raw, un-deduplicated instances across a page would + * see a different number. + */ + readonly chromeRate: number; + /** + * The shell token set `shellQuorum` discovered for this type within this + * cluster. Exposed as raw evidence — comparing two sibling clusters' + * `shellTokens` for the same type (e.g. via + * {@link ./jaccard-similarity.js | jaccardSimilarity}) is how a caller + * answers "do these two clusters share the same footer but differ in + * whether they have a sidebar nav". + */ + readonly shellTokens: readonly string[]; + /** How many member pages contributed at least one instance of this type. */ + readonly memberCountWithInstance: number; +}; + +/** + * Structured, uninterpreted explanation of why a final cluster's member + * pages ended up together, and which sibling clusters (same Pass-0 block, + * different final cluster) they were nonetheless split from. Deliberately + * carries no human-readable text or verdicts ("these are the same template + * except for X") — that judgment belongs to the caller, which has the + * product context (and the localization requirements) `@d-zero/page-cluster` + * itself does not. + */ +export type ClusterReason = { + /** Number of pages this final cluster contains. */ + readonly memberCount: number; + /** + * The distinct Pass-0 blocking keys that fed into this final cluster, and + * the evidence behind each. Usually one entry; more than one means Stage B + * merged pages that started in different blocks — itself a notable part + * of the explanation (e.g. two differently-styled URL sections turned out + * to share the same DOM template). + */ + readonly blocking: readonly { + readonly blockKey: string; + readonly reason: BlockingReason; + }[]; + /** + * The frequency-quorum core of this cluster's DOM structural tokens (see + * {@link ./merge-cross-block-clusters.js | computeQuorumCore}) — the + * "this DOM structure is common" evidence, independent of CSS or + * landmarks. + */ + readonly structuralCoreTokens: readonly string[]; + /** Per-landmark-type commonality. Types absent from every member page are omitted. */ + readonly landmarks: { readonly [K in LandmarkType]?: LandmarkClusterProfile }; + /** + * Final cluster keys that share at least one Pass-0 block with this + * cluster but were not merged into it by Stage A/B — candidates for "why + * did these split" comparison. Does not include clusters that started in + * a different block from this one. + */ + readonly siblingClusterKeys: readonly string[]; +}; + +/** + * Builds one final cluster's {@link ClusterReason} from data Stage A/B + * already computed for clustering itself — no re-tokenization, no re-running + * `shellQuorum`'s corpus-wide discovery pass, just re-deriving per-type shells + * and quorum cores from the same pooled member state + * {@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters} already + * built and would otherwise have discarded. + * @param input + * @param input.tokenSets + * @param input.landmarkInstances + * @param input.blocking + * @param input.siblingClusterKeys + * @param input.chromeThreshold + * @example + * ```ts + * const reason = buildClusterReason({ + * tokenSets: finalGroup.tokenSets, + * landmarkInstances: finalGroup.landmarkInstances, + * blocking: [{ blockKey: 'css:abc123', reason: { kind: 'css', distinctiveStylesheetHrefs: ['/a.css'] } }], + * siblingClusterKeys: ['["css:abc123","cluster:1"]'], + * }); + * ``` + */ +export function buildClusterReason(input: { + readonly tokenSets: readonly ReadonlySet[]; + readonly landmarkInstances: readonly (readonly PerPageLandmarkInstance[])[]; + readonly blocking: readonly { + readonly blockKey: string; + readonly reason: BlockingReason; + }[]; + readonly siblingClusterKeys: readonly string[]; + readonly chromeThreshold?: number; +}): ClusterReason { + const memberCount = input.tokenSets.length; + const chromeThreshold = input.chromeThreshold ?? DEFAULT_CHROME_OVERLAP_THRESHOLD; + + const structuralCoreTokens = [...computeQuorumCore(input.tokenSets)].toSorted(); + + const landmarks: { [K in LandmarkType]?: LandmarkClusterProfile } = {}; + for (const type of ALL_LANDMARK_TYPES) { + const perMemberInstancesOfType = input.landmarkInstances.map((instances) => + instances.filter((instance) => instance.type === type), + ); + const membersWithInstance = perMemberInstancesOfType.filter( + (instances) => instances.length > 0, + ); + if (membersWithInstance.length === 0) continue; + + const shellTokens = shellQuorum(perMemberInstancesOfType); + let chromeInstanceCount = 0; + let totalInstanceCount = 0; + for (const instances of membersWithInstance) { + for (const instance of instances) { + totalInstanceCount++; + if (isChromeLandmarkInstance(instance.tokens, shellTokens, chromeThreshold)) { + chromeInstanceCount++; + } + } + } + + landmarks[type] = { + presenceRate: membersWithInstance.length / memberCount, + chromeRate: totalInstanceCount === 0 ? 0 : chromeInstanceCount / totalInstanceCount, + shellTokens: [...shellTokens].toSorted(), + memberCountWithInstance: membersWithInstance.length, + }; + } + + return { + memberCount, + blocking: input.blocking, + structuralCoreTokens, + landmarks, + siblingClusterKeys: input.siblingClusterKeys, + }; +} diff --git a/packages/@d-zero/page-cluster/src/build-page-landmark-report.spec.ts b/packages/@d-zero/page-cluster/src/build-page-landmark-report.spec.ts deleted file mode 100644 index 33c1659a..00000000 --- a/packages/@d-zero/page-cluster/src/build-page-landmark-report.spec.ts +++ /dev/null @@ -1,64 +0,0 @@ -import { describe, expect, test } from 'vitest'; - -import { buildPageLandmarkReport } from './build-page-landmark-report.js'; -import { extractLandmarks } from './extract-landmarks.js'; -import { tokenize } from './tokenize.js'; - -describe('buildPageLandmarkReport', () => { - test('a header instance whose tokens are all shell tokens is reported as chrome', () => { - const landmarks = extractLandmarks( - '
content
', - ); - const shellTokens = new Set( - tokenize(`${landmarks.header[0]!.html}`).tokens, - ); - const report = buildPageLandmarkReport(landmarks, shellTokens); - expect(report.header).toHaveLength(1); - expect(report.header[0]!.isChrome).toBe(true); - }); - - test('an instance with no overlap with shell tokens is reported as content', () => { - const landmarks = extractLandmarks( - '

x

content
', - ); - const shellTokens = new Set(['body>totally>unrelated']); - const report = buildPageLandmarkReport(landmarks, shellTokens); - expect(report.header[0]!.isChrome).toBe(false); - }); - - test('main instances carry no isChrome field', () => { - const landmarks = extractLandmarks('
content
'); - const report = buildPageLandmarkReport(landmarks, new Set()); - expect(report.main).toHaveLength(1); - expect(Object.keys(report.main[0]!)).not.toContain('isChrome'); - }); - - test('a page with no landmarks yields all-empty arrays', () => { - const landmarks = extractLandmarks('content only'); - const report = buildPageLandmarkReport(landmarks, new Set()); - expect(report).toStrictEqual({ - header: [], - footer: [], - nav: [], - aside: [], - form: [], - search: [], - main: [], - }); - }); - - test('each reported instance carries the instance position, not its html', () => { - const landmarks = extractLandmarks('
H
M
'); - const report = buildPageLandmarkReport(landmarks, new Set()); - const source = landmarks.header[0]!; - expect(report.header[0]).toStrictEqual({ - startOffset: source.startOffset, - endOffset: source.endOffset, - startLine: source.startLine, - startColumn: source.startColumn, - endLine: source.endLine, - endColumn: source.endColumn, - isChrome: false, - }); - }); -}); diff --git a/packages/@d-zero/page-cluster/src/build-page-landmark-report.ts b/packages/@d-zero/page-cluster/src/build-page-landmark-report.ts deleted file mode 100644 index 6f401947..00000000 --- a/packages/@d-zero/page-cluster/src/build-page-landmark-report.ts +++ /dev/null @@ -1,107 +0,0 @@ -import type { - ExtractLandmarksResult, - LandmarkInstance, - LandmarkPosition, -} from './extract-landmarks.js'; -import type { TokenizeOptions } from './types.js'; - -import { isChromeLandmarkInstance } from './is-chrome-landmark-instance.js'; -import { ALL_LANDMARK_TYPES } from './per-page-landmark-signatures.js'; -import { tokenize } from './tokenize.js'; - -/** - * A landmark instance's position plus whether - * {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance} judged - * it shared site/section chrome (`true`) or page-specific content (`false`), - * against the unit's {@link ./shell-quorum.js | shellQuorum} shell tokens. - */ -export type ReportedLandmarkInstance = LandmarkPosition & { readonly isChrome: boolean }; - -/** - * Per-page landmark position report built by - * {@link ./build-page-landmark-report.js | buildPageLandmarkReport}. `main` - * carries no `isChrome` verdict — it never participates in chrome/shell - * discovery (see `extractLandmarks`'s "main handling" note) and is always - * content. - */ -export type PageLandmarkReport = { - header: ReportedLandmarkInstance[]; - footer: ReportedLandmarkInstance[]; - nav: ReportedLandmarkInstance[]; - aside: ReportedLandmarkInstance[]; - form: ReportedLandmarkInstance[]; - search: ReportedLandmarkInstance[]; - main: LandmarkPosition[]; -}; - -/** - * Strips `html` off a {@link LandmarkInstance}, keeping only its position. - * `buildPageLandmarkReport`'s output is meant to be serialized per page - * across a whole corpus (the CLI's JSONL output), so the report - * deliberately excludes each instance's raw HTML to keep that payload from - * scaling with markup size — callers who also need the HTML already have - * `ExtractLandmarksResult` in hand. - * @param instance - */ -function toPosition(instance: LandmarkInstance): LandmarkPosition { - return { - startOffset: instance.startOffset, - endOffset: instance.endOffset, - startLine: instance.startLine, - startColumn: instance.startColumn, - endLine: instance.endLine, - endColumn: instance.endColumn, - }; -} - -/** - * Builds a page's landmark position report: every landmark instance's - * location, with `header`/`footer`/`nav`/`aside`/`form`/`search` instances - * additionally classified as chrome or content against `shellTokens`. - * - * Reads `landmarks` directly — the full, non-deduplicated instance list - * `extractLandmarks` produced — rather than going through - * {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}'s - * per-page-deduplicated `PerPageLandmarkInstance[]`: that dedupe collapses - * same-signature instances to one entry, which would silently drop the - * position of every duplicate instance a position report needs to include. - * @param landmarks - * @param shellTokens The unit-level shell token set from - * {@link ./shell-quorum.js | shellQuorum}, computed once per final cluster - * and shared across every member page's report. - * @param tokenizeOptions - * @example - * ```ts - * const shellTokens = shellQuorum(clusterPerPageInstances); - * const report = buildPageLandmarkReport(extractLandmarks(page.html), shellTokens); - * ``` - */ -export function buildPageLandmarkReport( - landmarks: ExtractLandmarksResult, - shellTokens: ReadonlySet, - tokenizeOptions?: TokenizeOptions, -): PageLandmarkReport { - const report: PageLandmarkReport = { - header: [], - footer: [], - nav: [], - aside: [], - form: [], - search: [], - main: landmarks.main.map(toPosition), - }; - - for (const type of ALL_LANDMARK_TYPES) { - for (const instance of landmarks[type]) { - const tokens = instance.html - ? new Set(tokenize(`${instance.html}`, tokenizeOptions).tokens) - : new Set(); - report[type].push({ - ...toPosition(instance), - isChrome: isChromeLandmarkInstance(tokens, shellTokens), - }); - } - } - - return report; -} diff --git a/packages/@d-zero/page-cluster/src/cli.spec.ts b/packages/@d-zero/page-cluster/src/cli.spec.ts index 78cb49f4..0f63b4e5 100644 --- a/packages/@d-zero/page-cluster/src/cli.spec.ts +++ b/packages/@d-zero/page-cluster/src/cli.spec.ts @@ -1,3 +1,6 @@ +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; import { Readable, Writable } from 'node:stream'; import { describe, expect, test } from 'vitest'; @@ -68,9 +71,15 @@ describe('parseArgs', () => { }); }); - test('--include-landmark-positions', () => { - expect(parseArgs(['--include-landmark-positions'])).toEqual({ - includeLandmarkPositions: true, + test('--cluster-reasons-file takes a value', () => { + expect(parseArgs(['--cluster-reasons-file', 'reasons.json'])).toEqual({ + clusterReasonsFile: 'reasons.json', + }); + }); + + test('--cluster-reasons-file without a value flags an error', () => { + expect(parseArgs(['--cluster-reasons-file'])).toEqual({ + unknownFlag: '--cluster-reasons-file requires a value', }); }); @@ -149,7 +158,7 @@ describe('runCli', () => { expect(stdout.read()).toMatch(/Usage:/); }); - test('--help mentions --include-landmark-positions', async () => { + test('--help mentions --cluster-reasons-file', async () => { const stdout = makeCollector(); const stderr = makeCollector(); await runCli({ @@ -159,7 +168,7 @@ describe('runCli', () => { argv: ['--help'], version: '0.0.0', }); - expect(stdout.read()).toMatch(/--include-landmark-positions/); + expect(stdout.read()).toMatch(/--cluster-reasons-file/); }); test('--version prints version and exits 0', async () => { @@ -310,7 +319,7 @@ describe('runCli', () => { ); }); - test('--include-landmark-positions adds a landmarks field to each output line', async () => { + test('--cluster-reasons-file writes a clusterKey-keyed ClusterReason object, output lines stay landmarks-free', async () => { const input = [ JSON.stringify({ id: 'a', @@ -325,31 +334,61 @@ describe('runCli', () => { html: '
H
two
', }), ].join('\n'); + const stdout = makeCollector(); + const stderr = makeCollector(); + const dir = await mkdtemp(path.join(tmpdir(), 'page-cluster-cli-')); + const reasonsFile = path.join(dir, 'reasons.json'); + try { + const code = await runCli({ + stdin: makeStdin(input), + stdout: stdout.stream, + stderr: stderr.stream, + argv: ['--cluster-reasons-file', reasonsFile], + version: '0.0.0', + }); + expect(code).toBe(0); + + const lines = stdout.read().split('\n').filter(Boolean); + expect(lines).toHaveLength(2); + const parsed = lines.map( + (line) => JSON.parse(line) as { id: string; clusterKey: string }, + ); + expect(parsed[0]).not.toHaveProperty('landmarks'); + expect(parsed[0]!.clusterKey).toBe(parsed[1]!.clusterKey); + + const reasonsByKey = JSON.parse(await readFile(reasonsFile, 'utf8')) as Record< + string, + { memberCount: number; landmarks: { header?: { chromeRate: number } } } + >; + const reason = reasonsByKey[parsed[0]!.clusterKey]; + expect(reason?.memberCount).toBe(2); + expect(reason?.landmarks.header?.chromeRate).toBe(1); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + test('--cluster-reasons-file write failure is reported as a clean exit-1 error, not an unhandled rejection', async () => { const stdout = makeCollector(); const stderr = makeCollector(); const code = await runCli({ - stdin: makeStdin(input), + stdin: makeStdin( + JSON.stringify({ id: 'a', html: '
H
' }), + ), stdout: stdout.stream, stderr: stderr.stream, - argv: ['--include-landmark-positions'], + // A path under a nonexistent directory always fails ENOENT on write. + argv: [ + '--cluster-reasons-file', + path.join(tmpdir(), 'page-cluster-nonexistent-dir', 'reasons.json'), + ], version: '0.0.0', }); - expect(code).toBe(0); - const lines = stdout.read().split('\n').filter(Boolean); - expect(lines).toHaveLength(2); - const parsed = lines.map( - (line) => - JSON.parse(line) as { - id: string; - clusterKey: string; - landmarks: { header: { isChrome: boolean }[]; main: object[] }; - }, - ); - expect(parsed[0]!.landmarks.header[0]!.isChrome).toBe(true); - expect(parsed[0]!.landmarks.main).toHaveLength(1); + expect(code).toBe(1); + expect(stripAnsi(stderr.read())).toMatch(/\[page-cluster\] error: /); }); - test('without --include-landmark-positions, output lines carry no landmarks field', async () => { + test('without --cluster-reasons-file, no reasons file is written and output lines carry no landmarks field', async () => { const stdout = makeCollector(); const stderr = makeCollector(); const code = await runCli({ diff --git a/packages/@d-zero/page-cluster/src/cli.ts b/packages/@d-zero/page-cluster/src/cli.ts index c6d5e1af..99d935c8 100644 --- a/packages/@d-zero/page-cluster/src/cli.ts +++ b/packages/@d-zero/page-cluster/src/cli.ts @@ -5,13 +5,14 @@ // and streams progress to stderr via `@d-zero/dealer`'s `Lanes` — in-place // animated header on a TTY, appended `[page-cluster] …` lines otherwise. +import type { ClusterReason } from './build-cluster-reason.js'; import type { - PageClusterKeyResult, PageClusterSignals, ProgressEvent, ResolvePageClusterKeysOptions, } from './resolve-page-cluster-keys.js'; +import { writeFile } from 'node:fs/promises'; import process from 'node:process'; import { Lanes } from '@d-zero/dealer'; @@ -24,14 +25,14 @@ import { resolvePageClusterKeys } from './resolve-page-cluster-keys.js'; */ type CliArgs = { readonly contentBlockAttribute?: string; - readonly includeLandmarkPositions?: boolean; + readonly clusterReasonsFile?: string; readonly help?: boolean; readonly version?: boolean; readonly unknownFlag?: string; }; const HELP_TEXT = `Usage: - page-cluster [--content-block-attribute ] [--include-landmark-positions] < pages.jsonl > clusters.jsonl + page-cluster [--content-block-attribute ] [--cluster-reasons-file ] < pages.jsonl > clusters.jsonl Input (JSONL, one page per line): { @@ -45,20 +46,27 @@ Input (JSONL, one page per line): Output (JSONL, one line per input page, in input order): { "id": "...", "clusterKey": "..." } - With --include-landmark-positions, each line additionally carries a - \`landmarks\` field: every header/footer/nav/aside/form/search/main - instance's position (1-based line/column plus string offsets), with the - six excisable types (all but main) also carrying an \`isChrome\` verdict - against that page's final cluster: + With --cluster-reasons-file , a separate JSON file is written once + processing completes: an object keyed by clusterKey, one entry per final + cluster (not per page — a ClusterReason is sized by cluster count, so + this file has no page-count limit, unlike a per-page report would). + Each ClusterReason reports the blocking evidence that grouped the + cluster (shared stylesheet set or URL path prefix), the shared DOM- + structural token core, per-landmark-type (header/footer/nav/aside/form/ + search) commonality within the cluster, and the sibling cluster keys it + was split from within the same blocking group — e.g. "header/footer are + common chrome across both clusters, but they differ in whether a + sidebar nav is present": { - "id": "...", "clusterKey": "...", - "landmarks": { - "header": [{ "startLine": 1, "startColumn": 7, "endLine": 1, - "endColumn": 30, "startOffset": 6, "endOffset": 29, - "isChrome": true }], - "footer": [...], "nav": [...], "aside": [...], "form": [...], "search": [...], - "main": [{ "startLine": 2, "startColumn": 1, "endLine": 10, - "endColumn": 8, "startOffset": 40, "endOffset": 120 }] + "[\\"path:news\\",\\"cluster:0\\"]": { + "memberCount": 42, + "blocking": [{ "blockKey": "path:news", "reason": { "kind": "path", "pathKey": "news" } }], + "structuralCoreTokens": ["body>main>article", "..."], + "landmarks": { + "header": { "presenceRate": 1, "chromeRate": 1, "shellTokens": ["..."], "memberCountWithInstance": 42 }, + "aside": { "presenceRate": 0.3, "chromeRate": 0, "shellTokens": [], "memberCountWithInstance": 13 } + }, + "siblingClusterKeys": ["[\\"path:news\\",\\"cluster:1\\"]"] } } @@ -66,14 +74,10 @@ Options: --content-block-attribute CMS-provided attribute marking freeform content blocks that should be stripped before comparison (e.g. \`data-bgb\`). - --include-landmark-positions Add the \`landmarks\` field described - above to every output line. Not - supported for corpora over 20,000 pages - (throws instead of streaming). Disables - progress output on stderr — this option - always routes through the same - non-progress-emitting code path as a - run without progress. + --cluster-reasons-file Write the per-cluster ClusterReason + object described above to after + processing completes. No page-count + limit. --help Print this help and exit. --version Print the package version and exit. @@ -111,7 +115,7 @@ const VERBOSE_HEADER = '[page-cluster]'; export function parseArgs(argv: readonly string[]): CliArgs { const out: { contentBlockAttribute?: string; - includeLandmarkPositions?: boolean; + clusterReasonsFile?: string; help?: boolean; version?: boolean; unknownFlag?: string; @@ -139,8 +143,14 @@ export function parseArgs(argv: readonly string[]): CliArgs { i++; break; } - case '--include-landmark-positions': { - out.includeLandmarkPositions = true; + case '--cluster-reasons-file': { + const next = argv[i + 1]; + if (next === undefined) { + out.unknownFlag = `${arg} requires a value`; + return out; + } + out.clusterReasonsFile = next; + i++; break; } default: { @@ -432,30 +442,26 @@ export async function runCli(options: { renderProgress(lanes, useTty, readingDoneLine(pages.length)); + // Only worth collecting when the caller asked for the file — a + // ClusterReason Map costs bookkeeping proportional to cluster count, + // not page count, but there's no reason to pay even that when unused. + const reasonsByClusterKey = args.clusterReasonsFile + ? new Map() + : undefined; + const resolveOptions: ResolvePageClusterKeysOptions = { contentBlockAttribute: args.contentBlockAttribute, onProgress: (event) => { renderProgress(lanes, useTty, formatProgressLine(event, elapsed())); }, + onClusterReason: reasonsByClusterKey + ? (key, reason) => reasonsByClusterKey.set(key, reason) + : undefined, }; - // `includeLandmarkPositions` always routes resolvePageClusterKeys - // through its non-progress-emitting sync path (see that option's own - // JSDoc), so the onProgress callback above is set but never invoked - // in this branch — no separate "quiet" resolveOptions variant needed. let clusterKeys: string[]; - let landmarksByIndex: PageClusterKeyResult['landmarks'][] | undefined; try { - if (args.includeLandmarkPositions) { - const results = await resolvePageClusterKeys(() => pages, { - ...resolveOptions, - includeLandmarkPositions: true, - }); - clusterKeys = results.map((r) => r.clusterKey); - landmarksByIndex = results.map((r) => r.landmarks); - } else { - clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions); - } + clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions); } catch (error) { renderProgress(lanes, useTty, errorLine((error as Error).message)); return 1; @@ -465,13 +471,22 @@ export async function runCli(options: { renderProgress(lanes, useTty, doneLine(pages.length, clusterCount, elapsed())); for (const [index, key] of clusterKeys.entries()) { - const row: { id: string | number; clusterKey: string; landmarks?: unknown } = { - id: ids[index] ?? index, - clusterKey: key, - }; - if (landmarksByIndex) row.landmarks = landmarksByIndex[index]; + const row = { id: ids[index] ?? index, clusterKey: key }; options.stdout.write(`${JSON.stringify(row)}\n`); } + + if (args.clusterReasonsFile && reasonsByClusterKey) { + try { + await writeFile( + args.clusterReasonsFile, + JSON.stringify(Object.fromEntries(reasonsByClusterKey), null, 2), + ); + } catch (error) { + renderProgress(lanes, useTty, errorLine((error as Error).message)); + return 1; + } + } + return 0; } finally { lanes.close(); diff --git a/packages/@d-zero/page-cluster/src/derive-blocking-reason.ts b/packages/@d-zero/page-cluster/src/derive-blocking-reason.ts new file mode 100644 index 00000000..f08283a7 --- /dev/null +++ b/packages/@d-zero/page-cluster/src/derive-blocking-reason.ts @@ -0,0 +1,35 @@ +/** + * The evidence behind one Pass-0 blocking key — which signal + * {@link ./resolve-blocking-group-keys.js | resolveBlockingGroupKeys} (and, + * for the `orphanMerge` variant, {@link ./reassign-orphan-block-keys.js | + * reassignOrphanBlockKeys}) actually used to decide it, carried verbatim with + * no added interpretation. Every page sharing a `css:` blocking key + * shares the exact same `distinctiveStylesheetHrefs` set by construction (the + * hash is derived from that set), so one `BlockingReason` per distinct + * blocking key is enough — it does not need to vary per page. + */ +export type BlockingReason = + | { + readonly kind: 'css'; + /** + * The sorted, deduplicated, first-party stylesheet hrefs left after + * corpus-wide chrome removal — the exact set + * {@link ./derive-stylesheet-group-key.js | deriveStylesheetGroupKey} + * hashed into this blocking key. + */ + readonly distinctiveStylesheetHrefs: readonly string[]; + } + | { + readonly kind: 'path'; + /** The `derivePathGroupKey` result this blocking key was derived from. */ + readonly pathKey: string; + } + | { + readonly kind: 'orphanMerge'; + /** + * The confined `path:` key this stylesheet-less page was folded into a + * same-section `css:` block under — see + * {@link ./reassign-orphan-block-keys.js | reassignOrphanBlockKeys}. + */ + readonly pathKey: string; + }; diff --git a/packages/@d-zero/page-cluster/src/extract-landmarks.spec.ts b/packages/@d-zero/page-cluster/src/extract-landmarks.spec.ts index d5814a8f..286ebebf 100644 --- a/packages/@d-zero/page-cluster/src/extract-landmarks.spec.ts +++ b/packages/@d-zero/page-cluster/src/extract-landmarks.spec.ts @@ -312,8 +312,8 @@ describe('extractLandmarks (main handling)', () => { test('main is absent from ALL_LANDMARK_TYPES-driven chrome discovery by construction (position-only)', () => { // extractLandmarks itself has no chrome/shell notion — this test only // pins that main instances are reported at all, alongside the other - // six, with no isChrome-like field (that lives in - // build-page-landmark-report.ts, a separate layer downstream). + // six, with no isChrome-like field (that classification is derived + // downstream, e.g. by build-cluster-reason.ts). const html = '
content
'; const result = extractLandmarks(html); expect(result.main).toHaveLength(1); diff --git a/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.spec.ts b/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.spec.ts index d834826e..1ad41e51 100644 --- a/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.spec.ts +++ b/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.spec.ts @@ -66,8 +66,10 @@ const landmarksWith = ( }; describe('mergeCrossBlockClusters', () => { - test('empty input returns empty map', () => { - expect(mergeCrossBlockClusters([], {})).toEqual(new Map()); + test('empty input returns empty maps', () => { + const result = mergeCrossBlockClusters([], {}); + expect(result.rootByKey).toEqual(new Map()); + expect(result.finalGroupsByRoot).toEqual(new Map()); }); test('single unit maps to itself', () => { @@ -77,7 +79,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit], {}); - expect(result.get('k1')).toBe('k1'); + expect(result.rootByKey.get('k1')).toBe('k1'); }); test('two units with identical token sets merge into one', () => { @@ -93,7 +95,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('k1')).toBe(result.get('k2')); + expect(result.rootByKey.get('k1')).toBe(result.rootByKey.get('k2')); }); test('two units with disjoint token sets stay separate', () => { @@ -112,8 +114,8 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('k1')).toBe('k1'); - expect(result.get('k2')).toBe('k2'); + expect(result.rootByKey.get('k1')).toBe('k1'); + expect(result.rootByKey.get('k2')).toBe('k2'); }); test('multi-page units with same shape but different class names merge via shape-Jaccard', () => { @@ -148,7 +150,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks, noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('reports')).toBe(result.get('projects')); + expect(result.rootByKey.get('reports')).toBe(result.rootByKey.get('projects')); }); test('single-page units are excluded from shape-Jaccard comparison', () => { @@ -169,8 +171,8 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('solo-a')).toBe('solo-a'); - expect(result.get('solo-b')).toBe('solo-b'); + expect(result.rootByKey.get('solo-a')).toBe('solo-a'); + expect(result.rootByKey.get('solo-b')).toBe('solo-b'); }); test('result map has an entry for every input unit key', () => { @@ -190,9 +192,9 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2, unit3], {}); - expect(result.has('a')).toBe(true); - expect(result.has('b')).toBe(true); - expect(result.has('c')).toBe(true); + expect(result.rootByKey.has('a')).toBe(true); + expect(result.rootByKey.has('b')).toBe(true); + expect(result.rootByKey.has('c')).toBe(true); }); test('result is deterministic: same input twice produces identical output', () => { @@ -209,8 +211,8 @@ describe('mergeCrossBlockClusters', () => { }; const r1 = mergeCrossBlockClusters([unit1, unit2], {}); const r2 = mergeCrossBlockClusters([unit1, unit2], {}); - expect(r1.get('k1')).toBe(r2.get('k1')); - expect(r1.get('k2')).toBe(r2.get('k2')); + expect(r1.rootByKey.get('k1')).toBe(r2.rootByKey.get('k1')); + expect(r1.rootByKey.get('k2')).toBe(r2.rootByKey.get('k2')); }); test('merged result preserves an existing unit key (no freshly invented key)', () => { @@ -226,7 +228,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([noLandmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - const rootKey = result.get('block-a')!; + const rootKey = result.rootByKey.get('block-a')!; expect(rootKey === 'block-a' || rootKey === 'block-b').toBe(true); }); @@ -247,7 +249,7 @@ describe('mergeCrossBlockClusters', () => { memberLandmarkInstances: toInstances([landmarks]), }; const result = mergeCrossBlockClusters([unit1, unit2], {}); - expect(result.get('l1')).toBe(result.get('l2')); + expect(result.rootByKey.get('l1')).toBe(result.rootByKey.get('l2')); }); }); @@ -304,7 +306,7 @@ describe('mergeCrossBlockClusters shellQuorum (via mergeCrossBlockClusters exerc // have been at risk. The stronger correctness assertion is that // shellQuorum returns non-empty for these landmarks (exercised // implicitly by not crashing on the L2 shell lookup). - expect(result.get('A')).toBe(result.get('B')); + expect(result.rootByKey.get('A')).toBe(result.rootByKey.get('B')); }); test('landmark instances that vary per page do not falsely act as shell (histogram cuts them off)', () => { @@ -333,7 +335,7 @@ describe('mergeCrossBlockClusters shellQuorum (via mergeCrossBlockClusters exerc // separate. Precondition: shell histogram correctly filters out the // per-page varying headers rather than admitting them all via a // union fallback. - expect(result.get('A')).toBe('A'); - expect(result.get('B')).toBe('B'); + expect(result.rootByKey.get('A')).toBe('A'); + expect(result.rootByKey.get('B')).toBe('B'); }); }); diff --git a/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.ts b/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.ts index c01437ca..1444c38b 100644 --- a/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.ts +++ b/packages/@d-zero/page-cluster/src/merge-cross-block-clusters.ts @@ -117,10 +117,16 @@ const GENERIC_SEGMENTS = new Set([ // --------------------------------------------------------------------------- /** - * + * Frequency-based token core of a group of member pages: a token must be + * present in at least `QUORUM_FRACTION` of members to enter the core, with a + * full-union fallback when no token clears that bar (see + * {@link mergeCrossBlockClusters}'s JSDoc for why quorum beats strict + * intersection or full union). Exported so callers building a `ClusterReason` + * (`build-cluster-reason.ts`) can re-derive a final group's structural core + * from `finalGroupsByRoot` without duplicating this logic. * @param memberDistinctiveTokens */ -function quorumCore( +export function computeQuorumCore( memberDistinctiveTokens: readonly ReadonlySet[], ): ReadonlySet { const n = memberDistinctiveTokens.length; @@ -204,11 +210,38 @@ function l2Contained(xSig: Map, ySig: Map): bool // Main function // --------------------------------------------------------------------------- +/** + * One root key's pooled member state after Stage B converges: every + * `tokenSets`/`landmarkInstances` entry folded in from every unit merged into + * this root (down-sampled to `capMembers` when the caller opts in, same as + * during merging). This is the exact state `mergeCrossBlockClusters` already + * builds internally to run quorum-core/shell comparisons each round — it was + * discarded once `keyToRoot` was returned. Exposing it lets a `ClusterReason` + * (`build-cluster-reason.ts`) be built from data Stage B already computed, + * with no extra pass over the corpus. + */ +export type FinalGroupMembers = { + readonly tokenSets: readonly ReadonlySet[]; + readonly landmarkInstances: readonly (readonly PerPageLandmarkInstance[])[]; +}; + +/** + * `mergeCrossBlockClusters`'s result: the root-key mapping every caller + * needs for `clusterKey` resolution, plus each surviving root's final pooled + * member state for callers that also want to explain *why* (`ClusterReason`). + */ +export type MergeCrossBlockClustersResult = { + /** Every input unit's `key` mapped to its final root key. Units not absorbed into any other unit map to themselves. */ + readonly rootByKey: ReadonlyMap; + /** Every surviving root key's final pooled member state. */ + readonly finalGroupsByRoot: ReadonlyMap; +}; + /** * Merges cross-block clusters (Stage B) via recursive quorum-core comparison. * - * Returns a `Map` from each input unit's `key` to its final root key. Units - * not absorbed into any other unit map to themselves. + * Returns the root-key mapping (see {@link MergeCrossBlockClustersResult}). + * Units not absorbed into any other unit map to themselves. * * Three merge mechanisms run per round, in order: * 1. **Fine stage** — complete-linkage at `CROSS_BLOCK_THRESHOLD` on quorum @@ -247,9 +280,17 @@ export function mergeCrossBlockClusters( */ capMembers?: number; }, -): Map { +): MergeCrossBlockClustersResult { if (units.length <= 1) { - return new Map(units.map((u) => [u.key, u.key])); + return { + rootByKey: new Map(units.map((u) => [u.key, u.key])), + finalGroupsByRoot: new Map( + units.map((u) => [ + u.key, + { tokenSets: u.memberTokenSets, landmarkInstances: u.memberLandmarkInstances }, + ]), + ), + }; } const threshold = options?.similarityThreshold ?? CROSS_BLOCK_THRESHOLD; @@ -334,7 +375,7 @@ export function mergeCrossBlockClusters( // Quorum core per group const cores = new Map>(); for (const key of groupKeys) { - cores.set(key, quorumCore(groupDistinctive.get(key) ?? [])); + cores.set(key, computeQuorumCore(groupDistinctive.get(key) ?? [])); } // --------------------------------------------------------------- @@ -533,5 +574,11 @@ export function mergeCrossBlockClusters( applyMerges(l2Merges); } - return keyToRoot; + const finalGroupsByRoot = new Map( + [...groups.entries()].map(([root, g]) => [ + root, + { tokenSets: g.tokenSets, landmarkInstances: g.landmarkInstances }, + ]), + ); + return { rootByKey: keyToRoot, finalGroupsByRoot }; } diff --git a/packages/@d-zero/page-cluster/src/pass0-blocking.spec.ts b/packages/@d-zero/page-cluster/src/pass0-blocking.spec.ts index 369ffc94..0ecc4b51 100644 --- a/packages/@d-zero/page-cluster/src/pass0-blocking.spec.ts +++ b/packages/@d-zero/page-cluster/src/pass0-blocking.spec.ts @@ -105,6 +105,69 @@ describe('resolveBlockKeys', () => { }); }); +describe('resolveBlockKeys (includeReasons)', () => { + test('css/path reasons pass through resolveBlockingGroupKeys unchanged when no orphan reassignment happens', () => { + const { blockKeys, reasonsByBlockKey } = resolveBlockKeys( + [ + { paths: ['news', '1'], stylesheetHrefs: ['https://example.com/a.css'] }, + { paths: ['news', '2'], stylesheetHrefs: ['https://example.com/a.css'] }, + { paths: ['about'], stylesheetHrefs: ['https://example.com/b.css'] }, + ], + { includeReasons: true }, + ); + expect(reasonsByBlockKey.get(blockKeys[0]!)).toEqual({ + kind: 'css', + distinctiveStylesheetHrefs: ['https://example.com/a.css'], + }); + expect(reasonsByBlockKey.get('path:about')).toEqual({ + kind: 'path', + pathKey: 'about', + }); + }); + + test('an orphan-merged key gets an orphanMerge reason carrying the confined pathKey', () => { + const pages = [ + { + paths: ['news', '1'], + stylesheetHrefs: ['https://example.com/a.css', 'https://example.com/shared.css'], + }, + { + paths: ['news', '2'], + stylesheetHrefs: ['https://example.com/a.css', 'https://example.com/shared.css'], + }, + { + paths: ['about'], + stylesheetHrefs: ['https://example.com/b.css', 'https://example.com/shared.css'], + }, + // Orphan (no stylesheets) in the news section — reassigned to the + // news css: block, which should carry an `orphanMerge` reason + // distinct from the plain `css` reason of a non-reassigned key. + { paths: ['news', '4'], stylesheetHrefs: [] }, + ]; + const { blockKeys, reasonsByBlockKey } = resolveBlockKeys(pages, { + includeReasons: true, + }); + expect(blockKeys[0]).toMatch(/^orphan-merge:/); + expect(reasonsByBlockKey.get(blockKeys[0]!)).toEqual({ + kind: 'orphanMerge', + pathKey: 'news', + }); + }); + + test('reasonsByBlockKey has exactly one entry per distinct block key', () => { + const { blockKeys, reasonsByBlockKey } = resolveBlockKeys( + [ + { paths: ['news', '1'], stylesheetHrefs: ['https://example.com/a.css'] }, + { paths: ['news', '2'], stylesheetHrefs: ['https://example.com/a.css'] }, + { paths: ['about'], stylesheetHrefs: ['https://example.com/b.css'] }, + ], + { includeReasons: true }, + ); + expect(blockKeys).toHaveLength(3); + expect(reasonsByBlockKey.size).toBe(2); + }); +}); + describe('groupIndicesByBlockKey', () => { test('groups indices by key, preserving first-seen order of keys and input order within each', () => { const groups = groupIndicesByBlockKey(['a', 'b', 'a', 'c', 'a', 'b']); diff --git a/packages/@d-zero/page-cluster/src/pass0-blocking.ts b/packages/@d-zero/page-cluster/src/pass0-blocking.ts index c2a43071..cb5474e7 100644 --- a/packages/@d-zero/page-cluster/src/pass0-blocking.ts +++ b/packages/@d-zero/page-cluster/src/pass0-blocking.ts @@ -1,7 +1,11 @@ +import type { BlockingReason } from './derive-blocking-reason.js'; import type { ResolveBlockingGroupKeysOptions } from './resolve-blocking-group-keys.js'; import { filterFirstPartyStylesheetHrefs } from './filter-first-party-stylesheet-hrefs.js'; -import { reassignOrphanBlockKeys } from './reassign-orphan-block-keys.js'; +import { + REASSIGNED_KEY_PREFIX, + reassignOrphanBlockKeys, +} from './reassign-orphan-block-keys.js'; import { resolveBlockingGroupKeys } from './resolve-blocking-group-keys.js'; /** @@ -38,6 +42,13 @@ export type ResolveBlockKeysOptions = ResolveBlockingGroupKeysOptions & { readonly restrictStylesheetsToFirstParty?: boolean; }; +/** Return shape when `includeReasons: true` is passed to `resolveBlockKeys`. */ +export type BlockKeysWithReasons = { + readonly blockKeys: string[]; + /** One entry per distinct final block key produced, keyed by that key. */ + readonly reasonsByBlockKey: ReadonlyMap; +}; + /** * Splits `resolvePageClusterKeys` into a size-flat first pass so the driver * can decide per-block memory strategy before loading any page HTML. Runs the @@ -78,19 +89,24 @@ export type ResolveBlockKeysOptions = ResolveBlockingGroupKeysOptions & { * // ['css:', 'css:', 'path:about'] * ``` */ +export function resolveBlockKeys( + pages: readonly Pass0PageSignals[], + options: ResolveBlockKeysOptions & { includeReasons: true }, +): BlockKeysWithReasons; export function resolveBlockKeys( pages: readonly Pass0PageSignals[], options?: ResolveBlockKeysOptions, -): string[] { +): string[]; +export function resolveBlockKeys( + pages: readonly Pass0PageSignals[], + options?: ResolveBlockKeysOptions & { includeReasons?: boolean }, +): string[] | BlockKeysWithReasons { const restrictStylesheetsToFirstParty = options?.restrictStylesheetsToFirstParty ?? true; const blockingPages = restrictStylesheetsToFirstParty ? filterFirstPartyStylesheetHrefs(pages) : pages; - const rawBlockKeys = resolveBlockingGroupKeys(blockingPages, options); - const reassignOrphans = options?.reassignOrphans ?? true; - if (!reassignOrphans) return rawBlockKeys; // Orphan reassignment always uses a numeric `pathDepth`. When the caller // asked for `'auto'`, fall back to the historical default 1 here — a // future PR that wires the auto-cut depth through can compute it once @@ -98,7 +114,36 @@ export function resolveBlockKeys( // call to keep them consistent. const numericPathDepth = typeof options?.pathDepth === 'number' ? options.pathDepth : undefined; - return reassignOrphanBlockKeys(blockingPages, rawBlockKeys, numericPathDepth); + const reassignOrphans = options?.reassignOrphans ?? true; + + if (!options?.includeReasons) { + const rawBlockKeys = resolveBlockingGroupKeys(blockingPages, options); + if (!reassignOrphans) return rawBlockKeys; + return reassignOrphanBlockKeys(blockingPages, rawBlockKeys, numericPathDepth); + } + + const { keys: rawBlockKeys, reasonsByKey } = resolveBlockingGroupKeys(blockingPages, { + ...options, + includeReasons: true, + }); + if (!reassignOrphans) { + return { blockKeys: rawBlockKeys, reasonsByBlockKey: reasonsByKey }; + } + + const finalBlockKeys = reassignOrphanBlockKeys( + blockingPages, + rawBlockKeys, + numericPathDepth, + ); + const reasonsByBlockKey = new Map(reasonsByKey); + for (const blockKey of finalBlockKeys) { + if (reasonsByBlockKey.has(blockKey)) continue; + reasonsByBlockKey.set(blockKey, { + kind: 'orphanMerge', + pathKey: blockKey.slice(REASSIGNED_KEY_PREFIX.length), + }); + } + return { blockKeys: finalBlockKeys, reasonsByBlockKey }; } /** diff --git a/packages/@d-zero/page-cluster/src/per-page-landmark-signatures.ts b/packages/@d-zero/page-cluster/src/per-page-landmark-signatures.ts index c0be4c95..b089588a 100644 --- a/packages/@d-zero/page-cluster/src/per-page-landmark-signatures.ts +++ b/packages/@d-zero/page-cluster/src/per-page-landmark-signatures.ts @@ -32,11 +32,10 @@ export const ALL_LANDMARK_TYPES: readonly LandmarkType[] = [ * * `position` is the instance's location in the page it came from, computed * once by {@link ./extract-landmarks.js | extractLandmarks} and carried here - * unchanged — a back-reference for callers (e.g. - * {@link ./build-page-landmark-report.js | buildPageLandmarkReport}) that - * need to report where a chrome-classified instance actually sits, not just - * that it exists. It plays no part in `tokens`/`signature` computation or in - * the corpus-frequency logic that consumes this type. + * unchanged — a back-reference for callers that need to report where a + * chrome-classified instance actually sits, not just that it exists. It + * plays no part in `tokens`/`signature` computation or in the corpus- + * frequency logic that consumes this type. */ export type PerPageLandmarkInstance = { readonly type: LandmarkType; diff --git a/packages/@d-zero/page-cluster/src/reassign-orphan-block-keys.ts b/packages/@d-zero/page-cluster/src/reassign-orphan-block-keys.ts index 87302fd1..4e6f7a8a 100644 --- a/packages/@d-zero/page-cluster/src/reassign-orphan-block-keys.ts +++ b/packages/@d-zero/page-cluster/src/reassign-orphan-block-keys.ts @@ -5,9 +5,12 @@ import { derivePathGroupKey } from './derive-path-group-key.js'; /** * Prefix distinguishing a reassigned key from the `css:`/`path:` keys * {@link ./resolve-blocking-group-keys.js | resolveBlockingGroupKeys} itself - * produces, so the two families can never collide. + * produces, so the two families can never collide. Exported so + * {@link ./pass0-blocking.js | resolveBlockKeys} can recover the confined + * path key back out of a reassigned block key when building `BlockingReason`s, + * without duplicating this literal. */ -const REASSIGNED_KEY_PREFIX = 'orphan-merge:'; +export const REASSIGNED_KEY_PREFIX = 'orphan-merge:'; /** * Reads `values[index]`, throwing instead of returning `undefined`. Every diff --git a/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.spec.ts b/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.spec.ts index df3d6782..b2050e57 100644 --- a/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.spec.ts +++ b/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.spec.ts @@ -157,3 +157,48 @@ describe('resolveBlockingGroupKeys', () => { ); }); }); + +describe('resolveBlockingGroupKeys (includeReasons)', () => { + const pages = [ + { + paths: ['dept-a', 'news', '1'], + stylesheetHrefs: ['https://example.com/a.css', 'https://example.com/common.css'], + }, + { + paths: ['dept-a', 'news', '2'], + stylesheetHrefs: ['https://example.com/a.css', 'https://example.com/common.css'], + }, + { paths: ['dept-b', 'about'], stylesheetHrefs: ['https://example.com/common.css'] }, + { paths: ['dept-c', 'contact'], stylesheetHrefs: ['https://example.com/common.css'] }, + ]; + + test('a css: key reason carries the exact distinctive hrefs that were hashed (common.css excluded)', () => { + const { keys, reasonsByKey } = resolveBlockingGroupKeys(pages, { + includeReasons: true, + }); + const reason = reasonsByKey.get(keys[0]!); + expect(reason).toEqual({ + kind: 'css', + distinctiveStylesheetHrefs: ['https://example.com/a.css'], + }); + }); + + test('a path: key reason carries the pathKey it fell back to', () => { + const { reasonsByKey } = resolveBlockingGroupKeys(pages, { includeReasons: true }); + expect(reasonsByKey.get('path:dept-b')).toEqual({ kind: 'path', pathKey: 'dept-b' }); + expect(reasonsByKey.get('path:dept-c')).toEqual({ kind: 'path', pathKey: 'dept-c' }); + }); + + test('reasonsByKey has exactly one entry per distinct key, not one per page', () => { + const { keys, reasonsByKey } = resolveBlockingGroupKeys(pages, { + includeReasons: true, + }); + expect(keys).toHaveLength(4); + expect(reasonsByKey.size).toBe(3); + }); + + test('without includeReasons, the return value is still a plain string[]', () => { + const result = resolveBlockingGroupKeys(pages); + expect(Array.isArray(result)).toBe(true); + }); +}); diff --git a/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.ts b/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.ts index 182f3991..6b2295f3 100644 --- a/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.ts +++ b/packages/@d-zero/page-cluster/src/resolve-blocking-group-keys.ts @@ -1,3 +1,5 @@ +import type { BlockingReason } from './derive-blocking-reason.js'; + import { computeDocumentFrequency } from './compute-document-frequency.js'; import { derivePathClusterKeys } from './derive-path-cluster-keys.js'; import { derivePathGroupKey } from './derive-path-group-key.js'; @@ -44,6 +46,13 @@ export type ResolveBlockingGroupKeysOptions = { hrefCommonThreshold?: number; }; +/** Return shape when `includeReasons: true` is passed to `resolveBlockingGroupKeys`. */ +export type BlockingGroupKeysWithReasons = { + readonly keys: string[]; + /** One entry per distinct blocking key produced, keyed by that key. */ + readonly reasonsByKey: ReadonlyMap; +}; + const DEFAULT_MIN_CSS_GROUP_SIZE = 2; /** @@ -129,10 +138,18 @@ const DEFAULT_MIN_CSS_GROUP_SIZE = 2; * // common.css is loaded by all 3 pages and is filtered out as non-discriminative chrome. * ``` */ +export function resolveBlockingGroupKeys( + pages: readonly PageBlockingSignals[], + options: ResolveBlockingGroupKeysOptions & { includeReasons: true }, +): BlockingGroupKeysWithReasons; export function resolveBlockingGroupKeys( pages: readonly PageBlockingSignals[], options?: ResolveBlockingGroupKeysOptions, -): string[] { +): string[]; +export function resolveBlockingGroupKeys( + pages: readonly PageBlockingSignals[], + options?: ResolveBlockingGroupKeysOptions & { includeReasons?: boolean }, +): string[] | BlockingGroupKeysWithReasons { const pathDepthOption = options?.pathDepth; const minCssGroupSize = options?.minCssGroupSize ?? DEFAULT_MIN_CSS_GROUP_SIZE; const hrefCommonThreshold = options?.hrefCommonThreshold; @@ -187,15 +204,29 @@ export function resolveBlockingGroupKeys( } } - return pages.map((page, index) => { + const reasonsByKey = new Map(); + const keys = pages.map((page, index) => { const cssKey = cssKeys[index]; if (cssKey !== undefined && (cssKeyCounts.get(cssKey) ?? 0) >= minCssGroupSize) { - return `css:${cssKey}`; + const key = `css:${cssKey}`; + if (!reasonsByKey.has(key)) { + reasonsByKey.set(key, { + kind: 'css', + distinctiveStylesheetHrefs: [...(distinctiveHrefs[index] ?? [])].toSorted(), + }); + } + return key; } const pathKey = perPagePathKeys === null ? derivePathGroupKey(page.paths, pathDepthOption as number | undefined) : (perPagePathKeys[index] ?? ''); - return `path:${pathKey}`; + const key = `path:${pathKey}`; + if (!reasonsByKey.has(key)) { + reasonsByKey.set(key, { kind: 'path', pathKey }); + } + return key; }); + + return options?.includeReasons ? { keys, reasonsByKey } : keys; } diff --git a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys-streaming.spec.ts b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys-streaming.spec.ts index a42e1fcd..68169c1b 100644 --- a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys-streaming.spec.ts +++ b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys-streaming.spec.ts @@ -1,3 +1,4 @@ +import type { ClusterReason } from './build-cluster-reason.js'; import type { PageClusterSignals, ProgressEvent } from './resolve-page-cluster-keys.js'; import { describe, expect, test } from 'vitest'; @@ -166,31 +167,43 @@ describe('resolvePageClusterKeys onProgress on small corpus', () => { }); }); -describe('resolvePageClusterKeys (includeLandmarkPositions)', () => { - test('small corpus with includeLandmarkPositions matches resolvePageClusterKeysInMemory', async () => { +describe('resolvePageClusterKeys (onClusterReason)', () => { + test('small corpus with onClusterReason produces the same keys and reasons as resolvePageClusterKeysInMemory', async () => { const pages = buildTinyCorpus(); + const streamedReasons = new Map(); const streamed = await resolvePageClusterKeys(() => pages, { - includeLandmarkPositions: true, + onClusterReason: (key, reason) => streamedReasons.set(key, reason), }); + const inMemoryReasons = new Map(); const inMemory = resolvePageClusterKeysInMemory(pages, { - includeLandmarkPositions: true, + onClusterReason: (key, reason) => inMemoryReasons.set(key, reason), }); expect(streamed).toEqual(inMemory); + expect(streamedReasons).toEqual(inMemoryReasons); + expect(streamedReasons.size).toBeGreaterThan(0); }); - test('combined with onProgress, still returns reports but emits no progress events (routed through the sync path)', async () => { + test('combined with onProgress, still fires onClusterReason but emits no progress events (routed through the sync path)', async () => { const pages = buildTinyCorpus(); const events: ProgressEvent[] = []; + const reasons = new Map(); const result = await resolvePageClusterKeys(() => pages, { - includeLandmarkPositions: true, + onClusterReason: (key, reason) => reasons.set(key, reason), onProgress: (event) => events.push(event), }); expect(result).toHaveLength(pages.length); - expect(result[0]).toHaveProperty('landmarks'); + expect(reasons.size).toBeGreaterThan(0); expect(events).toStrictEqual([]); }); - test('a corpus above CORPUS_INLINE_THRESHOLD rejects with a RangeError instead of streaming', async () => { + test('a corpus above CORPUS_INLINE_THRESHOLD completes on the streaming path and fires onClusterReason once per final cluster, without throwing', async () => { + // Regression guard for the fixed design flaw this replaces: the old + // `includeLandmarkPositions` retained every member page's landmark + // data until final clustering completed, which is why it had to + // reject corpora over this threshold outright. `ClusterReason` is + // sized by cluster count, not page count, so it has no such ceiling — + // this test's whole point is proving that by actually crossing the + // threshold instead of asserting a thrown error. const bigCount = CORPUS_INLINE_THRESHOLD + 1; /** * @yields {PageClusterSignals} A minimal page, `bigCount` times. @@ -200,8 +213,13 @@ describe('resolvePageClusterKeys (includeLandmarkPositions)', () => { yield { paths: ['p', String(i)], stylesheetHrefs: [], html: '' }; } } - await expect( - resolvePageClusterKeys(() => generate(), { includeLandmarkPositions: true }), - ).rejects.toThrow(RangeError); - }); + const reasons = new Map(); + const keys = await resolvePageClusterKeys(() => generate(), { + onClusterReason: (key, reason) => reasons.set(key, reason), + }); + expect(keys).toHaveLength(bigCount); + const distinctKeys = new Set(keys); + expect(reasons.size).toBe(distinctKeys.size); + for (const key of distinctKeys) expect(reasons.has(key)).toBe(true); + }, 30_000); }); diff --git a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.spec.ts b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.spec.ts index 566c22db..0d51c06f 100644 --- a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.spec.ts +++ b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.spec.ts @@ -1,7 +1,8 @@ +import type { ClusterReason } from './build-cluster-reason.js'; + import { describe, expect, test } from 'vitest'; import { - assertLandmarkPositionsSupportedForPageCount, resolvePageClusterKeysInMemory, resolvePageClusterKeysInMemory as resolvePageClusterKeys, } from './resolve-page-cluster-keys.js'; @@ -644,8 +645,39 @@ describe('resolvePageClusterKeys (local-landmark pseudo-token injection)', () => }); }); -describe('resolvePageClusterKeysInMemory (includeLandmarkPositions)', () => { - test('omitting includeLandmarkPositions returns a plain string[], unchanged from before', () => { +describe('resolvePageClusterKeysInMemory (onClusterReason)', () => { + test('two clusters split within the same block are reported as siblings of each other', () => { + // Same fixture as "pages within the same block but with dissimilar + // structure get different final keys" above: header/footer are + // identical (and excluded from comparison by default), but the + //
vs
distinguishing element still splits Stage A into + // two clusters within the single `path:dept-a` block. + const cardHtml = + '
H
C
F
'; + const formHtml = + '
H
F
F
'; + const reasons = new Map(); + const result = resolvePageClusterKeysInMemory( + [ + { paths: ['dept-a', 'page1'], stylesheetHrefs: [], html: cardHtml }, + { paths: ['dept-a', 'page2'], stylesheetHrefs: [], html: cardHtml }, + { paths: ['dept-a', 'page3'], stylesheetHrefs: [], html: formHtml }, + ], + { onClusterReason: (key, reason) => reasons.set(key, reason) }, + ); + + // Precondition: the two clusters really did split within one block. + expect(result[2]).not.toBe(result[0]); + expect(reasons.size).toBe(2); + + const cardReason = reasons.get(result[0]!)!; + const formReason = reasons.get(result[2]!)!; + expect(cardReason.siblingClusterKeys).toEqual([result[2]]); + expect(formReason.siblingClusterKeys).toEqual([result[0]]); + expect(cardReason.blocking).toEqual(formReason.blocking); + }); + + test('omitting onClusterReason returns a plain string[], unchanged from before', () => { const html = '
H
content
'; const result = resolvePageClusterKeysInMemory([ { paths: ['a'], stylesheetHrefs: [], html }, @@ -653,47 +685,47 @@ describe('resolvePageClusterKeysInMemory (includeLandmarkPositions)', () => { expect(result).toStrictEqual(['["path:a","cluster:0"]']); }); - test("a header shared across the whole cluster is reported as chrome; each page's own unique nav is not", () => { + test("a header shared across the whole cluster reads as chrome (chromeRate 1); each page's own unique nav does not", () => { const sharedHeader = '
'; const html1 = `${sharedHeader}
one
`; const html2 = `${sharedHeader}
two
`; + const reasons = new Map(); const result = resolvePageClusterKeysInMemory( [ { paths: ['p', '1'], stylesheetHrefs: [], html: html1 }, { paths: ['p', '2'], stylesheetHrefs: [], html: html2 }, ], - { includeLandmarkPositions: true }, + { onClusterReason: (key, reason) => reasons.set(key, reason) }, ); - expect(result).toHaveLength(2); // Both pages land in the same final cluster (identical header + same - // article structure), so shellQuorum is computed over both together. - expect(result[0]!.clusterKey).toBe(result[1]!.clusterKey); - expect(result[0]!.landmarks.header[0]!.isChrome).toBe(true); - expect(result[1]!.landmarks.header[0]!.isChrome).toBe(true); + // article structure), so one ClusterReason covers both. + expect(result[0]).toBe(result[1]); + expect(reasons.size).toBe(1); + const reason = reasons.get(result[0]!)!; + expect(reason.memberCount).toBe(2); + expect(reason.landmarks.header?.chromeRate).toBe(1); // Each page's nav carries a page-unique class not shared by the other // — content, not shell. - expect(result[0]!.landmarks.nav[0]!.isChrome).toBe(false); - expect(result[1]!.landmarks.nav[0]!.isChrome).toBe(false); + expect(reason.landmarks.nav?.chromeRate).toBe(0); }); - test('main instances are reported with position but no isChrome verdict', () => { + test('main never appears in ClusterReason.landmarks — chrome discovery does not classify content', () => { const html = '
H
content
'; - const result = resolvePageClusterKeysInMemory( - [{ paths: ['a'], stylesheetHrefs: [], html }], - { includeLandmarkPositions: true }, - ); - expect(result[0]!.landmarks.main).toHaveLength(1); - expect(Object.keys(result[0]!.landmarks.main[0]!)).not.toContain('isChrome'); + const reasons = new Map(); + resolvePageClusterKeysInMemory([{ paths: ['a'], stylesheetHrefs: [], html }], { + onClusterReason: (key, reason) => reasons.set(key, reason), + }); + expect(Object.keys([...reasons.values()][0]!.landmarks)).not.toContain('main'); }); - test('two distinct clusters each get their own shellQuorum, not a corpus-wide one', () => { + test('two distinct clusters each get their own shellQuorum-derived ClusterReason, not a corpus-wide one', () => { // Cluster A's header/nav tokens never appear anywhere in cluster B's // pages, and vice versa. If shellQuorum were mistakenly computed once // over all 6 pages instead of once per final cluster, each header's // corpus-wide frequency would be 3/6 = 0.5 — below the single-distinct- - // signature fallback clamp (0.8) — and neither would be chrome. Per- - // cluster computation gives each header a 3/3 = 1.0 frequency within - // its own cluster, so both are correctly chrome. + // signature fallback clamp (0.8) — and neither would read as chrome. + // Per-cluster computation gives each header a 3/3 = 1.0 frequency + // within its own cluster, so both correctly read as chrome. const headerA = '
'; const headerB = '
'; const clusterAPages = Array.from({ length: 3 }, (_, i) => ({ @@ -706,34 +738,24 @@ describe('resolvePageClusterKeysInMemory (includeLandmarkPositions)', () => { stylesheetHrefs: [], html: `${headerB}
content ${i}
`, })); + const reasons = new Map(); const result = resolvePageClusterKeysInMemory([...clusterAPages, ...clusterBPages], { - includeLandmarkPositions: true, + onClusterReason: (key, reason) => reasons.set(key, reason), }); // Precondition: the two groups really are in different final clusters // (different blocking path AND different tag skeleton: article vs // section) — otherwise this test wouldn't be exercising the // per-cluster grouping at all. - const clusterKeys = new Set(result.map((r) => r.clusterKey)); - expect(clusterKeys.size).toBe(2); - expect(result[0]!.clusterKey).toBe(result[1]!.clusterKey); - expect(result[0]!.clusterKey).toBe(result[2]!.clusterKey); - expect(result[3]!.clusterKey).toBe(result[4]!.clusterKey); - expect(result[3]!.clusterKey).toBe(result[5]!.clusterKey); - - for (const r of result) { - expect(r.landmarks.header[0]!.isChrome).toBe(true); - } - }); -}); - -describe('assertLandmarkPositionsSupportedForPageCount', () => { - test('does not throw when pageCount is at or below threshold', () => { - expect(() => assertLandmarkPositionsSupportedForPageCount(5, 5)).not.toThrow(); - expect(() => assertLandmarkPositionsSupportedForPageCount(4, 5)).not.toThrow(); - }); + expect(reasons.size).toBe(2); + expect(result[0]).toBe(result[1]); + expect(result[0]).toBe(result[2]); + expect(result[3]).toBe(result[4]); + expect(result[3]).toBe(result[5]); - test('throws a RangeError when pageCount exceeds threshold', () => { - expect(() => assertLandmarkPositionsSupportedForPageCount(6, 5)).toThrow(RangeError); + for (const reason of reasons.values()) { + expect(reason.landmarks.header?.chromeRate).toBe(1); + expect(reason.memberCount).toBe(3); + } }); }); diff --git a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.ts b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.ts index 1b7c1f45..3a5af04e 100644 --- a/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.ts +++ b/packages/@d-zero/page-cluster/src/resolve-page-cluster-keys.ts @@ -1,15 +1,14 @@ +import type { ClusterReason } from './build-cluster-reason.js'; +import type { BlockingReason } from './derive-blocking-reason.js'; import type { ExtractLandmarksResult } from './extract-landmarks.js'; -import type { CrossBlockUnit } from './merge-cross-block-clusters.js'; +import type { CrossBlockUnit, FinalGroupMembers } from './merge-cross-block-clusters.js'; import type { PerPageLandmarkInstance } from './per-page-landmark-signatures.js'; import type { ResolveBlockingGroupKeysOptions } from './resolve-blocking-group-keys.js'; import type { ResolveStructuralClusterKeysOptions } from './resolve-structural-cluster-keys.js'; import type { TokenizeOptions } from './types.js'; import { autoCutThreshold } from './auto-cut-threshold.js'; -import { - type PageLandmarkReport, - buildPageLandmarkReport, -} from './build-page-landmark-report.js'; +import { buildClusterReason } from './build-cluster-reason.js'; import { capContentDepth } from './cap-content-depth.js'; import { detectContentDepthCap, @@ -22,7 +21,6 @@ import { mergeCrossBlockClusters } from './merge-cross-block-clusters.js'; import { groupIndicesByBlockKey, resolveBlockKeys } from './pass0-blocking.js'; import { computePerPageLandmarkInstances } from './per-page-landmark-signatures.js'; import { removeContentBlocks } from './remove-content-blocks.js'; -import { shellQuorum } from './shell-quorum.js'; import { stageAPerBlock } from './stage-a-per-block.js'; import { tokenize } from './tokenize.js'; @@ -287,41 +285,70 @@ export function computeLocalLandmarkTokens( } /** - * Builds every page's {@link PageLandmarkReport} for the `includeLandmarkPositions` - * result path. Groups pages by their *final* cluster key (post–Stage-B), runs - * {@link ./shell-quorum.js | shellQuorum} once per cluster over the pooled - * member `PerPageLandmarkInstance`s (the same "shell tokens" concept Stage B - * itself uses for L2 shell corroboration, just recomputed at the final- - * cluster granularity rather than the pre-merge unit granularity), then - * classifies each member page's own landmark instances against that - * cluster-level shell. - * @param finalKeys - * @param landmarks - * @param perPageInstances + * Builds and emits one {@link ClusterReason} per final cluster via + * `onClusterReason`, from data Stage A/B already computed for clustering + * itself: `crossBlockUnits` (Stage A's pre-merge units, each carrying its + * originating block key inside `JSON.parse(unit.key)[0]`), Stage B's + * `rootByKey`/`finalGroupsByRoot`, the per-block-key `BlockingReason`s Pass 0 + * derived, and the per-block sibling-unit-key lists the driver accumulated + * alongside its Stage A loop. No re-tokenization and no extra corpus pass — + * this only re-groups references the driver already held. + * @param crossBlockUnits + * @param rootByKey + * @param finalGroupsByRoot + * @param reasonsByBlockKey + * @param siblingUnitKeysByBlock + * @param onClusterReason */ -function buildLandmarkReportsByCluster( - finalKeys: readonly string[], - landmarks: readonly ExtractLandmarksResult[], - perPageInstances: readonly (readonly PerPageLandmarkInstance[])[], -): PageLandmarkReport[] { - const indicesByKey = new Map(); - for (const [i, key] of finalKeys.entries()) { - const indices = indicesByKey.get(key); - if (indices) { - indices.push(i); +function emitClusterReasons( + crossBlockUnits: readonly CrossBlockUnit[], + rootByKey: ReadonlyMap, + finalGroupsByRoot: ReadonlyMap, + reasonsByBlockKey: ReadonlyMap, + siblingUnitKeysByBlock: ReadonlyMap, + onClusterReason: (clusterKey: string, reason: ClusterReason) => void, +): void { + const unitKeysByRoot = new Map(); + for (const unit of crossBlockUnits) { + const root = rootByKey.get(unit.key) ?? unit.key; + const list = unitKeysByRoot.get(root); + if (list) { + list.push(unit.key); } else { - indicesByKey.set(key, [i]); + unitKeysByRoot.set(root, [unit.key]); } } - const reports: PageLandmarkReport[] = Array.from({ length: finalKeys.length }); - for (const indices of indicesByKey.values()) { - const shellTokens = shellQuorum(indices.map((i) => perPageInstances[i]!)); - for (const i of indices) { - reports[i] = buildPageLandmarkReport(landmarks[i]!, shellTokens); + for (const [rootKey, unitKeys] of unitKeysByRoot) { + const finalGroup = finalGroupsByRoot.get(rootKey); + if (!finalGroup) continue; + + const seenBlockKeys = new Set(); + const blocking: { blockKey: string; reason: BlockingReason }[] = []; + const siblingRoots = new Set(); + for (const unitKey of unitKeys) { + const blockKey = JSON.parse(unitKey)[0] as string; + if (!seenBlockKeys.has(blockKey)) { + seenBlockKeys.add(blockKey); + const reason = reasonsByBlockKey.get(blockKey); + if (reason) blocking.push({ blockKey, reason }); + } + for (const siblingUnitKey of siblingUnitKeysByBlock.get(blockKey) ?? []) { + const siblingRoot = rootByKey.get(siblingUnitKey) ?? siblingUnitKey; + if (siblingRoot !== rootKey) siblingRoots.add(siblingRoot); + } } + + onClusterReason( + rootKey, + buildClusterReason({ + tokenSets: finalGroup.tokenSets, + landmarkInstances: finalGroup.landmarkInstances, + blocking, + siblingClusterKeys: [...siblingRoots].toSorted(), + }), + ); } - return reports; } /** @@ -408,50 +435,44 @@ export type ResolvePageClusterKeysOptions = TokenizeOptions & * sync helper to running a per-block async loop that emits * `pass1-block-complete` and `stage-b-start`. Omitting `onProgress` * keeps the small-corpus branch on the pre-refactor sync path with - * zero yield overhead. Ignored when `includeLandmarkPositions` is - * `true` — that option always routes the small-corpus branch through - * the sync helper (see `includeLandmarkPositions`'s own JSDoc), so no - * progress events fire in that combination. + * zero yield overhead. Ignored when `onClusterReason` is set — that + * option always routes the small-corpus branch through the sync + * helper (see `onClusterReason`'s own JSDoc), so no progress events + * fire in that combination. */ onProgress?: (event: ProgressEvent) => void; /** - * When `true`, every result entry additionally carries a - * {@link PageLandmarkReport} — each landmark instance's position, plus - * a chrome/content verdict for the six excisable types (see - * {@link ./build-page-landmark-report.js | buildPageLandmarkReport}). - * Changes the return shape from `string[]` to - * `{@link PageClusterKeyResult}[]` (see the overloads on - * `resolvePageClusterKeysInMemory`/`resolvePageClusterKeys`/ - * `resolvePageClusterKeysFromArray`). Defaults to `false`, in which - * case every existing caller's behavior and return type are - * unchanged. + * Optional observability hook invoked once per **final cluster** (not + * per page) with that cluster's {@link ClusterReason} — the blocking + * signal that grouped it, its DOM-structural token core, its + * per-landmark-type commonality, and the sibling cluster keys it was + * split from within the same Pass-0 block. Unlike `onProgress`, this + * fires on every path — small-corpus and streaming alike — because a + * `ClusterReason` is sized by cluster count, not page count, so it + * carries no streaming-path memory risk the way a per-page report + * would. * - * Not supported on the streaming path - * (`pageCount > {@link CORPUS_INLINE_THRESHOLD}`): reservoir sampling - * and Jaccard-based non-sample assignment there have no notion of - * "this page's shell tokens" to classify chrome against, and - * retrofitting one is out of scope. `resolvePageClusterKeys` throws a - * `RangeError` up front if both apply to the same call, rather than - * silently degrading semantics. + * Building the reasons re-uses Stage A/B's own intermediate state (the + * quorum core, the per-unit landmark instances, the blocking + * evidence) — it does not re-tokenize pages or re-run corpus-wide + * discovery. Omitting `onClusterReason` skips that bookkeeping + * entirely, so existing callers pay nothing for this option. * - * On the async factory-based `resolvePageClusterKeys`, this option + * On the async factory-based `resolvePageClusterKeys`, setting this * forces the small-corpus branch through the sync * `resolvePageClusterKeysInMemory` helper regardless of `onProgress` * — see `onProgress`'s own JSDoc. + * @example + * ```ts + * const reasons = new Map(); + * const keys = await resolvePageClusterKeys(pages, { + * onClusterReason: (key, reason) => reasons.set(key, reason), + * }); + * ``` */ - includeLandmarkPositions?: boolean; + onClusterReason?: (clusterKey: string, reason: ClusterReason) => void; }; -/** - * One page's clustering result when `includeLandmarkPositions` is `true`: - * the same `clusterKey` every caller already gets, plus that page's - * {@link PageLandmarkReport}. - */ -export type PageClusterKeyResult = { - readonly clusterKey: string; - readonly landmarks: PageLandmarkReport; -}; - /** * Corpus size at or below which the async factory-based * `resolvePageClusterKeys` reads the entire input into an array and delegates @@ -475,27 +496,6 @@ export type PageClusterKeyResult = { */ export const CORPUS_INLINE_THRESHOLD = 20_000; -/** - * Throws when `includeLandmarkPositions` is combined with a corpus over - * `threshold` pages (the streaming path — see `includeLandmarkPositions`'s - * own JSDoc for why it has no sample-based equivalent there). Split out from - * its call site so tests can exercise the boundary with a small injected - * `threshold` instead of constructing a 20,001-page fixture to cross the - * real {@link CORPUS_INLINE_THRESHOLD}. - * @param pageCount - * @param threshold - */ -export function assertLandmarkPositionsSupportedForPageCount( - pageCount: number, - threshold: number, -): void { - if (pageCount > threshold) { - throw new RangeError( - `resolvePageClusterKeys: includeLandmarkPositions is not supported for corpora larger than ${threshold} pages (the streaming path) — this corpus has ${pageCount}`, - ); - } -} - /** * Reservoir-sample size per block on the streaming path. Blocks larger than * this have Stage A run on a random sample of `BLOCK_SAMPLE_SIZE` pages, @@ -549,18 +549,10 @@ export const BLOCK_SAMPLE_SIZE = 100; * @param pages * @param options */ -export function resolvePageClusterKeysInMemory( - pages: readonly PageClusterSignals[], - options: ResolvePageClusterKeysOptions & { includeLandmarkPositions: true }, -): PageClusterKeyResult[]; -export function resolvePageClusterKeysInMemory( - pages: readonly PageClusterSignals[], - options?: ResolvePageClusterKeysOptions, -): string[]; export function resolvePageClusterKeysInMemory( pages: readonly PageClusterSignals[], options?: ResolvePageClusterKeysOptions, -): string[] | PageClusterKeyResult[] { +): string[] { const excludeLandmarks = options?.excludeLandmarks ?? true; const similarityThreshold = options?.similarityThreshold ?? 0.8; @@ -571,18 +563,15 @@ export function resolvePageClusterKeysInMemory( } // Always computed: landmark fields are needed by Stage B's shell - // corroboration regardless of `excludeLandmarks`, and `remainderHtml` is - // needed whenever `excludeLandmarks` is true. + // corroboration regardless of `excludeLandmarks`. const landmarks: readonly ExtractLandmarksResult[] = pages.map((page) => extractLandmarks(page.html), ); - // Corpus-level chrome discovery. `perPageInstances` is only consumed - // below when `includeLandmarkPositions` is set — computed unconditionally - // anyway since `computeLocalChromeArtifacts` already builds it internally - // for chrome discovery, so exposing it here costs nothing extra. - const { localTokensByPage: localLandmarkTokensByPage, perPageInstances } = - computeLocalChromeArtifacts(landmarks, options); + const { localTokensByPage: localLandmarkTokensByPage } = computeLocalChromeArtifacts( + landmarks, + options, + ); const contentBlockAttribute = options?.contentBlockAttribute; const preparedHtml = pages.map((page, index) => { @@ -601,7 +590,19 @@ export function resolvePageClusterKeysInMemory( ? filterFirstPartyStylesheetHrefs(pages) : pages; - const blockKeys = resolveBlockKeys(blockingPages, options); + // Reasons (blocking evidence) are only worth deriving when a caller + // actually asked for `onClusterReason` — see that option's own JSDoc for + // why this is the only place ClusterReason bookkeeping is opt-in. + const onClusterReason = options?.onClusterReason; + let blockKeys: string[]; + let reasonsByBlockKey: ReadonlyMap | undefined; + if (onClusterReason) { + const result = resolveBlockKeys(blockingPages, { ...options, includeReasons: true }); + blockKeys = result.blockKeys; + reasonsByBlockKey = result.reasonsByBlockKey; + } else { + blockKeys = resolveBlockKeys(blockingPages, options); + } const indicesByBlockKey = groupIndicesByBlockKey(blockKeys); // Validated here, eagerly, because it's otherwise only reached from @@ -612,6 +613,9 @@ export function resolvePageClusterKeysInMemory( const finalKeys: string[] = Array.from({ length: pages.length }); const crossBlockUnits: CrossBlockUnit[] = []; + const siblingUnitKeysByBlock = onClusterReason + ? new Map() + : undefined; for (const [blockKey, indices] of indicesByBlockKey) { const result = stageAPerBlock( @@ -628,22 +632,37 @@ export function resolvePageClusterKeysInMemory( finalKeys[pageIndex] = key; } crossBlockUnits.push(...result.crossBlockUnits); + siblingUnitKeysByBlock?.set( + blockKey, + result.crossBlockUnits.map((u) => u.key), + ); } // Stage B: cross-block merge — always runs regardless of options - const stageBResult = mergeCrossBlockClusters(crossBlockUnits, options); + const { rootByKey, finalGroupsByRoot } = mergeCrossBlockClusters( + crossBlockUnits, + options, + ); for (let i = 0; i < finalKeys.length; i++) { const currentKey = finalKeys[i]!; - const rootKey = stageBResult.get(currentKey); + const rootKey = rootByKey.get(currentKey); if (rootKey !== undefined && rootKey !== currentKey) { finalKeys[i] = rootKey; } } - if (!options?.includeLandmarkPositions) return finalKeys; + if (onClusterReason && reasonsByBlockKey && siblingUnitKeysByBlock) { + emitClusterReasons( + crossBlockUnits, + rootByKey, + finalGroupsByRoot, + reasonsByBlockKey, + siblingUnitKeysByBlock, + onClusterReason, + ); + } - const reports = buildLandmarkReportsByCluster(finalKeys, landmarks, perPageInstances); - return finalKeys.map((clusterKey, i) => ({ clusterKey, landmarks: reports[i]! })); + return finalKeys; } /** @@ -746,10 +765,10 @@ async function resolveSmallCorpusWithProgress( onProgress({ phase: 'stage-b-start', unitCount: crossBlockUnits.length }); - const stageBResult = mergeCrossBlockClusters(crossBlockUnits, options); + const { rootByKey } = mergeCrossBlockClusters(crossBlockUnits, options); for (let i = 0; i < finalKeys.length; i++) { const currentKey = finalKeys[i]!; - const rootKey = stageBResult.get(currentKey); + const rootKey = rootByKey.get(currentKey); if (rootKey !== undefined && rootKey !== currentKey) { finalKeys[i] = rootKey; } @@ -821,18 +840,10 @@ export type PageFactory = () => * }); * ``` */ -export async function resolvePageClusterKeys( - pages: PageFactory, - options: ResolvePageClusterKeysOptions & { includeLandmarkPositions: true }, -): Promise; -export async function resolvePageClusterKeys( - pages: PageFactory, - options?: ResolvePageClusterKeysOptions, -): Promise; export async function resolvePageClusterKeys( pages: PageFactory, options?: ResolvePageClusterKeysOptions, -): Promise { +): Promise { const onProgress = options?.onProgress; // Pass 0: HTML-free — collect blocking signals (paths, stylesheetHrefs, // host) into an array. This is the only per-page state we keep across @@ -872,27 +883,18 @@ export async function resolvePageClusterKeys( // into per-block progress, so delegate to the untouched sync path — // keeping behavior byte-for-byte identical (and yield-overhead-free) // to how library-only consumers experienced this before the CLI - // progress work landed. `includeLandmarkPositions` always routes here - // too (see its own JSDoc): `resolveSmallCorpusWithProgress` has no - // landmark-report support, and duplicating that logic into the + // progress work landed. `onClusterReason` always routes here too (see + // its own JSDoc): `resolveSmallCorpusWithProgress` has no + // cluster-reason support, and duplicating that logic into the // progress-emitting path for a reporting feature that has nothing to // do with progress observability isn't worth the added surface. - if (onProgress === undefined || options?.includeLandmarkPositions) { + if (onProgress === undefined || options?.onClusterReason) { return resolvePageClusterKeysInMemory(fullPages, options); } return resolveSmallCorpusWithProgress(fullPages, onProgress, options); } - // Large corpus: streaming path. includeLandmarkPositions has no sample- - // based equivalent (see its own JSDoc) — fail fast rather than silently - // ignoring the option or returning a semantically-wrong report. - if (options?.includeLandmarkPositions) { - assertLandmarkPositionsSupportedForPageCount( - blockingSignals.length, - CORPUS_INLINE_THRESHOLD, - ); - } - + // Large corpus: streaming path. const excludeLandmarks = options?.excludeLandmarks ?? true; const similarityThreshold = options?.similarityThreshold ?? 0.8; if (!(similarityThreshold >= 0 && similarityThreshold <= 1)) { @@ -907,12 +909,31 @@ export async function resolvePageClusterKeys( const blockingPagesForKeys = restrictStylesheetsToFirstParty ? filterFirstPartyStylesheetHrefs(blockingSignals) : blockingSignals; - const blockKeys = resolveBlockKeys(blockingPagesForKeys, options); + + // Reasons (blocking evidence) cost nothing beyond bookkeeping — a Map + // keyed by distinct block key, not by page — but are only derived when a + // caller actually asked for `onClusterReason`. + const onClusterReason = options?.onClusterReason; + let blockKeys: string[]; + let reasonsByBlockKey: ReadonlyMap | undefined; + if (onClusterReason) { + const result = resolveBlockKeys(blockingPagesForKeys, { + ...options, + includeReasons: true, + }); + blockKeys = result.blockKeys; + reasonsByBlockKey = result.reasonsByBlockKey; + } else { + blockKeys = resolveBlockKeys(blockingPagesForKeys, options); + } const indicesByBlockKey = groupIndicesByBlockKey(blockKeys); const finalKeys: string[] = Array.from({ length: blockingSignals.length }); const crossBlockUnits: CrossBlockUnit[] = []; const contentBlockAttribute = options?.contentBlockAttribute; + const siblingUnitKeysByBlock = onClusterReason + ? new Map() + : undefined; /** * Per-block bucket accumulates a reservoir sample of the block's pages @@ -989,6 +1010,10 @@ export async function resolvePageClusterKeys( finalKeys[idx] = key; } crossBlockUnits.push(...result.crossBlockUnits); + siblingUnitKeysByBlock?.set( + bucket.blockKey, + result.crossBlockUnits.map((u) => u.key), + ); if (bucket.seenCount > bucket.reservoirIndices.length) { // Save assignment artifacts for Pass 1b. @@ -1134,17 +1159,28 @@ export async function resolvePageClusterKeys( if (onProgress) { onProgress({ phase: 'stage-b-start', unitCount: crossBlockUnits.length }); } - const stageBResult = mergeCrossBlockClusters(crossBlockUnits, { + const { rootByKey, finalGroupsByRoot } = mergeCrossBlockClusters(crossBlockUnits, { ...options, capMembers: BLOCK_SAMPLE_SIZE, }); for (let i = 0; i < finalKeys.length; i++) { const currentKey = finalKeys[i]!; - const rootKey = stageBResult.get(currentKey); + const rootKey = rootByKey.get(currentKey); if (rootKey !== undefined && rootKey !== currentKey) { finalKeys[i] = rootKey; } } + + if (onClusterReason && reasonsByBlockKey && siblingUnitKeysByBlock) { + emitClusterReasons( + crossBlockUnits, + rootByKey, + finalGroupsByRoot, + reasonsByBlockKey, + siblingUnitKeysByBlock, + onClusterReason, + ); + } return finalKeys; } @@ -1164,17 +1200,9 @@ export async function resolvePageClusterKeys( * ]); * ``` */ -export function resolvePageClusterKeysFromArray( - pages: readonly PageClusterSignals[], - options: ResolvePageClusterKeysOptions & { includeLandmarkPositions: true }, -): Promise; export function resolvePageClusterKeysFromArray( pages: readonly PageClusterSignals[], options?: ResolvePageClusterKeysOptions, -): Promise; -export function resolvePageClusterKeysFromArray( - pages: readonly PageClusterSignals[], - options?: ResolvePageClusterKeysOptions, -): Promise { +): Promise { return resolvePageClusterKeys(() => pages, options); } diff --git a/packages/@d-zero/page-cluster/src/shell-quorum.ts b/packages/@d-zero/page-cluster/src/shell-quorum.ts index e62b8899..367efcde 100644 --- a/packages/@d-zero/page-cluster/src/shell-quorum.ts +++ b/packages/@d-zero/page-cluster/src/shell-quorum.ts @@ -74,10 +74,10 @@ const SHELL_QUORUM_FALLBACK_FRACTION = 0.8; * * Originally private to {@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters}'s * Stage B L2 corroboration; exported from its own module so - * {@link ./resolve-page-cluster-keys.js | resolvePageClusterKeysInMemory}'s - * `includeLandmarkPositions` reporting path can run it once per final - * cluster to classify individual landmark instances as chrome (see - * {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance}). + * {@link ./build-cluster-reason.js | buildClusterReason} can run it once per + * final cluster, per landmark type, to classify individual landmark + * instances as chrome (see {@link ./is-chrome-landmark-instance.js | + * isChromeLandmarkInstance}). * @param perPageInstances */ export function shellQuorum(