Skip to content

feat(page-cluster)!: add landmark position + chrome reporting #926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
YusukeHirao merged 1 commit into from
Jul 24, 2026
Merged

feat(page-cluster)!: add landmark position + chrome reporting #926

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 47 additions & 7 deletions packages/@d-zero/page-cluster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ yarn add @d-zero/page-cluster
### CLI

```sh
page-cluster [--content-block-attribute <name>] < pages.jsonl > clusters.jsonl
page-cluster [--content-block-attribute <name>] [--include-landmark-positions] < pages.jsonl > clusters.jsonl
```

**入力**: JSONL 1 行 1 ページ。フィールドは以下。`html` 以外はすべて任意（`paths` / `stylesheetHrefs` がないと粗い分類になる）。
Expand All @@ -36,6 +36,45 @@ page-cluster [--content-block-attribute <name>] < pages.jsonl > clusters.jsonl
{ "id": "任意の識別子", "clusterKey": "..." }
```

`--include-landmark-positions` を指定すると、各行に `landmarks` フィールドが追加される。header / footer / nav / aside / form / search / main のインスタンスごとに、HTML 内の位置（1-based の line/column と文字列オフセットの両方）を返す。header 〜 search の 6 種は追加で、そのページが属する最終クラスタ内での頻度分析（`shellQuorum`）に基づく `isChrome`（サイト/セクション共通の chrome か、ページ固有のコンテンツか）を持つ。`main` は常にコンテンツなので `isChrome` を持たない。

```json
{
"id": "任意の識別子",
"clusterKey": "...",
"landmarks": {
"header": [
{
"startLine": 1,
"startColumn": 7,
"endLine": 1,
"endColumn": 30,
"startOffset": 6,
"endOffset": 29,
"isChrome": true
}
],
"footer": [],
"nav": [],
"aside": [],
"form": [],
"search": [],
"main": [
{
"startLine": 2,
"startColumn": 1,
"endLine": 10,
"endColumn": 8,
"startOffset": 40,
"endOffset": 120
}
]
}
}
```

20,000 ページを超えるコーパス（ストリーミング経路）では `--include-landmark-positions` は使えない（エラーで終了する）。ストリーミング経路はリザーバサンプリングと近似割当を使うため、ページ単位の chrome 判定に必要な「そのページが属する最終クラスタの shell トークン」という概念を持たないため。

クローラ出力が JSON 配列の場合は `jq` で line-delimited に変換して食わせる:

```sh
Expand All @@ -45,6 +84,7 @@ jq -c '.[]' crawl-output.json | page-cluster > clusters.jsonl
#### オプション

- `--content-block-attribute <name>` — CMS が自由編集コンテンツブロックに付与している属性名（例: `data-bgb`）が分かっている場合に指定する。指定すると比較前にその属性を持つ要素配下を無視するので、同じテンプレートで本文構成だけ違うページを混同しなくなる。唯一の site-specific なオプションで、未指定でも `<main>` / `role="main"` を起点にした自動深さキャップが常時働く（詳細は `resolve-page-cluster-keys.ts` の JSDoc を参照）
- `--include-landmark-positions` — 出力の各行に上記の `landmarks` フィールドを追加する。20,000 ページ超のコーパスでは使えない。指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため）
- `--help` / `-h` — ヘルプを表示する
- `--version` / `-v` — バージョンを表示する

Expand Down Expand Up @@ -76,12 +116,12 @@ silence したい場合は `2>/dev/null`。ログに残したい場合は `2> pr

サブパスエクスポート構成。import パスと提供関数の対応は以下。

| import パス | 提供関数 |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@d-zero/page-cluster` | `tokenize` — `<body>` 配下を構造トークン列に変換する低レベルプリミティブ |
| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力） |
| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` — 6 種の HTML5 ランドマーク（header / footer / nav / aside / form / search）を抽出 |
| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類 |
| import パス | 提供関数 |
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@d-zero/page-cluster` | `tokenize` — `<body>` 配下を構造トークン列に変換する低レベルプリミティブ |
| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。いずれも `includeLandmarkPositions: true` を渡すと `clusterKey` に加えて位置情報つきの `landmarks`（`PageLandmarkReport`）を返す |
| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す |
| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類 |

```ts
import { resolvePageClusterKeysFromArray } from '@d-zero/page-cluster/resolve-page-cluster-keys';
Expand Down
64 changes: 64 additions & 0 deletions packages/@d-zero/page-cluster/src/build-page-landmark-report.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import { describe, expect, test } from 'vitest';

import { buildPageLandmarkReport } from './build-page-landmark-report.js';
import { extractLandmarks } from './extract-landmarks.js';
import { tokenize } from './tokenize.js';

describe('buildPageLandmarkReport', () => {
test('a header instance whose tokens are all shell tokens is reported as chrome', () => {
const landmarks = extractLandmarks(
'<body><header><nav><a>Home</a></nav></header><main>content</main></body>',
);
const shellTokens = new Set(
tokenize(`<body>${landmarks.header[0]!.html}</body>`).tokens,
);
const report = buildPageLandmarkReport(landmarks, shellTokens);
expect(report.header).toHaveLength(1);
expect(report.header[0]!.isChrome).toBe(true);
});

test('an instance with no overlap with shell tokens is reported as content', () => {
const landmarks = extractLandmarks(
'<body><header><p class="unique-per-page">x</p></header><main>content</main></body>',
);
const shellTokens = new Set(['body>totally>unrelated']);
const report = buildPageLandmarkReport(landmarks, shellTokens);
expect(report.header[0]!.isChrome).toBe(false);
});

test('main instances carry no isChrome field', () => {
const landmarks = extractLandmarks('<body><main>content</main></body>');
const report = buildPageLandmarkReport(landmarks, new Set());
expect(report.main).toHaveLength(1);
expect(Object.keys(report.main[0]!)).not.toContain('isChrome');
});

test('a page with no landmarks yields all-empty arrays', () => {
const landmarks = extractLandmarks('<body>content only</body>');
const report = buildPageLandmarkReport(landmarks, new Set());
expect(report).toStrictEqual({
header: [],
footer: [],
nav: [],
aside: [],
form: [],
search: [],
main: [],
});
});

test('each reported instance carries the instance position, not its html', () => {
const landmarks = extractLandmarks('<body><header>H</header><main>M</main></body>');
const report = buildPageLandmarkReport(landmarks, new Set());
const source = landmarks.header[0]!;
expect(report.header[0]).toStrictEqual({
startOffset: source.startOffset,
endOffset: source.endOffset,
startLine: source.startLine,
startColumn: source.startColumn,
endLine: source.endLine,
endColumn: source.endColumn,
isChrome: false,
});
});
});
107 changes: 107 additions & 0 deletions packages/@d-zero/page-cluster/src/build-page-landmark-report.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
import type {
ExtractLandmarksResult,
LandmarkInstance,
LandmarkPosition,
} from './extract-landmarks.js';
import type { TokenizeOptions } from './types.js';

import { isChromeLandmarkInstance } from './is-chrome-landmark-instance.js';
import { ALL_LANDMARK_TYPES } from './per-page-landmark-signatures.js';
import { tokenize } from './tokenize.js';

/**
* A landmark instance's position plus whether
* {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance} judged
* it shared site/section chrome (`true`) or page-specific content (`false`),
* against the unit's {@link ./shell-quorum.js | shellQuorum} shell tokens.
*/
export type ReportedLandmarkInstance = LandmarkPosition & { readonly isChrome: boolean };

/**
* Per-page landmark position report built by
* {@link ./build-page-landmark-report.js | buildPageLandmarkReport}. `main`
* carries no `isChrome` verdict — it never participates in chrome/shell
* discovery (see `extractLandmarks`'s "main handling" note) and is always
* content.
*/
export type PageLandmarkReport = {
header: ReportedLandmarkInstance[];
footer: ReportedLandmarkInstance[];
nav: ReportedLandmarkInstance[];
aside: ReportedLandmarkInstance[];
form: ReportedLandmarkInstance[];
search: ReportedLandmarkInstance[];
main: LandmarkPosition[];
};

/**
* Strips `html` off a {@link LandmarkInstance}, keeping only its position.
* `buildPageLandmarkReport`'s output is meant to be serialized per page
* across a whole corpus (the CLI's JSONL output), so the report
* deliberately excludes each instance's raw HTML to keep that payload from
* scaling with markup size — callers who also need the HTML already have
* `ExtractLandmarksResult` in hand.
* @param instance
*/
function toPosition(instance: LandmarkInstance): LandmarkPosition {
return {
startOffset: instance.startOffset,
endOffset: instance.endOffset,
startLine: instance.startLine,
startColumn: instance.startColumn,
endLine: instance.endLine,
endColumn: instance.endColumn,
};
}

/**
* Builds a page's landmark position report: every landmark instance's
* location, with `header`/`footer`/`nav`/`aside`/`form`/`search` instances
* additionally classified as chrome or content against `shellTokens`.
*
* Reads `landmarks` directly — the full, non-deduplicated instance list
* `extractLandmarks` produced — rather than going through
* {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}'s
* per-page-deduplicated `PerPageLandmarkInstance[]`: that dedupe collapses
* same-signature instances to one entry, which would silently drop the
* position of every duplicate instance a position report needs to include.
* @param landmarks
* @param shellTokens The unit-level shell token set from
* {@link ./shell-quorum.js | shellQuorum}, computed once per final cluster
* and shared across every member page's report.
* @param tokenizeOptions
* @example
* ```ts
* const shellTokens = shellQuorum(clusterPerPageInstances);
* const report = buildPageLandmarkReport(extractLandmarks(page.html), shellTokens);
* ```
*/
export function buildPageLandmarkReport(
landmarks: ExtractLandmarksResult,
shellTokens: ReadonlySet<string>,
tokenizeOptions?: TokenizeOptions,
): PageLandmarkReport {
const report: PageLandmarkReport = {
header: [],
footer: [],
nav: [],
aside: [],
form: [],
search: [],
main: landmarks.main.map(toPosition),
};

for (const type of ALL_LANDMARK_TYPES) {
for (const instance of landmarks[type]) {
const tokens = instance.html
? new Set(tokenize(`<body>${instance.html}</body>`, tokenizeOptions).tokens)
: new Set<string>();
report[type].push({
...toPosition(instance),
isChrome: isChromeLandmarkInstance(tokens, shellTokens),
});
}
}

return report;
}
75 changes: 75 additions & 0 deletions packages/@d-zero/page-cluster/src/cli.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,12 @@ describe('parseArgs', () => {
});
});

test('--include-landmark-positions', () => {
expect(parseArgs(['--include-landmark-positions'])).toEqual({
includeLandmarkPositions: true,
});
});

test('unknown flag is captured', () => {
expect(parseArgs(['--nope'])).toEqual({ unknownFlag: '--nope' });
});
Expand Down Expand Up @@ -143,6 +149,19 @@ describe('runCli', () => {
expect(stdout.read()).toMatch(/Usage:/);
});

test('--help mentions --include-landmark-positions', async () => {
const stdout = makeCollector();
const stderr = makeCollector();
await runCli({
stdin: makeStdin(''),
stdout: stdout.stream,
stderr: stderr.stream,
argv: ['--help'],
version: '0.0.0',
});
expect(stdout.read()).toMatch(/--include-landmark-positions/);
});

test('--version prints version and exits 0', async () => {
const stdout = makeCollector();
const stderr = makeCollector();
Expand Down Expand Up @@ -290,4 +309,60 @@ describe('runCli', () => {
/\[page-cluster\] error: .*failed to parse JSONL/,
);
});

test('--include-landmark-positions adds a landmarks field to each output line', async () => {
const input = [
JSON.stringify({
id: 'a',
paths: ['news', '1'],
stylesheetHrefs: [],
html: '<body><header>H</header><main><article>one</article></main></body>',
}),
JSON.stringify({
id: 'b',
paths: ['news', '2'],
stylesheetHrefs: [],
html: '<body><header>H</header><main><article>two</article></main></body>',
}),
].join('\n');
const stdout = makeCollector();
const stderr = makeCollector();
const code = await runCli({
stdin: makeStdin(input),
stdout: stdout.stream,
stderr: stderr.stream,
argv: ['--include-landmark-positions'],
version: '0.0.0',
});
expect(code).toBe(0);
const lines = stdout.read().split('\n').filter(Boolean);
expect(lines).toHaveLength(2);
const parsed = lines.map(
(line) =>
JSON.parse(line) as {
id: string;
clusterKey: string;
landmarks: { header: { isChrome: boolean }[]; main: object[] };
},
);
expect(parsed[0]!.landmarks.header[0]!.isChrome).toBe(true);
expect(parsed[0]!.landmarks.main).toHaveLength(1);
});

test('without --include-landmark-positions, output lines carry no landmarks field', async () => {
const stdout = makeCollector();
const stderr = makeCollector();
const code = await runCli({
stdin: makeStdin(
JSON.stringify({ id: 'a', html: '<body><header>H</header></body>' }),
),
stdout: stdout.stream,
stderr: stderr.stream,
argv: [],
version: '0.0.0',
});
expect(code).toBe(0);
const line = stdout.read().split('\n').find(Boolean);
expect(JSON.parse(line!)).not.toHaveProperty('landmarks');
});
});
Loading
Loading