Skip to content

Commit 0744f94

Browse files
feat(scripts): a third test-budget class, "compiles the package type graph once" (#23897)
Fixes #23821 Clause-②: no The refresh diff's closed list of test-budget classes (`TEST_BUDGET_CLASSES` in `scripts/measure-test-shard-timings.mjs`) gains its third value, `compiles the package type graph once`. A test file whose irreducible cost is one compile of its package's type graph can now declare its own measured budget, under the same grammar, number rule and refusals as the first two classes. One file changed: +107 / −16. ## The ruling Ruling record `6212133431` (letter A, the maintainer's 「同意」), which extends ruling record `6181973363` on #23534 by one class. Verbatim: > 1. **The class name** is `compiles the package type graph once`. It is the third value of `TEST_BUDGET_CLASSES` (`scripts/measure-test-shard-timings.mjs:1255-1258` at this read, two values); the refusal at `:1295` and its text at `:1269` list three classes; the self-test at `:2997` pins three. > 2. **The number** stays ruling 6181973363 item 2: the measured CI reading × 1.25, rounded up to the next 10 s, rising only by a PR that re-measures and says why in the line. **The three prohibitions** (item 5) stand: no deletion without a named duplicate, no timeout raise, no move to a slower or skipped project. > 4. No `AGENTS.md` edit: `AGENTS.md:129-130` names the ruling record and no class list. Ruling item 3 (the two declarations themselves) is the spec lane's. **The two header lines in `packages/spec/src/contracts/scoped-context.test.ts` and `packages/spec/src/integration/connector-author-shape.test.ts` follow on #23180 in the spec lane once this lands.** This PR touches no `packages/**` file, no workflow and no `AGENTS.md`. ## Every place in the script that changed Line numbers are at `origin/main` `497518c3f`. They match the ruling's read. | Where (`origin/main`) | Before | After | |:--|:--|:--| | `:1225-1231`, the `readTestBudget` docblock | "(ruling record 6181973363, letter B)"; "Two classes …"; "A file of either class …" | Cites both records. "Three classes …" names the third (a single compile of the package's type graph, every probe after the first reading that one program; ruling record 6212133431, letter A, which adds this class and changes nothing else). "A file of any class …" | | `:1239`, the grammar's CLASS row | "spelled as the ruling names it" | "spelled as its ruling names it" | | `:1255-1258`, `TEST_BUDGET_CLASSES` | two values | a third value, `'compiles the package type graph once'` | | `:1268-1269`, the refusal's text | "(ruling record 6181973363)"; `CLASS is "a" or "b"` | "(ruling records 6181973363 and 6212133431)"; `CLASS is one of "a", "b", "c"`, so every refusal lists all three classes | | `:1295`, the class refusal | `the class "X" is neither class the ruling names` | `the class "X" is none of the 3 classes the rulings name` (the count is read from the list) | | `:1662-1664`, the battery roster | `'per-file budget declarations (ruling 6181973363)': 14` | `18`, and the comment names ruling record 6212133431's cases | | `:2982-2986`, self-test fixtures | `[producerClass, wholePackageClass]` | adds `typeGraphClass`; `ruledClasses` (the three spellings as **literals**); `specFileWith` (the two spec files' real header, declaration on line 2); `typeGraphLine` (the ruling's line) | | `:2996-2999`, the class pin | `length !== 2`, two spellings | `length !== 3`, three spellings in order, against the literals | | `:3019-3021`, the off-list class case | "a class naming neither" | "a class naming none of them" (wording only) | The only other places that cite the mechanism name no class and list none, so they are unchanged: - `:212-217` (`FILE_BUDGET_SECONDS`' doc); - `:1545` (the rendered "Held to the budget it declares" line prints each file's own class); - the self-test section banner at `:2980`. A tree-wide grep for `TEST_BUDGET_CLASSES`, either class phrase and `test-budget` outside this script found only the 12 declaring test files. It found nothing in `docs/`, `content/`, `.claude/`, `AGENTS.md` or `.github/workflows/`. ## Proof the new value is read end to end Four new self-test cases are in battery `per-file budget declarations (ruling 6181973363)`, which grows from 14 to 18: 1. **Accepted.** The ruling's line, `// test-budget: 30 — compiles the package type graph once: …`, sits under the exact header of the two spec files (copyright line, then the declaration on line 2, where all 12 declarations in the tree sit). It reads as `{ seconds: 30, cls: 'compiles the package type graph once', line: 2 }`. 2. **The two earlier classes still read** in that same header, at 40 s each. The third class took nothing from them. 3. **Near misses are refused loudly.** Six spellings are refused: `type-graph`, a dropped `once`, a dropped `package`, `packages`, a capital `C`, and a trailing space before the colon. Each refusal starts with `f.test.ts:3: `, quotes the bad spelling, and names **all three** classes. 4. **On disk, through `--diff`.** A file carrying the ruling's line goes through `declaredBudgetInTree` and the real `diffMain`, with the same CLI path as the existing round trip. - At the card's slowest CI reading (18.1 s), it is held, not listed: `` `src/integration/graph.test.ts` (`spec`) 18.10 s of its 30 s, compiles the package type graph once ``. - At 31 s it is listed against its own budget: `| ... | 31.00 (declared 30) | 1 / 1 |`. **Ablation.** This was one-off, on `f8bacd05b` (the head), through `scripts/ablation-replace.mjs` in WRAP mode, with a trap backstop on absolute paths. No permanent mutation test remains. All four legs restored to blob `5e2b82584d38`, which equals HEAD, and `git diff HEAD` was empty after each. | Leg | Mutation (anchor 1 → 0, blob changed) | Self-test reading | |:--|:--|:--| | 0 | battery pin 18 → 19 | exit 1: `registered 18 case(s), below its pinned floor of 19`. The battery runs exactly 18. | | 1 | delete the third value | exit 1 at the pin: `budget: the classes read ["spawns the real producer once per single-cause fixture","whole-package import"]` | | 2 | control: the `check` thunk catches each failure and prints `RED` instead of aborting; the value is present | exit 0, **zero** `RED` lines | | 3 | delete the third value **and** the leg-2 instrumentation | exactly **four** `RED` lines; every other case stayed green | The four `RED` lines in leg 3: - the pin; - case 1 (`scoped-context.test.ts:2: the class "compiles the package type graph once" is none of the 2 classes the rulings name`); - case 3 (the first near miss: the refusal no longer names a third class); - case 4 (the on-disk `--diff` refuses `graph.test.ts:2`). Case 2 stayed green, as a control should. **The 12 declarations on `main` read unchanged.** The script has no dry-run mode, so a read-only probe imported the script's own `readTestBudget` and read every file in `packages/`, `apps/` and `examples/` that carries a `// test-budget:` line. The reading is the same 12 files (10 producer-class, 2 whole-package), with identical seconds, class, line and reason. The rows digest is `f1975b9e4af0…` at both `497518c3f` (before) and `f8bacd05b` (after). The two spec files read `null` on disk today. With the ruling's line spliced in at line 2 **in memory only**, each reads as the third class at 30 s. ## Tests `node scripts/measure-test-shard-timings.mjs --self-test` printed `measure-test-shard-timings: self-test OK`, exit 0, at `f8bacd05b`. `pnpm exec eslint --no-inline-config scripts/measure-test-shard-timings.mjs` exited 0. The control-byte scan found nothing. ## Gates (local, at `f8bacd05b`, the final head) - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived **32** commands from git, against merge base `497518c3f`. - All 32 ran and all 32 exited **0**, each exit captured before any pipe. They include: - `pnpm check:pm-dispatch-gates`, run detached with its exit written to a file: `✓ dispatch-gates self-test (fast tier): 1884 cases pass`; - `check:nul-bytes`, `check:entry-guard` and `check:cross-package-test-inputs`; - the self-test wiring and anchor gates with their self-tests; - the three shard-timing self-tests (`measure-test-shard-timings`, `partition-test-shards`, `ci/select-shard-timings-run`). - `--ran`: `✓ dispatch-gates --ran: 32 derived famil(ies) accounted for — 32 run, 0 NOT-MEASURED (a DERIVED zero — all 32 recorded an exit code and none of them is 3)`. - `node scripts/pm/check-governed-merges.mjs --branch claude/issue-23821-type-graph-budget-class` reported `0 of 1 path(s) hit the register` (**NOT governed**), with 123 changed lines, under the 3,000 threshold. - Every family CI runs is read from this PR's check-runs. `scripts/**` ships in no package, so no changeset. ## Acceptance notes (not filed) - **The battery's name still reads `(ruling 6181973363)`.** Renaming it would only churn the roster. Its comment names both records. Carrier: none. - **No PR-time parse of a declaration (as in PR #23595).** The spec lane's two lines are parsed only by the next weekly refresh. The spec lane can check them locally before pushing: ```bash node -e "import('./scripts/measure-test-shard-timings.mjs').then((m) => console.log(m.readTestBudget(require('fs').readFileSync(process.argv[1], 'utf8'), process.argv[1])))" packages/spec/src/contracts/scoped-context.test.ts ``` --- _Generated by [Claude Code](https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent a13fd4d commit 0744f94

1 file changed

Lines changed: 107 additions & 16 deletions

File tree

‎scripts/measure-test-shard-timings.mjs‎

Lines changed: 107 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1222,21 +1222,25 @@ export function readWindowCaptures(dir) {
12221222
return { byRun, problems, present: true };
12231223
}
12241224

1225-
// -- A test file's own measured budget (ruling record 6181973363, letter B) --
1225+
// -- A test file's own measured budget (ruling records 6181973363, 6212133431) --
12261226
//
1227-
// Two classes of test file cannot fit the 10 s line without being weakened:
1227+
// Three classes of test file cannot fit the 10 s line without being weakened:
12281228
// one that spawns the real producer (a gate, a generator) once per
1229-
// single-cause fixture, and one whose subject is every module of its package.
1230-
// A file of either class carries its own budget in place of the line, as ONE
1231-
// line in its header, and is listed only when the window reads it above it:
1229+
// single-cause fixture, and one whose subject is every module of its package
1230+
// (both ruling record 6181973363, letter B); and one whose floor is a single
1231+
// compile of its package's type graph, every probe after the first reading
1232+
// that one program (ruling record 6212133431, letter A, which adds this class
1233+
// and changes nothing else). A file of any class carries its own budget in
1234+
// place of the line, as ONE line in its header, and is listed only when the
1235+
// window reads it above it:
12321236
//
12331237
// // test-budget: SECONDS — CLASS: REASON
12341238
//
12351239
// SECONDS the measured CI reading x CEILING_HEADROOM, rounded UP to the
12361240
// next 10 s: a whole number, a multiple of 10, above
12371241
// FILE_BUDGET_SECONDS (one at or under the line lifts nothing). It
12381242
// rises only by a PR that re-measures and says why in REASON.
1239-
// CLASS exactly one of TEST_BUDGET_CLASSES, spelled as the ruling names it.
1243+
// CLASS exactly one of TEST_BUDGET_CLASSES, spelled as its ruling names it.
12401244
// REASON why the file is of that class; not empty.
12411245
//
12421246
// The separators are an em dash between single spaces, and a colon and a space
@@ -1255,6 +1259,7 @@ export function readWindowCaptures(dir) {
12551259
export const TEST_BUDGET_CLASSES = Object.freeze([
12561260
'spawns the real producer once per single-cause fixture',
12571261
'whole-package import',
1262+
'compiles the package type graph once',
12581263
]);
12591264
const TEST_BUDGET_ATTEMPT = /^(?:\/\/+|\/\*+|\*)\s*test[-_ ]?budget\s*:/i;
12601265
const TEST_BUDGET_LINE = /^\/\/ test-budget: (\S+) — ([^:]+): (.*)$/u;
@@ -1265,8 +1270,8 @@ export function readTestBudget(text, label) {
12651270
const refuse = (line, why) => {
12661271
throw new Error(
12671272
`${label}:${line}: ${why}. A test file's budget is ONE header line, \`// test-budget: SECONDS — CLASS: REASON\` ` +
1268-
`(ruling record 6181973363): SECONDS is the measured CI reading x ${CEILING_HEADROOM} rounded up to the next 10 s, ` +
1269-
`above the ${FILE_BUDGET_SECONDS} s line; CLASS is ${TEST_BUDGET_CLASSES.map((c) => `"${c}"`).join(' or ')}; ` +
1273+
`(ruling records 6181973363 and 6212133431): SECONDS is the measured CI reading x ${CEILING_HEADROOM} rounded up to the next 10 s, ` +
1274+
`above the ${FILE_BUDGET_SECONDS} s line; CLASS is one of ${TEST_BUDGET_CLASSES.map((c) => `"${c}"`).join(', ')}; ` +
12701275
'REASON is not empty. Refusing to guess which budget the file is held to.'
12711276
);
12721277
};
@@ -1292,7 +1297,7 @@ export function readTestBudget(text, label) {
12921297
const seconds = Number(number);
12931298
if (seconds % 10 !== 0) refuse(line, `the budget ${seconds} s is not on the 10 s grid`);
12941299
if (seconds <= FILE_BUDGET_SECONDS) refuse(line, `the budget ${seconds} s does not lift the ${FILE_BUDGET_SECONDS} s line`);
1295-
if (!TEST_BUDGET_CLASSES.includes(cls)) refuse(line, `the class "${cls}" is neither class the ruling names`);
1300+
if (!TEST_BUDGET_CLASSES.includes(cls)) refuse(line, `the class "${cls}" is none of the ${TEST_BUDGET_CLASSES.length} classes the rulings name`);
12961301
if (reason.trim() === '') refuse(line, 'the reason is empty');
12971302
found = { seconds, cls, reason: reason.trim(), line };
12981303
}
@@ -1660,8 +1665,9 @@ const SELF_TEST_BATTERIES = Object.freeze({
16601665
// (the density-only and the code fixture first), and the per-file budget.
16611666
'refresh diff and per-file budget (#22949 E)': 22,
16621667
// Ruling record 6181973363: a test file's own measured budget, the
1663-
// declaration's grammar and its refusals, and the list held to it.
1664-
'per-file budget declarations (ruling 6181973363)': 14,
1668+
// declaration's grammar and its refusals, and the list held to it; ruling
1669+
// record 6212133431: its third class, read in memory and from disk.
1670+
'per-file budget declarations (ruling 6181973363)': 18,
16651671
});
16661672

16671673
// DELETING an entry silences that battery's floor exactly as effectively as
@@ -2979,11 +2985,20 @@ function selfTest() {
29792985

29802986
// -- A test file's own measured budget (ruling record 6181973363) ----------
29812987
battery('per-file budget declarations (ruling 6181973363)');
2982-
const [producerClass, wholePackageClass] = TEST_BUDGET_CLASSES;
2988+
const [producerClass, wholePackageClass, typeGraphClass] = TEST_BUDGET_CLASSES;
2989+
// The ruled spellings, as LITERALS: a case that read them back out of
2990+
// TEST_BUDGET_CLASSES would still pass with a value deleted from it.
2991+
const ruledClasses = ['spawns the real producer once per single-cause fixture', 'whole-package import', 'compiles the package type graph once'];
29832992
const declare = (seconds, cls = producerClass, reason = 'each case spawns the gate once') =>
29842993
`// test-budget: ${seconds} — ${cls}: ${reason}`;
29852994
// Line 3 of the file is `decl`; `rest` follows the header's blank line 4.
29862995
const fileWith = (decl, rest = "import { it } from 'vitest';\n") => `// Copyright (c) 2026 ObjectStack.\n//\n${decl}\n\n${rest}`;
2996+
// The header of the two spec files the third class was ruled for, with the
2997+
// declaration on line 2, where every declaration in the tree sits.
2998+
const specFileWith = (decl) =>
2999+
`// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.\n${decl}\n\nimport { describe, it, expect } from 'vitest';\nimport ts from 'typescript';\n`;
3000+
const typeGraphLine =
3001+
'// test-budget: 30 — compiles the package type graph once: one ts.createProgram over the published entries, and every probe after the first reads that one program';
29873002
const refusalOf = (fn) => {
29883003
try {
29893004
fn();
@@ -2993,11 +3008,49 @@ function selfTest() {
29933008
return null;
29943009
};
29953010
check(() => {
2996-
// The two classes, spelled as the ruling names them, and no third.
2997-
if (TEST_BUDGET_CLASSES.length !== 2 || producerClass !== 'spawns the real producer once per single-cause fixture' || wholePackageClass !== 'whole-package import') {
3011+
// The three classes, spelled as their rulings name them, in order, and no fourth.
3012+
if (
3013+
TEST_BUDGET_CLASSES.length !== 3 ||
3014+
producerClass !== ruledClasses[0] ||
3015+
wholePackageClass !== ruledClasses[1] ||
3016+
typeGraphClass !== ruledClasses[2]
3017+
) {
29983018
throw new Error(`budget: the classes read ${JSON.stringify(TEST_BUDGET_CLASSES)}`);
29993019
}
30003020
});
3021+
check(() => {
3022+
// The third class (ruling record 6212133431), in the line the ruling
3023+
// writes, under the header of the files it was ruled for: a declared budget.
3024+
const d = readTestBudget(specFileWith(typeGraphLine), 'packages/spec/src/contracts/scoped-context.test.ts');
3025+
if (d?.seconds !== 30 || d.cls !== ruledClasses[2] || d.line !== 2 || !d.reason.startsWith('one ts.createProgram over')) {
3026+
throw new Error(`budget: the third class's declaration read ${JSON.stringify(d)}`);
3027+
}
3028+
});
3029+
check(() => {
3030+
// The two earlier classes still read, in that same header: the third
3031+
// took nothing from them.
3032+
for (const cls of ruledClasses.slice(0, 2)) {
3033+
const d = readTestBudget(specFileWith(declare(40, cls, 'r')), 'f.test.ts');
3034+
if (d?.seconds !== 40 || d.cls !== cls || d.line !== 2) throw new Error(`budget: the class "${cls}" read ${JSON.stringify(d)}`);
3035+
}
3036+
});
3037+
check(() => {
3038+
// A near miss of the third class is REFUSED, never read as a nearby class:
3039+
// the refusal names the file and line, the spelling, and all three classes.
3040+
for (const near of [
3041+
'compiles the package type-graph once',
3042+
'compiles the package type graph',
3043+
'compiles the type graph once',
3044+
'compiles the packages type graph once',
3045+
'Compiles the package type graph once',
3046+
'compiles the package type graph once ',
3047+
]) {
3048+
const msg = refusalOf(() => readTestBudget(fileWith(declare(30, near)), 'f.test.ts'));
3049+
if (!msg?.startsWith('f.test.ts:3: ') || !msg.includes(`"${near}"`) || !ruledClasses.every((c) => msg.includes(`"${c}"`))) {
3050+
throw new Error(`budget: the near miss "${near}" read as ${msg}`);
3051+
}
3052+
}
3053+
});
30013054
check(() => {
30023055
// Each class parses, in a `//` header and below a header docblock.
30033056
const d = readTestBudget(fileWith(declare(1130)), 'f.test.ts');
@@ -3016,9 +3069,9 @@ function selfTest() {
30163069
}
30173070
});
30183071
check(() => {
3019-
// A class naming neither is REFUSED, naming the file and the line.
3072+
// A class naming none of them is REFUSED, naming the file and the line.
30203073
const msg = refusalOf(() => readTestBudget(fileWith(declare(1130, 'slow by nature')), 'packages/x/a.test.ts'));
3021-
if (!msg?.startsWith('packages/x/a.test.ts:3: ') || !msg.includes('slow by nature')) throw new Error(`budget: a class naming neither read as ${msg}`);
3074+
if (!msg?.startsWith('packages/x/a.test.ts:3: ') || !msg.includes('slow by nature')) throw new Error(`budget: a class naming none of them read as ${msg}`);
30223075
});
30233076
check(() => {
30243077
// A number off the 10 s grid, at or under the line, or not a whole number is REFUSED.
@@ -3154,6 +3207,44 @@ function selfTest() {
31543207
rmSync(dir, { recursive: true, force: true });
31553208
}
31563209
});
3210+
check(() => {
3211+
// The third class end to end, from a file on disk through --diff: at the
3212+
// card's slowest CI reading (18.1 s) the file is held to its 30 s and named
3213+
// with its class; read over the 30 s, it is listed against them.
3214+
const dir = mkdtempSync(path.join(tmpdir(), 'os-refresh-type-graph-'));
3215+
try {
3216+
const pkgDir = path.join(dir, 'spec');
3217+
mkdirSync(path.join(pkgDir, 'src', 'integration'), { recursive: true });
3218+
writeFileSync(path.join(pkgDir, 'src', 'integration', 'graph.test.ts'), specFileWith(typeGraphLine));
3219+
writeFileSync(path.join(dir, 'prior.json'), JSON.stringify(priorWindow));
3220+
writeFileSync(path.join(dir, 'refreshed.json'), JSON.stringify(codeWindow));
3221+
const run = (ms, out) => {
3222+
const caps = path.join(dir, `captures-${out}`);
3223+
mkdirSync(path.join(caps, 'n1', '1'), { recursive: true });
3224+
writeFileSync(path.join(caps, 'n1', '1', 'shard-1.json'), JSON.stringify(capture([fileRow('spec', 'src/integration/graph.test.ts', ms)])));
3225+
const argv = ['--diff', '--prior', path.join(dir, 'prior.json'), '--refreshed', path.join(dir, 'refreshed.json'), '--captures', caps, '--out', path.join(dir, out)];
3226+
const quiet = console.error;
3227+
console.error = () => {};
3228+
try {
3229+
return diffMain(argv, { packageDir: (name) => (name === 'spec' ? pkgDir : null) });
3230+
} finally {
3231+
console.error = quiet;
3232+
}
3233+
};
3234+
const heldCode = run(18_100, 'held.md');
3235+
const held = readFileSync(path.join(dir, 'held.md'), 'utf8');
3236+
if (heldCode !== 0 || held.includes('| `src/integration/graph.test.ts`') || !held.includes('`src/integration/graph.test.ts` (`spec`) 18.10 s of its 30 s, compiles the package type graph once')) {
3237+
throw new Error(`cli: a type-graph file under its declared budget exited ${heldCode} and wrote:\n${held}`);
3238+
}
3239+
const overCode = run(31_000, 'over.md');
3240+
const over = readFileSync(path.join(dir, 'over.md'), 'utf8');
3241+
if (overCode !== 0 || !over.includes('| `src/integration/graph.test.ts` | `spec` | `n1` | 31.00 (declared 30) | 1 / 1 |')) {
3242+
throw new Error(`cli: a type-graph file over its declared budget exited ${overCode} and wrote:\n${over}`);
3243+
}
3244+
} finally {
3245+
rmSync(dir, { recursive: true, force: true });
3246+
}
3247+
});
31573248

31583249
// -- The floor: every declared battery RAN, and ran its cases (#13489) ----
31593250
//

0 commit comments

Comments
 (0)