Skip to content

Commit 6892888

Browse files
fix(skills): four objectstack-platform eval cases agree with their own must_not_contain assertions (#27529)
<!-- claude-projects-attribution: {"github_login":"marchtian"} --> _Requested by **田騏** · [project thread](https://claude.ai/code/project/chan_01VmuUhXU9SSUzqJYyXSmYSi?thread=cmsg_01VmuUhXU9SSUzqJYyXSmYSi8irujFEnBnqcKvK4Kv1YDL)_ Fixes #27455 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:`skills/objectstack-platform/evals/config-plugins-ops.json` 四个用例(1、2、4、7)的标准答案与自身的 `must_not_contain` 断言不再互斥——用例 1 的断言收窄为键拼法 `tableName:`;用例 2、4、7 的标准答案改写警句,不再逐字拼出被拒字串。另加一张 `@objectstack/skills` 的 patch changeset。 - 为什么改:这份评测文件既是标准答案又是评分器;四个用例的标准答案逐字命中自己的拒收串,逐字写出理想答案会被判错,平台技能在配置、插件、运维题上的质量信号因此是噪声。 - 风险与回滚:只动一份评测 fixture 与一张 changeset,不动 SKILL.md、规则、spec 或任何运行时;无断言被删除,文件缩 13 token、天花板不动。回滚 = revert 这一个 commit。 - 先看:`skills/objectstack-platform/evals/config-plugins-ops.json` 用例 7 的 `expected_output`(两处改写:`role` 去冒号、具名平台集改为「no platform set」),其次用例 1 的 `tableName:` 收窄,再次用例 2。 - 两分钟验证:仓根跑 `python3 -c 'import json;d=json.load(open("skills/objectstack-platform/evals/config-plugins-ops.json"));[print(e["id"],[s for s in e["assertions"].get("must_not_contain",[]) if s in e["expected_output"]]) for e in d["evals"]]'`,预期九行、每行的列表都是空的 `[]`(main 上第 1、2、4、7 行非空)。 - 需要你做的:Tier H(`skills/**`)——`GOVERNED_APPROVERS` 账号一次 APPROVED review,其后由席位落地。 ## What changes One shipped fixture, `skills/objectstack-platform/evals/config-plugins-ops.json` (ships in `@objectstack/skills`), and one patch changeset. Each of the four cases was judged from the skill text, per the card's Wanted: drop the refused literal from the reference answer when the answer only names it to warn, or narrow the assertion to the wrong form it was meant to catch. | eval | refused string spelled in its own answer | judged from | fix | |:--|:--|:--|:--| | 1 | `tableName` | `SKILL.md` line 116 ("never set `tableName`"; also AGENTS.md Prime Directive 6) — the skill's own warning uses the bare word; the wrong form is the object key | assertion narrowed to the key spelling `tableName:`, the form its siblings `workflows:` / `approvals:` already use; reference answer unchanged | | 2 | `data:beforeInsert`, `onInstall` | `references/plugin-hooks.md` lines 73–76 (a `'data:beforeInsert'` handler on the kernel bus registers and silently never fires); AGENTS.md Plugin pattern (the contract is `init` / `start` / `destroy`; no install / enable / disable hooks) — both named only to warn, and the wrong forms have several spellings (method, property, call site), so no single narrowing holds | reference rephrased: "a `data:*` kernel hook registers and never fires", "No install / enable / disable hooks."; assertions unchanged | | 4 | `@objectstack/adapter-` | `SKILL.md` line 275 ("There are **no** `@objectstack/adapter-*` packages") — named only to warn; narrowing to a single-quoted import would miss the double-quoted `package.json` dependency form | reference rephrased: "no `adapter-*` packages exist"; assertion unchanged | | 7 | `role:`, `admin_full_access` | `DemoStaffSchema` refuses `role` as an alias of `position` (`packages/spec/src/system/demo-staff.zod.ts` line 71); `SKILL.md` lines 438–440 ("a platform set such as `admin_full_access` is refused") — `role:` is already the key spelling; `admin_full_access` is a value with the same quote-style leak as eval 4 | reference names the alias as `role` (no colon) and says "no platform set" (the skill's own term); assertions unchanged | - No assertion deleted, no case added or removed. Evals 3, 5, 6, 8 and 9, including the predecessor PR #27426 (merged `501bfc69`) eval 9, are unchanged byte for byte. - `.changeset/27455-platform-eval-self-agreement.md`: `@objectstack/skills` patch, carrying the two declaration lines above. - Not done, by ruling (triage comment 6491392918): no lint over evals — a self-check that would catch the class is a separate card. The file's ceiling row in `scripts/check-skills-token-ratchet.mjs` is not lowered either: that script is outside this card's declared file surface, so the new headroom (14 tokens) is reported here, not spent or banked. ## Tests No test file changes: nothing in the repo executes this fixture; its consumer is the skills eval grader that applies `must_contain` / `must_not_contain` as substrings to an answer (the card's repro models that). The diff touches no workspace package, so no package build or test is owed (local verification scope ① and ② are empty). <details><summary>The card's repro, before (origin/main ded67bf) and after (6f2257a)</summary> ```text $ python3 -c 'import json;d=json.load(open("skills/objectstack-platform/evals/config-plugins-ops.json"));[print(e["id"],[s for s in e["assertions"].get("must_not_contain",[]) if s in e["expected_output"]]) for e in d["evals"]]' # before 1 ['tableName'] 2 ['data:beforeInsert', 'onInstall'] 3 [] 4 ['@objectstack/adapter-'] 5 [] 6 [] 7 ['role:', 'admin_full_access'] 8 [] 9 [] # after 1 [] 2 [] 3 [] 4 [] 5 [] 6 [] 7 [] 8 [] 9 [] ``` </details> <details><summary>Per-case cross-check after (no must_not_contain string in its own answer, every must_contain string present, JSON parses)</summary> ```text 1 neg-in-answer: [] pos-missing: [] 2 neg-in-answer: [] pos-missing: [] 3 neg-in-answer: [] pos-missing: [] 4 neg-in-answer: [] pos-missing: [] 5 neg-in-answer: [] pos-missing: [] 6 neg-in-answer: [] pos-missing: [] 7 neg-in-answer: [] pos-missing: [] 8 neg-in-answer: [] pos-missing: [] 9 neg-in-answer: [] pos-missing: [] cases: 9 failures: 0 ``` </details> ## Acceptance notes Tier H (`skills/**`) readings, both required: - Changed file, whole file: 95 → 95 lines; 6756 → 6703 bytes; 1689 → 1676 tokens (`scripts/check-skills-token-ratchet.mjs`, ceiling 1690 unchanged, headroom 1 → 14). - Whole package: `skills/objectstack-platform/**` 88041 → 87988 bytes; the sum of every `skills/*/SKILL.md` 4610 → 4610 lines (no SKILL.md touched); the whole authored bundle 207870 → 207857 tokens (ceiling sum 217474). Observations, noted and not filed: - No gate in either root checks a `must_not_contain` string against the same case's `expected_output` (`ui/scripts/check-skill-eval-tokens.mjs` deliberately shape-checks `must_not_contain` only). Triage ruled that lint a separate card; the seat carries it. - Eval 3 also refuses `@objectstack/adapter-`; its answer never spelled the string, so it is untouched. ## Verification Readings taken on the final head `6f2257ad` (`git rev-parse --short HEAD`), in the worktree, after `pnpm install --frozen-lockfile`. - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 42 commands from the diff against the merge base `ded67bfb`. Each ran with its exit code captured before any pipe: 37 exit 0; 5 exit 3 (PREREQUISITE NOT MET — no package `dist/` in the tree; the diff touches no package, so the closure build is not owed locally and CI runs these): `pnpm --filter @objectstack/lint run check:doc-formula-expressions`, `pnpm check:dts-closure`, `pnpm check:dual-build-cjs-loads`, `pnpm check:lean-entry-closure`, `pnpm check:sourcemap-no-sources-content`. Reconciliation `node scripts/pm/dispatch-gates.mjs --ran FILE --repo objectstack-ai/objectstack`: "42 derived famil(ies) accounted for — 37 run, 5 NOT-MEASURED (5 DERIVED from a recorded exit 3)", exit 0. - `node scripts/check-skills-token-ratchet.mjs`: exit 0 — "skills/objectstack-platform/evals/config-plugins-ops.json is 1676 tokens (ceiling 1690; headroom 14)"; `--self-test`: exit 0, "79 cases pass". - Control-byte self-scan over both touched files: none; `pnpm check:nul-bytes` exit 0. - Dogfood battery: n/a — no `!` in the title and no `(narrowing)` arm. - CI check-runs: in progress at the time of opening; the seat reads them. <details><summary>Gate ledger (exit code, command) on 6f2257a</summary> ```text 0 node scripts/check-adr-0087-registration.mjs --base origin/main 0 node scripts/check-adr-0087-registration.mjs --self-test 0 node scripts/check-changeset-no-major.mjs --base origin/main 0 node scripts/check-closing-keyword-parity.mjs 0 node scripts/check-closing-keyword-parity.mjs --self-test 0 node scripts/check-comment-mask-corpus.mjs 0 node scripts/check-doc-route-spelling.mjs --advisory 0 node scripts/check-doc-route-spelling.mjs --self-test 0 node scripts/check-dts-emitted.mjs --self-test 0 node scripts/check-empty-changeset.mjs --base origin/main 0 node scripts/check-empty-changeset.mjs --self-test 0 node scripts/check-skills-token-ratchet.mjs 0 node scripts/check-skills-token-ratchet.mjs --self-test 0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test 0 node scripts/ci/plan-test-core-shards.mjs --self-test 0 node scripts/ci/prune-turbo-cache.mjs --self-test 0 node scripts/pm/release-rehearsal-clone.mjs --self-test 0 node scripts/release-pending-publish.mjs --self-test 3 pnpm --filter @objectstack/lint run check:doc-formula-expressions (PREREQUISITE NOT MET: @objectstack/lint not built) 0 pnpm check:agent-test-spelling 0 pnpm check:changeset-gate-self-tests 0 pnpm check:ci-filter-parity 0 pnpm check:closed-ceremony-text 0 pnpm check:corpus-claim-drift 0 pnpm check:cross-package-test-inputs 0 pnpm check:doc-authoring 0 pnpm check:driver-memory-census 3 pnpm check:dts-closure (PREREQUISITE NOT MET: no dist/) 3 pnpm check:dual-build-cjs-loads (PREREQUISITE NOT MET: no dist/) 0 pnpm check:gitlink-declared 3 pnpm check:lean-entry-closure (PREREQUISITE NOT MET: no dist/) 0 pnpm check:nul-bytes 0 pnpm check:pm-changeset-deadline-census 0 pnpm check:pm-governed-merges 0 pnpm check:published-files 0 pnpm check:refd-timer-probe 0 pnpm check:role-word 0 pnpm check:skill-compatibility 0 pnpm check:skill-frame-sync 0 pnpm check:skill-identifier-liveness 3 pnpm check:sourcemap-no-sources-content (PREREQUISITE NOT MET: no dist/) 0 pnpm check:watch-hint-literal ``` </details> --- _Generated by [Claude Code](https://claude.ai/code/session_017ihL2dPRZXfgE1A2Vct3FU)_ Co-authored-by: marchtian <noreply@anthropic.com>
1 parent 4d80569 commit 6892888

2 files changed

Lines changed: 16 additions & 4 deletions

File tree

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
'@objectstack/skills': patch
3+
---
4+
5+
fix(skills): four objectstack-platform eval cases agree with their own `must_not_contain` assertions
6+
7+
Clause-②: no
8+
Metadata-protocol: none
9+
10+
- **Before.** In `skills/objectstack-platform/evals/config-plugins-ops.json`, four cases refused a string their own `expected_output` spelled: eval 1 `tableName`, eval 2 `data:beforeInsert` and `onInstall`, eval 4 `@objectstack/adapter-`, eval 7 `role:` and `admin_full_access`. Graded against those assertions, the reference answer itself scored wrong, so the eval could not tell a good configuration, plugin or operations answer from a bad one.
11+
- **Now.** Judged per case from the skill text. Eval 1 narrows its assertion to the key spelling `tableName:` (the skill's own warning, "never set `tableName`", names the bare word; the wrong form is the key beside `workflows:` / `approvals:`). Eval 2 says "a `data:*` kernel hook registers and never fires" and "no install / enable / disable hooks" instead of spelling the phantom hook and method. Eval 4 says "no `adapter-*` packages exist". Eval 7 names the refused alias as `role` (the key spelling `role:` stays refused) and says "no platform set" (the skill's term for a set such as the refused platform-admin one). No assertion is deleted, no case added or removed; the file shrinks by 13 tokens under its unchanged ceiling.
12+
- No schema, key or behaviour changes.

‎skills/objectstack-platform/evals/config-plugins-ops.json‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@
88
"files": [],
99
"assertions": {
1010
"must_contain": ["defineStack", "manifest", "engines", "Object.values("],
11-
"must_not_contain": ["workflows:", "approvals:", "tableName"]
11+
"must_not_contain": ["workflows:", "approvals:", "tableName:"]
1212
}
1313
},
1414
{
1515
"id": 2,
1616
"prompt": "Build a plugin registering an `audit` service that records every metadata reload and every `task` insert.",
17-
"expected_output": "A `Plugin` with `name`, `init(ctx)` calling `ctx.registerService('audit', ...)` and `ctx.hook('metadata:reloaded', ...)`, `start(ctx)`, `destroy()`. Record events are not kernel hooks — `ctx.hook('data:beforeInsert')` registers and never fires — so the task insert is a data hook (objectstack-data). No `onInstall` / `onEnable` / `onDisable`.",
17+
"expected_output": "A `Plugin` with `name`, `init(ctx)` calling `ctx.registerService('audit', ...)` and `ctx.hook('metadata:reloaded', ...)`, `start(ctx)`, `destroy()`. Record events are not kernel hooks — a `data:*` kernel hook registers and never fires — so the task insert is a data hook (objectstack-data). No install / enable / disable hooks.",
1818
"files": [],
1919
"assertions": {
2020
"must_contain": ["registerService", "ctx.hook(", "metadata:reloaded", "init(", "destroy"],
@@ -34,7 +34,7 @@
3434
{
3535
"id": 4,
3636
"prompt": "We deploy on Vercel serverless functions in our own Hono app; mount the ObjectStack API under /api.",
37-
"expected_output": "`createHonoApp({ kernel })` from `@objectstack/hono` (prefix defaults to `/api`) in the host app — `HonoServerPlugin` only when ObjectStack owns the server; `LiteKernel` for a cold-start-sensitive host; `/auth/*` follows the auth `basePath`, not `prefix`; no `@objectstack/adapter-*` packages exist.",
37+
"expected_output": "`createHonoApp({ kernel })` from `@objectstack/hono` (prefix defaults to `/api`) in the host app — `HonoServerPlugin` only when ObjectStack owns the server; `LiteKernel` for a cold-start-sensitive host; `/auth/*` follows the auth `basePath`, not `prefix`; no `adapter-*` packages exist.",
3838
"files": [],
3939
"assertions": {
4040
"must_contain": ["createHonoApp", "@objectstack/hono", "LiteKernel"],
@@ -64,7 +64,7 @@
6464
{
6565
"id": 7,
6666
"prompt": "Prepare a demo of our contracts app: the seed assigns matters to 'Legal Counsel 1' and 'Paralegal 1', and the demo opens signed in as Legal Counsel 1.",
67-
"expected_output": "Both go in `demoStaff` — `name` as the seed references them, `email`, a `position` / `permissionSets` the stack declares — and `os dev` creates the accounts with the seed password: no `password` key, no `role:` alias, no platform set like `admin_full_access`, no hand-written `sys_user` seed row. `devLogins` creates nothing (at most a banner entry); `os build` drops `demoStaff`; production creates no demo account.",
67+
"expected_output": "Both go in `demoStaff` — `name` as the seed references them, `email`, a `position` / `permissionSets` the stack declares — and `os dev` creates the accounts with the seed password: no `password` key, no `role` alias, no platform set, no hand-written `sys_user` seed row. `devLogins` creates nothing (at most a banner entry); `os build` drops `demoStaff`; production creates no demo account.",
6868
"files": [],
6969
"assertions": {
7070
"must_contain": ["demoStaff", "position", "permissionSets", "os dev"],

0 commit comments

Comments
 (0)