Repository navigation
Commit 6892888
fix(skills): four objectstack-platform eval cases agree with their own must_not_contain assertions (#27529)
<!-- claude-projects-attribution: {"github_login":"marchtian"} -->
_Requested by **田騏** · [project
thread](https://claude.ai/code/project/chan_01VmuUhXU9SSUzqJYyXSmYSi?thread=cmsg_01VmuUhXU9SSUzqJYyXSmYSi8irujFEnBnqcKvK4Kv1YDL)_
Fixes #27455
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:`skills/objectstack-platform/evals/config-plugins-ops.json`
四个用例(1、2、4、7)的标准答案与自身的 `must_not_contain` 断言不再互斥——用例 1 的断言收窄为键拼法
`tableName:`;用例 2、4、7 的标准答案改写警句,不再逐字拼出被拒字串。另加一张 `@objectstack/skills` 的
patch changeset。
-
为什么改:这份评测文件既是标准答案又是评分器;四个用例的标准答案逐字命中自己的拒收串,逐字写出理想答案会被判错,平台技能在配置、插件、运维题上的质量信号因此是噪声。
- 风险与回滚:只动一份评测 fixture 与一张 changeset,不动 SKILL.md、规则、spec
或任何运行时;无断言被删除,文件缩 13 token、天花板不动。回滚 = revert 这一个 commit。
- 先看:`skills/objectstack-platform/evals/config-plugins-ops.json` 用例 7 的
`expected_output`(两处改写:`role` 去冒号、具名平台集改为「no platform set」),其次用例 1 的
`tableName:` 收窄,再次用例 2。
- 两分钟验证:仓根跑 `python3 -c 'import
json;d=json.load(open("skills/objectstack-platform/evals/config-plugins-ops.json"));[print(e["id"],[s
for s in e["assertions"].get("must_not_contain",[]) if s in
e["expected_output"]]) for e in d["evals"]]'`,预期九行、每行的列表都是空的 `[]`(main
上第 1、2、4、7 行非空)。
- 需要你做的:Tier H(`skills/**`)——`GOVERNED_APPROVERS` 账号一次 APPROVED
review,其后由席位落地。
## What changes
One shipped fixture,
`skills/objectstack-platform/evals/config-plugins-ops.json` (ships in
`@objectstack/skills`), and one patch changeset. Each of the four cases
was judged from the skill text, per the card's Wanted: drop the refused
literal from the reference answer when the answer only names it to warn,
or narrow the assertion to the wrong form it was meant to catch.
| eval | refused string spelled in its own answer | judged from | fix |
|:--|:--|:--|:--|
| 1 | `tableName` | `SKILL.md` line 116 ("never set `tableName`"; also
AGENTS.md Prime Directive 6) — the skill's own warning uses the bare
word; the wrong form is the object key | assertion narrowed to the key
spelling `tableName:`, the form its siblings `workflows:` / `approvals:`
already use; reference answer unchanged |
| 2 | `data:beforeInsert`, `onInstall` | `references/plugin-hooks.md`
lines 73–76 (a `'data:beforeInsert'` handler on the kernel bus registers
and silently never fires); AGENTS.md Plugin pattern (the contract is
`init` / `start` / `destroy`; no install / enable / disable hooks) —
both named only to warn, and the wrong forms have several spellings
(method, property, call site), so no single narrowing holds | reference
rephrased: "a `data:*` kernel hook registers and never fires", "No
install / enable / disable hooks."; assertions unchanged |
| 4 | `@objectstack/adapter-` | `SKILL.md` line 275 ("There are **no**
`@objectstack/adapter-*` packages") — named only to warn; narrowing to a
single-quoted import would miss the double-quoted `package.json`
dependency form | reference rephrased: "no `adapter-*` packages exist";
assertion unchanged |
| 7 | `role:`, `admin_full_access` | `DemoStaffSchema` refuses `role` as
an alias of `position` (`packages/spec/src/system/demo-staff.zod.ts`
line 71); `SKILL.md` lines 438–440 ("a platform set such as
`admin_full_access` is refused") — `role:` is already the key spelling;
`admin_full_access` is a value with the same quote-style leak as eval 4
| reference names the alias as `role` (no colon) and says "no platform
set" (the skill's own term); assertions unchanged |
- No assertion deleted, no case added or removed. Evals 3, 5, 6, 8 and
9, including the predecessor PR #27426 (merged `501bfc69`) eval 9, are
unchanged byte for byte.
- `.changeset/27455-platform-eval-self-agreement.md`:
`@objectstack/skills` patch, carrying the two declaration lines above.
- Not done, by ruling (triage comment 6491392918): no lint over evals —
a self-check that would catch the class is a separate card. The file's
ceiling row in `scripts/check-skills-token-ratchet.mjs` is not lowered
either: that script is outside this card's declared file surface, so the
new headroom (14 tokens) is reported here, not spent or banked.
## Tests
No test file changes: nothing in the repo executes this fixture; its
consumer is the skills eval grader that applies `must_contain` /
`must_not_contain` as substrings to an answer (the card's repro models
that). The diff touches no workspace package, so no package build or
test is owed (local verification scope ① and ② are empty).
<details><summary>The card's repro, before (origin/main ded67bf) and
after (6f2257a)</summary>
```text
$ python3 -c 'import json;d=json.load(open("skills/objectstack-platform/evals/config-plugins-ops.json"));[print(e["id"],[s for s in e["assertions"].get("must_not_contain",[]) if s in e["expected_output"]]) for e in d["evals"]]'
# before
1 ['tableName']
2 ['data:beforeInsert', 'onInstall']
3 []
4 ['@objectstack/adapter-']
5 []
6 []
7 ['role:', 'admin_full_access']
8 []
9 []
# after
1 []
2 []
3 []
4 []
5 []
6 []
7 []
8 []
9 []
```
</details>
<details><summary>Per-case cross-check after (no must_not_contain string
in its own answer, every must_contain string present, JSON
parses)</summary>
```text
1 neg-in-answer: [] pos-missing: []
2 neg-in-answer: [] pos-missing: []
3 neg-in-answer: [] pos-missing: []
4 neg-in-answer: [] pos-missing: []
5 neg-in-answer: [] pos-missing: []
6 neg-in-answer: [] pos-missing: []
7 neg-in-answer: [] pos-missing: []
8 neg-in-answer: [] pos-missing: []
9 neg-in-answer: [] pos-missing: []
cases: 9 failures: 0
```
</details>
## Acceptance notes
Tier H (`skills/**`) readings, both required:
- Changed file, whole file: 95 → 95 lines; 6756 → 6703 bytes; 1689 →
1676 tokens (`scripts/check-skills-token-ratchet.mjs`, ceiling 1690
unchanged, headroom 1 → 14).
- Whole package: `skills/objectstack-platform/**` 88041 → 87988 bytes;
the sum of every `skills/*/SKILL.md` 4610 → 4610 lines (no SKILL.md
touched); the whole authored bundle 207870 → 207857 tokens (ceiling sum
217474).
Observations, noted and not filed:
- No gate in either root checks a `must_not_contain` string against the
same case's `expected_output` (`ui/scripts/check-skill-eval-tokens.mjs`
deliberately shape-checks `must_not_contain` only). Triage ruled that
lint a separate card; the seat carries it.
- Eval 3 also refuses `@objectstack/adapter-`; its answer never spelled
the string, so it is untouched.
## Verification
Readings taken on the final head `6f2257ad` (`git rev-parse --short
HEAD`), in the worktree, after `pnpm install --frozen-lockfile`.
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived 42 commands from the diff against
the merge base `ded67bfb`. Each ran with its exit code captured before
any pipe: 37 exit 0; 5 exit 3 (PREREQUISITE NOT MET — no package `dist/`
in the tree; the diff touches no package, so the closure build is not
owed locally and CI runs these): `pnpm --filter @objectstack/lint run
check:doc-formula-expressions`, `pnpm check:dts-closure`, `pnpm
check:dual-build-cjs-loads`, `pnpm check:lean-entry-closure`, `pnpm
check:sourcemap-no-sources-content`. Reconciliation `node
scripts/pm/dispatch-gates.mjs --ran FILE --repo
objectstack-ai/objectstack`: "42 derived famil(ies) accounted for — 37
run, 5 NOT-MEASURED (5 DERIVED from a recorded exit 3)", exit 0.
- `node scripts/check-skills-token-ratchet.mjs`: exit 0 —
"skills/objectstack-platform/evals/config-plugins-ops.json is 1676
tokens (ceiling 1690; headroom 14)"; `--self-test`: exit 0, "79 cases
pass".
- Control-byte self-scan over both touched files: none; `pnpm
check:nul-bytes` exit 0.
- Dogfood battery: n/a — no `!` in the title and no `(narrowing)` arm.
- CI check-runs: in progress at the time of opening; the seat reads
them.
<details><summary>Gate ledger (exit code, command) on 6f2257a</summary>
```text
0 node scripts/check-adr-0087-registration.mjs --base origin/main
0 node scripts/check-adr-0087-registration.mjs --self-test
0 node scripts/check-changeset-no-major.mjs --base origin/main
0 node scripts/check-closing-keyword-parity.mjs
0 node scripts/check-closing-keyword-parity.mjs --self-test
0 node scripts/check-comment-mask-corpus.mjs
0 node scripts/check-doc-route-spelling.mjs --advisory
0 node scripts/check-doc-route-spelling.mjs --self-test
0 node scripts/check-dts-emitted.mjs --self-test
0 node scripts/check-empty-changeset.mjs --base origin/main
0 node scripts/check-empty-changeset.mjs --self-test
0 node scripts/check-skills-token-ratchet.mjs
0 node scripts/check-skills-token-ratchet.mjs --self-test
0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test
0 node scripts/ci/plan-test-core-shards.mjs --self-test
0 node scripts/ci/prune-turbo-cache.mjs --self-test
0 node scripts/pm/release-rehearsal-clone.mjs --self-test
0 node scripts/release-pending-publish.mjs --self-test
3 pnpm --filter @objectstack/lint run check:doc-formula-expressions (PREREQUISITE NOT MET: @objectstack/lint not built)
0 pnpm check:agent-test-spelling
0 pnpm check:changeset-gate-self-tests
0 pnpm check:ci-filter-parity
0 pnpm check:closed-ceremony-text
0 pnpm check:corpus-claim-drift
0 pnpm check:cross-package-test-inputs
0 pnpm check:doc-authoring
0 pnpm check:driver-memory-census
3 pnpm check:dts-closure (PREREQUISITE NOT MET: no dist/)
3 pnpm check:dual-build-cjs-loads (PREREQUISITE NOT MET: no dist/)
0 pnpm check:gitlink-declared
3 pnpm check:lean-entry-closure (PREREQUISITE NOT MET: no dist/)
0 pnpm check:nul-bytes
0 pnpm check:pm-changeset-deadline-census
0 pnpm check:pm-governed-merges
0 pnpm check:published-files
0 pnpm check:refd-timer-probe
0 pnpm check:role-word
0 pnpm check:skill-compatibility
0 pnpm check:skill-frame-sync
0 pnpm check:skill-identifier-liveness
3 pnpm check:sourcemap-no-sources-content (PREREQUISITE NOT MET: no dist/)
0 pnpm check:watch-hint-literal
```
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_017ihL2dPRZXfgE1A2Vct3FU)_
Co-authored-by: marchtian <noreply@anthropic.com>1 parent 4d80569 commit 6892888
2 files changed
Lines changed: 16 additions & 4 deletions
File tree
- .changeset
- skills/objectstack-platform/evals
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
| 67 | + | |
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
| |||
0 commit comments