Repository navigation
Skill Eval: add configurable skill addition and update comparisons #100
Description
Parent: #88 · Delivery 2 of 3 · Builds on #99 / merged #94
Turn the existing pilot into one local comparison tool for skill additions and updates.
flowchart LR
A[Config] --> B[Snapshots]
B --> C[Plan]
C --> D[Authorize]
D --> E[Compare]
E --> F[Report]
Deliver
| Part | Minimal implementation |
|---|---|
| Configuration | Target skill, baseline/candidate Git revisions, cases, grader, model/settings and bounded execution limits. Reuse existing metadata; no skill-version registry. |
| Snapshots | Materialize the two Git revisions, then read/export their complete skill bundles with the existing reader. Its revision argument validates a snapshot; it does not retrieve historical Git content. Record existing revisions/hashes automatically. |
| Execution | Reuse CodexClient, export_bundle, run_trial and the accepted error handling. Fresh sessions, matched inputs and balanced two-arm ordering. |
| Comparison | Replace hardcoded N / E* pairing with explicit baseline/candidate roles, paired by task ID and repeat. Keep discovery results separate. |
| Reporting | Generate JSON and Markdown from retained run artifacts: fixes, regressions, unchanged correct/incorrect cases, missing pairs and inconclusive outcomes, with outputs and grading evidence. Unknown usage stays unknown. |
Comparison contract
- Addition: no target skill versus new target skill.
- Update: old target skill versus new target skill.
- Freeze cases, grader, model/settings and relevant resources; only the declared skill bundle/dependency change differs. Surface dependency and test/grader changes rather than silently changing the exam.
- Planning performs no model calls. Execution requires explicit authorization bound to the frozen plan/revisions and limits.
- Report regeneration must not rerun trials or spend tokens. Use fresh output directories; no generalized resume service.
Start with the existing performance-report audit tasks and verifier. Do not bundle new DLRover/fastsafetensors scoring rules into this tool increment.
Close when
- One entry point prepares and runs both an addition and an update using configuration rather than skill-specific code changes.
- Tests establish distinct frozen old/new bytes and an absent-target baseline, including resource changes and snapshot mismatches.
- Offline fixed-output clients exercise both modes, paired fixes/regressions, missing pairs, timeouts and output/grader errors; these cannot become false passes.
- A deliberately wrong candidate output is detected against independently specified expectations; label this as an offline pipeline control, not proof of real skill efficacy.
- Generated JSON + Markdown retain both revisions, settings, case results, evidence links and usage limits; regenerating a report makes no model call.
- Existing pilot behavior and historical evidence remain intact, with concise usage and configuration examples.
- Closure includes a compact change diagram, verification evidence and limits.
Boundaries
No new runner, dashboard, version platform, automatic skill rewriting/promotion, GitHub posting or required efficacy gate. Real-agent experiments need separate authorization; offline acceptance here does not discharge #88's real-agent addition/update end-to-end acceptance.
Activity
Dong1017 commentedon Oct 23, 2026
Delivery 2 accepted: configurable local skill comparisons
Merged via PR #109, reviewed head
403d92e40b418788bc590d973f19813b91c83c83, squash commit48e0d58.Final scoped review found no remaining blockers. Reviewed-head CI passed both Python full-check and installed-package lanes; Python 3.11 reported 702 passed, 9 skipped, 70 subtests passed. Independent review ran deterministic exact-source closure probes; this is not a claim of a full local rerun or post-merge CI completion.
Owner confirmation: closure explicitly authorized on 2026-10-23; closed as completed after the scoped offline-tool acceptance. #88 and #101 remain open.
Limits: zero real-model/paid calls in this acceptance. Fixed-output controls validate tooling, not skill efficacy. #101 owns PR integration; #88 stays open for all three slices and separately authorized real-agent addition/update end-to-end acceptance. No new runner, version registry or required efficacy gate was added.