Repository navigation
Skill Eval: harden deterministic grading and offline failure handling #99
Description
Parent: #88 · Delivery 1 of 3
Make the existing Skill Eval grader and error reporting trustworthy before adding reusable comparisons.
This issue records the already completed first delivery, implemented by #94 and merged as b3ef95a6dd3abd85c19b5527fe46e80ccb3521d4. It is a retrospective tracking/acceptance record, not a new implementation request.
flowchart LR
A[Literal answers] --> B[Verifier]
C[Offline client] --> D[Eval pipeline]
D --> B
B --> E[Outcomes]
D --> E
E --> F[Summary]
Scope and acceptance
| Area | Required result | Status |
|---|---|---|
| Independent grading cases | Literal expected answers, negative cases, threshold boundaries, rule precedence and extract mode; expectations do not call production expected() |
Delivered |
| Offline pipeline checks | Fixed-output client exercises execution, grading, saved records and summary without model calls | Delivered |
| Error semantics | Correct rejection of a bad input report can be task PASS; completed incorrect answers are FAIL; unavailable execution/grading is INCONCLUSIVE | Delivered |
| Recovery and usage | New interrupted records carry evaluation_error="interrupted_attempt"; partial usage remains unknown; legacy records remain unchanged |
Delivered |
Evidence
- Implementation and verification: PR #94.
- Reviewed head:
a3b519377862a9fb8bf0f455280f863d7f84575a; ready CI. - Python 3.10/3.11 full-check and installed-package jobs passed. Inspected Python 3.11 log: 584 passed, 9 skipped, 70 subtests passed.
- The recovered-interruption counter defect was corrected, with persisted classification, repeated-recovery byte stability and legacy compatibility regressions.
Limits
Offline tests establish these specified grading and failure-handling contracts, not skill efficacy. Free-text reason semantics are not judged. No skill edits, real-agent/device experiment, new runner, paid evaluation or efficacy merge gate is delivered. Historical evidence remains unchanged.
The configurable comparison command and PR feedback are the remaining deliveries under #88; this issue's closure does not close the parent.
Activity
Dong1017 commentedon Oct 23, 2026
Acceptance / closure
flowchart LR A[Self-consistency tests] --> B[Independent fixtures] C[Incomplete error records] --> D[Explicit error categories] B --> E[Traceable outcomes] D --> Eb3ef95a; reviewed heada3b5193Accepted as delivery 1 of #88. This issue was created retrospectively to restore the agreed three-issue tracking structure. Remaining work: configurable comparisons #100, then PR planning/feedback #101. Parent #88 stays open.