Repository navigation
SkillEval: explain retained old/new results in automated PR feedback #246
Description
Parent tracking: #91. Bounded follow-up to completed #235, not a reopening of its infrastructure/pilot scope.
Goal
A PR participant should understand what changed, why the frozen result failed, and what remains unknown directly from the github-actions comment, without downloading artifacts or relying on a manually written review.
Scope / acceptance
- Extract bounded adapter-aware component results from trusted regrading of retained answers; preserve exact aggregate scores, distinguish unavailable data from failures, and do not infer explanations absent evidence.
- Lead automated feedback with baseline/candidate differences, component pass/fail and concrete failure reason. Use PR133 retained run40814444430 as acceptance fixture: old capacity/E4/allocator check versus new readiness/E2+E4/receive check; candidate remains strict failure for extra citation, both refuse reuse.
- State coverage and evidence limits visibly: one synthetic case, uncovered scope, exact head/base/case/evidence bindings; no general efficacy, automatic LGTM or acceptance claim.
- Separate historical retained-result validity from authorization/budget for starting a new run. A blocked new execution must not obscure available valid evidence.
- Allowlist, escape and bound rendered fields; never dump arbitrary model text, hidden oracle answers, secrets or unbounded content. Preserve stale/expired/mismatched evidence handling.
- Regression tests cover changed components with unchanged total outcome, regression, missing pair, invalid/oversized fields and stale/expired/mismatched artifacts. Existing grading and admission behavior remain unchanged.
- Independent review and applicable software CI pass. After merge, verify the real github-actions PR133 comment through the legitimate report/publisher path using retained evidence, with no new paid model execution; read back final output before closure.
Delivery boundaries
Prefer one implementation PR for extraction/rendering/security/tests together. Split only if a genuine independent data/publisher prerequisite is discovered. No new task family, grader relaxation, provider/model calls, hardware experiment, budget refund, or #91 ownership takeover. Full model efficacy and #133 integration remain separate. A preview, merged PR or green CI alone does not satisfy final user-facing acceptance.
Acceptance: verified delivery and closure receipt, 2026-10-29. Broader SkillEval work continues in #253.
Activity
feat: explain retained SkillEval answers and component results (#247)
Dong1017 commentedon Oct 29, 2026
Acceptance: retained-result explanation delivered
#247 merged as
5a46ad5a9e5bb93ec8111d47d97c7e341c476710. The current-head independent review and Ready checks cover extraction/rendering, bounded escaped fields, preserved strict grading and stale/missing/mismatched evidence regressions. Full Python 3.11 CI reports 2496 passed, 11 skipped and 284 subtests; all four applicable jobs passed.The remaining user-facing criterion is now verified: report-only run 40848815720 completed successfully and updated the actual github-actions PR133 comment at 09:21:08 UTC. Execute was skipped; this used retained source run
40814444430, with no new paid model execution or reservation.Readback confirms:
5b110389…, basea4a0fbd…, head7b3e2e4…and implementationa0a76b3…preserved. Original answer bytes were independently downloaded; hashes721357bb…/3f16ddc0…match report/comment.All bounded #246 acceptance items are satisfied. Close this issue as completed. This does not close #133, #91, or the broader #253 SkillEval evolution; strict failure and efficacy limits remain unchanged.