Repository navigation
Skill Eval: automate PR evaluation planning and feedback #101
Description
Parent: #88 · Delivery 3 of 3 · Depends on #100's local comparison/report contract
Connect the local evaluator to skill PRs without making maintainers copy reports.
flowchart LR
A[PR change] --> B[Plan]
B --> C[Authorize]
C --> D[Evaluate]
D --> E[PR report]
F[New revision] --> B
Planned increment
| Capability | Required behavior |
|---|---|
| Automatic planning | Identify changed skills/resources using a small maintained case mapping. Show selected cases, uncovered changes and estimated calls/limits; planning makes no model calls. |
| Authorized execution | Bind each execution to explicit approval of the frozen plan, PR head, compared revisions and budget. A new revision requires renewed authorization. |
| Automatic feedback | Generate/update one identifiable PR summary with outcome differences and evidence links. No manual report copying. |
| Freshness and retry | Mark old results stale after relevant input changes. Duplicate triggers do not duplicate paid work; publication retry reuses results without reevaluation. |
| Separation | Execute untrusted PR content without publication credentials; use trusted code to publish validated report data. |
Detailed trigger, authorization and artifact-retention choices should be finalized against the delivered local comparison interface, not by adding a new orchestration platform now.
Close when
- A skill addition and an update select the appropriate comparison; unmapped changes are visibly uncovered, never passed.
- New head/plan changes invalidate prior authorization/results; stale jobs cannot replace a current report.
- Offline integration tests cover duplicate triggers, publication failure/retry and untrusted report handling without real model calls.
- The tool publishes a revision-bound PR report automatically from retained evidence, with a documented retention window.
- The parent Skill Eval v1.1: automated evaluation for skill additions and updates #88 end-to-end exercise covers one addition and one update with separately authorized real-agent execution and automatic PR feedback. Neutral or negative results are valid outcomes.
- Closure records a concise change diagram, verification receipts and remaining limitations.
Boundaries
Report-only integration, not a required efficacy gate or automatic merge/promotion. Authorship admission remains separate. No new skill-version fields, manual result ledger, dashboard, runner, or automatic evaluation of local Create/Update. Issue creation grants no model budget, credentials or repository-security changes.
Activity
Dong1017 commentedon Oct 23, 2026
Next delivery after #109
The local comparison implementation is now on main at
48e0d58. #100 is accepted and closed with explicit owner confirmation (2026-10-23); its offline closure evidence is recorded. #101 is now the remaining implementation slice under #88.tools/skill_eval_compare.pyanddocs/skill-eval-comparison.mdat the merged revision; identify the existing plan hash, execute and report interfaces before designing glue.No new runner, skill-version registry, dashboard or required efficacy gate. Keep the PR-facing layer thin. Real-agent addition/update end-to-end exercises still require a separately fixed budget and authorization; this comment grants neither paid execution nor new credentials/settings. #88 remains open.
Dong1017 commentedon Oct 25, 2026
Issue owner and remaining delivery plan
Acting Issue-level owner: Codex, the agent continuing #101 across its delivery PRs, under the user's explicit delegation. Preserve the three implementation workstreams under #88. #101/#88 remain open until their real addition/update flows and automatic feedback satisfy acceptance and the human owner decides.
Current delivery head: #118 at
c716307df7c9dbd0674638ecea894bd912790882, branchcodex/issue101-pr-skill-eval, Ready, unmerged and mergeable. Main snapshot merged into it isea7ab7dae50c8c943fac54ca4a16b58f4402ab67. Later main increments0cffd207/a8e4c572are knowledge guidance/projections with no file overlap with the ten-file integration increment; reassess main immediately before any future merge.de54da1preserves both operation-table entries; #145 is already merged asea7ab7dand #146 as6c426e8by their respective integration ownersf1c422cCurrent integration gate: per-head
knowledge-model-policyis success / Not applicable, but associated aggregatedeclarationsjobs are cancelled. This is not a claim that all applicable checks pass. The earlier independent software LGTM covered the oldf1c422chead and is retained as history, not approval for this new delta. No reviewer-role approval has been authored by this Issue owner.The human permits an expected-head squash merge of #118 only when genuine independent current-head Dong1017 LGTM, every applicable current CI/check/status and integration conditions all hold. Do not auto-merge, bypass checks or treat a green per-head status as clearance of an unresolved applicable workflow result. #145 is already merged; do not repeat that operation.
Next action: obtain independent review for the current full head, address any new actionable finding in this same PR and inspect current CI. Continue actual trial-subject planning only when trusted-main workflow/mapping, runtime/automatic feedback and truthful author admission are usable. There is no additional execution authorization, new runner, credential/security change, fourth evaluator workflow or Issue closure in this update.