Repository navigation
[SkillEval] Bounded diagnostic PR pilot with cross-PR daily admission #235
Description
Goal and bounded scope
Prepare one source-grounded diagnostic old-bundle/new-bundle evaluation of the KV-readiness insight in PR #133. Existing performance-report audit evaluation must remain unchanged. This tracks evaluation infrastructure and the pilot, not acceptance or merger of #133 itself.
Delivery / acceptance
- Strict diagnostic task and answer contract with a frozen independent finite-state oracle, held-out counterfactuals, explicit citation rubric and safe abstention. No answer keys in model workspaces.
- Opt-in selected raw-Git bundle closure with immutable path/mode/blob provenance, existing Reader parity and unsafe-entry rejection. Preserve whole-library snapshot guards.
- Atomic repository-wide Asia/Shanghai daily budget admission in addition to permanent PR lifetime reservations; fail closed on unknown outcomes, duplicate claims, midnight, cap or quarantine. No automatic refunds or credential changes.
- Independently reviewed implementation and applicable software CI. Existing audit compatibility remains covered.
- Audited prior obligations, reviewed immutable pricing/configuration and a feasible frozen plan before any paid execution.
- Retained paired answers, usage/cost bounds and trusted regrading; report improvement/regression/unchanged/inconclusive without extrapolating this small synthetic pilot to production efficacy.
Limits
No hardware experiments, new provider, new credentials or subscriptions. Knowledge/source acceptance, software checks and measured agent utility are separate. No new registry or broad autonomous-history runner. A successful plan or zero comparisons is not efficacy evidence. #133 has a separate integration conflict and existing reviewer history; preserve its exact compared base/head.
Activity
Dong1017 commentedon Oct 29, 2026
Pre-activation accounting receipt (no model execution)
Independent bounded audit at cutoff 2026-10-29 02:29:00 UTC inspected 3,298 unique retained repository Actions runs across fully paginated daily created windows, all 154 SkillEval runs and all job attempts. Exactly five non-skipped execute jobs were found, all on PR159/160: 40487687090, 40496441263, 40499987048, 40700386355, 40700395804. Three attempt-2 records did not execute paid jobs. Raw audit SHA256: bf237d0274067e98859f4527fa6f171ddd741f7b483bc2d08d71e622549c68d3. This covers retained official repository workflows, not deleted/local/manual activity or unrelated provider-account use.
PR133's retained run40294888586 skipped execute; artifact12637493743 contains comparisons=[] and estimated_calls=0 (ZIP SHA256 a413128aa29beb2d52ede6247cb88dbc2f14ec60fef7bc2fc46a99fed2f15090). Thus zero legacy paid PR133 execution is supported within this bounded history.
Preserve immutable existing PR ledgers: 159@2e20abcdf1ec1bda6030682d4f1711933e95bfb7 and 160@2294bd4a173f6e84b8c22de2a0bcfc99687c9297. Import the five real historical plan/run bindings conservatively at USD0.68 + 61.02 + 2.51 + 122.04 + 122.04 = USD308.29. No refunds or settlement; all imported obligations remain charged across days. This is a conservative accounting burden, not an invoice or claim the five jobs are still running.
The bounded pilot reserves at most USD122.04 (12 requests × USD10.17), leaving USD569.67 daily capacity after the conservative import and USD77.96 PR133 lifetime capacity. All values depend on unchanged audited ledger tips and the frozen plan/provider bounds. Fresh unauthenticated public pricing observation at 02:24:28 UTC matches the existing reviewed version and rates; the existing 10:53:48 UTC expiry is not extended.
At the final 02:30:15 UTC snapshot no queued or active paid workflow existed; PR159/160 are closed and PR133 has no execute label. Current-main equality is already checked before reservation and each request by the deployed controller, so advancing trusted main plus a fresh no-active-paid-work check quiesces supported older writers.
Remaining gates: merge independently reviewed #237 with passing CI; publish/pin actual audited genesis commits; review schema-2 configuration and mapping; freeze a fresh exact plan from trusted main; refresh ledger/active-work state immediately before the one-shot label controller. Stop on new contrary evidence, unknown applicable billing or changed obligations. No model/provider call has been made for this pilot.
Dong1017 commentedon Oct 29, 2026
First live pilot: inconclusive, pair incomplete
Run https://github.com/InferMatrixLab/InferRSI/actions/runs/40809150775 completed its workflow successfully, but did not complete the diagnostic comparison. Frozen PR plan45909aaa7c70088e6c52a0d141397f688f281efcf6d88feb9ed2301789ba5189 at trusted main40fb3588e76ece654d4520d08d09e9a25aba7a63 was independently verified against original plan/results/report artifacts and trusted regenerated reporting.
Baseline made 5 API requests and 10 read-tool calls, reading 9 frozen knowledge files totaling49,772 bytes. Before request6, accumulated request payload exceeded131,072 bytes (
request_bytes_limit). No final baseline answer was retained; candidate was not run. Outcome ismissing_pair/ stopped-client-failed, not regression, wrong answer or evidence the skill is ineffective. This was not the dollar/token/request-count cap.Observed tokens:42,223 input and8,194 output; usage marked incomplete. Tariff arithmetic on observed tokens is USD0.665544, not an invoice or a complete cost claim. Both ledgers retain the full USD122.04 reservation, leaving PR133 USD77.96 available; no refund/reset. Execute opt-in was removed after terminal jobs and retained artifacts were confirmed.
ZIP SHA256 receipts: plan85a9f9bc9b609afadc9e587b45bacee64a0472ba98bb6284b4001de2cb73d961; results465aec28b1356ca7e87adcee27d18bf04aea0e1d1d92b2fc5ce2ab4cfd3916f0; report41cee24676ceaf15b30a620ef41eab0af2099e46382a5be050e1b49a0820f03f. Planartifact12869397979, results12870026051, report12869891673.
Keep this Issue open. Next work is offline diagnosis and a newly frozen, independently checked presentation/request-budget correction that fits remaining lifetime allowance; no blind retry or relabeling of this result. One synthetic case and65 uncovered paths cannot establish whole-PR efficacy.
Dong1017 commentedon Oct 29, 2026
Full-context attempt: authorization transport failed before provider HTTP
Run https://github.com/InferMatrixLab/InferRSI/actions/runs/40812282302 completed the workflow, but baseline failed
request_authorization_deniedafter10.087s; no provider response or final answer was retained and candidate did not run. Trusted outcome remains missing_pair, not a skill-quality regression.The baseline full-context manifest and67018-byte payload match the frozen plan exactly. The agent.request entry records an attempt, not a delivered provider call: trusted code invokes authorization before provider HTTP. Job123245290879 shows a later successful grant-body write failing with BrokenPipe at03:29:44.716Z, consistent with the loopback client's10-second timeout expiring while live ledger/GitHub checks completed. #242 corrects this timing without bypassing checks or extending the remaining trial budget.
Plan7bbca9849d48e5f1216ba13950ce4bd3ea5effd4a53dd9d321f0acab70a50d0d; trusted main5011ea67f70bc83d170d1b21cbc845fa213c2338. Verified ZIP digests: plan12871085935=95e5c8f3666733031805d677855702c624fa59b213cf9a957709943026bb9cc1; results12871131484=3725c66a03921ffe0cb70524c3541224b9790da81ec4bf6d2736c9767c65931c; report12870608446=dd89cfbd903403eb6bb60ea573d973a0dc1eac9061683f9683720ab9a8e11c56. Published report matches independent trusted regeneration.
Execute opt-in removed after terminal jobs. Keep both reservations (USD122.04+20.34=142.38), leaving USD57.62 PR lifetime capacity; no refund or rerun. A subsequent corrected plan must be freshly frozen/reviewed and fit that remainder. #235 remains open; no model-efficacy conclusion yet.
Dong1017 commentedon Oct 29, 2026
Valid paired pilot retained; compatibility acceptance still open
Run https://github.com/InferMatrixLab/InferRSI/actions/runs/40814444430 completed the actual old/new pair. Detailed head-bound interpretation: #133 (review) . Compared base a4a0fbd and head 7b3e2e4 using trusted runtime a0a76b3; plan 5b110389eec2ac85f7fd5e2d4fb91e715a67f9e4c67906a412b706664bd3e16a.
Baseline answered capacity / [E4] / inspect_allocator_accounting / reuse=false; candidate answered readiness / [E2,E4] / check_receive_completion / reuse=false. Frozen oracle requires readiness / [E2] / check_receive_completion / reuse=false. Strict result is FAIL→FAIL (unchanged_incorrect): candidate diagnosis and next check align with the policy but the extra citation fails the contract. Both retain safe refusal. This is one synthetic policy case, not unique physical-cause proof, isolated C1 attribution, whole-PR efficacy or #133/#131 acceptance; 65 paths remain uncovered.
Independent trusted regrading and published-report equality were verified, together with resource manifests and frozen inputs. Two provider responses, 104261 input + 5912 output tokens. Conservative observed-usage tariff estimate USD1.279090 is not an invoice. Retained artifacts: plan12871358110 SHA256 6bb4da01c9b5b9a0eeacf9d2d3f1aeb97cca5f5a91e031840613a570960caeca; results12872245696 152c94182c96abebf84e859f8b740cbdace449c5ad871870821bfea4ebfa82f1; report12871464282 218dcece2fde5ca638cb9ff01c6b719a61ae6ffc582553c05f6cdae8ca2e35b2.
Fresh ledger read at 05:29 UTC: PR133 revision6 at49dcedff34a9547a3cd7db6332f85f69969ae8dd retains all three reservations, USD162.72 total, USD37.28 remaining. Daily revision6 at9f58055fc49386ab9927079b93a1979799bd5bb4 retains USD471.01 including historical obligations. No refunds; earlier incomplete attempts remain recorded. Execute label is absent. No additional paid run is needed for the compatibility repair.
Keep #235 open: #237 (comment) remains valid on current main. The eager skill_catalog import makes dependency-light reporting require optional PyYAML. Next scoped correction is to defer that import to selected snapshot preparation and cover no-PyYAML reporting/checkpoint/controller paths. Closure requires this existing compatibility acceptance item to be verified, not a positive pilot score.
fix: keep evaluation reporting free of optional parser imports (#244)
Dong1017 commentedon Oct 29, 2026
Bounded acceptance complete
The compatibility hold in comment6488825415 is now cleared: #244 merged as 861a2bb, preserving the exact independently reviewed two-file repair. Original review opinion4408125903 was accepted, reproduced and answered in #237 (comment) . Review #244 (review) records focused no-PyYAML verification and four passing Ready jobs (40823944277); postmerge policy/declaration checks also pass.
All six accepted items have been reconciled: frozen diagnostic contract/oracle and heldout cases; raw-Git bundle provenance and guards; atomic daily/lifetime admission; independent review/software compatibility; audited obligations and reviewed frozen plans; retained actual paired answers/usage and trusted regrading. Implementation/activation/recovery evidence is in #237, #240, #241 and #242; accounting audit6486272232 and final pilot receipt6488825415 remain authoritative with their stated limits.
Closing this bounded infrastructure-and-pilot issue as completed. Strict result remains FAIL→FAIL / unchanged_incorrect. One synthetic case, 65 uncovered paths, no unique physical-cause or general efficacy claim. This does not accept/merge #133, close #131, or promote skill maturity. Both incomplete attempts and all USD162.72 PR reservations remain retained; no refunds and no additional paid run required.