Skip to content

config: bind audited diagnostic pilot admission - #240

Merged
Dong1017 merged 2 commits into
mainfrom
codex/skill-eval-pr133-activation
Oct 29, 2026
Merged

Dong1017 merged 2 commits into
mainfrom
codex/skill-eval-pr133-activation

Conversation

@Dong1017

Copy link
Copy Markdown
Collaborator

Governing Issue and delivery slice

Refs #235

Three-file activation configuration for the bounded PR133 diagnostic pilot after #237. No credential, provider, workflow or knowledge-library changes.

  • Migrate public admission configuration to schema2 with actual audited global/PR133 genesis and checkpoint identities; preserve existing PR159/160 anchors and all provider rates/settings.
  • Add the explicit diagnostic mapping for optimization-search-policy and one fixed held-out synthetic case.
  • Existing old/new bundle exposure, request bounds, per-PR lifetime and cross-PR Shanghai daily cap stay enforced by trusted main. No label or model request is added by this PR.

Accounting / review evidence

Audit: #235 (comment)
Daily genesis: 1d3df4e (USD308.29 conservative historical/open obligations).
PR133 genesis: be31645 (zero historical paid execution within audited retained official workflow scope).
Both actual refs, commit parents, one-file trees and ledger bodies were independently read back. Existing 159/160 tips were unchanged. No reset/refund or universal provider-account absence claim.

Independent configuration review passes schema validation and verifies unchanged provider fingerprint253cf88a7ff04311fd1dae897cb68a8c986f136e837dd00dbe5f8872bd094b5f, preserved mappings, exact case bytes and adapter propagation. Worst-case pilot reservation: USD122.04 (12 requests × USD10.17). Current quote expires2026-10-29 10:53:48UTC; final admission must recheck freshness and current ledgers.

Limits and next step

Remote CI and exact-head review required before merge. Then regenerate the exact plan from clean trusted main and refresh labels/head/base/ledger/active-work state before the one-shot controller. Proposed pilot covers one synthetic diagnostic under equal selective-reading budgets: 3 mapped paths and65 uncovered remain explicit. No full-PR coverage, significance, hardware or efficacy result claimed. No provider/model/device calls made during preparation. #235 stays open for actual retained paired results.

Pin existing obligations and exact budget genesis identities; add one frozen diagnostic case without changing credentials or provider rates.

Agent-assisted-by: OpenAI Codex
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
@Dong1017
Dong1017 marked this pull request as ready for review October 29, 2026 02:39
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T02:47:21.793857Z a13e0a1 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Skill evaluation

Automatic execution / cumulative PR budget: {"reason":"Audited PR/global daily ledger missing, unavailable, quarantined or unsafe","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.

  • Retained evidence is planned / unexecuted.
  • No supported comparison cases were selected.
  • Uncovered/unsupported scope needs a case or explicit reviewer disposition.

PLANNED: execution is not authorized by knowing this hash.
Outcomes: {"fix":0,"inconclusive":0,"missing_pair":0,"regression":0,"unchanged_correct":0,"unchanged_incorrect":0}
Earlier revision results are stale for this plan.

Selected comparisons: 0 · Estimated task calls: 0.
Uncovered/unsupported changes: 5. Gaps never count as passes.

Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Evidence and logs · Retention: 30 days.
Outcomes are historical; verifiable only before Unix time 1795833991 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

Per-case outcomes (discovery excluded; regressions and gaps first):

[]

Showing at most 20 of 0 pairs; the evidence link contains all outcomes, including omitted regressions/gaps.

Plan SHA-256: 66867cda86acb2170e33249a5a53a97946c387e7ef10183524e15d806e7a6eda
Base: 29a52e586152d0973cdff79771987994e2f0f1c4 · Head: a13e0a143a5b84e0d63a12b954bee13324efd2c2
Trusted implementation: 29a52e586152d0973cdff79771987994e2f0f1c4

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"253cf88a7ff04311fd1dae897cb68a8c986f136e837dd00dbe5f8872bd094b5f","reservation":null}

Skill Comparison Cases Estimated calls

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-agent-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":131072,"max_requests":6,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (5)
  • uncovered: "docs/examples/skill-comparison/pr133-diagnostic-case05.json"
  • uncovered: "tests/test_skill_eval_auto_daily.py"
  • uncovered: "tests/test_skill_eval_deployment.py"
  • uncovered: "tools/skill_eval_auto_config.json"
  • uncovered: "tools/skill_eval_pr_cases.json"

Retain strict anchor and expiry assertions while testing schema-one migration independently of live configuration.

Agent-assisted-by: OpenAI Codex
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the bounded activation configuration at a13e0a1 against 29a52e5.

Independent review verified actual remote genesis refs, one-file ledger trees and audit receipt6486272232; global historical burden USD308.29 is retained without refunds, existing PR159/160 anchors and provider fingerprint/rates are unchanged. PR133 anchor and schema2 daily cap/timezone are exact. The diagnostic mapping and fixed case propagate to the merged trusted adapter; 12 maximum requests reserve USD122.04.

The two test-only follow-up changes preserve legacy-schema rejection while asserting the exact deployed three-anchor and daily-ledger identities; provider-expiry checks remain. Fresh Ready https://github.com/InferMatrixLab/InferRSI/actions/runs/40808299283 passes all four full-check/installed-package jobs on Python3.10/3.11; policy and draft checks pass, integration clean.

Before any paid admission regenerate the plan at actual merged main and refresh PR/head/base, ledger tips, label authority, active official writers and pricing expiry. This review is configuration/source/software evidence, not a paid result or formal approval. The proposed case tests narrow selective-reading diagnosis; 65 uncovered paths remain, no whole-PR efficacy or production claim. No provider/model/device call made in review.

@Dong1017
Dong1017 merged commit 40fb358 into main Oct 29, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant