Repository navigation
[Status 02] InferRSI evolution: ac702729 → 48e0d589 (2026-10-23) #116
Description
Incremental status snapshot using the six-section convention in Status 01. This brief records delivery, acceptance and evidence separately; it is not a new execution authorization.
1. Snapshot
- Checked: 2026-10-23, 09:05 UTC / 17:05 Shanghai.
- Range: previous published head
ac702729→ inspected main48e0d589, 38 commits ahead. Pinned comparison. - Scope: integrated changes since Status 01, plus material issue/acceptance changes and clearly separated open work. Integration dates below are not necessarily experiment dates.
- Bottom line: InferRSI now ships a more usable packaged knowledge library, stronger software/evidence checks, model-aware research preparation, and a reusable local skill-comparison path. The first real-agent pilot was accepted with a neutral efficacy result. Automated PR evaluation and broader task/hardware effectiveness remain unestablished.
2. What changed
| Change | Before → now | Why it matters | Delivery |
|---|---|---|---|
| Knowledge access and packaging | Split checkout/library material and incomplete text output → canonical categorized English bundles, generated discovery records, explicit omissions/no-match/material output and a documented base-versus-skills install path | Installed readers receive complete procedures/resources; the usability gaps identified in Status 01 have a merged correction | #44, #45, #46, #42, Oct 20–21 |
| Reliability and admission | Incomplete portability/evidence/package coverage → explicit Linux server-helper contract, reduced hostname collection, evidence inventory, frozen-byte checks, clean-wheel and minimum-Python CI; report-only knowledge authorship checks | More failures become visible before reuse; admission compliance still does not establish content correctness or enforced merge blocking | #52, #62, #64, #86, Oct 21–22 |
| Skill evaluation | Proposal → real-client pilot, corrected future ordering, independent negative grading controls and configurable absent/new or old/new comparisons | Reviewers can retain fixes, regressions, missing pairs and inconclusive outcomes without mistaking execution errors for success | #69, #82, #94, #109, Oct 21–23 |
| Bounded source knowledge | Broad source inventory → additional repository-lifetime procedures and targeted TP-transfer, checkpoint, communication, compiler and CannBot evidence rules | More actionable source-grounded guidance, with original pins, candidate maturity and untested boundaries retained | Living coverage tracker #91; recent #96, #95, #108, #107, #114 |
| Task preparation | Manual exact-AND searches and generic benchmark guidance → opt-in model/framework/task/modality/mechanism query union, attributed results and task-specific benchmark selection | Easier preparation without changing legacy search semantics or implying engine execution/hardware qualification | #102, #111, Oct 23 |
3. Architecture delta
The shared Harness → Model/Infra experiment architecture remains. New functionality extends knowledge consumption and evaluation around it rather than adding another model runner or autonomous optimizer.
flowchart LR
K[Canonical packaged knowledge] --> R[Collect / read / research]
R --> A[Human or external coding agent]
A --> W[Existing experiment workbench]
K --> S[Frozen Git skill snapshots]
S --> P[Local comparison plan]
P --> E[Authorized existing client and grader]
E --> O[Retained JSON / Markdown evidence]
O -. proposed integration .-> G[Automatic PR feedback: issue 101]
- Implemented: complete packaged bundles with generated projections; query expansion for preparation; revision-bound comparison planning, execution and report regeneration. Git 2.45+ is required by the new comparison path.
- Unchanged boundaries: the caller chooses goals, runtime and permissions. Reference and SearchSpace remain the two primary matrices. Report regeneration makes no model call. Knowledge promotion, engine execution and paid evaluation are not automatic.
- Sources: architecture, comparison contract. The dashed PR-feedback path remains planned, not delivered.
4. Capability status
| Capability / workstream | Code delivery | Goal acceptance | Validation / effect |
|---|---|---|---|
| Agent Usability v1, #32 | Corrective #42 merged at 035aa600 |
Closed; corrective record supplements the earlier assessment | Functional output/install checks; no measured agent productivity claim |
| Source-audit reliability, #43 | Seven bounded fixes in #52, d87aead9 |
Umbrella remains open | Software/package/inventory checks; unresolved license and product decisions are not erased |
| Skill Eval v1, #40 | #69 plus #82, accepted revision 407dd2a4 |
Accepted and closed | Runnable pilot accepted; no observed incremental correctness benefit |
| Skill Eval v1.1, #88 | Grading #94 at b3ef95a6; local comparisons #109 at 48e0d589 |
#99 closed; #100 closure-ready evidence recorded, owner closure confirmation pending; #100, #88 and #101 open | Offline controls demonstrate pipeline behavior. Real-agent addition/update-to-PR acceptance and automatic feedback remain outstanding |
| Coverage / consolidation, #91 | Multiple scoped deliveries, including CannBot #114 at 4ac039c1 |
Reopened as a maintained tracker; scoped merges do not complete it | Source/packaging checks; proposed old/new cases remain uncovered, not efficacy results |
| Model-aware preparation, #103 | #111 merged at d600992d |
Delivery does not establish broader scenario acceptance | Search/attribution and installed-package checks; no web crawler, new runtime or target-capacity proof |
| Knowledge admission, #50 | Report-only checker and lifecycle fixes merged; #98 still open at 6350b051 |
Rollout acceptance and enforcement remain separate | #98 narrows stale-status windows; it does not deliver required-check configuration or real merge-blocking validation |
| Campaign Plan / MLX, #30 / #38 | No accepted Campaign Plan integration established; MLX/MLX-LM source skills did land | Both issues remain open | Source knowledge does not satisfy MLX discovery/backend or Apple Silicon qualification criteria |
5. Evidence and limitations
- Accepted pilot: retained report and corrigendum records 18 final matched trials plus three discovery probes, all passing, with zero recoveries/regressions across 12 pairs. The no-skill baseline was equally correct. Historical ordering was unbalanced; elapsed differences are descriptive and confounded, not causal skill overhead. Earlier failures and the interrupted attempt remain retained.
- New comparison tool: feat: add configurable offline skill addition and update comparisons #109 reports an offline example with one fix, one regression, one inconclusive pair and one missing pair, with zero real model calls. Ready CI succeeded for reviewed head
403d92e4; its recorded Python 3.10/3.11 jobs each report 702 passed, 9 skipped and 70 subtests passed, plus clean-wheel checks. These are PR validation results, not a claim that post-merge CI on48e0d589has completed. - Source delivery: CannBot knowledge: ground CannBot stage evidence in Ascend analysis #114 retains a nineteen-file primary readset and packaged license; source binding and ready CI support its scoped delivery. Hooks, agent host, model/NPU execution and agent efficacy were not tested. The same source-versus-effect distinction applies to the other new distillations.
- Negative hardware evidence remains visible: open #97 records the frozen 8K FP8 comparison against vLLM: median paired TTFT ratio 1.700627 and TPOT ratio 1.153777, both failing the ≤1.03 performance gates. Whole-campaign status remains INCONCLUSIVE with quality/retrieval incomplete. Individual engineering gains do not override those failed gates; this PR is not merged or campaign-accepted in the snapshot.
- Report method: repository sources, merge flags, issue records and CI status were inspected. No new tests, model calls, device runs or benchmarks were executed for this brief. Green CI and source import do not establish task efficacy or hardware performance. Private operational paths and identifiers are excluded.
6. Next decisions
- Finish the bounded evaluation workflow: confirm Skill Eval: add configurable skill addition and update comparisons #100 closure, then implement #101's revision-bound planning/publication contract. Keep #88 open until its authorized real-agent addition/update and end-to-end criteria are met.
- Keep coverage honest: use #91 for missing sources and mechanism-level consolidation, preserve existing pins and candidate states, and select a genuine content change before proposing an efficacy comparison. Intake-only docs: record bounded vLLM-RLT intake #104/docs: record bounded AFD plugin intake #105 does not count as completed mechanism extraction.
- Separate delivery from operational acceptance: resolve #98 and [Tracking] Knowledge PR admission: model and effort policy #50's residual-window/enforcement decisions on their own evidence; assess evidence: freeze approved 8K search and retain FP8 optimization results #97's remaining quality/retrieval and frozen gates separately from PR readiness. No new spending, repository settings or campaign acceptance is authorized by this brief.
Activity
Dong1017 commentedon Oct 27, 2026
In-week architecture / delivery update — 2026-10-27, 09:57 Asia/Shanghai
This is an observed catch-up since the published snapshot, not reconstructed daily observations. The weekly body remains unchanged.
48e0d5894117be7c57f8b4ce793c0ec986f219e8.97f77a91373f3e2b61b03afcfd227e21edbffa2f; pinned comparison.Delivered capability versus acceptance
Architecture: extensions around the existing runtime
The CLI/RunSpec → Harness → Model/Infra separation remains. Developers should navigate from architecture.md, checking its claims against pinned implementation:
knowledge_operations.py,knowledge_store.py, canonicaldata/skill_library/; operation contract. Selection does not grant execution or promotion.tools/skill_eval_compare.py,tools/skill_eval_pr.py, the existing trusted workflow; workflow contract.tools/goal_checkpoint.pyderives a view from evidence; it does not create a new authority or engine runtime.No architecture diagram here should be read as proof that proposed Ceiling APIs, cross-host adapters or a complete autonomous optimization loop already exist.
In-flight work, blockers and where to contribute
Method: static repository/merge/acceptance/CI inspection and explicitly reported evidence. No new model, hardware or benchmark experiments were run for this update. Proposed reporting format is discussed separately in OmniRSI #2; it creates no new execution authority.
Dong1017 commentedon Oct 27, 2026
Current system architecture — 2026-10-27
Pinned to main
38d5883, including merged #174. This complements the earlier weekly update; it is an as-is implementation map, not a claim of autonomous optimization or completed live Skill Eval.Read the graph: blue = knowledge/indexing; green = Skill Eval; amber = GitHub jobs; purple = offline/future provider path. solid arrows show implemented data/control paths, subject to the gates printed in their boxes. Dashed arrows are not live-wired. A code path existing does not mean it has run successfully.
Where each box lives
run_trial, client invocation and evidence; skill_eval_verifier.py independently grades supported tasks. Missing/invalid evidence stays missing or inconclusive, not a pass.read_file/submit_answerand mockable transport. The comparison controller can freeze its identity, butrequire_live_readiness()refuses live execution. The external endpoint is yibuapi, using a Messages-compatible protocol—not an established direct connection to Anthropic's own service.Operational status: publication works; live evaluation is still separate
codex --version; it does not install/provision the Codex client or model access. The execute job has read-only GitHub repository grants and no PR publication credentials; only the separate publisher requests PR write permission. Exact-plan/revision/budget authorization remains mandatory. See execution and retention contract.Indexer → usable knowledge; Skill Eval → controlled evidence; GitHub jobs → trusted planning and feedback. These are connected components, not an automatic learn-and-deploy loop.