feat: Diagnoser (Slice 4) — grounded cause attribution for flagged areas#24
Merged
Conversation
Pure-rule, opt-in (`scan --diagnose`) cause classification for flagged areas: doc | config-doc | test-gap | refactor-flag | inherent-complexity (+ unclassified). Grounded by signal profile + doc-existence + a new opt-in `evidence` projection (failed-exec cmd-class + edited-file type buckets). Default output byte-identical; no LLM, no network, no git. Co-Authored-By: Claude <noreply@anthropic.com>
Bite-sized tasks: types+config → classifiers → evidence projection → profile → repo-context → rule engine → orchestration → pipeline → CLI → output → snapshot/privacy → docs → verify. Self-contained contracts per task; byte-identical-default invariant pinned. Co-Authored-By: Claude <noreply@anthropic.com>
Pure single-pass evidence projection for the Diagnoser (Slice 4, Task 3). Buckets a NormalizedEvent stream into the SessionEvidence shape that Task 4 and downstream cause rules consume: - failed exec tool_calls -> failures[classifyCmd(cmd, patterns)]++ - edit tool_call files -> edit_kinds[classifyFile(file)]++ All seven buckets are zero-filled; null-cmd failed execs are skipped per spec. No I/O, no mutation of inputs. Co-Authored-By: Claude <noreply@anthropic.com>
Thread Task 3's computeEvidence through the detector as an opt-in 4th param on runDetector: opts?.collectEvidence === true computes SessionEvidence per-envelope and passes it to assembleStruggleRecord's new 5th evidence? param; otherwise the field is omitted from the returned object literal so JSON.stringify produces no 'evidence' key. Default path (no opts / collectEvidence:false) is byte-identical: existing 3-arg callers (runScan at pipeline.ts:178, runReflect at pipeline.ts:436) are unchanged and the snapshot test passes without regeneration. 322 prior tests + 8 new tests in test/detector-evidence.test.ts all green. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Top-level docsDir symlink guard: if a configured docsDir is itself a
symlink to a directory (e.g. <repoRoot>/docs -> /etc), the lexical
path.resolve confinement check cannot see the target, but readdirSync
would follow the link at the kernel level and leak target files in as
fabricated matches (e.g. billing.md -> docs/sub/billing.md). Add an
lstatSync(dirAbs).isSymbolicLink() check at the docsDir entry point
(before collectFiles) and skip the dir if true; the docsDir is still
recorded in `checked`. Wrapped in try/catch to preserve the never-throws
/ fail-open contract. Mirrors the per-file lstat check already applied
inside collectFiles, just lifted to the entry point.
Test: repo-context test 10 mirrors the existing nested-symlink test at
the TOP level — creates <repoRoot>/docs as a symlink to an outside dir
containing sub/billing.md, asserts gatherRepoContext returns
{ docExists: false, matchedPath: null, checked: ['docs'] }.
Co-Authored-By: Claude <noreply@anthropic.com>
classify(profile, repoContext, cfg) → Diagnosis implements the four specific-
cause gates (doc, config-doc, test-gap, refactor-flag) verbatim from the
brief, scores each eligible cause as the fraction of the shared 5-signal
signature {explore_ratio, reread, failure_streak, corrections, oscillation}
currently elevated (refactor-flag +0.2 boost when repoContext.docExists,
clamped to 1.0), picks the winner with fixed precedence doc > config-doc >
test-gap > refactor-flag, then falls through confidence_floor →
inherent-complexity (wall_clock elevated + meanScore ≥ score_floor) →
unclassified.
Confidence = winner score for specific causes, wall_clock/(2·threshold)
clamped to [0,1] for inherent-complexity, 0 for unclassified. Rationale +
evidence_refs are derived-only (medians, share ratios, repo-relative paths,
integer counts) — no transcript prose, commands, or file bodies.
Pure: no I/O, no mutation; deterministic (gates + fixed precedence → same
inputs give same cause). Style mirrors src/output/hook.ts + src/detector/
ambient.ts.
Tests: 10 cases covering all 8 brief scenarios (doc fires; doc gated off
when docExists → refactor-flag; config-doc via config-share; test-gap via
test-share + low corrections; inherent-complexity via expense; unclassified
when nothing elevated; precedence tie doc>refactor; weak doc score 0.4 <
0.5 floor → unclassified) + purity + determinism. RED → GREEN. typecheck
clean; full suite 367/367 green.
Co-Authored-By: Claude <noreply@anthropic.com>
Thin coordinator tying the Diagnoser slice together: buildProfiles → per-unit gatherRepoContext → classify, with a per-unit try/catch boundary that degrades a throwing unit to a derived-only `unclassified` Diagnosis and never aborts the batch. Output sorted by unit.key ascending; empty flagged set → []. Co-Authored-By: Claude <noreply@anthropic.com>
…ult) Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Task 11 of the Diagnoser slice. Surfaces Diagnoser output in the user-visible leaderboard, opt-in via `--diagnose`: - formatHuman: when `diagnoses` is non-empty, renders a CAUSE column on each flagged row as `cause(confidence)` (e.g. `doc(0.78)`); unmatched or `unclassified` rows render `-`. When `diagnoses` is undefined/empty, the table is byte-identical to Slice 3 (no CAUSE column). - buildJsonEnvelope: includes `diagnoses` ONLY when a non-undefined value is passed (key absent otherwise — default `--json` byte-identical). - runScan: threads `diagnoses` into both builders conditionally. Default-path snapshot (no `--diagnose`) is unchanged — passes without regeneration. Updated the obsolete Task-9 pipeline invariant that asserted `--diagnose` doesn't change the output (it now does, by design). Co-Authored-By: Claude <noreply@anthropic.com>
…t + privacy
Slice 4, Task 12 — locks two guarantees before docs/verify:
(1) Default-scan snapshot stays byte-identical (Slice 1/2/3 invariant). The
original test case + snapshot entry are preserved verbatim — no `-u`
needed there. A second test case + snapshot entry lock the new CAUSE
column added under `scan --diagnose` (all 7 corpus areas currently
classify as `unclassified`, so every CAUSE cell renders `-`; locked here
so future cause drift is deliberate).
(2) New privacy section (e): seeds a distinctive prose marker through every
event a diagnosis leaf could leak from — 2 user_text corrections, 1
failing exec cmd, 1 marker-named read file path — then asserts:
- at least one Diagnosis is produced (non-vacuous; the fixture trips
cause=doc at confidence 0.6);
- the marker is absent from `Diagnosis.rationale` and every
`evidence_refs` member's string-valued leaf (`doc_absent.checked[]`,
`doc_present.path`);
- every leaf is a primitive / enum / closed-union literal (no arbitrary
transcript strings);
- `doc_absent.checked` entries are exactly cfg.docs_dirs literals;
- the marker is absent from the serialized diagnoses array as a whole.
Confirms Task 7's derived-only-by-construction contract holds end-to-end.
Co-Authored-By: Claude <noreply@anthropic.com>
Sync §9 testing-strategy with the current test code: section (d) was added in Slice 2 (baseline/repo_findings/calibrate prose guard), section (e) was added in Slice 4 Task 12 (diagnosis-leaf prose guard). Updates the section count and adds concise (d)/(e) summaries derived from the test assertions. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ocs_dirs validation, doc consistency
Consolidated fix wave from the whole-branch review (Minor-Important; no Critical).
Tests (honesty/stronger):
- config.test.ts: share_floor validators now assert field-named messages
- cli.test.ts: removed stale Task-11 NOTE; assert parsed.diagnoses is array
- snapshot.test.ts: softened overstated "CAUSE column locked" framing
(snapshot content unchanged; only the key string updated to match the
softened it() title — DEFAULT snapshot byte-identical)
- classify-util.test.ts: precedence test now exercises config>build
('npm setup tsc' has both tokens; was 'npm setup install' with no build)
- output.test.ts: formatHuman diagnoses:[] byte-identity + new test 25
(buildJsonEnvelope diagnoses:[] includes the key in JSON)
- privacy.test.ts: off-by-one comment (30 reads -> 31 with marker read)
Code (robustness):
- src/diagnoser/index.ts: outer try/catch around buildProfiles+per-unit loop
returns [] on throw — the "never throws" contract now holds unconditionally
(runScan calls diagnoseUnits unguarded). New test case (h).
- src/config.ts: validate docs_dirs as array of strings (was unvalidated;
scalar/non-string element would silently replace default and surface as
confusing internal error later). 3 new tests.
Docs:
- README/ARCHITECTURE/CONSUMER_GUIDE: stale "This is Slice 1" framing softened
to "default detection-only path + opt-in diagnosis".
- Diagnoser spec §7: computeEvidence path corrected
(src/detector/evidence.ts -> src/diagnoser/evidence.ts) to match code.
Verified: npm run typecheck clean; npm test 394/394 green; DEFAULT snapshot
byte-identical (snapshot tests pass without -u).
Co-Authored-By: Claude <noreply@anthropic.com>
…gline docs-watcher follow-ups after the fix-wave commit (00d2a54): - ARCHITECTURE.md §modules row + §Fail-open: now describe BOTH fail-open layers in src/diagnoser/index.ts — outer batch-level try/catch (degrades to []) + inner per-unit try/catch (degrades to one unclassified Diagnosis). Previously under-documented the new outer boundary added in 00d2a54. - CLAUDE.md: drop "Slice 1:" tagline label (inconsistent with the Diagnoser / Slice 4 sections in the same docs). Keep the accurate "no writes, no network" half for the default path; note --diagnose is the opt-in layer. Doc-only; no code or test changes. Typecheck clean; 394/394 tests still green. Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Jul 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the Diagnoser — the next component in the closed loop (
detect → diagnose).scan --diagnoseclassifies each flagged area into a typed cause with a derived-only rationale, so a human knows why an area struggles, not just that it does.Pure-rule, opt-in, and byte-identical when off — the Slice 1/2/3 invariant holds. No LLM, no network, no
git, no raw prose.Why each cause (grounding)
docconfig-doctest-gaprefactor-flaginherent-complexityDoc-existence is the key new fact — it stops the tool from recommending a doc for a code problem.
What's new
src/diagnoser/—classify-util(cmd/file catalogs),evidence(projection),profile(per-unit signal medians + elevation),repo-context(doc-existence, path-confined, fail-open, never-follow-symlinks),classify(rule engine),index(orchestration, per-unit fail-open).evidenceprojection onStruggleRecord(failed-exec cmd-class + edited-file type buckets), computed in the detector only under--diagnose.scan --diagnoseflag →causecolumn in the human table +diagnosesin--json(both opt-in only).docs_dirs(default['docs']) +diagnoseblock (floors).evidence/diagnosesexist only opt-in; snapshot+18/−0(only a new--diagnosesnapshot added).Contracts held
test/egress.test.ts), nogit, no raw prose in any diagnosis leaf (new privacy test, non-vacuous), path-confined fs reads, fail-open throughout.commander+yaml(test/packaging.test.ts).Verification
docsDirno longer followed — #215c88d).scan --diagnoseruns end-to-end on real transcripts without error.Deferred (tracked as issues)
LLM refinement +
docs_dirspurpose-form → Synthesizer (#19); git churn grounding (#11); per-area evidence attribution (#12); reflect integration (#14); configurable catalogs (#13); structural-absence repo unit (#18). Cause-floor calibration is a dogfood gate (#15). Spec: `docs/superpowers/specs/active/2026-07-18-harnessgap-diagnoser-design.md`.🤖 Generated with Claude Code