feat(eval): online trace scorer — nuvel eval (v1)

[Folken2]

Summary

First end-to-end implementation of the eval harness per spec 2026-05-20-eval-harness-v1-design.md and plan 2026-05-20-eval-harness.md. Online trace scorer: heuristics-first, judge-on-pass, scored.jsonl siblings, full nuvel eval CLI, dashboard score column, drift detection.

• Heuristics (8 flags): tool_error, llm_error, no_assistant_output, incomplete_trace, excessive_turns, tool_loop, cost_outlier, latency_outlier, token_bloat — deterministic, pure functions over Run.events.
• Judge: litellm.acompletion via an injection seam (tests never make real network calls). Model resolves: rubric → EVAL_JUDGE_MODEL env → DEFAULT_FAST_MODEL. Tolerant JSON parse, one retry.
• Scorer orchestrator: ScoreSession owns idempotency (skip on trace_id + scorer_version), concurrency (semaphore=5), and the cost budget (judges disable cleanly once spend crosses --max-cost-usd; heuristics keep running so nothing is dropped silently).
• CLI: nuvel eval score | report | worst | drift. drift exits non-zero when any agent crosses threshold, ready for a future webhook layer.
• Dashboard: loader side-loads scored.jsonl; home view shows a monochrome-friendly score pill (good/warn/bad/none) with the flag list in the tooltip.

Code preservation note

One change to existing code in nuvel/traces_cli.py: _iter_trace_files now skips reserved companion files (initially just scored.jsonl). Without this, the CLI/dashboard would re-ingest scored output as if it were trace data — a latent bug surfaced by the scorer's tests. Tested against the existing test_dashboard_watcher.py and test_dashboard_loader suites — no regressions.

Behavior changes

• New CLI subcommand nuvel eval — see docs/reference/cli.md.
• New env var EVAL_JUDGE_MODEL — documented in docs/reference/env-vars.md.
• Dashboard run cards gain a Score column. Unscored runs render as a muted em-dash; no visual regression when scored.jsonl doesn't exist.

Module layout

nuvel/eval/
  __init__.py     # public surface
  schema.py       # ScoredRun, JudgeResult, Flag, SCORER_VERSION
  heuristics.py   # 8 deterministic flag rules
  stats.py        # per-agent rolling percentiles (p95 cost/latency, p99 tokens)
  rubric.py       # default + YAML override loader; model priority chain
  judge.py        # litellm wrapper with retry + tolerant JSON parse
  scorer.py       # score_run + ScoreSession orchestrator
  writer.py       # append-only scored.jsonl + tolerant index loader
  drift.py        # rolling-window mean comparison per agent
  report.py       # render_report / render_worst / render_drift
  cli.py          # argparse subparsers

Test coverage

+88 new tests, all passing. Full suite: 471 passing.

• test_eval_schema.py — round-trip, constants, judge ok property
• test_eval_stats.py — percentile math, agent grouping, window cap
• test_eval_heuristics.py — every flag rule + stacked-penalty floor
• test_eval_rubric.py — default + YAML override + model priority chain + malformed YAML
• test_eval_judge.py — prompt assembly, JSON-from-prose, retry, error
• test_eval_writer.py — append/load round-trip, tolerance, last-wins
• test_eval_scorer.py — orchestration paths (heuristic-floor, judge-disabled, force, version-bump, budget exhaust, idempotent)
• test_eval_drift.py — window math + threshold edges + naive-tz handling
• test_eval_cli.py — full argparse paths via top-level nuvel.cli.build_parser
• test_dashboard_score_integration.py — loader join + view rendering + full FastAPI endpoint

What's NOT in v1 (deferred per spec)

• Alert delivery (Slack/webhook) — drift exit code is the seam.
• Score migration tooling for large scorer_version bumps.
• Golden-set / regression CI gate (separate harness).
• Replay with perturbed config (separate harness).
• Storing scored output in Postgres (stays JSONL until OrgMemoryService justifies a shared DB).

Test plan

• All new and existing unit tests pass locally (471/471).
• CLI nuvel eval --help and subcommand helps render correctly.
• Real-data smoke test: EVAL_JUDGE_MODEL=openrouter/moonshotai/kimi-k2.5 nuvel eval score --max-cost-usd 0.20 against existing generated-agents/*/traces. Not run overnight — judge calls cost money and require OPENROUTER_API_KEY.
• Dashboard smoke: nuvel dashboard with at least one scored run — confirm Score pill renders and tooltip lists flags.

🤖 Generated with Claude Code