You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs/EVAL_HARNESS.md documents the protocol but doesn't frame the "why would I add a case?" question.
A reader expecting an LLM-eval story finds infrastructure without conviction.
Proposed solution
Pick one of the following:
Option A — Add real cases. Add 3–4 realistic golden cases (e.g. a Q/A pair, a code-generation prompt with a deterministic expected substring, a tool-use trace check). Wire the nightly workflow to run them by default. Update EVAL_HARNESS.md with a "how to add a case" section.
Option B — Reframe as scaffold. Add a paragraph at the top of EVAL_HARNESS.md explicitly framing the eval slice as "scaffold for your project's eval cases, not a benchmark." Document the contract for adding cases. Keep the toy case as a smoke test and explain why the nightly is opt-in.
Acceptance criteria
One of Option A or Option B chosen and implemented
EVAL_HARNESS.md tells a reader either how to add cases or why this is a scaffold
Nightly workflow status (enabled / opt-in) is consistent with the chosen framing
Priority rationale
Currently the weakest organ of the harness story; either path closes the gap.
Activity
test: add adapter unit tests + adapters README (#94 review fixes)
docs: add Key interfaces section to adapters README (#94 review)
feat: eval pattern examples calling Azure OpenAI (#94)
test: add adapter unit tests + adapters README (#94 review fixes)
docs: add Key interfaces section to adapters README (#94 review)
feat: eval pattern examples calling Azure OpenAI (#104)
constk commentedon May 26, 2026
Closed via #104 (merged to develop as commit 18b4d30). Awaiting the next
release:PR to reach main.