agentic skills for academic demo

[vedika-saravanan]

Description

Adds the cudaq-academic-vqe-qaoa skill plus a self-contained, reproducible
evaluation harness for a workshop that demonstrates the value of agent skills
on CUDA-QX Solvers questions (VQE, QAOA, ADAPT-VQE, GQE). Attendees run each
prompt with and without the skill and compare three things: which
algorithm the agent recommends (recognition), how much it explores
(files/tool calls), and token usage.

This revision also removes the earlier baseline-collection helper scripts
(which the previous description itself flagged for removal), so the PR now
contains the skill and a lean, reusable eval harness.

Changes

Skill (.agents/skills/cudaq-academic-vqe-qaoa/)

• Install, VQE, and QAOA references, plus new ADAPT-VQE and GQE references.
• A problem-recognition routing table in SKILL.md that maps plain-domain
  problems (e.g. "split deliveries to cut fuel", "rank molecules by stability")
  to the right quantum problem class and reference, rather than assuming
  everything is MaxCut.

Eval harness (.agents/evals/academic-vqe-qaoa/)

• prompts.json — 8 prompts: P1–P6 technical footguns (cost benefit),
  P7–P8 domain-translation prompts that never name a quantum algorithm
  (recognition benefit).
• assertions.json + evaluate_metrics.py — deterministic substring scorer
  with token summing.
• measure_tokens.py — real recorder: invokes an installed agent CLI
  (Claude Code / Codex / Cursor) headlessly and parses the actual token
  usage and cost from its structured output; manual entry only as a tagged
  fallback (tokens_source is meter or manual, never conflated).
• context_size.py — offline, deterministic context sizing via tiktoken
  (no API key or agent CLI needed).
• tests/ — unit tests for the scorer and the CLI parsers.
• README.md — reproducible demo steps, a per-runtime skill on/off toggle,
  and the measured results below.

Removed (superseded process artifacts): compare_runs.py,
record_responses.py, BASELINE_COLLECTION.md, references/metrics.md.

Results (measured)

Context and response tokens are real tiktoken (cl100k_base) counts; files,
tool calls, and recognition come from live agent runs in both conditions.
Ratio is no-skill ÷ with-skill (>1 = skill is cheaper).

Prompt	Type	Maps to	Ctx in: skill / no	Resp out: skill / no	Total: skill / no	Ratio
P1	technical	QAOA	1,676 / 3,797	844 / 1,248	2,520 / 5,045	2.00×
P2	technical	QAOA	1,648 / 3,769	727 / 1,084	2,375 / 4,853	2.04×
P3	technical	QAOA	1,647 / 3,768	382 / 901	2,029 / 4,669	2.30×
P4	technical	ADAPT	1,779 / 1,302	914 / 1,844	2,693 / 3,146	1.17×
P5	technical	GQE	2,035 / 3,517	931 / 1,923	2,966 / 5,440	1.83×
P6	technical	install	1,797 / 815	441 / 702	2,238 / 1,517	0.68×
P7	domain	MaxCut/QAOA	1,639 / 3,760	892 / 1,381	2,531 / 5,141	2.03×
P8	domain	VQE	1,485 / 1,305	1,222 / 2,162	2,707 / 3,467	1.28×
All 8	—	—	13,706 / 22,033	6,353 / 11,245	20,059 / 33,278	1.66×

Behavior (live runs; N=1 except P5/P7 which are 5-run means):

Prompt	Files: no→skill	Calls: no→skill	Recognition: no→skill
P3	3→2	6→3	quantum→quantum
P4	9→2	12→5	quantum→quantum
P7	4.2→5.0	10.8→10.2	classical→quantum (0/5→5/5)
P8	3→3	8→5	quantum→quantum (no gap)

Honest read: overall 1.66× fewer tokens with the skill; P6 (one-line
install fix) is the one row where the ~1k-token skill overhead doesn't pay
off. The recognition win is cleanest on P7 (no-skill reframes the fleet split
as classical OR-Tools and never surfaces QAOA); P8's gap is weak because an
agent exploring this repo finds VQE on its own. Full per-prompt detail,
methodology, and caveats are in the eval README.md.

How to run

# offline context sizing (no API key / CLI needed; needs `pip install tiktoken`)
python3 .agents/evals/academic-vqe-qaoa/context_size.py

# real token meter, if an agent CLI is installed
python3 .agents/evals/academic-vqe-qaoa/measure_tokens.py --config with_skill --prompt-ids P5

# score recorded runs
python3 .agents/evals/academic-vqe-qaoa/evaluate_metrics.py runs/with_skill.json runs/without_skill.json

Validation

• measure_tokens.py --selftest passes (parsers verified against sample
  Claude/Codex/Cursor payloads).
• pytest .agents/evals/academic-vqe-qaoa/tests/ — 12 passed.
• context_size.py and evaluate_metrics.py run cleanly.
• All 8 prompts were run live in both conditions to produce the tables above.

Notes

Generated runs/*.json are gitignored local artifacts and are not included.

Runtime / performance impact

N/A — workshop/eval tooling and a skill; no library or runtime code changes.

Self-review checklist

Before requesting review

• I reviewed my own full diff.
• PR is in Draft if it is not yet ready for review.
• Temporary / debugging changes have been removed.
• Local test logs reviewed; no unexplained warnings or errors.
• CI logs reviewed; no unexplained warnings or errors.
• Full CI has been run.

Scope and size

• PR is under ~1000 lines, or an exception is justified in the description.
• Refactoring-only changes are isolated in their own PR(s).
• No existing tests were disabled or modified just to make this PR pass.

Tests

• New functionality has new tests.
• Tests fail if the new functionality is broken, not just when it is missing.
• Negative tests added where exceptions are expected.
• Truth data added where simple checks are insufficient for algorithmic correctness.
• CI runtime impact considered; team notified if significant.

Documentation

• Public-facing APIs have Doxygen docs.
• User-visible behavior changes have public docs, or a follow-up is tracked.

Code style

• Naming follows the existing convention for the area being modified.

Dependencies

• No new third-party dependencies. Exception: context_size.py uses
  tiktoken (offline tokenizer) as an eval-only dependency. It is not
  imported by the skill or any library/runtime code; the import is local
  to the eval harness.