[TonsOfFun]

Refs #619. Targets 1.9.4: the framework and the dashboard engine move together, with no migration.

The activeagent CLI: an executable in the activeagent gem (exe/activeagent, stdlib OptionParser + Net::HTTP, no new runtime dependencies). It loads neither Rails nor the framework. HTTP, auth and error handling follow ActiveAgent::Evals::Publisher: bounded timeouts, no redirects, HTTPS except on loopback, and server messages sanitized with the keys filtered out.

Merged with main after 1.9.3: evals report reads the report export that shipped in #621, and the three commands that exited 5 for lack of an API-key route (catalogs export, catalogs import --preview, agents versions) now have one, as new tools on the MCP facade. Against a 1.9.4 dashboard with its dashboard tools on, nothing exits 5; a 1.9.3 dashboard answers those three and traces export with exit 5.

Two keys

• the MCP facade (POST <mount>/mcp) takes a dashboard API key (ApiKey, aa_...): every read and run
• the ingest endpoints the framework's own reporters post to take the ingest key (ActionAgent.ingest_api_key, or the tenant's telemetry_api_key): evals import / evals report (<mount>/api/evaluation_reports) and traces import (<mount>/api/traces)

A profile stores api_key plus an optional ingest_key (defaults to api_key); --ingest-key and ACTIVEAGENT_INGEST_KEY set it. status checks both.

Commands

Command	Endpoint	Key
login [--url --key --ingest-key --profile --no-verify], logout, status	MCP initialize; Publisher#verify! on the collector	both
evals list [--agent --limit]	MCP evaluations_list	API key
evals runs EVALUATION (id or name; shows each run's RUN_ID)	MCP evaluations_get	API key
evals run EVALUATION [--models --keys --group --sandbox --wait --min-pass-rate --timeout --interval]	MCP evaluations_run, then evaluation_runs_get; exit 4 below the gate, prints the evals report command for the run	API key
evals report RUN --format json|html|md [-o FILE]	GET <mount>/api/evaluation_reports/:run_id	ingest key
evals import FILE [--run-id --source --agent --suite]	POST <mount>/api/evaluation_reports	ingest key
catalogs list	MCP catalogs_list	API key
catalogs import FILE [--name KEY] [--mode replace|merge]	MCP catalogs_import (now takes mode)	API key
catalogs import FILE --preview	MCP catalogs_preview (new)	API key
catalogs export KEY [--product KEY [--set KEY]] [--format yaml|json] [-o FILE]	MCP catalogs_export (new)	API key
agents list	MCP resources/list	API key
agents versions AGENT [--limit]	MCP agents_versions (new)	API key
traces list [--agent --since 6h --status --service --limit]	MCP traces_search	API key
traces get TRACE	MCP traces_get	API key
traces export TRACE [-o FILE]	MCP traces_export (new)	API key
traces import FILE	POST <mount>/api/traces	ingest key

Every command takes --json, --profile and --url. Exit codes: 0 ok, 1 failed, 2 usage, 3 not configured, 4 evaluation gate failed, 5 the dashboard does not offer what the command needs (an older engine, or ActionAgent.mcp_dashboard_tools = false). Profiles are stored in ~/.config/activeagent/credentials.yml ($XDG_CONFIG_HOME when set), written 0600 in a 0700 directory. ACTIVEAGENT_URL, ACTIVEAGENT_API_KEY, ACTIVEAGENT_INGEST_KEY and ACTIVEAGENT_PROFILE override the stored profile.

JSON import/export between installs

A run, a catalog and a trace each move between dashboards as one file:

• Runs: evals report RUN --format json writes the version-1 envelope evals import posts, so a run copied to another install and posted back where it came from is an identical retry (200). --format html is the standalone report page and --format md its GitHub-flavored Markdown (... --format md | gh pr comment N --body-file -). A run id the dashboard holds no complete run under exits 1 (the collector's own 404); a dashboard with no export route exits 5 (Rails' 404).
• Scenario catalogs: catalogs export KEY writes the canonical document from the records (YAML, or --format json; a .json output file implies it), whole or one product or set, which catalogs import reads back (YAML reads JSON). --preview shows what the import would do (catalog created/updated/kept, counts of products, sets and scenarios created/updated/removed/kept, each change by path) and writes nothing; --mode merge keeps what the document leaves out.
• Traces: traces export TRACE writes the trace whole as the { traces: [...], sdk } payload trace ingest takes (spans untruncated, the owner's credentials masked), and traces import FILE posts it with the ingest key; an install skips a trace_id it already holds, so importing twice stores nothing more.

Engine changes (actionagent)

• Four dashboard tools on the MCP facade: catalogs_export, catalogs_preview, agents_versions and traces_export; catalogs_import takes mode as POST /api/scenario_catalogs does. Listed in the facade's initialize instructions and the docs table.
• EvaluationSerializer.run_summary carries run_id (EvaluationReportExport.run_id_for: a published run's own, a dashboard run's dashboard-run-<id>-<created_at>), so GET /api/evaluations, evaluations_get, evaluations_run and evaluation_runs_get all name the handle the report export takes. Nil on an install whose runs predate migration 010.
• AgentVersionSerializer (extracted from AgentsController#version_json, same JSON) and ScenarioCatalog#export_catalog (the ActiveAgent::Evals::Catalog that #export_yaml now wraps).

Tests

• test/cli/cli_test.rb: 39 standalone WebMock tests (no Rails loaded, as the executable runs): profiles and permissions, environment precedence, keys filtered from errors, every command, the gate, the report formats and their 404/422 cases, catalog preview/export, agent versions, trace get/export/import.
• actionagent/test/cli_contract_test.rb: 10 contract tests that hand the CLI's requests to the dummy app (WebMock to_rack): status, import → list → runs → report (JSON round trip posted back as a duplicate, Markdown, HTML), catalog import → export (whole and one set, YAML and JSON) → preview → merge, agent versions with diffs, trace export → import → get → list.
• actionagent/test/mcp_dashboard_tools_test.rb: the four new tools (export re-ingests through POST /api/traces, tenant scoping, secret masking, narrowing, import mode, every refusal).
• Full bin/test on the merged tree under CI's placeholder keys: 4150 runs, 0 failures, 0 errors, 25 skips. bin/rubocop on every changed file: no offenses. gem build packages exe/activeagent and lib/active_agent/cli/*. vitepress build docs passes.

Docs

• docs/framework/cli.md (sidebar: Framework → Command Line) is the page, with the dashboard version each command needs; a Command Line card on the docs index.
• AGENTS.md: the CLI and the evals library in the quick reference; a section on the command line as the gem's third surface (what it may load, which endpoint each command answers through, the rule from Add an activeagent CLI for core dashboard features (evals, catalogs, reports) #619 that a dashboard feature ships with its CLI command, API-key endpoint and UI together, how a command and a facade tool are added, where the tests live); the facade's dashboard tools and the run_id every run summary carries; the CLI test files and the placeholder provider keys a full local bin/test needs.
• The gem and engine READMEs, and the dashboard (downloads, run fields, the facade's tools table, the catalogs section), evaluations (catalogs, report URLs, run fields), self-hosted observability (runs and traces) and telemetry (trace ingest) pages point at the CLI commands beside the rake tasks, downloads and routes they already describe; docs/evals/publication.md gets the evals report/evals import round trip.
• CHANGELOG under Unreleased, with the 1.9.4 lead and the upgrading note (no migration; which dashboard version each command needs), for the release commit to date.

Left as is, on purpose: the collector and trace ingest take the ingest key rather than a dashboard ApiKey, so one key does not cover every command; the profile's ingest_key covers the difference.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RSuPEbVwjxwJub8eAmGM9N