From ada7f4510a5e02dbe15bbe38acac1420aa88e489 Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Wed, 21 Oct 2026 15:08:57 -0700 Subject: [PATCH 1/4] Add the activeagent command line for evaluations, catalogs, agents and traces An `activeagent` executable in the gem (stdlib OptionParser and Net::HTTP, no new dependencies) that works with a dashboard over HTTP: login/logout with 0600 profiles and env overrides for CI, status, evals list/runs/run/report/import, catalogs list/import, agents list and traces list, with --json output and distinct exit codes. `evals run --min-pass-rate` waits for the run and exits 4 below the threshold. Reads and runs go through the MCP facade's dashboard tools with a dashboard API key; import and report use the report collector with the ingest key, which the dashboard authenticates separately. `evals report --format json|html|md` targets the export endpoint the report export branch adds. Commands whose routes are session-only today (catalog export and import preview, agent versions) exit 5 and name the route they need. Refs #619 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01KhpgqyUrmSDK8cLDSheH3W --- CHANGELOG.md | 16 + actionagent/test/cli_contract_test.rb | 135 +++++++ activeagent.gemspec | 5 +- docs/.vitepress/config.mts | 1 + docs/framework/cli.md | 183 ++++++++++ exe/activeagent | 8 + lib/active_agent/cli.rb | 274 ++++++++++++++ lib/active_agent/cli/client.rb | 195 ++++++++++ lib/active_agent/cli/credentials.rb | 79 ++++ lib/active_agent/cli/eval_commands.rb | 275 ++++++++++++++ lib/active_agent/cli/resource_commands.rb | 137 +++++++ lib/active_agent/cli/session_commands.rb | 106 ++++++ test/cli/cli_test.rb | 422 ++++++++++++++++++++++ 13 files changed, 1835 insertions(+), 1 deletion(-) create mode 100644 actionagent/test/cli_contract_test.rb create mode 100644 docs/framework/cli.md create mode 100755 exe/activeagent create mode 100644 lib/active_agent/cli.rb create mode 100644 lib/active_agent/cli/client.rb create mode 100644 lib/active_agent/cli/credentials.rb create mode 100644 lib/active_agent/cli/eval_commands.rb create mode 100644 lib/active_agent/cli/resource_commands.rb create mode 100644 lib/active_agent/cli/session_commands.rb create mode 100644 test/cli/cli_test.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 1157e4447..355c26524 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,22 @@ engine: the file chooser relay and the per-turn browser tokens are in the ### Added +- **`activeagent` command line** (`activeagent`, refs #619). The gem ships + an `activeagent` executable (stdlib `OptionParser` and `Net::HTTP`, no new + dependencies) that works with a dashboard over HTTP: `login`/`logout` + (named profiles in `~/.config/activeagent/credentials.yml`, `0600`; + `ACTIVEAGENT_URL`, `ACTIVEAGENT_API_KEY` and `ACTIVEAGENT_INGEST_KEY` + override them for CI), `status`, `evals list|runs|run|report|import`, + `catalogs list|import`, `agents list` and `traces list`, with `--json` + output and distinct exit codes. `evals run --min-pass-rate 0.8` waits for + the run and exits 4 below the threshold, for CI gating. Reads and runs go + through the MCP facade's dashboard tools with a dashboard API key; `evals + import` and `evals report` use the report collector with the ingest key. + `evals report --format json|html|md` needs the report export endpoint + (`GET /api/evaluation_reports/:run_id`). `catalogs export`, + `catalogs import --preview` and `agents versions` exit 5 until the + dashboard offers API-key routes for them. See `docs/framework/cli.md`. + - **Upload files to a page through the sandbox browser's live view** (`actionagent`, `@activeagents/browser-sidecar`). A file chooser the page opens while someone drives is answered from the driver's own machine diff --git a/actionagent/test/cli_contract_test.rb b/actionagent/test/cli_contract_test.rb new file mode 100644 index 000000000..03378317b --- /dev/null +++ b/actionagent/test/cli_contract_test.rb @@ -0,0 +1,135 @@ +# frozen_string_literal: true + +require "test_helper" +require "active_agent/cli" +require "stringio" +require "tmpdir" + +# The `activeagent` CLI against the engine itself, with WebMock handing each +# request to the dummy app rather than the network: the commands read and +# write what the MCP facade and the report collector actually serve, so a +# change to either that breaks the CLI fails here. +class CliContractTest < ActiveSupport::TestCase + URL = "http://localhost/activeagents" + INGEST_KEY = "install-key-for-cli" + + setup do + ActionAgent::Agent.delete_all + ActionAgent::ApiKey.delete_all + ActionAgent.ingest_api_key = INGEST_KEY + @key = ActionAgent::ApiKey.create!(name: "CLI") + @agent = ActionAgent::Agent.create!(name: "Support", slug: "support", provider: "mock", model: "mock-model", + instructions: "Answer from data.", status: :active) + @dir = Dir.mktmpdir("activeagent-cli-contract") + stub_request(:any, %r{\A#{Regexp.escape(URL)}/}).to_rack(Rails.application) + end + + teardown do + ActionAgent.ingest_api_key = nil + FileUtils.rm_rf(@dir) + end + + def cli(*argv) + out = StringIO.new + err = StringIO.new + env = { "ACTIVEAGENT_URL" => URL, "ACTIVEAGENT_API_KEY" => @key.token, "ACTIVEAGENT_INGEST_KEY" => INGEST_KEY } + code = ActiveAgent::CLI.start(argv, out: out, err: err, env: env, credentials_path: File.join(@dir, "credentials.yml")) + [ code, out.string, err.string ] + end + + def json(*argv) + code, out, err = cli(*argv, "--json") + assert_equal 0, code, err + JSON.parse(out) + end + + # A one-scenario, one-model report as ActiveAgent::Evals::Report writes it. + def report(run_id) + evals = ActiveAgent::Evals + spec = evals::ModelSpec.new(label: "mock-model", provider: "mock", model: "mock-model") + scenario = evals::Scenario.from_hash({ "key" => "status_1", "group" => "status", "prompt" => "Where is order 1?" }) + result = evals::Result.new( + scenario: scenario, spec: spec, scores: { "response_present" => 1.0 }, score: 1.0, status: "passed", + replay: evals::Replay.new(answer: "Shipped", tool_calls: [], duration_ms: 10, input_tokens: 5, output_tokens: 2, + metadata: { "run_id" => run_id }) + ) + evals::Report.new(results: [ result ], models: [ spec ], metadata: { "run_id" => run_id, "suite" => "orders" }) + end + + test "status accepts the dashboard key on the facade and the ingest key on the collector" do + status = json("status") + assert status["ok"], status.inspect + + ActionAgent.ingest_api_key = "another-key" + code, out, = cli("status") + assert_equal ActiveAgent::CLI::EXIT_FAILURE, code + assert_match(/Collector: FAILED: .*HTTP 401/, out) + end + + test "evals import stores a report that evals list and evals runs then read" do + path = File.join(@dir, "report.json") + File.write(path, JSON.generate( + "version" => 1, "run_id" => "cli-run-1", "source" => "support-app", "agent_name" => "Support", "suite" => "orders", + "report" => report("cli-run-1").to_h + )) + + code, out, err = cli("evals", "import", path) + assert_equal 0, code, err + assert_match(/Imported run cli-run-1/, out) + + evaluations = json("evals", "list")["evaluations"] + assert_equal 1, evaluations.size, evaluations.inspect + evaluation = evaluations.first + + runs = json("evals", "runs", evaluation["name"])["runs"] + assert_equal 1, runs.size + assert_equal "complete", runs.first["status"] + end + + test "catalogs import and list round-trip a YAML catalog" do + path = File.join(@dir, "shop.yml") + File.write(path, <<~YAML) + catalog: shop + products: + - key: support + agent: support + sets: + - key: orders + scenarios: + - key: status_1 + prompt: Where is order 1? + YAML + + code, out, err = cli("catalogs", "import", path) + assert_equal 0, code, err + assert_includes out, "Imported catalog shop" + + catalogs = json("catalogs", "list")["catalogs"] + assert_equal [ "shop" ], catalogs.map { |catalog| catalog["key"] } + assert_equal 1, catalogs.first["scenario_count"] + end + + test "evals run starts a scenario suite's run through the facade" do + evaluation = @agent.evaluations.new(name: "Support lookups", judge_kind: "rules", criteria: []) + evaluation.scenarios.build(key: "order_lookup", prompt: "Where is order 1?", group: "orders") + evaluation.save! + + started = json("evals", "run", "Support lookups") + + assert_equal evaluation.id, started.dig("evaluation", "id") + assert_equal evaluation.evaluation_runs.sole.id, started.dig("run", "id") + assert started["background"] + end + + test "agents list and traces list read the key's agents and telemetry" do + assert_equal [ "support" ], json("agents", "list")["agents"].map { |agent| agent["slug"] } + assert_equal 60, json("traces", "list", "--since", "1h")["window_minutes"] + end + + test "a refused dashboard key fails with the facade's explanation" do + @key.destroy! + code, _, err = cli("agents", "list") + assert_equal ActiveAgent::CLI::EXIT_FAILURE, code + assert_includes err, "refused the API key" + end +end diff --git a/activeagent.gemspec b/activeagent.gemspec index 86b941c0b..83835b3a1 100644 --- a/activeagent.gemspec +++ b/activeagent.gemspec @@ -9,8 +9,11 @@ Gem::Specification.new do |spec| spec.email = "jusbowen@gmail.com" # The dashboard lives in the sibling `actionagent` gem (actionagent/), so # nothing under it ships here. - spec.files = Dir["CHANGELOG.md", "README.md", "LICENSE", "lib/**/*"] + spec.files = Dir["CHANGELOG.md", "README.md", "LICENSE", "lib/**/*", "exe/*"] spec.require_paths = "lib" + # The `activeagent` command line (lib/active_agent/cli.rb). + spec.bindir = "exe" + spec.executables = [ "activeagent" ] spec.homepage = "https://activeagents.ai" spec.license = "MIT" # activeagents-telemetry, a hard dependency, requires Ruby >= 3.2 in every diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 585cba503..ad3bcea30 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -105,6 +105,7 @@ export default defineConfig({ { text: 'Browser Sessions', link: '/framework/browser-sessions' }, { text: 'Evaluations', link: '/framework/evaluations' }, { text: 'Self-Hosted Dashboard', link: '/framework/self-hosted-observability' }, + { text: 'Command Line', link: '/framework/cli' }, { text: 'Dashboard for RubyLLM Apps', link: '/framework/ruby_llm_dashboard' }, { text: 'Retries', link: '/framework/retries' }, { text: 'Asking the User', link: '/framework/input_requests' }, diff --git a/docs/framework/cli.md b/docs/framework/cli.md new file mode 100644 index 000000000..aa7474640 --- /dev/null +++ b/docs/framework/cli.md @@ -0,0 +1,183 @@ +# Command Line (`activeagent`) + +The `activeagent` gem ships an `activeagent` executable that works with a +dashboard (a Rails app mounting the `actionagent` engine) over HTTP: list and +run evaluations, gate CI on a pass rate, move evaluation reports between +installs, import scenario catalogs, and read agents and traces. It needs only +the gem — not the host app's checkout or database — and loads neither Rails +nor the framework. + +```bash +gem install activeagent # or: bundle exec activeagent ... +activeagent login --url https://ops.example.com/activeagents --key aa_... +activeagent status +activeagent evals list +``` + +## Keys and profiles + +The dashboard authenticates two kinds of key, and the CLI sends each where it +belongs: + +| Key | Where it comes from | Used by | +|---|---|---| +| Dashboard API key (`--key`) | The dashboard's Settings → API Keys (`aa_...`) | Everything that reads or runs: the MCP facade at `/mcp` | +| Ingest key (`--ingest-key`) | `ActionAgent.ingest_api_key`, or the tenant's telemetry key on a multi-tenant install | `evals import` and `evals report`: the collector at `/api/evaluation_reports` | + +The ingest key defaults to the API key, so set it only when the two differ. +A single-tenant install with no `ingest_api_key` accepts any ingest key. + +`login` checks the API key against the dashboard (skip with `--no-verify`) +and stores the profile in `~/.config/activeagent/credentials.yml` +(`$XDG_CONFIG_HOME` when set), created `0600` in a `0700` directory. Without +`--key` it reads the key from a prompt with echo off, or from standard input, +so the key need not reach shell history: + +```bash +activeagent login --url https://ops.example.com/activeagents --profile staging +activeagent evals list --profile staging +activeagent logout --profile staging +``` + +The URL is the engine's mount, including its path. HTTPS is required except +on `localhost`. + +In CI, set the environment instead of a profile. It overrides any stored +profile: + +| Variable | Meaning | +|---|---| +| `ACTIVEAGENT_URL` | The dashboard mount | +| `ACTIVEAGENT_API_KEY` | The dashboard API key | +| `ACTIVEAGENT_INGEST_KEY` | The ingest key, when it differs | +| `ACTIVEAGENT_PROFILE` | The stored profile to use (default `default`) | + +`activeagent status` checks both keys: the API key against the MCP facade and +the ingest key against the collector (`Publisher#verify!`, which stores +nothing). + +## Output and exit codes + +Commands print tables. `--json` prints the dashboard's JSON instead, for +`jq` and scripts. Options can go before or after the command. + +| Code | Meaning | +|---|---| +| 0 | Success | +| 1 | A request failed or the dashboard refused it | +| 2 | Bad command line | +| 3 | No URL or key configured | +| 4 | An evaluation run failed, or passed below `--min-pass-rate` | +| 5 | The dashboard has no API-key endpoint for the command yet | + +## Evaluations + +```bash +activeagent evals list [--agent SLUG] [--limit N] +activeagent evals runs EVALUATION +activeagent evals run EVALUATION [--models a,b] [--keys k1,k2] [--group G] [--sandbox ID] + [--wait] [--min-pass-rate 0.8] [--timeout 1800] [--interval 5] +activeagent evals report RUN [--format json|html|md] [-o FILE] +activeagent evals import FILE [--run-id ID --source NAME --agent NAME --suite NAME] +``` + +`EVALUATION` is an evaluation's id or its exact name. + +### Gating CI on a pass rate + +`evals run --min-pass-rate` starts the run, waits for it (polling every +`--interval` seconds, up to `--timeout`), prints the failed results with +their traces, and exits 4 when the run failed or passed less than the +threshold. The rate is `0.8` or `80%`. + +```yaml +# .github/workflows/evals.yml +- run: bundle exec activeagent evals run "Support orders" --models gpt-5-mini --min-pass-rate 0.9 + env: + ACTIVEAGENT_URL: https://ops.example.com/activeagents + ACTIVEAGENT_API_KEY: ${{ secrets.ACTIVEAGENT_API_KEY }} +``` + +A scenario suite's replays run the agent on the dashboard, so the run needs +execution enabled there and counts against its execution quota. + +### Reports + +`evals import` posts a saved report to the collector with +`ActiveAgent::Evals::Publisher`. The file is a version-1 envelope, or a bare +`Report#to_h` hash with the identities given as options (`run_id` defaults to +the report's `metadata.run_id`). Importing the same report again is a no-op; +a different report under the same `run_id` is refused. + +`evals report` reads one back from `GET /api/evaluation_reports/:run_id`: + +- `--format json` — the version-1 envelope, so `evals report` on one install + then `evals import` on another copies a run between them. +- `--format html` — the standalone HTML report. +- `--format md` — GitHub-flavored Markdown, for a pull request comment. + +Without `--format`, a terminal gets Markdown and anything else (a pipe, or +`-o FILE`) gets JSON, so scripts always see JSON unless they ask otherwise. + +```bash +activeagent evals report run-2026-10-21 --format json -o report.json +ACTIVEAGENT_PROFILE=production activeagent evals import report.json + +activeagent evals report run-2026-10-21 --format md | gh pr comment 123 --body-file - +``` + +The report endpoint arrives with the dashboard's report export; on a +dashboard that predates it, `evals report` exits 5. + +## Scenario catalogs + +```bash +activeagent catalogs list +activeagent catalogs import FILE [--name KEY] +``` + +`catalogs import` takes the YAML document (`catalog:`, `products:` with +`sets:` of `scenarios:`). Products, sets and scenarios are matched by key, so +importing the same file twice changes nothing. Importing needs the +`replace_scenarios` permission for the key's user. + +## Agents and traces + +```bash +activeagent agents list +activeagent traces list [--agent NAME] [--since 24h] [--status error|ok] [--service NAME] [--limit N] +``` + +`--since` takes minutes or a duration such as `30m`, `6h` or `7d` (at most +the dashboard's trace window). + +## Not yet available + +These commands exist but exit 5, because their dashboard routes only accept +a browser session today. Each needs an API-key-authenticated endpoint +([#619](https://github.com/activeagents/activeagent/issues/619)): + +| Command | Needs | +|---|---| +| `catalogs export KEY` | `GET /api/scenario_catalogs/:id/export` | +| `catalogs import FILE --preview` | `POST /api/scenario_catalogs/preview` | +| `agents versions AGENT` | `GET /api/agents/:id/versions` | + +## Where each command goes + +| Command | Endpoint | Key | +|---|---|---| +| `login`, `status` | `POST /mcp` `initialize` | API key | +| `status` | `POST /api/evaluation_reports` (empty envelope) | Ingest key | +| `evals list` | MCP `evaluations_list` | API key | +| `evals runs` | MCP `evaluations_get` | API key | +| `evals run` | MCP `evaluations_run`, then `evaluation_runs_get` | API key | +| `evals import` | `POST /api/evaluation_reports` | Ingest key | +| `evals report` | `GET /api/evaluation_reports/:run_id` | Ingest key | +| `catalogs list` / `import` | MCP `catalogs_list` / `catalogs_import` | API key | +| `agents list` | MCP `resources/list` | API key | +| `traces list` | MCP `traces_search` | API key | + +The MCP commands use the facade's dashboard tools. An install that sets +`ActionAgent.mcp_dashboard_tools = false` turns them off, and those commands +then exit 5. diff --git a/exe/activeagent b/exe/activeagent new file mode 100755 index 000000000..34d138ac9 --- /dev/null +++ b/exe/activeagent @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# The activeagent CLI. Loads only lib/active_agent/cli, not Rails or the +# framework. See docs/framework/cli.md. +require_relative "../lib/active_agent/cli" + +exit ActiveAgent::CLI.start(ARGV) diff --git a/lib/active_agent/cli.rb b/lib/active_agent/cli.rb new file mode 100644 index 000000000..672f104ed --- /dev/null +++ b/lib/active_agent/cli.rb @@ -0,0 +1,274 @@ +# frozen_string_literal: true + +require "json" +require "optparse" +require_relative "version" + +module ActiveAgent + # The `activeagent` executable: a dashboard's evaluations, scenario + # catalogs, agents and traces from a terminal or a CI job, over HTTP with a + # dashboard API key. Plain Ruby (OptionParser and Net::HTTP); it loads + # neither Rails nor the framework, so it starts fast and runs anywhere the + # gem is installed. + # + # activeagent login --url https://ops.example.com/activeagents --key aa_... + # activeagent evals run "Support orders" --wait --min-pass-rate 0.8 + # + # Every command answers through an API-key-authenticated endpoint: the MCP + # facade's dashboard tools (/mcp) or the report collector + # (/api/evaluation_reports). A command whose dashboard route only + # takes a browser session today says so and exits EXIT_UNAVAILABLE rather + # than scraping the UI's API. + # + # CLI.start returns the exit code rather than exiting, so it can be called + # from Ruby and tested in process. + class CLI + EXIT_SUCCESS = 0 + # A request failed or the dashboard refused it. + EXIT_FAILURE = 1 + # The command line was wrong. + EXIT_USAGE = 2 + # No dashboard URL or key to use. + EXIT_CONFIG = 3 + # An evaluation run failed, or passed fewer scenarios than --min-pass-rate. + EXIT_GATE = 4 + # The dashboard has no API-key route for the command yet. + EXIT_UNAVAILABLE = 5 + + # Every failure a command reports: the message is printed and the exit + # code returned. +status+ is the HTTP status or JSON-RPC code behind it. + class Error < StandardError + attr_reader :exit_code, :status + + def initialize(message = nil, exit_code: EXIT_FAILURE, status: nil) + super(message) + @exit_code = exit_code + @status = status + end + end + + ISSUE_URL = "https://github.com/activeagents/activeagent/issues/619" + + def self.version + ActiveAgent::VERSION + end + + def self.start(argv = ARGV, **options) + new(**options).run(argv.dup) + end + + # +sleeper+ and +clock+ let tests drive `evals run --wait` without + # waiting. + def initialize(out: $stdout, err: $stderr, input: $stdin, env: ENV, credentials_path: nil, + sleeper: ->(seconds) { sleep(seconds) }, clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }) + @out = out + @err = err + @input = input + @env = env + @credentials = Credentials.new(credentials_path || Credentials.default_path(env)) + @sleeper = sleeper + @clock = clock + @options = {} + end + + def run(argv) + global_parser.order!(argv) + return print_version if @options[:version] + + command = argv.shift + return print_help(global_parser) if command.nil? || command == "help" || @options[:help] + + dispatch(command, argv) + rescue OptionParser::ParseError => e + @err.puts "activeagent: #{e.message}" + @err.puts "Run `activeagent help` for usage." + EXIT_USAGE + rescue Error => e + @err.puts "activeagent: #{e.message}" + e.exit_code + end + + COMMANDS = { + "login" => "Store a dashboard URL and API key as a profile", + "logout" => "Remove a stored profile", + "status" => "Check the URL and keys against the dashboard", + "evals" => "Evaluations: list, runs, run, report, import", + "catalogs" => "Scenario catalogs: list, import, export", + "agents" => "Agents: list, versions", + "traces" => "Telemetry traces: list", + "version" => "Print the version" + }.freeze + + private + + def dispatch(command, argv) + case command + when "login" then login(argv) + when "logout" then logout(argv) + when "status" then status(argv) + when "version" then print_version + when "evals" then subcommand("evals", argv, %w[list runs run report import]) + when "catalogs" then subcommand("catalogs", argv, %w[list import export]) + when "agents" then subcommand("agents", argv, %w[list versions]) + when "traces" then subcommand("traces", argv, %w[list]) + else + raise Error.new("unknown command '#{command}'. Commands: #{COMMANDS.keys.join(", ")}", exit_code: EXIT_USAGE) + end + end + + def subcommand(group, argv, names) + name = argv.shift + if name.nil? || %w[help -h --help].include?(name) + @out.puts "Usage: activeagent #{group} <#{names.join("|")}> [options]" + return name.nil? ? EXIT_USAGE : EXIT_SUCCESS + end + unless names.include?(name) + raise Error.new("unknown command '#{group} #{name}'. Try: #{names.map { |n| "#{group} #{n}" }.join(", ")}", exit_code: EXIT_USAGE) + end + + send(:"#{group}_#{name}", argv) + end + + def global_parser + @global_parser ||= OptionParser.new do |parser| + parser.banner = "Usage: activeagent [options]" + parser.separator "" + parser.separator "Commands:" + COMMANDS.each { |name, description| parser.separator format(" %-10s %s", name, description) } + parser.separator "" + parser.separator "Options:" + common_options(parser) + parser.on("-v", "--version", "Print the version") { @options[:version] = true } + parser.separator "" + parser.separator "Environment: ACTIVEAGENT_URL, ACTIVEAGENT_API_KEY, ACTIVEAGENT_INGEST_KEY and" + parser.separator "ACTIVEAGENT_PROFILE override the stored profile (for CI)." + parser.separator "Exit codes: 0 ok, 1 failed, 2 usage, 3 not configured, 4 evaluation gate failed," + parser.separator "5 not yet available on this dashboard." + end + end + + # Options every command takes, before or after the command name. + def common_options(parser) + parser.on("--profile NAME", "Use this stored profile (default: ACTIVEAGENT_PROFILE or 'default')") { |value| @options[:profile] = value } + parser.on("--url URL", "The dashboard mount, e.g. https://ops.example.com/activeagents") { |value| @options[:url] = value } + parser.on("--json", "Print JSON for scripts instead of tables") { @options[:json] = true } + parser.on("-h", "--help", "Show help") { @options[:help] = true } + end + + # Parses +argv+ with the command's own options plus the common ones and + # returns the remaining positional arguments, printing the command's help + # and returning nil for --help. + def parse(argv, banner) + parser = OptionParser.new do |p| + p.banner = "Usage: activeagent #{banner}" + yield p if block_given? + common_options(p) + end + parser.parse!(argv) + if @options[:help] + @out.puts parser + return nil + end + argv + end + + def print_help(parser) + @out.puts parser + EXIT_SUCCESS + end + + def print_version + @out.puts(@options[:json] ? JSON.generate(version: self.class.version) : "activeagent #{self.class.version}") + EXIT_SUCCESS + end + + def profile_name + @options[:profile] || @env["ACTIVEAGENT_PROFILE"].to_s.then { |name| name.empty? ? "default" : name } + end + + # The profile's stored values with the environment and --url over them. + def profile + stored = @credentials[profile_name] + if stored.empty? && @options[:profile] && !@env["ACTIVEAGENT_URL"] + raise Error.new("No profile '#{profile_name}' in #{@credentials.path}; run `activeagent login --profile #{profile_name}`", + exit_code: EXIT_CONFIG) + end + + { + "url" => @options[:url] || env_value("ACTIVEAGENT_URL") || stored["url"], + "api_key" => env_value("ACTIVEAGENT_API_KEY") || stored["api_key"], + "ingest_key" => env_value("ACTIVEAGENT_INGEST_KEY") || stored["ingest_key"] + } + end + + def env_value(name) + value = @env[name].to_s.strip + value.empty? ? nil : value + end + + def client + @client ||= begin + values = profile + if values["url"].to_s.empty? + raise Error.new("No dashboard URL: run `activeagent login --url `, or set ACTIVEAGENT_URL", exit_code: EXIT_CONFIG) + end + Client.new(url: values["url"], api_key: values["api_key"], ingest_key: values["ingest_key"]) + end + end + + # Prints +data+ as JSON with --json, otherwise yields for the human form. + def emit(data) + if @options[:json] + @out.puts JSON.pretty_generate(data) + else + yield + end + EXIT_SUCCESS + end + + # Prints +rows+ as aligned columns. +columns+ maps each header to a + # proc reading a row's cell. + def table(rows, columns, empty: "Nothing to show.") + return @out.puts(empty) if rows.empty? + + cells = rows.map { |row| columns.values.map { |reader| cell(reader.call(row)) } } + widths = columns.keys.each_with_index.map { |header, i| ([ header ] + cells.map { |c| c[i] }).map(&:length).max } + line = ->(values) { values.each_with_index.map { |value, i| value.ljust(widths[i]) }.join(" ").rstrip } + @out.puts line.call(columns.keys) + cells.each { |c| @out.puts line.call(c) } + end + + def cell(value) + value.nil? ? "-" : value.to_s.gsub(/[[:space:]\p{C}]+/, " ") + end + + def unavailable(command, endpoint) + raise Error.new("`#{command}` is not yet available on this dashboard: it needs an API-key-authenticated " \ + "#{endpoint}, which only takes a browser session today. See #{ISSUE_URL}.", exit_code: EXIT_UNAVAILABLE) + end + + def require_argument(argv, name, usage) + value = argv.shift + raise Error.new("missing #{name}. Usage: activeagent #{usage}", exit_code: EXIT_USAGE) if value.to_s.empty? + raise Error.new("unexpected arguments: #{argv.join(" ")}", exit_code: EXIT_USAGE) unless argv.empty? + + value + end + + def positive_integer(value, name) + Integer(value, 10).tap { |n| raise ArgumentError unless n.positive? } + rescue ArgumentError, TypeError + raise Error.new("#{name} must be a positive integer", exit_code: EXIT_USAGE) + end + + def list_option(value) + value.split(",").map(&:strip).reject(&:empty?) + end + end +end + +require_relative "cli/credentials" +require_relative "cli/client" +require_relative "cli/session_commands" +require_relative "cli/eval_commands" +require_relative "cli/resource_commands" diff --git a/lib/active_agent/cli/client.rb b/lib/active_agent/cli/client.rb new file mode 100644 index 000000000..57a19fcc3 --- /dev/null +++ b/lib/active_agent/cli/client.rb @@ -0,0 +1,195 @@ +# frozen_string_literal: true + +require "json" +require "net/http" +require "openssl" +require "uri" +require "zlib" +require_relative "../evals/publisher" + +module ActiveAgent + class CLI + # HTTP to one dashboard mount, with the conventions of + # ActiveAgent::Evals::Publisher: blocking, bounded timeouts, no redirects + # followed with a bearer key, and errors that never quote a response body + # beyond its sanitized +error+ string, with the keys filtered out. + # + # The dashboard authenticates two kinds of key, and this client sends + # each where it belongs: + # + # - +api_key+, a dashboard API key (Settings -> API Keys, "aa_..."), for + # the MCP facade at /mcp, whose dashboard tools back most + # commands. + # - +ingest_key+, the install's ingest key (ActionAgent.ingest_api_key, + # or the tenant's telemetry key), for the collector endpoints under + # /api/evaluation_reports. Defaults to +api_key+. + class Client + DETAIL_LIMIT = 200 + JSONRPC_INVALID_PARAMS = -32602 + + attr_reader :url + + def initialize(url:, api_key: nil, ingest_key: nil, timeout: 30, open_timeout: 10) + @uri = self.class.parse_url(url) + @url = @uri.to_s + @api_key = api_key.to_s.strip + @ingest_key = ingest_key.to_s.strip.empty? ? @api_key : ingest_key.to_s.strip + @timeout = timeout + @open_timeout = open_timeout + @request_id = 0 + end + + # The dashboard mount as a URI, without a trailing slash. HTTPS is + # required except on loopback hosts, as for the publisher. + def self.parse_url(url) + uri = URI.parse(url.to_s.strip.sub(%r{/+\z}, "")) + unless uri.is_a?(URI::HTTP) && uri.host && !uri.userinfo && !uri.query && !uri.fragment + raise Error.new("The dashboard URL must be an HTTP(S) URL without credentials, query or fragment, " \ + "such as https://ops.example.com/activeagents", exit_code: EXIT_USAGE) + end + unless uri.scheme == "https" || %w[localhost 127.0.0.1 ::1].include?(uri.hostname) + raise Error.new("The dashboard URL requires HTTPS except on loopback hosts", exit_code: EXIT_USAGE) + end + uri + rescue URI::InvalidURIError + raise Error.new("The dashboard URL is not a valid URL", exit_code: EXIT_USAGE) + end + + def mcp_endpoint + "#{url}/mcp" + end + + def reports_endpoint + "#{url}/api/evaluation_reports" + end + + # A Publisher for the mount's collector, authenticated with the ingest key. + def publisher + require_key!(@ingest_key, "an ingest key") + ActiveAgent::Evals::Publisher.new(api_key: @ingest_key, endpoint: reports_endpoint, timeout: @timeout, open_timeout: @open_timeout) + rescue ArgumentError => e + raise Error.new(e.message, exit_code: EXIT_CONFIG) + end + + # Sends one JSON-RPC request to the MCP facade and returns its result. + def mcp(method, params = {}) + require_key!(@api_key, "a dashboard API key") + @request_id += 1 + body = JSON.generate(jsonrpc: "2.0", id: @request_id, method: method, params: params) + response = exchange(:post, URI(mcp_endpoint), @api_key, body: body) + + case response.code + when "200" then nil + when "401" then raise refused("the dashboard refused the API key; create one under Settings -> API Keys", response, @api_key, 401) + when "404" then raise refused("nothing at #{mcp_endpoint} answers MCP; check the dashboard URL (the engine's mount)", response, @api_key, 404) + else raise refused("the dashboard's MCP endpoint answered HTTP #{response.code}", response, @api_key, response.code.to_i) + end + + payload = parse_json(response.body) + if (error = payload["error"]) + raise Error.new("#{method} failed: #{sanitize(error["message"], @api_key) || "error #{error["code"]}"}", status: error["code"]) + end + payload["result"] + end + + # Calls one of the dashboard's MCP tools and returns its structured + # result. A tool's own refusal (isError) raises with its message; a + # dashboard tool the facade does not list means the install turned + # them off (ActionAgent.mcp_dashboard_tools = false) or predates them. + def tool(name, arguments = {}) + result = mcp("tools/call", name: name, arguments: arguments.compact) + structured = result.is_a?(Hash) ? result["structuredContent"] : nil + if result.is_a?(Hash) && result["isError"] + message = structured.is_a?(Hash) && structured["error"] || Array(result["content"]).filter_map { |part| part["text"] if part.is_a?(Hash) }.join(" ") + raise Error.new("#{name}: #{sanitize(message, @api_key) || "the tool refused the call"}") + end + structured.is_a?(Hash) ? structured : {} + rescue Error => e + raise unless e.status == JSONRPC_INVALID_PARAMS && e.message.include?("Unknown tool") + + raise Error.new("This dashboard does not offer the #{name} tool over MCP: it predates the dashboard tools, or " \ + "the install sets ActionAgent.mcp_dashboard_tools = false", exit_code: EXIT_UNAVAILABLE) + end + + # GETs +path+ under the mount with the ingest key and returns the + # response, raising on anything but 200. + def get(path, accept:, query: nil, not_found: nil) + require_key!(@ingest_key, "an ingest key") + uri = URI("#{url}#{path}") + uri.query = query if query + response = exchange(:get, uri, @ingest_key, accept: accept) + return response if response.code == "200" + + if response.code == "404" && not_found + raise refused(not_found, response, @ingest_key, 404, exit_code: EXIT_UNAVAILABLE) + end + if response.code == "401" + raise refused("the dashboard refused the ingest key; use the mount's ingest_api_key (--ingest-key)", response, @ingest_key, 401) + end + raise refused("GET #{path} answered HTTP #{response.code}", response, @ingest_key, response.code.to_i) + end + + private + + def require_key!(key, label) + return unless key.empty? + + raise Error.new("No #{label}: run `activeagent login`, or set ACTIVEAGENT_API_KEY", exit_code: EXIT_CONFIG) + end + + def exchange(verb, uri, key, body: nil, accept: "application/json") + http = Net::HTTP.new(uri.hostname, uri.port) + http.use_ssl = uri.scheme == "https" + http.open_timeout = @open_timeout + http.read_timeout = @timeout + http.write_timeout = @timeout + request = (verb == :post ? Net::HTTP::Post : Net::HTTP::Get).new(uri.request_uri) + request["Authorization"] = "Bearer #{key}" + request["Accept"] = accept + request["User-Agent"] = "activeagent-cli/#{ActiveAgent::CLI.version}" + if body + request["Content-Type"] = "application/json" + request.body = body + end + http.request(request) + rescue Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Zlib::Error => e + # These messages can quote the response's status line, headers or body. + raise Error.new("The dashboard returned a malformed response (#{e.class})"), cause: nil + rescue IOError, SocketError, SystemCallError, Timeout::Error, OpenSSL::SSL::SSLError => e + raise Error.new("Could not reach #{uri.scheme}://#{uri.host} (#{e.class}); check the URL and that the dashboard is up") + end + + def parse_json(body) + parsed = JSON.parse(body.to_s) + raise Error, "The dashboard returned JSON that is not an object" unless parsed.is_a?(Hash) + + parsed + rescue JSON::ParserError + # The parser's message quotes the body. + raise Error.new("The dashboard returned invalid JSON"), cause: nil + end + + def refused(message, response, key, status, exit_code: EXIT_FAILURE) + detail = begin + parsed = JSON.parse(response.body.to_s) + sanitize(parsed["error"], key) if parsed.is_a?(Hash) + rescue JSON::ParserError, EncodingError + nil + end + Error.new(detail ? "#{message} (#{detail})" : message, status: status, exit_code: exit_code) + end + + # Control characters and runs of whitespace become one space, the keys + # become [FILTERED], and the result is cut to DETAIL_LIMIT characters. + def sanitize(message, key) + return unless message.is_a?(String) + + detail = message.scrub.gsub(/[[:space:]\p{C}]+/, " ").strip + [ key, @api_key, @ingest_key ].uniq.reject(&:empty?).each { |secret| detail = detail.gsub(secret, "[FILTERED]") } + return if detail.empty? + + detail.length > DETAIL_LIMIT ? "#{detail[0, DETAIL_LIMIT - 1].rstrip}…" : detail + end + end + end +end diff --git a/lib/active_agent/cli/credentials.rb b/lib/active_agent/cli/credentials.rb new file mode 100644 index 000000000..ca79afe51 --- /dev/null +++ b/lib/active_agent/cli/credentials.rb @@ -0,0 +1,79 @@ +# frozen_string_literal: true + +require "fileutils" +require "yaml" + +module ActiveAgent + class CLI + # Named dashboard profiles in ~/.config/activeagent/credentials.yml + # ($XDG_CONFIG_HOME when set), written 0600 inside a 0700 directory: + # + # default: + # url: https://ops.example.com/activeagents + # api_key: aa_... + # ingest_key: ... # only when it differs from api_key + # + # A profile's values are what +login+ stored; the environment + # (ACTIVEAGENT_URL, ACTIVEAGENT_API_KEY, ACTIVEAGENT_INGEST_KEY) overrides + # them in CLI#profile, so CI never needs the file. + class Credentials + FIELDS = %w[url api_key ingest_key].freeze + + attr_reader :path + + def self.default_path(env = ENV) + base = env["XDG_CONFIG_HOME"].to_s.empty? ? File.join(Dir.home, ".config") : env["XDG_CONFIG_HOME"] + File.join(base, "activeagent", "credentials.yml") + end + + def initialize(path) + @path = path + end + + # Returns the profile's stored values, or an empty hash when the file or + # the profile does not exist. + def [](name) + profiles.fetch(name.to_s, {}) + end + + def names + profiles.keys + end + + def save(name, values) + data = profiles.merge(name.to_s => values.transform_keys(&:to_s).slice(*FIELDS).compact) + FileUtils.mkdir_p(File.dirname(path), mode: 0o700) + # Written beside the target and renamed over it, so a reader never + # sees half a file, and created 0600 so the key is never readable by + # others, even for a moment. + temp = "#{path}.#{Process.pid}.tmp" + File.open(temp, File::WRONLY | File::CREAT | File::TRUNC, 0o600) { |file| file.write(YAML.dump(data)) } + File.chmod(0o600, temp) + File.rename(temp, path) + ensure + FileUtils.rm_f(temp) if temp && File.exist?(temp) + end + + def delete(name) + data = profiles + return false unless data.delete(name.to_s) + + data.empty? ? File.delete(path) : File.open(path, "w", 0o600) { |file| file.write(YAML.dump(data)) } + true + end + + private + + def profiles + return {} unless File.exist?(path) + + data = YAML.safe_load_file(path) || {} + raise Error.new("#{path} is not a profile map; fix or remove it", exit_code: EXIT_CONFIG) unless data.is_a?(Hash) + + data.select { |_, values| values.is_a?(Hash) } + rescue Psych::Exception => e + raise Error.new("#{path} is not valid YAML (#{e.class}); fix or remove it", exit_code: EXIT_CONFIG) + end + end + end +end diff --git a/lib/active_agent/cli/eval_commands.rb b/lib/active_agent/cli/eval_commands.rb new file mode 100644 index 000000000..28099bb2f --- /dev/null +++ b/lib/active_agent/cli/eval_commands.rb @@ -0,0 +1,275 @@ +# frozen_string_literal: true + +require "uri" + +module ActiveAgent + class CLI + # `activeagent evals ...`. list, runs and run go through the MCP facade's + # evaluation tools; import posts a version-1 report to the collector with + # ActiveAgent::Evals::Publisher; report reads one back from the + # collector's export endpoint. + module EvalCommands + # How `evals report` asks GET /api/evaluation_reports/:run_id + # for each format. The JSON form is the version-1 envelope the + # collector takes, so `evals report --format json` then `evals import` + # copies a run between installs. + REPORT_FORMATS = { + "json" => { accept: "application/json", query: nil }, + "html" => { accept: "text/html", query: "format=html" }, + # GitHub-flavored Markdown, for a pull request comment: + # `activeagent evals report RUN --format md | gh pr comment 12 --body-file -` + "md" => { accept: "text/markdown", query: "format=md" } + }.freeze + + RUN_FINISHED = %w[complete failed].freeze + DEFAULT_WAIT_TIMEOUT = 30 * 60 + DEFAULT_POLL_INTERVAL = 5 + FAILURES_SHOWN = 20 + + private + + def evals_list(argv) + arguments = {} + argv = parse(argv, "evals list [--agent SLUG] [--limit N]") do |p| + p.on("--agent AGENT", "Only this agent's evaluations, by slug or id") { |v| arguments[:agent] = v } + p.on("--limit N", "At most N evaluations (max 50)") { |v| arguments[:limit] = positive_integer(v, "--limit") } + end + return EXIT_SUCCESS unless argv + + evaluations = client.tool("evaluations_list", arguments).fetch("evaluations", []) + emit(evaluations: evaluations) do + table(evaluations, { + "ID" => ->(e) { e["id"] }, + "NAME" => ->(e) { e["name"] }, + "AGENT" => ->(e) { e.dig("agent", "slug") }, + "SCENARIOS" => ->(e) { e["scenario_count"] }, + "RUNS" => ->(e) { e["run_count"] }, + "LATEST" => ->(e) { e.dig("latest_run", "status") }, + "PASSED" => ->(e) { passed(e["latest_run"]) } + }, empty: "No evaluations.") + end + end + + def evals_runs(argv) + argv = parse(argv, "evals runs EVALUATION") + return EXIT_SUCCESS unless argv + + evaluation_id = resolve_evaluation(require_argument(argv, "EVALUATION (an id or name)", "evals runs EVALUATION")) + payload = client.tool("evaluations_get", evaluation_id: evaluation_id) + runs = payload.fetch("runs", []) + emit(evaluation: payload["evaluation"], runs: runs) do + @out.puts "#{payload.dig("evaluation", "name")} (evaluation #{evaluation_id})" + table(runs, { + "RUN" => ->(r) { r["id"] }, + "#" => ->(r) { r["number"] }, + "STATUS" => ->(r) { r["status"] }, + "PASSED" => ->(r) { passed(r) }, + "SCORE" => ->(r) { r["average_score"] }, + "CREATED" => ->(r) { r["created_at"] } + }, empty: "No runs yet.") + end + end + + # Starts a run. With --wait (implied by --min-pass-rate), polls it to + # the end and exits EXIT_GATE when it failed or passed fewer scenarios + # than the threshold, so a CI step fails the build. + def evals_run(argv) + arguments = {} + wait = false + min_pass_rate = nil + timeout = DEFAULT_WAIT_TIMEOUT + interval = DEFAULT_POLL_INTERVAL + argv = parse(argv, "evals run EVALUATION [--models a,b] [--wait] [--min-pass-rate 0.8]") do |p| + p.on("--models LIST", "Candidate models to compare, comma-separated") { |v| arguments[:models] = list_option(v) } + p.on("--keys LIST", "Replay only these scenario keys, comma-separated") { |v| arguments[:keys] = list_option(v) } + p.on("--group GROUP", "Replay only this scenario group") { |v| arguments[:group] = v } + p.on("--sandbox ID", "Replay against this ready checkout sandbox") { |v| arguments[:sandbox_id] = v } + p.on("--wait", "Wait for the run to finish") { wait = true } + p.on("--min-pass-rate RATE", "Fail (exit #{EXIT_GATE}) below this pass rate: 0.8 or 80%; implies --wait") do |v| + min_pass_rate = pass_rate_option(v) + wait = true + end + p.on("--timeout SECONDS", "Give up waiting after this long (default #{DEFAULT_WAIT_TIMEOUT})") { |v| timeout = positive_integer(v, "--timeout") } + p.on("--interval SECONDS", "Poll this often (default #{DEFAULT_POLL_INTERVAL})") { |v| interval = positive_integer(v, "--interval") } + end + return EXIT_SUCCESS unless argv + + evaluation_id = resolve_evaluation(require_argument(argv, "EVALUATION (an id or name)", "evals run EVALUATION")) + started = client.tool("evaluations_run", arguments.merge(evaluation_id: evaluation_id)) + run = started.fetch("run", {}) + @err.puts "Started run #{run["id"]} of #{started.dig("evaluation", "name")} (evaluation #{evaluation_id})." unless @options[:json] + unless wait + return emit(started) { @out.puts "Status: #{run["status"]}. Follow it with `activeagent evals runs #{evaluation_id}`." } + end + + finished = wait_for_run(evaluation_id, run["id"], timeout: timeout, interval: interval) + gate(finished, min_pass_rate) + end + + # Without --format, a terminal gets Markdown to read and anything else + # (a pipe, a file, -o) gets the JSON envelope, so scripts see JSON. + def evals_report(argv) + format = nil + output = nil + argv = parse(argv, "evals report RUN [--format json|html|md] [-o FILE]") do |p| + p.on("--format FORMAT", REPORT_FORMATS.keys, + "json (a version-1 envelope), html or md; default md on a terminal, json otherwise") { |v| format = v } + p.on("-o", "--output FILE", "Write to FILE instead of standard output") { |v| output = v } + end + return EXIT_SUCCESS unless argv + + run_id = require_argument(argv, "RUN (the report's run_id)", "evals report RUN") + format ||= !output && @out.respond_to?(:tty?) && @out.tty? ? "md" : "json" + request = REPORT_FORMATS.fetch(format) + response = client.get("/api/evaluation_reports/#{URI.encode_www_form_component(run_id).gsub("+", "%20")}", + accept: request[:accept], query: request[:query], + not_found: "No report for run #{run_id}, or this dashboard has no report export " \ + "(GET /api/evaluation_reports/:run_id) yet") + body = response.body.to_s + body = JSON.pretty_generate(report_envelope(body)) if format == "json" + + if output + File.write(output, body.end_with?("\n") ? body : "#{body}\n") + @err.puts "Wrote #{format} report for run #{run_id} to #{output}." unless @options[:json] + else + @out.puts body + end + EXIT_SUCCESS + end + + # Posts a saved report to the collector. FILE is a version-1 envelope + # (what `evals report --format json` writes) or a bare Report#to_h + # hash, whose identities then come from the options. + def evals_import(argv) + identities = {} + argv = parse(argv, "evals import FILE [--run-id ID --source NAME --agent NAME --suite NAME]") do |p| + p.on("--run-id ID", "The report's run_id (kept across retries)") { |v| identities["run_id"] = v } + p.on("--source NAME", "The application that ran it") { |v| identities["source"] = v } + p.on("--agent NAME", "The agent's name") { |v| identities["agent_name"] = v } + p.on("--suite NAME", "The suite's name") { |v| identities["suite"] = v } + end + return EXIT_SUCCESS unless argv + + path = require_argument(argv, "FILE", "evals import FILE") + document = read_json_file(path) + envelope = document.key?("version") && document.key?("report") ? document : { "report" => document } + if envelope.key?("version") && envelope["version"] != 1 + raise Error.new("#{path} is a version #{envelope["version"].inspect} envelope; this CLI sends version 1", exit_code: EXIT_USAGE) + end + envelope = envelope.merge(identities) + envelope["run_id"] ||= envelope["report"].is_a?(Hash) ? envelope["report"].dig("metadata", "run_id") : nil + + receipt = client.publisher.call(report: envelope["report"], run_id: envelope["run_id"], source: envelope["source"], + agent_name: envelope["agent_name"], suite: envelope["suite"]) + emit(receipt) do + @out.puts "#{receipt["duplicate"] ? "Already stored" : "Imported"} run #{receipt["run_id"]} " \ + "(evaluation #{receipt["evaluation_id"]}, run #{receipt["id"]})." + end + rescue ArgumentError => e + raise Error.new("#{e.message}; give it with --run-id, --source, --agent or --suite", exit_code: EXIT_USAGE) + rescue ActiveAgent::Evals::Publisher::Error => e + raise Error.new(e.message, status: e.status) + end + + # An evaluation's id, from an id or its exact name among the first 50 + # evaluations the key can reach. + def resolve_evaluation(value) + return Integer(value, 10) if value.match?(/\A\d+\z/) + + matches = client.tool("evaluations_list", limit: 50).fetch("evaluations", []).select { |e| e["name"] == value } + case matches.size + when 1 then matches.first["id"] + when 0 then raise Error, "No evaluation named #{value.inspect}; `activeagent evals list` shows them" + else + raise Error.new("#{matches.size} evaluations are named #{value.inspect} (ids #{matches.map { |e| e["id"] }.join(", ")}); " \ + "pass the id", exit_code: EXIT_USAGE) + end + end + + def wait_for_run(evaluation_id, run_id, timeout:, interval:) + deadline = @clock.call + timeout + loop do + payload = client.tool("evaluation_runs_get", evaluation_id: evaluation_id, run_id: run_id, failed_only: true, limit: FAILURES_SHOWN) + return payload if RUN_FINISHED.include?(payload.dig("run", "status")) + + if @clock.call >= deadline + raise Error, "Run #{run_id} is still #{payload.dig("run", "status")} after #{timeout}s; " \ + "raise --timeout or follow it with `activeagent evals runs #{evaluation_id}`" + end + @sleeper.call(interval) + end + end + + def gate(payload, min_pass_rate) + run = payload.fetch("run", {}) + rate = pass_rate(run) + failed = run["status"] == "failed" + below = min_pass_rate && (rate.nil? || rate < min_pass_rate) + verdict = failed || below ? "fail" : "pass" + failures = payload.fetch("results", []) + + emit(payload.merge("pass_rate" => rate, "min_pass_rate" => min_pass_rate, "verdict" => verdict)) do + @out.puts "Run #{run["id"]} #{run["status"]}: #{passed(run) || "no results"}" \ + "#{rate ? format(" (%.1f%%)", rate * 100) : ""}#{min_pass_rate ? format(", threshold %.1f%%", min_pass_rate * 100) : ""}." + @out.puts "Error: #{run["error_message"]}" if run["error_message"] + unless failures.empty? + @out.puts "Failed results:" + table(failures, { + "SCENARIO" => ->(r) { r["scenario_key"] }, + "MODEL" => ->(r) { r["model"] }, + "STATUS" => ->(r) { r["status"] }, + "FAULT" => ->(r) { r["fault"] }, + "TRACE" => ->(r) { r["trace_id"] } + }) + end + @out.puts(verdict == "pass" ? "PASS" : "FAIL") + end + verdict == "pass" ? EXIT_SUCCESS : EXIT_GATE + end + + def pass_rate(run) + evaluated = run["samples_evaluated"].to_i + evaluated.positive? ? run["samples_passed"].to_f / evaluated : nil + end + + def passed(run) + return unless run.is_a?(Hash) && run["samples_evaluated"] + + "#{run["samples_passed"].to_i}/#{run["samples_evaluated"]}" + end + + def pass_rate_option(value) + rate = value.end_with?("%") ? Float(value.delete_suffix("%")) / 100 : Float(value) + raise ArgumentError unless rate.between?(0, 1) + + rate + rescue ArgumentError, TypeError + raise Error.new("--min-pass-rate must be a rate between 0 and 1, or a percentage such as 80%", exit_code: EXIT_USAGE) + end + + # The collector's export must be the envelope `evals import` sends back. + def report_envelope(body) + envelope = JSON.parse(body) + unless envelope.is_a?(Hash) && envelope["version"] == 1 && envelope["report"].is_a?(Hash) + raise Error, "The dashboard's report is not a version-1 envelope" + end + envelope + rescue JSON::ParserError + raise Error.new("The dashboard returned invalid JSON for the report"), cause: nil + end + + def read_json_file(path) + document = JSON.parse(File.read(path)) + raise Error.new("#{path} must hold a JSON object", exit_code: EXIT_USAGE) unless document.is_a?(Hash) + + document + rescue Errno::ENOENT, Errno::EACCES, Errno::EISDIR => e + raise Error.new("Cannot read #{path} (#{e.class})", exit_code: EXIT_USAGE) + rescue JSON::ParserError + raise Error.new("#{path} is not valid JSON", exit_code: EXIT_USAGE), cause: nil + end + end + + include EvalCommands + end +end diff --git a/lib/active_agent/cli/resource_commands.rb b/lib/active_agent/cli/resource_commands.rb new file mode 100644 index 000000000..c91fe1528 --- /dev/null +++ b/lib/active_agent/cli/resource_commands.rb @@ -0,0 +1,137 @@ +# frozen_string_literal: true + +module ActiveAgent + class CLI + # `activeagent catalogs|agents|traces ...`, through the MCP facade. The + # commands whose dashboard route only takes a browser session today + # (catalog export and import preview, agent versions) exit + # EXIT_UNAVAILABLE naming the route they need. + module ResourceCommands + DURATION = /\A(\d+)([smhd]?)\z/ + DURATION_MINUTES = { "" => 1, "m" => 1, "h" => 60, "d" => 24 * 60 }.freeze + + private + + def catalogs_list(argv) + argv = parse(argv, "catalogs list") + return EXIT_SUCCESS unless argv + + catalogs = client.tool("catalogs_list").fetch("catalogs", []) + emit(catalogs: catalogs) do + table(catalogs, { + "KEY" => ->(c) { c["key"] }, + "NAME" => ->(c) { c["name"] }, + "PRODUCTS" => ->(c) { c["product_count"] }, + "SCENARIOS" => ->(c) { c["scenario_count"] }, + "SOURCE" => ->(c) { c["source_path"] || c["source_kind"] }, + "UPDATED" => ->(c) { c["updated_at"] } + }, empty: "No scenario catalogs.") + end + end + + # Imports a catalog's YAML document. Products, sets and scenarios are + # matched by key, so importing the same file twice changes nothing. + def catalogs_import(argv) + name = nil + preview = false + argv = parse(argv, "catalogs import FILE [--name KEY] [--preview]") do |p| + p.on("--name KEY", "The catalog key when the document sets no `catalog:`") { |v| name = v } + p.on("--preview", "Show what the import would change, without applying it") { preview = true } + end + return EXIT_SUCCESS unless argv + + path = require_argument(argv, "FILE", "catalogs import FILE") + unavailable("catalogs import --preview", "POST /api/scenario_catalogs/preview") if preview + document = begin + File.read(path) + rescue Errno::ENOENT, Errno::EACCES, Errno::EISDIR => e + raise Error.new("Cannot read #{path} (#{e.class})", exit_code: EXIT_USAGE) + end + + payload = client.tool("catalogs_import", document: document, name: name) + emit(payload) do + catalog = payload.fetch("catalog", {}) + @out.puts "Imported catalog #{catalog["key"]} (#{catalog["product_count"]} products, #{catalog["scenario_count"]} scenarios)." + Array(payload["products"]).each do |product| + sets = Array(product["sets"]).map { |set| "#{set["key"]} (#{set["scenario_count"]})" } + @out.puts " #{product["key"]}#{" -> #{product["agent"]}" if product["agent"]}: #{sets.join(", ")}" + end + end + end + + def catalogs_export(argv) + argv = parse(argv, "catalogs export KEY [-o FILE]") { |p| p.on("-o", "--output FILE", "Write to FILE") } + return EXIT_SUCCESS unless argv + + require_argument(argv, "KEY", "catalogs export KEY") + unavailable("catalogs export", "GET /api/scenario_catalogs/:id/export") + end + + # The agents the key reaches, from the facade's agent:// resources. + def agents_list(argv) + argv = parse(argv, "agents list") + return EXIT_SUCCESS unless argv + + resources = Array(client.mcp("resources/list")&.fetch("resources", [])) + agents = resources.map do |resource| + { "slug" => resource["uri"].to_s.delete_prefix("agent://"), "name" => resource["name"], "description" => resource["description"] } + end + emit(agents: agents) do + table(agents, { "SLUG" => ->(a) { a["slug"] }, "NAME" => ->(a) { a["name"] }, "DESCRIPTION" => ->(a) { a["description"] } }, + empty: "No agents.") + end + end + + def agents_versions(argv) + argv = parse(argv, "agents versions AGENT") + return EXIT_SUCCESS unless argv + + require_argument(argv, "AGENT", "agents versions AGENT") + unavailable("agents versions", "GET /api/agents/:id/versions") + end + + def traces_list(argv) + arguments = {} + argv = parse(argv, "traces list [--agent NAME] [--since 24h] [--status error|ok] [--limit N]") do |p| + p.on("--agent AGENT", "An agent class name or a dashboard agent's slug") { |v| arguments[:agent] = v } + p.on("--since DURATION", "How far back: minutes, or 30m, 6h, 7d (default 24h)") { |v| arguments[:since_minutes] = since_minutes(v) } + p.on("--status STATUS", %w[error ok], "Only failed (error) or successful (ok) traces") { |v| arguments[:status] = v } + p.on("--service NAME", "Only this reporting service") { |v| arguments[:service] = v } + p.on("--limit N", "At most N traces (max 100)") { |v| arguments[:limit] = positive_integer(v, "--limit") } + end + return EXIT_SUCCESS unless argv + raise Error.new("unexpected arguments: #{argv.join(" ")}", exit_code: EXIT_USAGE) unless argv.empty? + + payload = client.tool("traces_search", arguments) + traces = payload.fetch("traces", []) + emit(payload) do + table(traces, { + "TRACE" => ->(t) { t["short_id"] }, + "TIME" => ->(t) { t["timestamp"] }, + "AGENT" => ->(t) { [ t["agent"], t["action"] ].compact.join("#") }, + "MODEL" => ->(t) { t["model"] }, + "STATUS" => ->(t) { t["status"] }, + "MS" => ->(t) { t["duration_ms"] }, + "TOKENS" => ->(t) { t.dig("tokens", "total") } + }, empty: "No traces in the last #{payload["window_minutes"]} minutes.") + @out.puts "(more traces match; raise --limit or narrow the filters)" if payload["truncated"] + end + end + + # Minutes in a duration such as 90, 30m, 6h or 7d; seconds round up + # to a whole minute. + def since_minutes(value) + match = DURATION.match(value.to_s.strip.downcase) + raise Error.new("--since must be minutes or a duration such as 30m, 6h or 7d", exit_code: EXIT_USAGE) unless match + + amount = match[1].to_i + minutes = match[2] == "s" ? (amount / 60.0).ceil : amount * DURATION_MINUTES.fetch(match[2]) + raise Error.new("--since must be at least a minute", exit_code: EXIT_USAGE) unless minutes.positive? + + minutes + end + end + + include ResourceCommands + end +end diff --git a/lib/active_agent/cli/session_commands.rb b/lib/active_agent/cli/session_commands.rb new file mode 100644 index 000000000..0462e87ec --- /dev/null +++ b/lib/active_agent/cli/session_commands.rb @@ -0,0 +1,106 @@ +# frozen_string_literal: true + +module ActiveAgent + class CLI + # login, logout and status: the profile a command runs as, and whether + # the dashboard accepts it. + module SessionCommands + MCP_PROTOCOL_VERSION = "2025-03-26" + + private + + # Checks the key against the dashboard before storing it, unless + # --no-verify. The key comes from --key, ACTIVEAGENT_API_KEY, or a + # prompt (echo off on a terminal), so it need not sit in shell history. + def login(argv) + values = {} + verify = true + argv = parse(argv, "login --url URL [--key KEY] [--ingest-key KEY] [--profile NAME]") do |p| + p.on("--key KEY", "A dashboard API key (Settings -> API Keys); prompted for when omitted") { |v| values["api_key"] = v } + p.on("--ingest-key KEY", "The mount's ingest key, for evals import/report, when it differs from --key") { |v| values["ingest_key"] = v } + p.on("--[no-]verify", "Check the key against the dashboard first (default: on)") { |v| verify = v } + end + return EXIT_SUCCESS unless argv + raise Error.new("unexpected arguments: #{argv.join(" ")}", exit_code: EXIT_USAGE) unless argv.empty? + + stored = @credentials[profile_name] + url = @options[:url] || env_value("ACTIVEAGENT_URL") || stored["url"] + raise Error.new("missing --url, the dashboard's mount (e.g. https://ops.example.com/activeagents)", exit_code: EXIT_USAGE) if url.to_s.empty? + + values["url"] = Client.parse_url(url).to_s + values["api_key"] ||= env_value("ACTIVEAGENT_API_KEY") || prompt_for_key + values["ingest_key"] ||= env_value("ACTIVEAGENT_INGEST_KEY") || stored["ingest_key"] + raise Error.new("an API key is required", exit_code: EXIT_USAGE) if values["api_key"].to_s.strip.empty? + values["api_key"] = values["api_key"].strip + + server = verify ? mcp_initialize(Client.new(url: values["url"], api_key: values["api_key"])) : nil + @credentials.save(profile_name, values) + + emit(profile: profile_name, url: values["url"], verified: verify, server: server, credentials: @credentials.path) do + @out.puts "Logged in to #{values["url"]} as profile '#{profile_name}'#{" (not verified)" unless verify}." + @out.puts "Saved to #{@credentials.path}" + end + end + + def logout(argv) + argv = parse(argv, "logout [--profile NAME]") + return EXIT_SUCCESS unless argv + + removed = @credentials.delete(profile_name) + emit(profile: profile_name, removed: removed) do + @out.puts(removed ? "Removed profile '#{profile_name}'." : "No profile '#{profile_name}' was stored.") + end + end + + # Two checks, one per key: the MCP facade's initialize with the API key, + # and Publisher#verify! against the report collector with the ingest + # key. Exits EXIT_FAILURE when either fails. + def status(argv) + argv = parse(argv, "status [--profile NAME]") + return EXIT_SUCCESS unless argv + + client + checks = { + "dashboard" => check { { "server" => mcp_initialize(client) } }, + "collector" => check { client.publisher.verify! && { "endpoint" => client.reports_endpoint } } + } + ok = checks.values.all? { |result| result["ok"] } + + emit(profile: profile_name, url: client.url, ok: ok, checks: checks) do + @out.puts "Profile: #{profile_name}" + @out.puts "Dashboard: #{client.url}" + @out.puts "API key: #{checks["dashboard"]["ok"] ? "accepted (#{client.mcp_endpoint})" : "FAILED: #{checks["dashboard"]["error"]}"}" + @out.puts "Collector: #{checks["collector"]["ok"] ? "accepted (#{client.reports_endpoint})" : "FAILED: #{checks["collector"]["error"]}"}" + end + ok ? EXIT_SUCCESS : EXIT_FAILURE + end + + def check + { "ok" => true }.merge(yield || {}) + rescue Error, ActiveAgent::Evals::Publisher::Error => e + { "ok" => false, "error" => e.message } + end + + # The facade's serverInfo, which an accepted key gets back. + def mcp_initialize(client) + result = client.mcp("initialize", protocolVersion: MCP_PROTOCOL_VERSION, capabilities: {}, + clientInfo: { name: "activeagent-cli", version: self.class.version }) + result.is_a?(Hash) ? result["serverInfo"] : nil + end + + def prompt_for_key + if @input.respond_to?(:tty?) && @input.tty? + require "io/console" + @err.print "Dashboard API key: " + key = @input.noecho(&:gets) + @err.puts + key + else + @input.gets + end + end + end + + include SessionCommands + end +end diff --git a/test/cli/cli_test.rb b/test/cli/cli_test.rb new file mode 100644 index 000000000..e098467be --- /dev/null +++ b/test/cli/cli_test.rb @@ -0,0 +1,422 @@ +# frozen_string_literal: true + +# The CLI loads without Rails or the framework, as the executable does. +require "minitest/autorun" +require "webmock/minitest" +require "stringio" +require "tmpdir" +require_relative "../../lib/active_agent/cli" + +class ActiveAgentCLITest < Minitest::Test + CLI = ActiveAgent::CLI + URL = "https://dashboard.example.test/activeagents" + MCP = "#{URL}/mcp" + REPORTS = "#{URL}/api/evaluation_reports" + KEY = "aa_cli_test_key" + INGEST_KEY = "ingest-cli-test-key" + + class TTYStringIO < StringIO + def tty? = true + end + + def setup + super + @dir = Dir.mktmpdir("activeagent-cli") + @credentials = File.join(@dir, "activeagent", "credentials.yml") + @env = { "ACTIVEAGENT_URL" => URL, "ACTIVEAGENT_API_KEY" => KEY, "ACTIVEAGENT_INGEST_KEY" => INGEST_KEY } + @sleeps = [] + end + + def teardown + FileUtils.rm_rf(@dir) + super + end + + # Runs the CLI in process and returns [exit code, stdout, stderr]. + def cli(*argv, env: @env, input: StringIO.new, out: StringIO.new, now: nil) + err = StringIO.new + clock = now || -> { @sleeps.sum } + code = CLI.start(argv, out: out, err: err, input: input, env: env, credentials_path: @credentials, + sleeper: ->(seconds) { @sleeps << seconds }, clock: clock) + [ code, out.string, err.string ] + end + + def rpc_body(result) + { status: 200, body: { jsonrpc: "2.0", id: 1, result: result }.to_json, headers: { "Content-Type" => "application/json" } } + end + + def stub_rpc(method, result) + stub_request(:post, MCP).with(headers: { "Authorization" => "Bearer #{KEY}" }) { |request| JSON.parse(request.body)["method"] == method } + .to_return(rpc_body(result)) + end + + # Stubs one dashboard tool; +responses+ answer successive calls in turn. + def stub_tool(name, *responses, arguments: nil, error: nil) + stub = stub_request(:post, MCP).with(headers: { "Authorization" => "Bearer #{KEY}" }) do |request| + body = JSON.parse(request.body) + body["method"] == "tools/call" && body.dig("params", "name") == name && + (arguments.nil? || arguments.all? { |key, value| body.dig("params", "arguments", key.to_s) == value }) + end + return stub.to_return(rpc_body(content: [], structuredContent: { error: error }, isError: true)) if error + + responses.each_with_index { |payload, i| (i.zero? ? stub : stub.then).to_return(rpc_body(content: [], structuredContent: payload)) } + stub + end + + def run_payload(status, passed: 0, evaluated: 0, results: []) + { evaluation: { id: 7, name: "Orders" }, + run: { id: 41, status: status, samples_evaluated: evaluated, samples_passed: passed }, + results: results } + end + + def envelope(run_id: "run-1") + { "version" => 1, "run_id" => run_id, "source" => "support-app", "agent_name" => "SupportBot", "suite" => "orders", + "report" => { "results" => [ { "scenario_key" => "status_1", "answer" => "Shipped" } ], "metadata" => { "run_id" => run_id } } } + end + + # -- core --------------------------------------------------------------- + + def test_version_help_and_usage_errors + assert_equal [ 0, "activeagent #{ActiveAgent::VERSION}\n", "" ], cli("--version") + code, out, = cli("help") + assert_equal 0, code + assert_includes out, "evals" + assert_includes out, "Exit codes" + + code, _, err = cli("deploy") + assert_equal CLI::EXIT_USAGE, code + assert_includes err, "unknown command 'deploy'" + assert_equal CLI::EXIT_USAGE, cli("evals", "frobnicate").first + assert_equal CLI::EXIT_USAGE, cli("evals", "list", "--bogus").first + end + + def test_no_url_is_a_configuration_error + code, _, err = cli("agents", "list", env: {}) + assert_equal CLI::EXIT_CONFIG, code + assert_includes err, "activeagent login" + end + + def test_an_http_url_is_refused_except_on_loopback + code, _, err = cli("agents", "list", env: @env.merge("ACTIVEAGENT_URL" => "http://dashboard.example.test/activeagents")) + assert_equal CLI::EXIT_USAGE, code + assert_includes err, "HTTPS" + end + + def test_login_verifies_the_key_and_stores_a_private_profile + stub_rpc("initialize", serverInfo: { name: "activeagents", version: "1.0" }) + + code, out, = cli("login", "--url", "#{URL}/", "--key", KEY, "--profile", "staging", env: {}) + + assert_equal 0, code, out + assert_includes out, "profile 'staging'" + assert_equal 0o600, File.stat(@credentials).mode & 0o777 + assert_equal 0o700, File.stat(File.dirname(@credentials)).mode & 0o777 + assert_equal({ "staging" => { "url" => URL, "api_key" => KEY } }, YAML.safe_load_file(@credentials)) + + # A later command finds the profile by name, without the environment. + stub_rpc("resources/list", resources: []) + assert_equal 0, cli("agents", "list", "--profile", "staging", env: {}).first + assert_equal CLI::EXIT_CONFIG, cli("agents", "list", "--profile", "missing", env: {}).first + end + + def test_login_reads_the_key_from_standard_input + stub_rpc("initialize", serverInfo: {}) + code, = cli("login", "--url", URL, env: {}, input: StringIO.new("#{KEY}\n")) + assert_equal 0, code + assert_equal KEY, YAML.safe_load_file(@credentials).dig("default", "api_key") + end + + def test_a_refused_key_is_not_stored + stub_request(:post, MCP).to_return(status: 401, body: { error: "Unauthorized: #{KEY} is unknown" }.to_json) + + code, _, err = cli("login", "--url", URL, "--key", KEY, env: {}) + + assert_equal CLI::EXIT_FAILURE, code + assert_includes err, "refused the API key" + refute_includes err, KEY + refute File.exist?(@credentials) + end + + def test_the_environment_overrides_the_stored_profile + stub_rpc("initialize", serverInfo: {}) + cli("login", "--url", "https://old.example.test/activeagents", "--key", "aa_old", "--no-verify", env: {}) + stub_rpc("resources/list", resources: [ { uri: "agent://support", name: "Support" } ]) + + code, out, = cli("agents", "list") + + assert_equal 0, code + assert_includes out, "support" + end + + def test_logout_removes_the_profile + cli("login", "--url", URL, "--key", KEY, "--no-verify", env: {}) + assert_equal [ 0, "Removed profile 'default'.\n", "" ], cli("logout", env: {}) + refute File.exist?(@credentials) + end + + def test_status_checks_both_keys + stub_rpc("initialize", serverInfo: { name: "activeagents" }) + stub_request(:post, REPORTS).with(headers: { "Authorization" => "Bearer #{INGEST_KEY}" }, body: "{}") + .to_return(status: 422, body: { error: "version must be 1" }.to_json) + + code, out, = cli("status", "--json") + + assert_equal 0, code + status = JSON.parse(out) + assert status["ok"] + assert status.dig("checks", "dashboard", "ok") + assert_equal REPORTS, status.dig("checks", "collector", "endpoint") + end + + def test_status_fails_when_the_collector_refuses_the_ingest_key + stub_rpc("initialize", serverInfo: {}) + stub_request(:post, REPORTS).to_return(status: 401, body: { error: "Invalid API key" }.to_json) + + code, out, = cli("status") + + assert_equal CLI::EXIT_FAILURE, code + assert_match(/API key:\s+accepted/, out) + assert_match(/Collector: FAILED: .*HTTP 401/, out) + end + + def test_a_tool_the_dashboard_does_not_offer_is_unavailable + stub_request(:post, MCP).to_return(rpc_body(nil).merge(body: { jsonrpc: "2.0", id: 1, error: { code: -32602, message: "Unknown tool: evaluations_list" } }.to_json)) + + code, _, err = cli("evals", "list") + + assert_equal CLI::EXIT_UNAVAILABLE, code + assert_includes err, "mcp_dashboard_tools" + end + + def test_a_tool_refusal_is_reported_without_the_key + stub_tool("evaluations_list", error: "No agent #{KEY}") + code, _, err = cli("evals", "list", "--agent", "nobody") + assert_equal CLI::EXIT_FAILURE, code + assert_includes err, "No agent [FILTERED]" + end + + # -- evals -------------------------------------------------------------- + + def test_evals_list_prints_a_table_or_json + evaluations = [ { id: 7, name: "Orders", agent: { slug: "support" }, scenario_count: 3, run_count: 2, + latest_run: { status: "complete", samples_evaluated: 3, samples_passed: 2 } } ] + stub_tool("evaluations_list", { evaluations: evaluations }, arguments: { "agent" => "support", "limit" => 5 }) + + code, out, = cli("evals", "list", "--agent", "support", "--limit", "5") + assert_equal 0, code + assert_match(/^ID\s+NAME\s+AGENT/, out) + assert_match(/^7\s+Orders\s+support\s+3\s+2\s+complete\s+2\/3$/, out) + + code, out, = cli("--json", "evals", "list", "--agent", "support", "--limit", "5") + assert_equal 0, code + assert_equal "Orders", JSON.parse(out).dig("evaluations", 0, "name") + end + + def test_evals_runs_resolves_an_evaluation_by_name + stub_tool("evaluations_list", { evaluations: [ { id: 7, name: "Orders" }, { id: 8, name: "Refunds" } ] }) + stub_tool("evaluations_get", { evaluation: { id: 7, name: "Orders" }, + runs: [ { id: 41, number: 2, status: "complete", samples_evaluated: 4, samples_passed: 4 } ] }, + arguments: { "evaluation_id" => 7 }) + + code, out, = cli("evals", "runs", "Orders") + + assert_equal 0, code, out + assert_match(/^41\s+2\s+complete\s+4\/4/, out) + end + + def test_evals_run_without_wait_returns_once_started + stub_tool("evaluations_run", run_payload("pending"), arguments: { "evaluation_id" => 7, "models" => [ "a", "b" ] }) + + code, out, err = cli("evals", "run", "7", "--models", "a,b") + + assert_equal 0, code + assert_includes err, "Started run 41" + assert_includes out, "Status: pending" + end + + def test_evals_run_waits_and_passes_the_gate + stub_tool("evaluations_run", run_payload("pending")) + stub_tool("evaluation_runs_get", run_payload("running", evaluated: 1), run_payload("complete", passed: 9, evaluated: 10), + arguments: { "run_id" => 41, "failed_only" => true }) + + code, out, = cli("evals", "run", "7", "--min-pass-rate", "80%", "--interval", "2") + + assert_equal 0, code, out + assert_equal [ 2 ], @sleeps + assert_includes out, "(90.0%), threshold 80.0%" + assert_match(/PASS$/, out) + end + + def test_evals_run_fails_the_gate_below_the_threshold + failures = [ { scenario_key: "refund_1", model: "mock/beta", status: "failed", fault: "missing_capability", trace_id: "abc" } ] + stub_tool("evaluations_run", run_payload("pending")) + stub_tool("evaluation_runs_get", run_payload("complete", passed: 7, evaluated: 10, results: failures)) + + code, out, = cli("evals", "run", "7", "--min-pass-rate", "0.8", "--json") + + assert_equal CLI::EXIT_GATE, code + result = JSON.parse(out) + assert_equal "fail", result["verdict"] + assert_in_delta 0.7, result["pass_rate"] + assert_equal "refund_1", result.dig("results", 0, "scenario_key") + end + + def test_evals_run_fails_the_gate_when_the_run_failed + stub_tool("evaluations_run", run_payload("pending")) + stub_tool("evaluation_runs_get", run_payload("failed")) + code, out, = cli("evals", "run", "7", "--wait") + assert_equal CLI::EXIT_GATE, code + assert_match(/FAIL$/, out) + end + + def test_evals_run_gives_up_after_the_timeout + stub_tool("evaluations_run", run_payload("pending")) + stub_tool("evaluation_runs_get", run_payload("running")) + + code, _, err = cli("evals", "run", "7", "--wait", "--timeout", "10", "--interval", "4") + + assert_equal CLI::EXIT_FAILURE, code + assert_includes err, "still running after 10s" + assert_equal [ 4, 4, 4 ], @sleeps + end + + def test_an_invalid_pass_rate_is_a_usage_error + assert_equal CLI::EXIT_USAGE, cli("evals", "run", "7", "--min-pass-rate", "1.5").first + end + + def test_evals_report_writes_the_json_envelope + stub_request(:get, "#{REPORTS}/run-1").with(headers: { "Authorization" => "Bearer #{INGEST_KEY}", "Accept" => "application/json" }) + .to_return(status: 200, body: envelope.to_json) + path = File.join(@dir, "report.json") + + code, _, err = cli("evals", "report", "run-1", "-o", path) + + assert_equal 0, code, err + assert_equal envelope, JSON.parse(File.read(path)) + end + + def test_evals_report_asks_for_html_and_markdown + stub_request(:get, "#{REPORTS}/run%201?format=html").with(headers: { "Accept" => "text/html" }) + .to_return(status: 200, body: "report") + stub_request(:get, "#{REPORTS}/run%201?format=md").with(headers: { "Accept" => "text/markdown" }) + .to_return(status: 200, body: "## Orders\n") + + assert_equal [ 0, "report\n", "" ], cli("evals", "report", "run 1", "--format", "html") + assert_equal [ 0, "## Orders\n", "" ], cli("evals", "report", "run 1", "--format", "md") + end + + def test_evals_report_defaults_to_markdown_on_a_terminal_and_json_otherwise + stub_request(:get, "#{REPORTS}/run-1?format=md").to_return(status: 200, body: "## Orders\n") + stub_request(:get, "#{REPORTS}/run-1").to_return(status: 200, body: envelope.to_json) + + assert_equal "## Orders\n", cli("evals", "report", "run-1", out: TTYStringIO.new)[1] + assert_equal envelope, JSON.parse(cli("evals", "report", "run-1")[1]) + end + + def test_evals_report_without_the_export_endpoint_is_unavailable + stub_request(:get, "#{REPORTS}/run-1").to_return(status: 404, body: "") + code, _, err = cli("evals", "report", "run-1") + assert_equal CLI::EXIT_UNAVAILABLE, code + assert_includes err, "GET /api/evaluation_reports/:run_id" + end + + def test_evals_report_refuses_a_body_that_is_not_an_envelope + stub_request(:get, "#{REPORTS}/run-1").to_return(status: 200, body: { results: [] }.to_json) + assert_equal CLI::EXIT_FAILURE, cli("evals", "report", "run-1").first + end + + def test_evals_import_posts_an_envelope_with_the_ingest_key + path = File.join(@dir, "report.json") + File.write(path, envelope.to_json) + request = stub_request(:post, REPORTS).with(headers: { "Authorization" => "Bearer #{INGEST_KEY}" }, body: envelope) + .to_return(status: 201, body: { run_id: "run-1", status: "complete", id: 5, evaluation_id: 3 }.to_json) + + code, out, = cli("evals", "import", path) + + assert_equal 0, code, out + assert_includes out, "Imported run run-1 (evaluation 3, run 5)" + assert_requested request + end + + def test_evals_import_takes_a_bare_report_with_identities + path = File.join(@dir, "report.json") + File.write(path, envelope["report"].to_json) + stub_request(:post, REPORTS).with { |request| JSON.parse(request.body).slice("run_id", "suite") == { "run_id" => "run-1", "suite" => "orders" } } + .to_return(status: 200, body: { run_id: "run-1", status: "complete", id: 5, evaluation_id: 3, duplicate: true }.to_json) + + code, out, = cli("evals", "import", path, "--source", "app", "--agent", "Bot", "--suite", "orders") + assert_equal 0, code, out + assert_includes out, "Already stored" + + code, _, err = cli("evals", "import", path, "--source", "app") + assert_equal CLI::EXIT_USAGE, code + assert_includes err, "agent_name must be a nonempty string" + end + + def test_evals_import_reports_a_rejection + path = File.join(@dir, "report.json") + File.write(path, envelope.to_json) + stub_request(:post, REPORTS).to_return(status: 409, body: { error: "run-1 holds a different report" }.to_json) + + code, _, err = cli("evals", "import", path) + + assert_equal CLI::EXIT_FAILURE, code + assert_includes err, "HTTP 409" + end + + def test_evals_import_of_a_missing_file_is_a_usage_error + assert_equal CLI::EXIT_USAGE, cli("evals", "import", File.join(@dir, "nope.json")).first + end + + # -- catalogs, agents, traces -------------------------------------------- + + def test_catalogs_list_and_import + stub_tool("catalogs_list", { catalogs: [ { key: "shop", name: "Shop", product_count: 1, scenario_count: 4, source_kind: "api" } ] }) + path = File.join(@dir, "shop.yml") + File.write(path, "catalog: shop\nproducts: []\n") + stub_tool("catalogs_import", { catalog: { key: "shop", product_count: 1, scenario_count: 4 }, + products: [ { key: "support", agent: "Support", sets: [ { key: "orders", scenario_count: 4 } ] } ] }, + arguments: { "document" => "catalog: shop\nproducts: []\n" }) + + code, out, = cli("catalogs", "list") + assert_equal 0, code + assert_match(/^shop\s+Shop\s+1\s+4\s+api/, out) + + code, out, = cli("catalogs", "import", path) + assert_equal 0, code, out + assert_includes out, "Imported catalog shop" + assert_includes out, "support -> Support: orders (4)" + end + + def test_commands_without_an_api_key_route_say_so + path = File.join(@dir, "shop.yml") + File.write(path, "catalog: shop\n") + + [ %w[catalogs export shop], [ "catalogs", "import", path, "--preview" ], %w[agents versions support] ].each do |argv| + code, _, err = cli(*argv) + assert_equal CLI::EXIT_UNAVAILABLE, code, argv.inspect + assert_includes err, "not yet available on this dashboard" + end + assert_not_requested :post, MCP + end + + def test_agents_list_reads_the_agent_resources + stub_rpc("resources/list", resources: [ { uri: "agent://support", name: "Support", description: "Answers orders" } ]) + code, out, = cli("agents", "list", "--json") + assert_equal 0, code + assert_equal [ { "slug" => "support", "name" => "Support", "description" => "Answers orders" } ], JSON.parse(out)["agents"] + end + + def test_traces_list_converts_since_to_minutes + stub_tool("traces_search", { traces: [ { short_id: "abcd1234", agent: "SupportAgent", action: "reply", status: "error", + tokens: { total: 120 } } ], truncated: true, window_minutes: 360 }, + arguments: { "since_minutes" => 360, "status" => "error" }) + + code, out, = cli("traces", "list", "--since", "6h", "--status", "error") + + assert_equal 0, code, out + assert_match(/^abcd1234\s+-\s+SupportAgent#reply/, out) + assert_includes out, "more traces match" + assert_equal CLI::EXIT_USAGE, cli("traces", "list", "--since", "soon").first + end +end From e7c3428674c0638ae73c7b1b6d9d76754803361d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 14:19:03 +0000 Subject: [PATCH 2/4] Finish the activeagent CLI: catalog export and preview, agent versions, trace export and import The three commands that exited 5 for lack of an API-key route now work. The MCP facade gains catalogs_export (a catalog's canonical document as YAML or JSON, whole or one product or set), catalogs_preview (what an import would change, writing nothing) and agents_versions (what GET /api/agents/:id/versions lists), and catalogs_import takes mode as the API does. Traces move between dashboards the way runs do: traces_export returns one trace whole, as the payload trace ingest takes, and `traces export` and `traces import` write and post it, the import with the ingest key; `traces get` reads one. Every run summary carries run_id, the handle the report export and `evals report` take, which `evals runs` shows and the CI gate prints. `evals report` runs against the export that shipped in 1.9.3: a run id the dashboard holds no complete run under exits 1, a dashboard that predates the export exits 5. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01RSuPEbVwjxwJub8eAmGM9N --- CHANGELOG.md | 48 ++-- .../action_agent/api/agents_controller.rb | 18 +- .../action_agent/api/mcp_controller.rb | 7 +- .../action_agent/api/mcp_dashboard_tools.rb | 223 ++++++++++++++++-- .../models/action_agent/scenario_catalog.rb | 10 +- .../action_agent/agent_version_serializer.rb | 29 +++ .../action_agent/evaluation_serializer.rb | 9 + actionagent/test/cli_contract_test.rb | 136 +++++++++++ actionagent/test/mcp_dashboard_tools_test.rb | 136 ++++++++++- docs/evals/publication.md | 10 + docs/framework/cli.md | 156 ++++++++---- docs/framework/dashboard.md | 14 +- lib/active_agent/cli.rb | 54 +++-- lib/active_agent/cli/client.rb | 58 ++++- lib/active_agent/cli/eval_commands.rb | 35 +-- lib/active_agent/cli/resource_commands.rb | 139 ++++++----- lib/active_agent/cli/trace_commands.rb | 129 ++++++++++ test/cli/cli_test.rb | 168 +++++++++++-- 18 files changed, 1138 insertions(+), 241 deletions(-) create mode 100644 actionagent/app/serializers/action_agent/agent_version_serializer.rb create mode 100644 lib/active_agent/cli/trace_commands.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index b0e813e0e..d46d8a330 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,21 +9,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **`activeagent` command line** (`activeagent`, refs #619). The gem ships - an `activeagent` executable (stdlib `OptionParser` and `Net::HTTP`, no new - dependencies) that works with a dashboard over HTTP: `login`/`logout` - (named profiles in `~/.config/activeagent/credentials.yml`, `0600`; - `ACTIVEAGENT_URL`, `ACTIVEAGENT_API_KEY` and `ACTIVEAGENT_INGEST_KEY` - override them for CI), `status`, `evals list|runs|run|report|import`, - `catalogs list|import`, `agents list` and `traces list`, with `--json` - output and distinct exit codes. `evals run --min-pass-rate 0.8` waits for - the run and exits 4 below the threshold, for CI gating. Reads and runs go - through the MCP facade's dashboard tools with a dashboard API key; `evals - import` and `evals report` use the report collector with the ingest key. - `evals report --format json|html|md` needs the report export endpoint - (`GET /api/evaluation_reports/:run_id`). `catalogs export`, - `catalogs import --preview` and `agents versions` exit 5 until the - dashboard offers API-key routes for them. See `docs/framework/cli.md`. +- **`activeagent` command line** (`activeagent`, `actionagent`; refs #619). + The gem ships an `activeagent` executable (stdlib `OptionParser` and + `Net::HTTP`, no new dependencies) that works with a dashboard over HTTP: + `login`/`logout` (named profiles in `~/.config/activeagent/credentials.yml`, + `0600`; `ACTIVEAGENT_URL`, `ACTIVEAGENT_API_KEY` and + `ACTIVEAGENT_INGEST_KEY` override them for CI), `status`, + `evals list|runs|run|report|import`, `catalogs list|import|export` + (`import --preview` shows what a document would change, `--mode merge` + keeps what it leaves out), `agents list|versions` and + `traces list|get|export|import`, with `--json` output and distinct exit + codes. `evals run --min-pass-rate 0.8` waits for the run and exits 4 + below the threshold, for CI gating. Reads and runs go through the MCP + facade's dashboard tools with a dashboard API key; the imports and + `evals report` use the endpoints the framework's own reporters post to, + with the ingest key. A run, a catalog and a trace each move between + installs as one file: `evals report RUN --format json` writes the + version-1 envelope `evals import` posts (`--format html|md` the report + page or its Markdown, for a pull request comment), `catalogs export KEY` + the canonical document (YAML, or `--format json`) that `catalogs import` + reads, and `traces export TRACE` the payload `traces import` sends to + trace ingest. See `docs/framework/cli.md`. + +- **Four more dashboard tools on the MCP facade** (`actionagent`), the + API-key routes the CLI needed: `catalogs_export` (a catalog's canonical + document as YAML or JSON, whole or one product or set), `catalogs_preview` + (what `catalogs_import` would change, writing nothing), `agents_versions` + (an agent's versions with their diffs, as `GET /api/agents/:id/versions`) + and `traces_export` (one trace, whole and untruncated, as the payload + `POST /api/traces` takes, so a trace moves between installs). + `catalogs_import` takes `mode: replace|merge` as the API does. Every run + summary (`GET /api/evaluations`, the facade's `evaluations_get`, + `evaluations_run` and `evaluation_runs_get`) now carries `run_id`, the + handle `GET /api/evaluation_reports/:run_id` takes. ## [1.9.3] - 2026-10-21 diff --git a/actionagent/app/controllers/action_agent/api/agents_controller.rb b/actionagent/app/controllers/action_agent/api/agents_controller.rb index f9b864c3e..dea1b97f8 100644 --- a/actionagent/app/controllers/action_agent/api/agents_controller.rb +++ b/actionagent/app/controllers/action_agent/api/agents_controller.rb @@ -591,23 +591,7 @@ def mcp_server_params end def version_json(version, include_diff: false) - json = { - id: version.id, - version_number: version.version_number, - change_summary: version.change_summary, - created_by: version.created_by, - created_at: version.created_at, - is_latest: version.latest?, - release: version.release?, - release_digest: version.release_digest, - revision: version.revision - } - - if include_diff && version.previous - json[:diff] = version.diff(version.previous) - end - - json + AgentVersionSerializer.call(version, include_diff: include_diff) end def default_appearance_for(preset) diff --git a/actionagent/app/controllers/action_agent/api/mcp_controller.rb b/actionagent/app/controllers/action_agent/api/mcp_controller.rb index 8c56914f3..c9e236383 100644 --- a/actionagent/app/controllers/action_agent/api/mcp_controller.rb +++ b/actionagent/app/controllers/action_agent/api/mcp_controller.rb @@ -195,8 +195,11 @@ def initialize_instructions "failures). A run_ call that pauses to ask for input returns its input request ids: " \ "input_requests_list shows what paused runs are waiting on, and input_requests_answer answers a text or " \ "choice request. Scenarios kept as a YAML catalog of products and sets (an .activeagents/evals file) are " \ - "imported with catalogs_import, listed with catalogs_list, and one set is run as an evaluation named " \ - "catalog/product/set with catalog_set_run (pass project_id to run against a project's running sandbox)." + "imported with catalogs_import (catalogs_preview shows what an import would change first), listed with " \ + "catalogs_list, written back out with catalogs_export, and one set is run as an evaluation named " \ + "catalog/product/set with catalog_set_run (pass project_id to run against a project's running sandbox). " \ + "agents_versions lists an agent's versions with what each changed, and traces_export reads one trace " \ + "whole, as the payload the trace ingest endpoint takes." end MESSAGE_INPUT_SCHEMA = { diff --git a/actionagent/app/controllers/concerns/action_agent/api/mcp_dashboard_tools.rb b/actionagent/app/controllers/concerns/action_agent/api/mcp_dashboard_tools.rb index 300cf71b8..5d99d79e9 100644 --- a/actionagent/app/controllers/concerns/action_agent/api/mcp_dashboard_tools.rb +++ b/actionagent/app/controllers/concerns/action_agent/api/mcp_dashboard_tools.rb @@ -37,8 +37,8 @@ module MCPDashboardTools NAMES = %w[ evaluations_list evaluations_get evaluations_create scenarios_merge explorations_submit evaluations_run - evaluation_runs_get evaluation_runs_compare traces_search traces_get input_requests_list input_requests_answer - catalogs_list catalogs_import catalog_set_run + evaluation_runs_get evaluation_runs_compare traces_search traces_get traces_export input_requests_list + input_requests_answer catalogs_list catalogs_import catalogs_preview catalogs_export catalog_set_run agents_versions ].freeze # The kinds input_requests_answer answers. An approval and a secret are @@ -320,6 +320,17 @@ module MCPDashboardTools required: [ "trace_id" ] } }, + { + name: "traces_export", + description: "One trace, whole, as the payload POST /api/traces ingests ({ traces: [trace], sdk }): " \ + "its spans untruncated, with the owner's credentials masked, so the trace can be sent to another " \ + "dashboard with that dashboard's ingest key. Read one to work with it instead: traces_get.", + inputSchema: { + type: "object", + properties: { trace_id: { type: "string", description: "The trace's id, its OpenTelemetry trace id, or that id's first 8 characters" } }, + required: [ "trace_id" ] + } + }, { name: "input_requests_list", description: "List the requests for input this key's paused agent runs are waiting on, newest first: each " \ @@ -364,11 +375,50 @@ module MCPDashboardTools type: "object", properties: { document: { type: "string", description: "The catalog as YAML" }, - name: { type: "string", description: "The catalog key when the document sets no `catalog:`" } + name: { type: "string", description: "The catalog key when the document sets no `catalog:`" }, + mode: { + type: "string", enum: ScenarioCatalogImport::MODES, + description: "replace (the default) makes the catalog the document; merge keeps what the document leaves out" + } + }, + required: [ "document" ] + } + }, + { + name: "catalogs_preview", + description: "What catalogs_import of this document would change, without writing anything: whether the " \ + "catalog would be created, updated or left unchanged, how many products, sets and scenarios " \ + "would be created, updated, removed (or kept, in merge mode), and the first " \ + "#{ScenarioCatalogImport::PREVIEW_LIMIT} changes by path.", + inputSchema: { + type: "object", + properties: { + document: { type: "string", description: "The catalog as YAML" }, + name: { type: "string", description: "The catalog key when the document sets no `catalog:`" }, + mode: { + type: "string", enum: ScenarioCatalogImport::MODES, + description: "replace (the default) makes the catalog the document; merge keeps what the document leaves out" + } }, required: [ "document" ] } }, + { + name: "catalogs_export", + description: "A catalog's canonical document from its records, as YAML (the default) or JSON: the whole " \ + "catalog, or one product or one set of it, as a document of the catalog's key that a merge " \ + "import reads back. The same document the dashboard's Export YAML downloads.", + inputSchema: { + type: "object", + properties: { + catalog: { type: "string", description: "The catalog's key" }, + product: { type: "string", description: "Only this product, by key" }, + set: { type: "string", description: "Only this set of the product, by key" }, + format: { type: "string", enum: %w[yaml json], description: "yaml (the default) or json" } + }, + required: [ "catalog" ] + } + }, { name: "catalog_set_run", description: "Run one set of a catalog as an evaluation of the agent under test: the product's agent, the " \ @@ -388,6 +438,20 @@ module MCPDashboardTools }, required: %w[catalog product set] } + }, + { + name: "agents_versions", + description: "An agent's versions, newest first: each with its number, what it changed from the version " \ + "before (diff), who made it, whether it is the agent's current version, and whether it was cut " \ + "from the agent's code on deploy (release) rather than edited on the dashboard.", + inputSchema: { + type: "object", + properties: { + agent: { type: "string", description: "The agent's slug or id" }, + limit: { type: "integer", description: "At most this many (default #{LIST_LIMIT}, max #{MAX_LIST_LIMIT})" } + }, + required: [ "agent" ] + } } ].freeze @@ -426,11 +490,15 @@ def dispatch_dashboard_tool(name) when "evaluation_runs_compare" then evaluation_runs_compare_tool when "traces_search" then traces_search_tool when "traces_get" then traces_get_tool + when "traces_export" then traces_export_tool when "input_requests_list" then input_requests_list_tool when "input_requests_answer" then input_requests_answer_tool when "catalogs_list" then catalogs_list_tool when "catalogs_import" then catalogs_import_tool + when "catalogs_preview" then catalogs_preview_tool + when "catalogs_export" then catalogs_export_tool when "catalog_set_run" then catalog_set_run_tool + when "agents_versions" then agents_versions_tool end end @@ -821,18 +889,8 @@ def traces_search_tool } end - # Looked up as Api::TraceReportsController#show looks a trace up: the - # OpenTelemetry trace id, then a prefix of it, then the record id. def traces_get_tool - id = tool_argument(:trace_id).to_s - raise MCPController::McpError.new("Missing required argument: trace_id", MCPController::JSONRPC_INVALID_PARAMS) if id.blank? - - scope = owned_traces - trace = scope.find_by(trace_id: id) || - scope.where("trace_id LIKE ?", "#{ActionAgent.trace_model.sanitize_sql_like(id)}%").first || - (id.match?(/\A\d+\z/) && scope.find_by(id: id)) or - raise ToolError, "No trace #{id.truncate(64)} was found" - + trace = find_tool_trace! detail = TelemetryTraceSerializer.detail(trace) spans = detail.delete(:spans) failed_spans = spans.select { |span| span[:error] }.map do |span| @@ -849,6 +907,43 @@ def traces_get_tool PayloadBounds.bound(payload, max_string: MAX_STRING, max_items: MAX_TRACE_SPANS) end + # The trace as Api::TracesController#create takes it, whole: the + # stored trace's spans and resource attributes with the SDK that + # reported it, so a POST of the result to another install's + # /api/traces stores the same trace there (an install skips a + # trace_id it already holds). Not bounded as traces_get bounds a trace, + # since the point is a copy; the owner's credentials are masked from + # it as from every tool's result. + def traces_export_tool + trace = find_tool_trace! + + { + traces: [ { + trace_id: trace.trace_id, + service_name: trace.service_name, + environment: trace.environment, + timestamp: trace.timestamp&.iso8601(6), + resource_attributes: trace.resource_attributes.presence || {}, + spans: Array(trace.spans) + } ], + sdk: trace.sdk_info.presence || {} + } + end + + # The trace `trace_id` names, looked up as Api::TraceReportsController#show + # looks a trace up: the OpenTelemetry trace id, then a prefix of it, + # then the record id. + def find_tool_trace! + id = tool_argument(:trace_id).to_s + raise MCPController::McpError.new("Missing required argument: trace_id", MCPController::JSONRPC_INVALID_PARAMS) if id.blank? + + scope = owned_traces + scope.find_by(trace_id: id) || + scope.where("trace_id LIKE ?", "#{ActionAgent.trace_model.sanitize_sql_like(id)}%").first || + (id.match?(/\A\d+\z/) && scope.find_by(id: id)) or + raise ToolError, "No trace #{id.truncate(64)} was found" + end + def input_requests_list_tool InputRequest.expire_overdue!(owner_input_requests) scope = owner_input_requests.pending.for_listing.includes(:subject).recent @@ -948,19 +1043,16 @@ def catalogs_list_tool end # Imports a catalog document as POST /api/scenario_catalogs does: the - # owner's agents and projects resolve the products' targets. + # owner's agents and projects resolve the products' targets, and `mode` + # is replace (the default) or merge, as there. def catalogs_import_tool - document = required_tool_argument!(:document).to_s - raise ToolError, "document is larger than #{ScenarioCatalog::MAX_BYTES} bytes" if document.bytesize > ScenarioCatalog::MAX_BYTES - + document = catalog_document_argument! authorize_catalog_write!(ScenarioCatalog.new) - catalog = ScenarioCatalogImport.new( - owner: current_owner, document: document, name: tool_argument(:name).to_s.presence, - source_kind: "api", agents: owner_agents, projects: owned(Project) - ).call + catalog = catalog_import(document).call { catalog: ScenarioCatalogSerializer.summary(catalog), + mode: catalog_mode_argument, products: catalog.products.map do |product| { key: product.key, agent: product.agent&.name, project_id: product.project_id, sets: product.sets.map { |set| { key: set.key, scenario_count: set.scenarios.size } } } @@ -970,12 +1062,95 @@ def catalogs_import_tool raise ToolError, e.message end + # What catalogs_import of the document would change, as POST + # /api/scenario_catalogs/preview answers it, writing nothing. + def catalogs_preview_tool + catalog_import(catalog_document_argument!).preview + rescue ScenarioCatalogImport::Invalid => e + raise ToolError, e.message + end + + # The catalog's canonical document from its records, as GET + # /api/scenario_catalogs/:id/export serves it, whole or narrowed to one + # product or one set. + def catalogs_export_tool + catalog = find_tool_catalog! + format = tool_argument(:format).to_s.presence || "yaml" + raise ToolError, "format is yaml or json" unless %w[yaml json].include?(format) + + selection = {} + product = set = nil + if (product_key = tool_argument(:product).to_s.presence) + product = catalog.products.find_by(key: product_key) || + raise(ToolError, "No product #{product_key.inspect} in catalog #{catalog.key}") + if (set_key = tool_argument(:set).to_s.presence) + set = product.sets.find_by(key: set_key) || raise(ToolError, "No set #{set_key.inspect} in product #{product.key}") + selection[:set_ids] = [ set.id ] + else + selection[:product_ids] = [ product.id ] + end + elsif tool_argument(:set).present? + raise ToolError, "set names a set of a product: give product too" + end + + exported = catalog.export_catalog(**selection) + { + catalog: { id: catalog.id, key: catalog.key, name: catalog.name, digest: catalog.digest }, + product: product&.key, + set: set&.key, + format: format, + digest: exported.digest, + document: format == "json" ? JSON.pretty_generate(exported.to_h) : exported.to_yaml + } + end + + def catalog_import(document) + ScenarioCatalogImport.new( + owner: current_owner, document: document, name: tool_argument(:name).to_s.presence, + source_kind: "api", agents: owner_agents, projects: owned(Project), mode: catalog_mode_argument + ) + end + + def catalog_document_argument! + document = required_tool_argument!(:document).to_s + raise ToolError, "document is larger than #{ScenarioCatalog::MAX_BYTES} bytes" if document.bytesize > ScenarioCatalog::MAX_BYTES + + document + end + + # replace unless the call says merge; ScenarioCatalogImport refuses + # anything else. + def catalog_mode_argument + tool_argument(:mode).to_s.presence || "replace" + end + + # The owner's catalog `catalog` names by key. + def find_tool_catalog! + key = required_tool_argument!(:catalog).to_s + owned(ScenarioCatalog).find_by(key: key) || raise(ToolError, "No catalog #{key.inspect}; catalogs_list names them") + end + + # An agent's versions as GET /api/agents/:id/versions lists them, each + # with its diff from the version before, newest first. + def agents_versions_tool + agent = resolve_tool_agent!(required_tool_argument!(:agent)) + limit = integer_argument(:limit, default: LIST_LIMIT, min: 1, max: MAX_LIST_LIMIT) + versions = agent.agent_versions.recent.limit(limit + 1).to_a + + { + agent: { id: agent.id, slug: agent.slug, name: agent.name, version_count: agent.agent_versions.count }, + versions: versions.first(limit).map do |version| + PayloadBounds.bound(AgentVersionSerializer.call(version, include_diff: true), max_string: MAX_STRING) + end, + truncated: versions.size > limit + } + end + # Runs one set as POST /api/scenario_catalogs/:id/sets/:set_id/run # does, gated as evaluations_run is. A project's sandbox must already # be running: booting one is a dashboard action, not a tool's. def catalog_set_run_tool - catalog = owned(ScenarioCatalog).find_by(key: required_tool_argument!(:catalog).to_s) || - raise(ToolError, "No catalog #{tool_argument(:catalog).inspect}; catalogs_list names them") + catalog = find_tool_catalog! product = catalog.products.find_by(key: required_tool_argument!(:product).to_s) || raise(ToolError, "No product #{tool_argument(:product).inspect} in catalog #{catalog.key}") set = product.sets.find_by(key: required_tool_argument!(:set).to_s) || diff --git a/actionagent/app/models/action_agent/scenario_catalog.rb b/actionagent/app/models/action_agent/scenario_catalog.rb index bc25f7803..e87c028a5 100644 --- a/actionagent/app/models/action_agent/scenario_catalog.rb +++ b/actionagent/app/models/action_agent/scenario_catalog.rb @@ -69,10 +69,16 @@ def to_catalog # that hold them), as a document of this catalog's key that a merge # import (ScenarioCatalogImport mode "merge") reads back into it. def export_yaml(product_ids: nil, set_ids: nil, scenario_ids: nil) + export_catalog(product_ids: product_ids, set_ids: set_ids, scenario_ids: scenario_ids).to_yaml + end + + # The records as an ActiveAgent::Evals::Catalog, whole or narrowed as + # #export_yaml narrows it, for the YAML or the JSON of the document. + def export_catalog(product_ids: nil, set_ids: nil, scenario_ids: nil) selection = { product_ids: ids(product_ids), set_ids: ids(set_ids), scenario_ids: ids(scenario_ids) } - return to_catalog.to_yaml if selection.values.all?(&:nil?) + return to_catalog if selection.values.all?(&:nil?) - ActiveAgent::Evals::Catalog.new([ to_document(**selection) ], name: key).to_yaml + ActiveAgent::Evals::Catalog.new([ to_document(**selection) ], name: key) end # A key for a new product, set or scenario: +text+ as a key, made unique diff --git a/actionagent/app/serializers/action_agent/agent_version_serializer.rb b/actionagent/app/serializers/action_agent/agent_version_serializer.rb new file mode 100644 index 000000000..140fbcd72 --- /dev/null +++ b/actionagent/app/serializers/action_agent/agent_version_serializer.rb @@ -0,0 +1,29 @@ +# frozen_string_literal: true + +module ActionAgent + # JSON for one of an agent's versions: what GET /api/agents/:id/versions + # lists and the MCP facade's agents_versions returns. + module AgentVersionSerializer + module_function + + # +include_diff+ adds `diff`, what the version changed from the one + # before it ({ key => { from:, to: } }), when there is one. + def call(version, include_diff: false) + json = { + id: version.id, + version_number: version.version_number, + change_summary: version.change_summary, + created_by: version.created_by, + created_at: version.created_at, + is_latest: version.latest?, + release: version.release?, + release_digest: version.release_digest, + revision: version.revision + } + if include_diff && (previous = version.previous) + json[:diff] = version.diff(previous) + end + json + end + end +end diff --git a/actionagent/app/serializers/action_agent/evaluation_serializer.rb b/actionagent/app/serializers/action_agent/evaluation_serializer.rb index 7f1211e56..588cbafef 100644 --- a/actionagent/app/serializers/action_agent/evaluation_serializer.rb +++ b/actionagent/app/serializers/action_agent/evaluation_serializer.rb @@ -86,6 +86,7 @@ def run_summary(run, number: nil) { id: run.id, number: number, + run_id: export_run_id(run), status: run.status, average_score: average_score(run), samples_evaluated: run.samples_evaluated, @@ -98,6 +99,14 @@ def run_summary(run, number: nil) } end + # The run_id the run exports under (EvaluationReportExport.run_id_for), + # the handle `GET /api/evaluation_reports/:run_id` and the CLI's + # `evals report` take; nil on an install whose runs predate the report + # identity migration. + def export_run_id(run) + EvaluationRun.column_names.include?("external_run_id") ? EvaluationReportExport.run_id_for(run) : nil + end + # Returns up to +limit+ of the evaluation's runs, newest first. Sorts the # preloaded association when one was loaded rather than issuing one # ORDER BY query per evaluation. diff --git a/actionagent/test/cli_contract_test.rb b/actionagent/test/cli_contract_test.rb index 03378317b..9b22b0bac 100644 --- a/actionagent/test/cli_contract_test.rb +++ b/actionagent/test/cli_contract_test.rb @@ -13,9 +13,27 @@ class CliContractTest < ActiveSupport::TestCase URL = "http://localhost/activeagents" INGEST_KEY = "install-key-for-cli" + CATALOG = <<~YAML + catalog: shop + products: + - key: support + agent: support + sets: + - key: orders + scenarios: + - key: status_1 + prompt: Where is order 1? + - key: refunds + scenarios: + - key: refund_1 + prompt: Refund order 1. + YAML + setup do ActionAgent::Agent.delete_all ActionAgent::ApiKey.delete_all + ActionAgent::ScenarioCatalog.delete_all + ActionAgent::TelemetryTrace.delete_all ActionAgent.ingest_api_key = INGEST_KEY @key = ActionAgent::ApiKey.create!(name: "CLI") @agent = ActionAgent::Agent.create!(name: "Support", slug: "support", provider: "mock", model: "mock-model", @@ -126,6 +144,124 @@ def report(run_id) assert_equal 60, json("traces", "list", "--since", "1h")["window_minutes"] end + test "evals report reads a run back as the envelope evals import sends, and as Markdown and HTML" do + path = File.join(@dir, "report.json") + File.write(path, JSON.generate( + "version" => 1, "run_id" => "cli-run-2", "source" => "support-app", "agent_name" => "Support", "suite" => "orders", + "report" => report("cli-run-2").to_h + )) + assert_equal 0, cli("evals", "import", path).first + + # evals runs names the run_id the report is read back under. + evaluation = json("evals", "list")["evaluations"].sole + assert_equal "cli-run-2", json("evals", "runs", evaluation["id"].to_s)["runs"].sole["run_id"] + + exported = File.join(@dir, "exported.json") + code, _, err = cli("evals", "report", "cli-run-2", "-o", exported) + assert_equal 0, code, err + envelope = JSON.parse(File.read(exported)) + assert_equal [ 1, "cli-run-2", "support-app", "Support", "orders" ], envelope.values_at("version", "run_id", "source", "agent_name", "suite") + assert_equal "Shipped", envelope.dig("report", "results", 0, "answer") + + # Posting the export back is an identical retry, so the file moves a run + # between installs. + code, out, = cli("evals", "import", exported) + assert_equal 0, code, out + assert_match(/Already stored run cli-run-2/, out) + + code, out, = cli("evals", "report", "cli-run-2", "--format", "md") + assert_equal 0, code, out + assert_match(/\A# /, out) + assert_includes out, "status_1" + + code, out, = cli("evals", "report", "cli-run-2", "--format", "html") + assert_equal 0, code, out + assert_includes out, " "Answer from data.", "to" => "Answer from data, briefly." }, versions.first.dig("diff", "instructions")) + + code, out, = cli("agents", "versions", "support") + assert_equal 0, code, out + assert_includes out, "Support (support): 2 versions" + assert_match(/^2\s+.*\binstructions$/, out) + end + + test "traces export and import move a trace between dashboards" do + ActionAgent::TelemetryTrace.create_from_payload({ + "trace_id" => "cli-trace-1", "service_name" => "support-app", "environment" => "test", + "timestamp" => Time.current.iso8601(6), "resource_attributes" => { "host.name" => "web-1" }, + "spans" => [ { + "span_id" => "root", "parent_span_id" => nil, "name" => "SupportAgent.answer", "type" => "root", + "duration_ms" => 120.0, "status" => "OK", "tokens" => { "input" => 10, "output" => 5 }, + "attributes" => { "agent.class" => "SupportAgent", "agent.action" => "answer" } + } ] + }, { "name" => "activeagent", "version" => "1.9.3" }) + path = File.join(@dir, "trace.json") + + assert_equal 0, cli("traces", "export", "cli-trace-1", "-o", path).first + payload = JSON.parse(File.read(path)) + assert_equal "cli-trace-1", payload.dig("traces", 0, "trace_id") + assert_equal({ "host.name" => "web-1" }, payload.dig("traces", 0, "resource_attributes")) + assert_equal "1.9.3", payload.dig("sdk", "version") + + ActionAgent::TelemetryTrace.delete_all + code, out, = cli("traces", "import", path) + assert_equal 0, code, out + assert_includes out, "Sent 1 trace to #{URL}/api/traces: cli-trace-1." + stored = ActionAgent::TelemetryTrace.find_by!(trace_id: "cli-trace-1") + assert_equal "SupportAgent", stored.agent_class + assert_equal 15, stored.total_tokens + assert_equal({ "host.name" => "web-1" }, stored.resource_attributes) + assert_equal "1.9.3", stored.sdk_info["version"] + + # Importing the same file again stores nothing more. + assert_equal 0, cli("traces", "import", path).first + assert_equal 1, ActionAgent::TelemetryTrace.where(trace_id: "cli-trace-1").count + + assert_equal "cli-trace-1", json("traces", "get", "cli-trac")["trace_id"] + assert_equal [ "cli-trace-1" ], json("traces", "list")["traces"].map { |trace| trace["trace_id"] } + end + test "a refused dashboard key fails with the facade's explanation" do @key.destroy! code, _, err = cli("agents", "list") diff --git a/actionagent/test/mcp_dashboard_tools_test.rb b/actionagent/test/mcp_dashboard_tools_test.rb index 7a99515d2..e1ca8fbfb 100644 --- a/actionagent/test/mcp_dashboard_tools_test.rb +++ b/actionagent/test/mcp_dashboard_tools_test.rb @@ -9,8 +9,8 @@ class McpDashboardToolsTest < ActionDispatch::IntegrationTest RUNTIME_TOKEN = "aa_runtime_dashboard_tools_s3cret" DASHBOARD_TOOLS = %w[ evaluations_list evaluations_get evaluations_create scenarios_merge explorations_submit evaluations_run - evaluation_runs_get evaluation_runs_compare traces_search traces_get input_requests_list input_requests_answer - catalogs_list catalogs_import catalog_set_run + evaluation_runs_get evaluation_runs_compare traces_search traces_get traces_export input_requests_list + input_requests_answer catalogs_list catalogs_import catalogs_preview catalogs_export catalog_set_run agents_versions ].freeze # A trace model whose tenant is its service name, so a multi-tenant scope @@ -427,6 +427,61 @@ def structured(body) assert_equal nothing_body.dig("result", "content", 0, "text").sub("f" * 32, "ID"), theirs_body.dig("result", "content", 0, "text").sub(other.trace_id, "ID") assert_equal true, call_tool("traces_get", { trace_id: other.id.to_s }).dig("result", "isError") + + assert_equal own.trace_id, structured(call_tool("traces_export", { trace_id: own.trace_id })).dig("traces", 0, "trace_id") + assert_equal true, call_tool("traces_export", { trace_id: other.trace_id }).dig("result", "isError") + end + + test "traces_export returns the trace whole, as the payload trace ingest stores" do + trace = create_trace(agent_class: "SupportAgent", spans: 150, big_attribute: "y" * 5_000) + + result = structured(call_tool("traces_export", { trace_id: trace.trace_id.first(8) })) + + exported = result["traces"].sole + assert_equal trace.trace_id, exported["trace_id"] + assert_equal [ "support-hub", "test" ], exported.values_at("service_name", "environment") + assert_equal trace.reload.timestamp.iso8601(6), exported["timestamp"] + assert_equal 151, exported["spans"].size + assert_equal "y" * 5_000, exported["spans"][1].dig("attributes", "tool.output.result") + assert_equal({ "agent.class" => "SupportAgent" }, exported["resource_attributes"]) + assert_equal({}, result["sdk"]) + + # Posted to an install that lacks the trace, the result stores the same trace. + trace.destroy! + post "/activeagents/api/traces", params: result, as: :json + assert_response :accepted + copy = ActionAgent::TelemetryTrace.find_by!(trace_id: trace.trace_id) + assert_equal 151, copy.spans.size + assert_equal exported["spans"], copy.spans + assert_equal({ "agent.class" => "SupportAgent" }, copy.resource_attributes) + + # The owner's credentials are masked from it, as from every tool's result. + leaky = create_trace(agent_class: "SupportAgent", big_attribute: "Bearer #{@key.token}") + masked = structured(call_tool("traces_export", { trace_id: leaky.trace_id })) + assert_equal "Bearer [REDACTED]", masked.dig("traces", 0, "spans", 1, "attributes", "tool.output.result") + + assert_equal true, call_tool("traces_export", { trace_id: "does-not-exist" }).dig("result", "isError") + assert_equal(-32602, call_tool("traces_export").dig("error", "code")) + end + + test "agents_versions lists an agent's versions with their diffs, newest first" do + @agent.update!(instructions: "Answer from data, briefly.") + + result = structured(call_tool("agents_versions", { agent: "support" })) + + assert_equal({ "id" => @agent.id, "slug" => "support", "name" => "Support", "version_count" => 2 }, result["agent"]) + assert_equal [ 2, 1 ], result["versions"].map { |version| version["version_number"] } + assert_equal({ "from" => "Answer from data.", "to" => "Answer from data, briefly." }, result["versions"].first.dig("diff", "instructions")) + assert_equal true, result["versions"].first["is_latest"] + assert_nil result["versions"].last["diff"] + assert_equal false, result["truncated"] + + limited = structured(call_tool("agents_versions", { agent: @agent.id.to_s, limit: 1 })) + assert_equal [ 2 ], limited["versions"].map { |version| version["version_number"] } + assert_equal true, limited["truncated"] + + assert_equal true, call_tool("agents_versions", { agent: "nobody" }).dig("result", "isError") + assert_equal(-32602, call_tool("agents_versions").dig("error", "code")) end test "dashboard reads need no execution switch and spend no execution quota" do @@ -506,6 +561,83 @@ def structured(body) assert_match(/catalogs_list names them/, body.dig("result", "content", 0, "text")) end + test "catalogs_preview shows what an import would change and writes nothing" do + ActionAgent::ScenarioCatalog.delete_all + + preview = structured(call_tool("catalogs_preview", { document: CATALOG_DOCUMENT })) + assert_equal "create", preview.dig("catalog", "action") + assert_equal "replace", preview["mode"] + assert_equal 2, preview["scenario_count"] + assert_equal 0, ActionAgent::ScenarioCatalog.count + + structured(call_tool("catalogs_import", { document: CATALOG_DOCUMENT })) + changed = CATALOG_DOCUMENT + " - key: cancel\n prompt: Cancel order ABC-123.\n" + preview = structured(call_tool("catalogs_preview", { document: changed, mode: "merge" })) + # The catalog record itself (key, name, description) is unchanged; the + # new scenario is in the change list. + assert_equal "unchanged", preview.dig("catalog", "action") + assert_equal "merge", preview["mode"] + assert_equal 1, preview.dig("counts", "create") + assert_includes preview["changes"], { "level" => "scenario", "path" => "support/smoke/cancel", "action" => "create" } + assert_equal 2, ActionAgent::ScenarioCatalog.sole.scenario_count + + body = call_tool("catalogs_preview", { document: "catalog: x\nproducts: [{ name: nameless }]" }) + assert_equal true, body.dig("result", "isError") + body = call_tool("catalogs_preview", { document: CATALOG_DOCUMENT, mode: "sideways" }) + assert_equal true, body.dig("result", "isError") + assert_match(/replace or merge/, body.dig("result", "content", 0, "text")) + assert_equal(-32602, call_tool("catalogs_preview").dig("error", "code")) + end + + test "catalogs_import merges when asked, keeping what the document leaves out" do + ActionAgent::ScenarioCatalog.delete_all + structured(call_tool("catalogs_import", { document: CATALOG_DOCUMENT })) + without_refund = CATALOG_DOCUMENT.sub(" - key: refund\n prompt: Refund order ABC-123.\n", "") + assert_not_includes without_refund, "key: refund" + + merged = structured(call_tool("catalogs_import", { document: without_refund, mode: "merge" })) + assert_equal "merge", merged["mode"] + assert_equal 2, ActionAgent::ScenarioCatalog.sole.scenario_count + + replaced = structured(call_tool("catalogs_import", { document: without_refund })) + assert_equal "replace", replaced["mode"] + assert_equal 1, ActionAgent::ScenarioCatalog.sole.scenario_count + end + + test "catalogs_export writes the catalog back as YAML or JSON, whole or one product or set" do + ActionAgent::ScenarioCatalog.delete_all + two_sets = CATALOG_DOCUMENT + " - key: nightly\n scenarios:\n - key: cancel\n prompt: Cancel order ABC-123.\n" + structured(call_tool("catalogs_import", { document: two_sets })) + + exported = structured(call_tool("catalogs_export", { catalog: "support_desk" })) + assert_equal "yaml", exported["format"] + document = YAML.safe_load(exported["document"]) + assert_equal "support_desk", document["catalog"] + assert_equal %w[smoke nightly], document.dig("products", 0, "sets").map { |set| set["key"] } + assert_equal %w[order_lookup refund], document.dig("products", 0, "sets", 0, "scenarios").map { |scenario| scenario["key"] } + assert_equal exported.dig("catalog", "digest"), exported["digest"] + assert_nil exported["product"] + + narrowed = structured(call_tool("catalogs_export", { catalog: "support_desk", product: "support", set: "nightly", format: "json" })) + assert_equal [ "json", "support", "nightly" ], narrowed.values_at("format", "product", "set") + as_json = JSON.parse(narrowed["document"]) + assert_equal "support_desk", as_json["catalog"] + assert_equal [ "nightly" ], as_json.dig("products", 0, "sets").map { |set| set["key"] } + + # The JSON document imports as the YAML would: a merge of it changes nothing. + before = ActionAgent::ScenarioCatalog.sole.digest + structured(call_tool("catalogs_import", { document: narrowed["document"], mode: "merge" })) + assert_equal before, ActionAgent::ScenarioCatalog.sole.digest + + body = call_tool("catalogs_export", { catalog: "nope" }) + assert_equal true, body.dig("result", "isError") + assert_match(/catalogs_list names them/, body.dig("result", "content", 0, "text")) + assert_match(/give product too/, call_tool("catalogs_export", { catalog: "support_desk", set: "smoke" }).dig("result", "content", 0, "text")) + assert_match(/No set "nope"/, call_tool("catalogs_export", { catalog: "support_desk", product: "support", set: "nope" }).dig("result", "content", 0, "text")) + assert_match(/yaml or json/, call_tool("catalogs_export", { catalog: "support_desk", format: "toml" }).dig("result", "content", 0, "text")) + assert_equal(-32602, call_tool("catalogs_export").dig("error", "code")) + end + test "catalog_set_run honours the execution switch and quota, and asks for an agent when the product names none" do ActionAgent::ScenarioCatalog.delete_all agentless = CATALOG_DOCUMENT.sub(" agent: Support\n", "") diff --git a/docs/evals/publication.md b/docs/evals/publication.md index bc0185a40..f9b81067a 100644 --- a/docs/evals/publication.md +++ b/docs/evals/publication.md @@ -337,6 +337,16 @@ bin/rails action_agent:evaluations:import FILE=tmp/run.json # => Stored: run ci-2026-10-21 as run #41 of evaluation #7 (orders (eu)), complete ``` +From any machine with the gem, the [`activeagent` CLI](/framework/cli#reports) +does the same over HTTP with the ingest key, and `evals runs` shows each +run's `run_id`: + +```bash +activeagent evals report ci-2026-10-21 --format json -o run.json +activeagent evals import run.json --profile other-install +activeagent evals report ci-2026-10-21 --format md | gh pr comment 123 --body-file - +``` + ### Metering A host that meters its install answers the new `:evaluation_report` kind from diff --git a/docs/framework/cli.md b/docs/framework/cli.md index aa7474640..22b2af5ba 100644 --- a/docs/framework/cli.md +++ b/docs/framework/cli.md @@ -2,10 +2,10 @@ The `activeagent` gem ships an `activeagent` executable that works with a dashboard (a Rails app mounting the `actionagent` engine) over HTTP: list and -run evaluations, gate CI on a pass rate, move evaluation reports between -installs, import scenario catalogs, and read agents and traces. It needs only -the gem — not the host app's checkout or database — and loads neither Rails -nor the framework. +run evaluations, gate CI on a pass rate, move evaluation runs, scenario +catalogs and traces between installs as files, and read agents and traces. It +needs only the gem — not the host app's checkout or database — and loads +neither Rails nor the framework. ```bash gem install activeagent # or: bundle exec activeagent ... @@ -14,6 +14,11 @@ activeagent status activeagent evals list ``` +It needs `actionagent` 1.9.3 or later on the dashboard, with the facade's +dashboard tools on (the default). An older dashboard, or one that set +`ActionAgent.mcp_dashboard_tools = false`, answers the commands that need +them with exit code 5. + ## Keys and profiles The dashboard authenticates two kinds of key, and the CLI sends each where it @@ -22,10 +27,13 @@ belongs: | Key | Where it comes from | Used by | |---|---|---| | Dashboard API key (`--key`) | The dashboard's Settings → API Keys (`aa_...`) | Everything that reads or runs: the MCP facade at `/mcp` | -| Ingest key (`--ingest-key`) | `ActionAgent.ingest_api_key`, or the tenant's telemetry key on a multi-tenant install | `evals import` and `evals report`: the collector at `/api/evaluation_reports` | +| Ingest key (`--ingest-key`) | `ActionAgent.ingest_api_key`, or the tenant's telemetry key on a multi-tenant install | What the framework's own reporters post with: `evals import` and `evals report` (the report collector at `/api/evaluation_reports`) and `traces import` (trace ingest at `/api/traces`) | The ingest key defaults to the API key, so set it only when the two differ. -A single-tenant install with no `ingest_api_key` accepts any ingest key. +A single-tenant install with no `ingest_api_key` accepts any ingest key for +the imports, but refuses `evals report` until one is set (every stored run +would otherwise be readable under its run id by anyone who reaches the +mount). `login` checks the API key against the dashboard (skip with `--no-verify`) and stores the profile in `~/.config/activeagent/credentials.yml` @@ -59,7 +67,9 @@ nothing). ## Output and exit codes Commands print tables. `--json` prints the dashboard's JSON instead, for -`jq` and scripts. Options can go before or after the command. +`jq` and scripts. Options can go before or after the command. A command that +writes a document (`evals report`, `catalogs export`, `traces export`) +prints it, or writes it to `-o FILE` and says so on standard error. | Code | Meaning | |---|---| @@ -68,7 +78,7 @@ Commands print tables. `--json` prints the dashboard's JSON instead, for | 2 | Bad command line | | 3 | No URL or key configured | | 4 | An evaluation run failed, or passed below `--min-pass-rate` | -| 5 | The dashboard has no API-key endpoint for the command yet | +| 5 | The dashboard does not offer what the command needs: an older engine, or its dashboard tools turned off | ## Evaluations @@ -81,14 +91,18 @@ activeagent evals report RUN [--format json|html|md] [-o FILE] activeagent evals import FILE [--run-id ID --source NAME --agent NAME --suite NAME] ``` -`EVALUATION` is an evaluation's id or its exact name. +`EVALUATION` is an evaluation's id or its exact name. `evals runs` lists an +evaluation's runs with their `RUN_ID`, the handle `evals report` takes: a run +an application published keeps its own `run_id`, and a run the dashboard +executed is `dashboard-run--`. ### Gating CI on a pass rate `evals run --min-pass-rate` starts the run, waits for it (polling every `--interval` seconds, up to `--timeout`), prints the failed results with -their traces, and exits 4 when the run failed or passed less than the -threshold. The rate is `0.8` or `80%`. +their traces and the `evals report` command for the run, and exits 4 when +the run failed or passed less than the threshold. The rate is `0.8` or +`80%`. ```yaml # .github/workflows/evals.yml @@ -103,21 +117,28 @@ execution enabled there and counts against its execution quota. ### Reports -`evals import` posts a saved report to the collector with -`ActiveAgent::Evals::Publisher`. The file is a version-1 envelope, or a bare -`Report#to_h` hash with the identities given as options (`run_id` defaults to -the report's `metadata.run_id`). Importing the same report again is a no-op; -a different report under the same `run_id` is refused. - -`evals report` reads one back from `GET /api/evaluation_reports/:run_id`: +`evals report` reads a complete run back from +`GET /api/evaluation_reports/:run_id` +(see [Exporting a stored run](/evals/publication#exporting-a-stored-run)): -- `--format json` — the version-1 envelope, so `evals report` on one install - then `evals import` on another copies a run between them. -- `--format html` — the standalone HTML report. +- `--format json` — the version-1 envelope `ActiveAgent::Evals::Publisher` + sends, so `evals report` on one install then `evals import` on another + copies a run between them, and importing it back where it came from is + an identical retry. +- `--format html` — the standalone HTML report page, following the viewer's + light or dark preference. - `--format md` — GitHub-flavored Markdown, for a pull request comment. Without `--format`, a terminal gets Markdown and anything else (a pipe, or `-o FILE`) gets JSON, so scripts always see JSON unless they ask otherwise. +A run id the dashboard holds no complete run under exits 1; a dashboard +with no report export at all (before `actionagent` 1.9.3) exits 5. + +`evals import` posts a saved report to the collector with +`ActiveAgent::Evals::Publisher`. The file is a version-1 envelope, or a bare +`Report#to_h` hash with the identities given as options (`run_id` defaults to +the report's `metadata.run_id`). Importing the same report again is a no-op; +a different report under the same `run_id` is refused. ```bash activeagent evals report run-2026-10-21 --format json -o report.json @@ -126,42 +147,75 @@ ACTIVEAGENT_PROFILE=production activeagent evals import report.json activeagent evals report run-2026-10-21 --format md | gh pr comment 123 --body-file - ``` -The report endpoint arrives with the dashboard's report export; on a -dashboard that predates it, `evals report` exits 5. - ## Scenario catalogs ```bash activeagent catalogs list -activeagent catalogs import FILE [--name KEY] +activeagent catalogs import FILE [--name KEY] [--mode replace|merge] [--preview] +activeagent catalogs export KEY [--product KEY [--set KEY]] [--format yaml|json] [-o FILE] ``` -`catalogs import` takes the YAML document (`catalog:`, `products:` with -`sets:` of `scenarios:`). Products, sets and scenarios are matched by key, so -importing the same file twice changes nothing. Importing needs the -`replace_scenarios` permission for the key's user. +`catalogs import` takes the catalog document (`catalog:`, `products:` with +`sets:` of `scenarios:`; see [Catalogs](/framework/evaluations#catalogs)), +as YAML or as the JSON `catalogs export --format json` writes. Products, sets +and scenarios are matched by key, so importing the same file twice changes +nothing; `--mode replace` (the default) makes the catalog the document, and +`--mode merge` keeps what the document leaves out. `--preview` shows what the +import would do — whether the catalog would be created, how many products, +sets and scenarios would be created, updated, removed or kept, and each +change by path — and writes nothing. Importing needs the `replace_scenarios` +permission for the key's user; previewing and exporting need only the key. + +`catalogs export` writes the canonical document from the dashboard's records +(the same document the Catalogs page's Export YAML downloads): the whole +catalog, or one product or one set of it, as a document of the catalog's key +that `--mode merge` reads back in. `-o FILE` ending in `.json` writes JSON. -## Agents and traces +```bash +activeagent catalogs export support_desk -o .activeagents/evals/support_desk.yml +activeagent catalogs import .activeagents/evals/support_desk.yml --preview +activeagent catalogs export support_desk --product triage --set smoke --format json +``` + +## Agents ```bash activeagent agents list -activeagent traces list [--agent NAME] [--since 24h] [--status error|ok] [--service NAME] [--limit N] +activeagent agents versions AGENT [--limit N] ``` -`--since` takes minutes or a duration such as `30m`, `6h` or `7d` (at most -the dashboard's trace window). +`agents versions` lists an agent's versions newest first: who made each, +whether it was cut from the agent's code on deploy (a release, with its +revision) or edited on the dashboard, and which fields it changed +(`--json` has the full diff). + +## Traces -## Not yet available +```bash +activeagent traces list [--agent NAME] [--since 24h] [--status error|ok] [--service NAME] [--limit N] +activeagent traces get TRACE +activeagent traces export TRACE [-o FILE] +activeagent traces import FILE +``` -These commands exist but exit 5, because their dashboard routes only accept -a browser session today. Each needs an API-key-authenticated endpoint -([#619](https://github.com/activeagents/activeagent/issues/619)): +`--since` takes minutes or a duration such as `30m`, `6h` or `7d` (at most +the dashboard's trace window). `TRACE` is a trace's id, its OpenTelemetry +trace id, or that id's first 8 characters, as `traces list` shows it. + +`traces get` shows a trace as the dashboard's trace page does: its spans, +tool calls and errors, with long values cut. `traces export` writes the trace +whole, as the payload `POST /api/traces` ingests (`{ "traces": [...], +"sdk": {...} }`, with the owner's credentials masked), and `traces import` +posts such a file — or one trace object, or a reporter's batch — to the +dashboard's trace ingest with the ingest key. A trace id the dashboard +already holds is skipped there, so importing the same file twice stores +nothing more. -| Command | Needs | -|---|---| -| `catalogs export KEY` | `GET /api/scenario_catalogs/:id/export` | -| `catalogs import FILE --preview` | `POST /api/scenario_catalogs/preview` | -| `agents versions AGENT` | `GET /api/agents/:id/versions` | +```bash +activeagent traces export 4f1c9b2a -o trace.json +ACTIVEAGENT_PROFILE=local activeagent traces import trace.json +activeagent traces get 4f1c9b2a --profile local +``` ## Where each command goes @@ -174,10 +228,14 @@ a browser session today. Each needs an API-key-authenticated endpoint | `evals run` | MCP `evaluations_run`, then `evaluation_runs_get` | API key | | `evals import` | `POST /api/evaluation_reports` | Ingest key | | `evals report` | `GET /api/evaluation_reports/:run_id` | Ingest key | -| `catalogs list` / `import` | MCP `catalogs_list` / `catalogs_import` | API key | -| `agents list` | MCP `resources/list` | API key | -| `traces list` | MCP `traces_search` | API key | - -The MCP commands use the facade's dashboard tools. An install that sets -`ActionAgent.mcp_dashboard_tools = false` turns them off, and those commands -then exit 5. +| `catalogs list` / `import` / `import --preview` / `export` | MCP `catalogs_list` / `catalogs_import` / `catalogs_preview` / `catalogs_export` | API key | +| `agents list` / `versions` | MCP `resources/list` / `agents_versions` | API key | +| `traces list` / `get` / `export` | MCP `traces_search` / `traces_get` / `traces_export` | API key | +| `traces import` | `POST /api/traces` | Ingest key | + +The MCP commands use the facade's +[dashboard tools](/framework/dashboard#evaluations-and-telemetry-from-your-coding-harness). +An install that sets `ActionAgent.mcp_dashboard_tools = false` turns them +off, and those commands then exit 5. The rake tasks +(`action_agent:evaluations:export|import`, `action_agent:catalogs:import|export|run`) +do the same work inside the host app, without a key. diff --git a/docs/framework/dashboard.md b/docs/framework/dashboard.md index b0866aee6..bc3e53fc2 100644 --- a/docs/framework/dashboard.md +++ b/docs/framework/dashboard.md @@ -879,9 +879,11 @@ answering `202` with the set, its evaluation and the pending run. A set whose product names no agent answers `422` with `code: "no_target"`. The same operations are MCP tools for a coding harness (`catalogs_list`, -`catalogs_import`, `catalog_set_run`, see -[the MCP facade](#evaluations-and-telemetry-from-your-coding-harness)) and -rake tasks for a deploy or a cron: +`catalogs_import`, `catalogs_preview`, `catalogs_export`, `catalog_set_run`, +see [the MCP facade](#evaluations-and-telemetry-from-your-coding-harness)), +commands of the [`activeagent` CLI](/framework/cli#scenario-catalogs) +(`catalogs list`, `catalogs import [--preview]`, `catalogs export`, over +HTTP with a dashboard API key) and rake tasks for a deploy or a cron: ```bash bin/rails action_agent:catalogs:import FILE=.activeagents/evals/support_desk.yml # ACCOUNT_ID= or USER_ID= for the owner @@ -948,11 +950,15 @@ calls. | `evaluation_runs_compare` | Two runs of one evaluation, result by result: fixed, regressed, still failing, added, removed. Defaults to the latest run against the one before it | | `traces_search` | Summary rows of traces, newest first, filtered by `agent` (class name or dashboard slug), `status` (`error` or `ok`), `service`, `since_minutes`, `min_tokens` and `min_duration_ms`; at most 100 | | `traces_get` | One trace by id, OpenTelemetry trace id or its first 8 characters: spans, tool calls with their arguments and results, tokens, estimated cost and failed spans | +| `traces_export` | The same trace whole, as the payload `POST /api/traces` ingests (`{ traces: [trace], sdk }`), its spans untruncated and the owner's credentials masked, so it can be sent to another dashboard with that dashboard's ingest key | | `input_requests_list` | The pending [input requests](#input-requests) of the key's runs, newest first, in the API's shape; `run_id` filters to one run | | `input_requests_answer` | Answers a `text` or `choice` request as the key's caller, under the same permission check as the API. A `confirm` or `secret` request is refused with an error that points to the dashboard | | `catalogs_list` | The owner's [scenario catalogs](#scenario-catalogs): each with its products, sets, scenario count, digest and whether its document is in Active Storage | -| `catalogs_import` | Imports a catalog `document` (YAML) under the key's owner, resolving each product's `agent` by name among the owner's agents. Re-importing the same key updates products, sets and scenarios in place. Needs the `replace_scenarios` permission | +| `catalogs_import` | Imports a catalog `document` (YAML, or the JSON `catalogs_export` writes) under the key's owner, resolving each product's `agent` by name among the owner's agents. Re-importing the same key updates products, sets and scenarios in place; `mode` is `replace` (the default: the document is the catalog) or `merge` (what the document leaves out is kept). Needs the `replace_scenarios` permission | +| `catalogs_preview` | What `catalogs_import` of the `document` would change, with the same `name` and `mode`, writing nothing: whether the catalog record would be created, updated or left as it is, counts of products, sets and scenarios created, updated, removed or kept, and the first 500 changes by path | +| `catalogs_export` | A catalog's canonical document from its records (`catalog` key), as YAML or `format: json`: the whole catalog, or one `product` or one `set` of it, as a document of the catalog's key that a merge import reads back. What the Catalogs page's Export YAML downloads | | `catalog_set_run` | Runs one set (`catalog`, `product`, `set` keys) as an evaluation named `catalog/product/set`, against the product's agent, an `agent` (slug or id) or a `project_id` whose sandbox is already running; `models`, `keys`, `browser` and `sandbox_id` narrow the run as `evaluations_run` does. Gated like `evaluations_run` | +| `agents_versions` | An agent's versions (`agent`, slug or id), newest first, as `GET /api/agents/:id/versions` lists them: each with its number, its diff from the version before, who made it, whether it is the current version and whether it was cut from the agent's code on deploy; `limit` at most 50 | A typical loop: `evaluations_run`, poll `evaluation_runs_get` until the run is `complete`, read the fix items and a failing result's trace with diff --git a/lib/active_agent/cli.rb b/lib/active_agent/cli.rb index 672f104ed..dde070b50 100644 --- a/lib/active_agent/cli.rb +++ b/lib/active_agent/cli.rb @@ -15,10 +15,12 @@ module ActiveAgent # activeagent evals run "Support orders" --wait --min-pass-rate 0.8 # # Every command answers through an API-key-authenticated endpoint: the MCP - # facade's dashboard tools (/mcp) or the report collector - # (/api/evaluation_reports). A command whose dashboard route only - # takes a browser session today says so and exits EXIT_UNAVAILABLE rather - # than scraping the UI's API. + # facade's dashboard tools (/mcp) with the dashboard API key, or the + # ingest endpoints the framework's own reporters post to (the report + # collector at /api/evaluation_reports, trace ingest at + # /api/traces) with the ingest key. A dashboard that does not offer + # what a command needs (an older engine, or one that turned the dashboard + # tools off) exits EXIT_UNAVAILABLE saying so. # # CLI.start returns the exit code rather than exiting, so it can be called # from Ruby and tested in process. @@ -32,7 +34,8 @@ class CLI EXIT_CONFIG = 3 # An evaluation run failed, or passed fewer scenarios than --min-pass-rate. EXIT_GATE = 4 - # The dashboard has no API-key route for the command yet. + # The dashboard does not offer what the command needs: an older engine, + # or one with its dashboard tools turned off. EXIT_UNAVAILABLE = 5 # Every failure a command reports: the message is printed and the exit @@ -47,8 +50,6 @@ def initialize(message = nil, exit_code: EXIT_FAILURE, status: nil) end end - ISSUE_URL = "https://github.com/activeagents/activeagent/issues/619" - def self.version ActiveAgent::VERSION end @@ -93,9 +94,9 @@ def run(argv) "logout" => "Remove a stored profile", "status" => "Check the URL and keys against the dashboard", "evals" => "Evaluations: list, runs, run, report, import", - "catalogs" => "Scenario catalogs: list, import, export", + "catalogs" => "Scenario catalogs: list, import (or preview), export", "agents" => "Agents: list, versions", - "traces" => "Telemetry traces: list", + "traces" => "Telemetry traces: list, get, export, import", "version" => "Print the version" }.freeze @@ -110,7 +111,7 @@ def dispatch(command, argv) when "evals" then subcommand("evals", argv, %w[list runs run report import]) when "catalogs" then subcommand("catalogs", argv, %w[list import export]) when "agents" then subcommand("agents", argv, %w[list versions]) - when "traces" then subcommand("traces", argv, %w[list]) + when "traces" then subcommand("traces", argv, %w[list get export import]) else raise Error.new("unknown command '#{command}'. Commands: #{COMMANDS.keys.join(", ")}", exit_code: EXIT_USAGE) end @@ -143,7 +144,7 @@ def global_parser parser.separator "Environment: ACTIVEAGENT_URL, ACTIVEAGENT_API_KEY, ACTIVEAGENT_INGEST_KEY and" parser.separator "ACTIVEAGENT_PROFILE override the stored profile (for CI)." parser.separator "Exit codes: 0 ok, 1 failed, 2 usage, 3 not configured, 4 evaluation gate failed," - parser.separator "5 not yet available on this dashboard." + parser.separator "5 not available on this dashboard (an older engine, or its dashboard tools turned off)." end end @@ -242,9 +243,33 @@ def cell(value) value.nil? ? "-" : value.to_s.gsub(/[[:space:]\p{C}]+/, " ") end - def unavailable(command, endpoint) - raise Error.new("`#{command}` is not yet available on this dashboard: it needs an API-key-authenticated " \ - "#{endpoint}, which only takes a browser session today. See #{ISSUE_URL}.", exit_code: EXIT_UNAVAILABLE) + # Writes +body+ to +path+, or prints it, ending in one newline either + # way. With a path, the block's line is said on standard error (not with + # --json, whose standard output is the only thing a script reads). + def write_output(body, path) + body = "#{body}\n" unless body.end_with?("\n") + if path + File.write(path, body) + @err.puts yield if block_given? && !@options[:json] + else + @out.print body + end + EXIT_SUCCESS + end + + def read_text_file(path) + File.read(path) + rescue Errno::ENOENT, Errno::EACCES, Errno::EISDIR => e + raise Error.new("Cannot read #{path} (#{e.class})", exit_code: EXIT_USAGE) + end + + def read_json_file(path) + document = JSON.parse(read_text_file(path)) + raise Error.new("#{path} must hold a JSON object", exit_code: EXIT_USAGE) unless document.is_a?(Hash) + + document + rescue JSON::ParserError + raise Error.new("#{path} is not valid JSON", exit_code: EXIT_USAGE), cause: nil end def require_argument(argv, name, usage) @@ -272,3 +297,4 @@ def list_option(value) require_relative "cli/session_commands" require_relative "cli/eval_commands" require_relative "cli/resource_commands" +require_relative "cli/trace_commands" diff --git a/lib/active_agent/cli/client.rb b/lib/active_agent/cli/client.rb index 57a19fcc3..c6ee50b08 100644 --- a/lib/active_agent/cli/client.rb +++ b/lib/active_agent/cli/client.rb @@ -21,8 +21,10 @@ class CLI # the MCP facade at /mcp, whose dashboard tools back most # commands. # - +ingest_key+, the install's ingest key (ActionAgent.ingest_api_key, - # or the tenant's telemetry key), for the collector endpoints under - # /api/evaluation_reports. Defaults to +api_key+. + # or the tenant's telemetry key), for the endpoints the framework's + # reporters post to: the report collector at + # /api/evaluation_reports and trace ingest at /api/traces. + # Defaults to +api_key+. class Client DETAIL_LIMIT = 200 JSONRPC_INVALID_PARAMS = -32602 @@ -63,6 +65,10 @@ def reports_endpoint "#{url}/api/evaluation_reports" end + def traces_endpoint + "#{url}/api/traces" + end + # A Publisher for the mount's collector, authenticated with the ingest key. def publisher require_key!(@ingest_key, "an ingest key") @@ -111,22 +117,45 @@ def tool(name, arguments = {}) "the install sets ActionAgent.mcp_dashboard_tools = false", exit_code: EXIT_UNAVAILABLE) end + # Rails' own 404 body, from a dashboard whose routes have no such path. + GENERIC_NOT_FOUND = /\Anot found\z/i + # GETs +path+ under the mount with the ingest key and returns the - # response, raising on anything but 200. - def get(path, accept:, query: nil, not_found: nil) + # response, raising on anything but 200. A 404 is one of two things: + # the endpoint's own refusal, a JSON error naming what is missing, + # raised as +missing+ with that detail; or nothing at the path at all + # (an older engine), raised as +unavailable+ with EXIT_UNAVAILABLE. + def get(path, accept:, query: nil, missing: nil, unavailable: nil) require_key!(@ingest_key, "an ingest key") uri = URI("#{url}#{path}") uri.query = query if query response = exchange(:get, uri, @ingest_key, accept: accept) return response if response.code == "200" - if response.code == "404" && not_found - raise refused(not_found, response, @ingest_key, 404, exit_code: EXIT_UNAVAILABLE) + case response.code + when "404" + detail = error_detail(response, @ingest_key) + raise Error.new("#{missing} (#{detail})", status: 404) if missing && detail && !detail.match?(GENERIC_NOT_FOUND) + + raise Error.new(unavailable || "Nothing at #{path} answers; check the dashboard URL", status: 404, exit_code: EXIT_UNAVAILABLE) + when "401" + raise refused("the dashboard refused the ingest key; use the mount's ingest_api_key (--ingest-key)", response, @ingest_key, 401) + else + raise refused("GET #{path} answered HTTP #{response.code}", response, @ingest_key, response.code.to_i) end + end + + # POSTs +body+, a JSON string, to +path+ under the mount with the + # ingest key and returns the response, raising on anything but a 2xx. + def post(path, body) + require_key!(@ingest_key, "an ingest key") + response = exchange(:post, URI("#{url}#{path}"), @ingest_key, body: body) + return response if response.code.start_with?("2") + if response.code == "401" raise refused("the dashboard refused the ingest key; use the mount's ingest_api_key (--ingest-key)", response, @ingest_key, 401) end - raise refused("GET #{path} answered HTTP #{response.code}", response, @ingest_key, response.code.to_i) + raise refused("POST #{path} answered HTTP #{response.code}", response, @ingest_key, response.code.to_i) end private @@ -170,15 +199,18 @@ def parse_json(body) end def refused(message, response, key, status, exit_code: EXIT_FAILURE) - detail = begin - parsed = JSON.parse(response.body.to_s) - sanitize(parsed["error"], key) if parsed.is_a?(Hash) - rescue JSON::ParserError, EncodingError - nil - end + detail = error_detail(response, key) Error.new(detail ? "#{message} (#{detail})" : message, status: status, exit_code: exit_code) end + # The sanitized `error` string of a JSON error body, or nil. + def error_detail(response, key) + parsed = JSON.parse(response.body.to_s) + sanitize(parsed["error"], key) if parsed.is_a?(Hash) + rescue JSON::ParserError, EncodingError + nil + end + # Control characters and runs of whitespace become one space, the keys # become [FILTERED], and the result is cut to DETAIL_LIMIT characters. def sanitize(message, key) diff --git a/lib/active_agent/cli/eval_commands.rb b/lib/active_agent/cli/eval_commands.rb index 28099bb2f..3ad05cc97 100644 --- a/lib/active_agent/cli/eval_commands.rb +++ b/lib/active_agent/cli/eval_commands.rb @@ -65,7 +65,9 @@ def evals_runs(argv) "STATUS" => ->(r) { r["status"] }, "PASSED" => ->(r) { passed(r) }, "SCORE" => ->(r) { r["average_score"] }, - "CREATED" => ->(r) { r["created_at"] } + "CREATED" => ->(r) { r["created_at"] }, + # The handle `evals report` takes. + "RUN_ID" => ->(r) { r["run_id"] } }, empty: "No runs yet.") end end @@ -108,6 +110,9 @@ def evals_run(argv) # Without --format, a terminal gets Markdown to read and anything else # (a pipe, a file, -o) gets the JSON envelope, so scripts see JSON. + # RUN is the run's run_id: the RUN_ID column of `evals runs`, which is + # a published run's own run_id and a dashboard run's + # dashboard-run--. def evals_report(argv) format = nil output = nil @@ -118,23 +123,17 @@ def evals_report(argv) end return EXIT_SUCCESS unless argv - run_id = require_argument(argv, "RUN (the report's run_id)", "evals report RUN") + run_id = require_argument(argv, "RUN (the run's run_id, the RUN_ID of `evals runs`)", "evals report RUN") format ||= !output && @out.respond_to?(:tty?) && @out.tty? ? "md" : "json" request = REPORT_FORMATS.fetch(format) response = client.get("/api/evaluation_reports/#{URI.encode_www_form_component(run_id).gsub("+", "%20")}", accept: request[:accept], query: request[:query], - not_found: "No report for run #{run_id}, or this dashboard has no report export " \ - "(GET /api/evaluation_reports/:run_id) yet") + missing: "No report for run #{run_id}; `activeagent evals runs EVALUATION` shows each run's RUN_ID", + unavailable: "This dashboard has no report export (GET /api/evaluation_reports/:run_id); " \ + "it predates actionagent 1.9.3") body = response.body.to_s body = JSON.pretty_generate(report_envelope(body)) if format == "json" - - if output - File.write(output, body.end_with?("\n") ? body : "#{body}\n") - @err.puts "Wrote #{format} report for run #{run_id} to #{output}." unless @options[:json] - else - @out.puts body - end - EXIT_SUCCESS + write_output(body, output) { "Wrote #{format} report for run #{run_id} to #{output}." } end # Posts a saved report to the collector. FILE is a version-1 envelope @@ -222,6 +221,7 @@ def gate(payload, min_pass_rate) "TRACE" => ->(r) { r["trace_id"] } }) end + @out.puts "Report: activeagent evals report #{run["run_id"]}" if run["run_id"] @out.puts(verdict == "pass" ? "PASS" : "FAIL") end verdict == "pass" ? EXIT_SUCCESS : EXIT_GATE @@ -257,17 +257,6 @@ def report_envelope(body) rescue JSON::ParserError raise Error.new("The dashboard returned invalid JSON for the report"), cause: nil end - - def read_json_file(path) - document = JSON.parse(File.read(path)) - raise Error.new("#{path} must hold a JSON object", exit_code: EXIT_USAGE) unless document.is_a?(Hash) - - document - rescue Errno::ENOENT, Errno::EACCES, Errno::EISDIR => e - raise Error.new("Cannot read #{path} (#{e.class})", exit_code: EXIT_USAGE) - rescue JSON::ParserError - raise Error.new("#{path} is not valid JSON", exit_code: EXIT_USAGE), cause: nil - end end include EvalCommands diff --git a/lib/active_agent/cli/resource_commands.rb b/lib/active_agent/cli/resource_commands.rb index c91fe1528..45758feb2 100644 --- a/lib/active_agent/cli/resource_commands.rb +++ b/lib/active_agent/cli/resource_commands.rb @@ -2,13 +2,13 @@ module ActiveAgent class CLI - # `activeagent catalogs|agents|traces ...`, through the MCP facade. The - # commands whose dashboard route only takes a browser session today - # (catalog export and import preview, agent versions) exit - # EXIT_UNAVAILABLE naming the route they need. + # `activeagent catalogs|agents ...`, through the MCP facade's dashboard + # tools: scenario catalogs are listed, imported (or the import + # previewed) and exported as the dashboard's Catalogs page does it, and + # agents are listed with their versions. module ResourceCommands - DURATION = /\A(\d+)([smhd]?)\z/ - DURATION_MINUTES = { "" => 1, "m" => 1, "h" => 60, "d" => 24 * 60 }.freeze + CATALOG_FORMATS = %w[yaml json].freeze + CATALOG_MODES = %w[replace merge].freeze private @@ -29,26 +29,28 @@ def catalogs_list(argv) end end - # Imports a catalog's YAML document. Products, sets and scenarios are - # matched by key, so importing the same file twice changes nothing. + # Imports a catalog's document (YAML, or the JSON `catalogs export + # --format json` writes, which YAML reads). Products, sets and + # scenarios are matched by key, so importing the same file twice + # changes nothing; --mode merge keeps what the document leaves out. + # --preview shows what the import would change and writes nothing. def catalogs_import(argv) - name = nil + arguments = {} preview = false - argv = parse(argv, "catalogs import FILE [--name KEY] [--preview]") do |p| - p.on("--name KEY", "The catalog key when the document sets no `catalog:`") { |v| name = v } + argv = parse(argv, "catalogs import FILE [--name KEY] [--mode replace|merge] [--preview]") do |p| + p.on("--name KEY", "The catalog key when the document sets no `catalog:`") { |v| arguments[:name] = v } + p.on("--mode MODE", CATALOG_MODES, "replace (the default) makes the catalog the document; merge keeps what it leaves out") do |v| + arguments[:mode] = v + end p.on("--preview", "Show what the import would change, without applying it") { preview = true } end return EXIT_SUCCESS unless argv path = require_argument(argv, "FILE", "catalogs import FILE") - unavailable("catalogs import --preview", "POST /api/scenario_catalogs/preview") if preview - document = begin - File.read(path) - rescue Errno::ENOENT, Errno::EACCES, Errno::EISDIR => e - raise Error.new("Cannot read #{path} (#{e.class})", exit_code: EXIT_USAGE) - end + arguments[:document] = read_text_file(path) + return catalogs_preview(arguments) if preview - payload = client.tool("catalogs_import", document: document, name: name) + payload = client.tool("catalogs_import", arguments) emit(payload) do catalog = payload.fetch("catalog", {}) @out.puts "Imported catalog #{catalog["key"]} (#{catalog["product_count"]} products, #{catalog["scenario_count"]} scenarios)." @@ -59,12 +61,51 @@ def catalogs_import(argv) end end + def catalogs_preview(arguments) + payload = client.tool("catalogs_preview", arguments) + emit(payload) do + catalog = payload.fetch("catalog", {}) + # The catalog's own action is about its record (name, description); + # what its products, sets and scenarios would do is the change list. + verdict = { "create" => "created", "update" => "updated" }.fetch(catalog["action"], "kept as it is") + @out.puts "Preview (#{payload["mode"]}): catalog #{catalog["key"]} would be #{verdict}; " \ + "#{payload["scenario_count"]} scenarios in the document." + counts = payload.fetch("counts", {}).reject { |action, _| action == "unchanged" } + @out.puts "Changes: #{counts.empty? ? "none" : counts.map { |action, count| "#{count} #{action}" }.join(", ")}" + table(Array(payload["changes"]), { + "ACTION" => ->(c) { c["action"] }, + "LEVEL" => ->(c) { c["level"] }, + "PATH" => ->(c) { c["path"] } + }, empty: "Nothing would change.") + end + end + + # Writes a catalog's canonical document from the dashboard's records: + # the whole catalog, or one product or set of it. The same document + # the Catalogs page's Export YAML downloads; --format json writes it + # as JSON instead. def catalogs_export(argv) - argv = parse(argv, "catalogs export KEY [-o FILE]") { |p| p.on("-o", "--output FILE", "Write to FILE") } + arguments = {} + output = nil + argv = parse(argv, "catalogs export KEY [--product KEY [--set KEY]] [--format yaml|json] [-o FILE]") do |p| + p.on("--product KEY", "Only this product") { |v| arguments[:product] = v } + p.on("--set KEY", "Only this set of --product") { |v| arguments[:set] = v } + p.on("--format FORMAT", CATALOG_FORMATS, "yaml (the default; json when FILE ends in .json) or json") { |v| arguments[:format] = v } + p.on("-o", "--output FILE", "Write to FILE instead of standard output") { |v| output = v } + end return EXIT_SUCCESS unless argv - require_argument(argv, "KEY", "catalogs export KEY") - unavailable("catalogs export", "GET /api/scenario_catalogs/:id/export") + arguments[:catalog] = require_argument(argv, "KEY", "catalogs export KEY") + arguments[:format] ||= output.to_s.end_with?(".json") ? "json" : "yaml" + payload = client.tool("catalogs_export", arguments) + if @options[:json] + @out.puts JSON.pretty_generate(payload) + return EXIT_SUCCESS + end + + write_output(payload["document"].to_s, output) do + "Wrote catalog #{payload.dig("catalog", "key")} (#{payload["format"]}) to #{output}." + end end # The agents the key reaches, from the facade's agent:// resources. @@ -82,54 +123,30 @@ def agents_list(argv) end end + # An agent's versions, newest first, with what each changed. def agents_versions(argv) - argv = parse(argv, "agents versions AGENT") - return EXIT_SUCCESS unless argv - - require_argument(argv, "AGENT", "agents versions AGENT") - unavailable("agents versions", "GET /api/agents/:id/versions") - end - - def traces_list(argv) arguments = {} - argv = parse(argv, "traces list [--agent NAME] [--since 24h] [--status error|ok] [--limit N]") do |p| - p.on("--agent AGENT", "An agent class name or a dashboard agent's slug") { |v| arguments[:agent] = v } - p.on("--since DURATION", "How far back: minutes, or 30m, 6h, 7d (default 24h)") { |v| arguments[:since_minutes] = since_minutes(v) } - p.on("--status STATUS", %w[error ok], "Only failed (error) or successful (ok) traces") { |v| arguments[:status] = v } - p.on("--service NAME", "Only this reporting service") { |v| arguments[:service] = v } - p.on("--limit N", "At most N traces (max 100)") { |v| arguments[:limit] = positive_integer(v, "--limit") } + argv = parse(argv, "agents versions AGENT [--limit N]") do |p| + p.on("--limit N", "At most N versions (max 50)") { |v| arguments[:limit] = positive_integer(v, "--limit") } end return EXIT_SUCCESS unless argv - raise Error.new("unexpected arguments: #{argv.join(" ")}", exit_code: EXIT_USAGE) unless argv.empty? - payload = client.tool("traces_search", arguments) - traces = payload.fetch("traces", []) + arguments[:agent] = require_argument(argv, "AGENT (a slug or id)", "agents versions AGENT") + payload = client.tool("agents_versions", arguments) emit(payload) do - table(traces, { - "TRACE" => ->(t) { t["short_id"] }, - "TIME" => ->(t) { t["timestamp"] }, - "AGENT" => ->(t) { [ t["agent"], t["action"] ].compact.join("#") }, - "MODEL" => ->(t) { t["model"] }, - "STATUS" => ->(t) { t["status"] }, - "MS" => ->(t) { t["duration_ms"] }, - "TOKENS" => ->(t) { t.dig("tokens", "total") } - }, empty: "No traces in the last #{payload["window_minutes"]} minutes.") - @out.puts "(more traces match; raise --limit or narrow the filters)" if payload["truncated"] + agent = payload.fetch("agent", {}) + @out.puts "#{agent["name"]} (#{agent["slug"]}): #{agent["version_count"]} versions" + table(payload.fetch("versions", []), { + "VERSION" => ->(v) { v["version_number"] }, + "CREATED" => ->(v) { v["created_at"] }, + "BY" => ->(v) { v["created_by"] }, + "RELEASE" => ->(v) { v["release"] ? (v["revision"] || v["release_digest"].to_s[0, 12]) : nil }, + "SUMMARY" => ->(v) { v["change_summary"] }, + "CHANGED" => ->(v) { v["diff"].is_a?(Hash) ? v["diff"].keys.join(", ") : nil } + }, empty: "No versions.") + @out.puts "(more versions; raise --limit)" if payload["truncated"] end end - - # Minutes in a duration such as 90, 30m, 6h or 7d; seconds round up - # to a whole minute. - def since_minutes(value) - match = DURATION.match(value.to_s.strip.downcase) - raise Error.new("--since must be minutes or a duration such as 30m, 6h or 7d", exit_code: EXIT_USAGE) unless match - - amount = match[1].to_i - minutes = match[2] == "s" ? (amount / 60.0).ceil : amount * DURATION_MINUTES.fetch(match[2]) - raise Error.new("--since must be at least a minute", exit_code: EXIT_USAGE) unless minutes.positive? - - minutes - end end include ResourceCommands diff --git a/lib/active_agent/cli/trace_commands.rb b/lib/active_agent/cli/trace_commands.rb new file mode 100644 index 000000000..ca7b152a8 --- /dev/null +++ b/lib/active_agent/cli/trace_commands.rb @@ -0,0 +1,129 @@ +# frozen_string_literal: true + +module ActiveAgent + class CLI + # `activeagent traces ...`. list and get read through the MCP facade's + # trace tools; export reads one trace whole as the payload trace ingest + # takes (traces_export), and import posts such a payload to + # /api/traces with the ingest key, so a trace moves between + # dashboards the way a run report does with `evals report` and `evals + # import`. + module TraceCommands + DURATION = /\A(\d+)([smhd]?)\z/ + DURATION_MINUTES = { "" => 1, "m" => 1, "h" => 60, "d" => 24 * 60 }.freeze + + private + + def traces_list(argv) + arguments = {} + argv = parse(argv, "traces list [--agent NAME] [--since 24h] [--status error|ok] [--limit N]") do |p| + p.on("--agent AGENT", "An agent class name or a dashboard agent's slug") { |v| arguments[:agent] = v } + p.on("--since DURATION", "How far back: minutes, or 30m, 6h, 7d (default 24h)") { |v| arguments[:since_minutes] = since_minutes(v) } + p.on("--status STATUS", %w[error ok], "Only failed (error) or successful (ok) traces") { |v| arguments[:status] = v } + p.on("--service NAME", "Only this reporting service") { |v| arguments[:service] = v } + p.on("--limit N", "At most N traces (max 100)") { |v| arguments[:limit] = positive_integer(v, "--limit") } + end + return EXIT_SUCCESS unless argv + raise Error.new("unexpected arguments: #{argv.join(" ")}", exit_code: EXIT_USAGE) unless argv.empty? + + payload = client.tool("traces_search", arguments) + traces = payload.fetch("traces", []) + emit(payload) do + table(traces, { + "TRACE" => ->(t) { t["short_id"] }, + "TIME" => ->(t) { t["timestamp"] }, + "AGENT" => ->(t) { [ t["agent"], t["action"] ].compact.join("#") }, + "MODEL" => ->(t) { t["model"] }, + "STATUS" => ->(t) { t["status"] }, + "MS" => ->(t) { t["duration_ms"] }, + "TOKENS" => ->(t) { t.dig("tokens", "total") } + }, empty: "No traces in the last #{payload["window_minutes"]} minutes.") + @out.puts "(more traces match; raise --limit or narrow the filters)" if payload["truncated"] + end + end + + # One trace as the dashboard's trace page shows it: its spans, tool + # calls and errors, with long values cut (traces_get). + def traces_get(argv) + argv = parse(argv, "traces get TRACE") + return EXIT_SUCCESS unless argv + + trace_id = require_argument(argv, "TRACE (a trace id, or its first 8 characters)", "traces get TRACE") + payload = client.tool("traces_get", trace_id: trace_id) + emit(payload) do + @out.puts "Trace #{payload["trace_id"]}: #{[ payload["agent"], payload["action"] ].compact.join("#")} " \ + "on #{payload["model"] || "-"} (#{payload["status"]}, #{payload["duration_ms"]} ms, " \ + "#{payload.dig("tokens", "total")} tokens)" + @out.puts "Error: #{payload["error"]}" if payload["error"] + table(Array(payload["spans"]), { + "SPAN" => ->(s) { s["span_id"] }, + "TYPE" => ->(s) { s["type"] }, + "NAME" => ->(s) { s["name"] }, + "START" => ->(s) { s["start"] }, + "MS" => ->(s) { s["duration"] }, + "STATUS" => ->(s) { s["status"] } + }, empty: "No spans.") + omitted = payload["spans_omitted"].to_i + @out.puts "(#{omitted} more spans; `activeagent traces export #{payload["trace_id"]}` has them all)" if omitted.positive? + end + end + + # Writes the trace, whole, as the payload POST /api/traces + # ingests ({ traces: [trace], sdk }), which `traces import` sends. + def traces_export(argv) + output = nil + argv = parse(argv, "traces export TRACE [-o FILE]") do |p| + p.on("-o", "--output FILE", "Write to FILE instead of standard output") { |v| output = v } + end + return EXIT_SUCCESS unless argv + + trace_id = require_argument(argv, "TRACE (a trace id, or its first 8 characters)", "traces export TRACE") + payload = client.tool("traces_export", trace_id: trace_id) + write_output(JSON.pretty_generate(payload), output) do + "Wrote trace #{payload.dig("traces", 0, "trace_id")} to #{output}." + end + end + + # Posts a saved trace payload to the dashboard's trace ingest with the + # ingest key. FILE is what `traces export` writes, or one trace object, + # or the batch a reporter would send. A trace_id the dashboard already + # holds is skipped there, so importing the same file twice stores + # nothing more. + def traces_import(argv) + argv = parse(argv, "traces import FILE") + return EXIT_SUCCESS unless argv + + path = require_argument(argv, "FILE", "traces import FILE") + document = read_json_file(path) + payload = document.key?("traces") ? document : { "traces" => [ document ] } + traces = payload["traces"] + unless traces.is_a?(Array) && traces.any? && traces.all? { |trace| trace.is_a?(Hash) && trace["trace_id"].is_a?(String) } + raise Error.new("#{path} holds no traces: it is the { \"traces\": [...] } payload `traces export` writes, " \ + "or one trace with a trace_id", exit_code: EXIT_USAGE) + end + payload["sdk"] = { "name" => "activeagent-cli", "version" => self.class.version } unless payload["sdk"].is_a?(Hash) && !payload["sdk"].empty? + + client.post("/api/traces", JSON.generate(payload)) + trace_ids = traces.map { |trace| trace["trace_id"] } + emit(accepted: true, trace_ids: trace_ids, endpoint: client.traces_endpoint) do + @out.puts "Sent #{trace_ids.size} trace#{"s" unless trace_ids.size == 1} to #{client.traces_endpoint}: #{trace_ids.join(", ")}." + end + end + + # Minutes in a duration such as 90, 30m, 6h or 7d; seconds round up + # to a whole minute. + def since_minutes(value) + match = DURATION.match(value.to_s.strip.downcase) + raise Error.new("--since must be minutes or a duration such as 30m, 6h or 7d", exit_code: EXIT_USAGE) unless match + + amount = match[1].to_i + minutes = match[2] == "s" ? (amount / 60.0).ceil : amount * DURATION_MINUTES.fetch(match[2]) + raise Error.new("--since must be at least a minute", exit_code: EXIT_USAGE) unless minutes.positive? + + minutes + end + end + + include TraceCommands + end +end diff --git a/test/cli/cli_test.rb b/test/cli/cli_test.rb index e098467be..9adb9eb76 100644 --- a/test/cli/cli_test.rb +++ b/test/cli/cli_test.rb @@ -65,7 +65,7 @@ def stub_tool(name, *responses, arguments: nil, error: nil) def run_payload(status, passed: 0, evaluated: 0, results: []) { evaluation: { id: 7, name: "Orders" }, - run: { id: 41, status: status, samples_evaluated: evaluated, samples_passed: passed }, + run: { id: 41, run_id: "dashboard-run-41-20261022100000", status: status, samples_evaluated: evaluated, samples_passed: passed }, results: results } end @@ -215,13 +215,15 @@ def test_evals_list_prints_a_table_or_json def test_evals_runs_resolves_an_evaluation_by_name stub_tool("evaluations_list", { evaluations: [ { id: 7, name: "Orders" }, { id: 8, name: "Refunds" } ] }) stub_tool("evaluations_get", { evaluation: { id: 7, name: "Orders" }, - runs: [ { id: 41, number: 2, status: "complete", samples_evaluated: 4, samples_passed: 4 } ] }, + runs: [ { id: 41, number: 2, status: "complete", samples_evaluated: 4, samples_passed: 4, + run_id: "dashboard-run-41-20261022100000" } ] }, arguments: { "evaluation_id" => 7 }) code, out, = cli("evals", "runs", "Orders") assert_equal 0, code, out - assert_match(/^41\s+2\s+complete\s+4\/4/, out) + assert_match(/^RUN\s+#\s+STATUS\s+PASSED\s+SCORE\s+CREATED\s+RUN_ID$/, out) + assert_match(/^41\s+2\s+complete\s+4\/4\s+-\s+-\s+dashboard-run-41-20261022100000$/, out) end def test_evals_run_without_wait_returns_once_started @@ -244,6 +246,7 @@ def test_evals_run_waits_and_passes_the_gate assert_equal 0, code, out assert_equal [ 2 ], @sleeps assert_includes out, "(90.0%), threshold 80.0%" + assert_includes out, "Report: activeagent evals report dashboard-run-41-20261022100000" assert_match(/PASS$/, out) end @@ -313,11 +316,25 @@ def test_evals_report_defaults_to_markdown_on_a_terminal_and_json_otherwise assert_equal envelope, JSON.parse(cli("evals", "report", "run-1")[1]) end - def test_evals_report_without_the_export_endpoint_is_unavailable - stub_request(:get, "#{REPORTS}/run-1").to_return(status: 404, body: "") + def test_evals_report_tells_a_missing_run_from_a_dashboard_without_the_export + # The collector's own 404 names the run. + stub_request(:get, "#{REPORTS}/run-1").to_return(status: 404, body: { error: "No evaluation report with that run_id" }.to_json) code, _, err = cli("evals", "report", "run-1") - assert_equal CLI::EXIT_UNAVAILABLE, code - assert_includes err, "GET /api/evaluation_reports/:run_id" + assert_equal CLI::EXIT_FAILURE, code + assert_includes err, "No report for run run-1; `activeagent evals runs EVALUATION` shows each run's RUN_ID (No evaluation report with that run_id)" + + # An older engine has no such route: Rails answers for it. + [ "", { status: 404, error: "Not Found" }.to_json ].each do |body| + stub_request(:get, "#{REPORTS}/run-1").to_return(status: 404, body: body) + code, _, err = cli("evals", "report", "run-1") + assert_equal CLI::EXIT_UNAVAILABLE, code, body + assert_includes err, "GET /api/evaluation_reports/:run_id" + end + + stub_request(:get, "#{REPORTS}/run-1").to_return(status: 422, body: { error: "Only a complete run can be exported" }.to_json) + code, _, err = cli("evals", "report", "run-1") + assert_equal CLI::EXIT_FAILURE, code + assert_includes err, "HTTP 422 (Only a complete run can be exported)" end def test_evals_report_refuses_a_body_that_is_not_an_envelope @@ -388,16 +405,137 @@ def test_catalogs_list_and_import assert_includes out, "support -> Support: orders (4)" end - def test_commands_without_an_api_key_route_say_so + def test_catalogs_import_preview_shows_the_changes_without_importing path = File.join(@dir, "shop.yml") - File.write(path, "catalog: shop\n") + File.write(path, "catalog: shop\nproducts: []\n") + stub_tool("catalogs_preview", { mode: "merge", catalog: { key: "shop", action: "update" }, scenario_count: 4, + counts: { "create" => 1, "unchanged" => 3 }, + changes: [ { level: "scenario", path: "support/orders/refund_2", action: "create" } ] }, + arguments: { "document" => "catalog: shop\nproducts: []\n", "mode" => "merge" }) - [ %w[catalogs export shop], [ "catalogs", "import", path, "--preview" ], %w[agents versions support] ].each do |argv| - code, _, err = cli(*argv) - assert_equal CLI::EXIT_UNAVAILABLE, code, argv.inspect - assert_includes err, "not yet available on this dashboard" - end - assert_not_requested :post, MCP + code, out, = cli("catalogs", "import", path, "--preview", "--mode", "merge") + + assert_equal 0, code, out + assert_includes out, "Preview (merge): catalog shop would be updated; 4 scenarios in the document." + assert_includes out, "Changes: 1 create" + assert_match(%r{^create\s+scenario\s+support/orders/refund_2$}, out) + assert_not_requested(:post, MCP) { |request| JSON.parse(request.body).dig("params", "name") == "catalogs_import" } + assert_equal CLI::EXIT_USAGE, cli("catalogs", "import", path, "--mode", "sideways").first + end + + def test_catalogs_export_writes_the_document_as_yaml_or_json + stub_tool("catalogs_export", { catalog: { key: "shop" }, format: "yaml", document: "catalog: shop\nproducts: []\n" }, + arguments: { "catalog" => "shop", "format" => "yaml" }) + stub_tool("catalogs_export", { catalog: { key: "shop" }, format: "json", product: "support", document: "{\n \"catalog\": \"shop\"\n}" }, + arguments: { "catalog" => "shop", "format" => "json", "product" => "support" }) + + assert_equal [ 0, "catalog: shop\nproducts: []\n", "" ], cli("catalogs", "export", "shop") + + # A .json file asks for JSON, and --json prints the tool's whole answer. + path = File.join(@dir, "shop.json") + code, out, err = cli("catalogs", "export", "shop", "--product", "support", "-o", path) + assert_equal 0, code, err + assert_equal "", out + assert_equal({ "catalog" => "shop" }, JSON.parse(File.read(path))) + assert_includes err, "Wrote catalog shop (json) to #{path}." + assert_equal "support", JSON.parse(cli("catalogs", "export", "shop", "--product", "support", "--format", "json", "--json")[1])["product"] + end + + def test_agents_versions_lists_each_version_with_what_it_changed + stub_tool("agents_versions", { agent: { id: 3, slug: "support", name: "Support", version_count: 2 }, + versions: [ { version_number: 2, created_at: "2026-10-22T10:00:00Z", created_by: "ops", release: true, + revision: "abc1234", change_summary: "Deploy", diff: { "model" => { "from" => "a", "to" => "b" } } }, + { version_number: 1, created_at: "2026-10-21T10:00:00Z", release: false, change_summary: "Initial version" } ], + truncated: true }, + arguments: { "agent" => "support", "limit" => 2 }) + + code, out, = cli("agents", "versions", "support", "--limit", "2") + + assert_equal 0, code, out + assert_includes out, "Support (support): 2 versions" + assert_match(/^2\s+2026-10-22T10:00:00Z\s+ops\s+abc1234\s+Deploy\s+model$/, out) + assert_match(/^1\s+2026-10-21T10:00:00Z\s+-\s+-\s+Initial version\s+-$/, out) + assert_includes out, "raise --limit" + end + + def test_traces_get_prints_the_trace_and_its_spans + stub_tool("traces_get", { trace_id: "abcd1234ef", agent: "SupportAgent", action: "reply", model: "gpt-5-mini", status: "ERROR", + duration_ms: 120.5, tokens: { total: 30 }, error: "boom", + spans: [ { span_id: "root", type: "root", name: "SupportAgent.reply", start: 0, duration: 120.5, status: "ERROR" } ], + spans_total: 101, spans_omitted: 100 }, + arguments: { "trace_id" => "abcd1234" }) + + code, out, = cli("traces", "get", "abcd1234") + + assert_equal 0, code, out + assert_includes out, "Trace abcd1234ef: SupportAgent#reply on gpt-5-mini (ERROR, 120.5 ms, 30 tokens)" + assert_includes out, "Error: boom" + assert_match(/^root\s+root\s+SupportAgent\.reply\s+0\s+120\.5\s+ERROR$/, out) + assert_includes out, "(100 more spans; `activeagent traces export abcd1234ef` has them all)" + end + + def test_traces_export_writes_the_ingest_payload + payload = { traces: [ { trace_id: "abcd1234ef", service_name: "app", timestamp: "2026-10-22T10:00:00.000000Z", spans: [] } ], + sdk: { name: "activeagent", version: "1.9.3" } } + stub_tool("traces_export", payload, arguments: { "trace_id" => "abcd1234ef" }) + path = File.join(@dir, "trace.json") + + code, out, err = cli("traces", "export", "abcd1234ef", "-o", path) + + assert_equal 0, code, err + assert_equal "", out + assert_equal JSON.parse(payload.to_json), JSON.parse(File.read(path)) + assert_includes err, "Wrote trace abcd1234ef to #{path}." + assert_equal JSON.parse(payload.to_json), JSON.parse(cli("traces", "export", "abcd1234ef")[1]) + end + + def test_traces_import_posts_the_payload_with_the_ingest_key + payload = { "traces" => [ { "trace_id" => "abcd1234ef", "service_name" => "app", "spans" => [] } ], + "sdk" => { "name" => "activeagent", "version" => "1.9.3" } } + path = File.join(@dir, "trace.json") + File.write(path, payload.to_json) + request = stub_request(:post, "#{URL}/api/traces").with(headers: { "Authorization" => "Bearer #{INGEST_KEY}" }, body: payload) + .to_return(status: 202, body: "") + + code, out, = cli("traces", "import", path) + + assert_equal 0, code, out + assert_includes out, "Sent 1 trace to #{URL}/api/traces: abcd1234ef." + assert_requested request + + # One bare trace is wrapped, and sent as this CLI's own SDK. + File.write(path, payload["traces"].first.to_json) + bare = stub_request(:post, "#{URL}/api/traces").with do |r| + body = JSON.parse(r.body) + body["traces"] == payload["traces"] && body.dig("sdk", "name") == "activeagent-cli" + end.to_return(status: 202) + code, out, = cli("traces", "import", path, "--json") + assert_equal 0, code, out + assert_equal [ "abcd1234ef" ], JSON.parse(out)["trace_ids"] + assert_requested bare + + File.write(path, { "nope" => true }.to_json) + code, _, err = cli("traces", "import", path) + assert_equal CLI::EXIT_USAGE, code + assert_includes err, "holds no traces" + end + + def test_traces_import_reports_a_refusal_without_the_key + path = File.join(@dir, "trace.json") + File.write(path, { "traces" => [ { "trace_id" => "x" } ] }.to_json) + stub_request(:post, "#{URL}/api/traces").to_return(status: 401, body: { error: "Invalid API key #{INGEST_KEY}" }.to_json) + + code, _, err = cli("traces", "import", path) + + assert_equal CLI::EXIT_FAILURE, code + assert_includes err, "refused the ingest key" + assert_includes err, "[FILTERED]" + refute_includes err, INGEST_KEY + + stub_request(:post, "#{URL}/api/traces").to_return(status: 429, body: { error: "Trace ingest limit reached" }.to_json) + code, _, err = cli("traces", "import", path) + assert_equal CLI::EXIT_FAILURE, code + assert_includes err, "HTTP 429 (Trace ingest limit reached)" end def test_agents_list_reads_the_agent_resources From 49ccd97e995a949ad5b4d519eea4995e40d3e21a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 15:13:08 +0000 Subject: [PATCH 3/4] Document the command line where the docs describe the same work by rake or UI AGENTS.md gains the CLI and the evals library in its quick reference, a section on the command line as the gem's third surface (what it may load, which endpoint each command answers through, the rule that a dashboard feature ships with its CLI command, API-key endpoint and UI together, how a command and a facade tool are added, where the tests and docs live), the facade's dashboard tools and the run_id every run summary carries, the CLI test files, and the placeholder provider keys a full local run needs. The gem and engine READMEs, the docs index, the dashboard, evaluations, self-hosted observability and telemetry pages point at the CLI commands beside the rake tasks, downloads and ingest routes they already describe, and the run fields list run_id. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01RSuPEbVwjxwJub8eAmGM9N --- AGENTS.md | 76 +++++++++++++++++++++ README.md | 8 ++- actionagent/README.md | 10 ++- docs/framework/dashboard.md | 10 ++- docs/framework/evaluations.md | 14 +++- docs/framework/self-hosted-observability.md | 15 ++-- docs/framework/telemetry.md | 5 +- docs/index.md | 4 ++ 8 files changed, 130 insertions(+), 12 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7ee842836..5c16360a4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,6 +24,8 @@ Keep each private customer's material within its own organization context. | Agent concerns/mixins | `lib/active_agent/concerns/` | | Agent-as-tool delegation | `lib/active_agent/delegation/` | | Rails generators | `lib/generators/active_agent/` | +| Scenario evaluations (parser, runner, scorer, report, publisher) | `lib/active_agent/evals/` | +| Command line (`activeagent` executable) | `exe/activeagent`, `lib/active_agent/cli/` | | Dashboard engine (`actionagent` gem) | `actionagent/` | | Dashboard React source | `actionagent/frontend/` | | Test suite | `test/` | @@ -283,6 +285,9 @@ bin/test test/path/to/test.rb # Run tests for a specific provider bin/test test/integration/open_ai/ +# The command line: standalone (no Rails loaded), then against the engine's dummy app +bin/test test/cli/cli_test.rb actionagent/test/cli_contract_test.rb + # Run both trees — the framework's test/ and the engine's actionagent/test/ bundle exec rake test ``` @@ -290,6 +295,13 @@ bundle exec rake test The engine's tests live in `actionagent/test/`. Both `bin/test` and the Rakefile's `test` task collect the framework and engine suites by default. +A full `bin/test` with no `.env.test` needs the placeholder provider keys CI +sets (`ANTHROPIC_API_KEY=ANTHROPIC_API_KEY OPEN_AI_API_KEY=OPEN_AI_API_KEY +OPEN_ROUTER_API_KEY=OPEN_ROUTER_API_KEY`); without them every provider test +whose client refuses to initialize without a key errors, a few hundred of +them, before any request is stubbed or replayed. The suite's own exit code +is the signal: piping it through `tail` hides it. + ### Test Fixtures - VCR cassettes in `test/fixtures/vcr_cassettes/` @@ -375,6 +387,65 @@ cycle. (Both gemspecs declare `railties`; that one is not part of the split.) (authentication, `current_user_resolver`, `multi_tenant`, `table_name_prefix`, `execution_enabled`, sandbox backends, quotas); all are optional and unset means single-user self-hosted behaviour. +- The MCP facade (`Api::MCPController`, `POST /mcp`, a dashboard API + key as the Bearer token) also serves the dashboard's own tools + (`Api::MCPDashboardTools`: `evaluations_*`, `evaluation_runs_*`, + `scenarios_merge`, `explorations_submit`, `traces_search|get|export`, + `catalogs_list|import|preview|export`, `catalog_set_run`, + `agents_versions`, `input_requests_*`). A new one is added in four + places — `NAMES`, `DEFINITIONS` (its JSON schema), + `dispatch_dashboard_tool`, and `DASHBOARD_TOOLS` in + `actionagent/test/mcp_dashboard_tools_test.rb` — plus the tools table in + `docs/framework/dashboard.md`. A tool reads under the key's owner exactly + as the JSON API reads for a signed-in owner (`owned`, `owner_agents`, + `owned_traces`), raises `ToolError` for anything the client can correct, + and every result is scrubbed of the owner's credentials before it leaves. +- Every serialized evaluation run (`EvaluationSerializer.run_summary`) + carries `run_id`, the handle `GET /api/evaluation_reports/:run_id` + takes: a published run's own, a dashboard run's + `dashboard-run--` (`EvaluationReportExport.run_id_for`). + +## The command line: the gem's third surface + +`exe/activeagent` (`lib/active_agent/cli.rb` and `lib/active_agent/cli/`) is +the `activeagent` executable: it works with a dashboard over HTTP from a +terminal or a CI job, and ships in the `activeagent` gem. Read +`docs/framework/cli.md` first. The shape to keep: + +- It loads neither Rails nor the framework: stdlib `OptionParser` and + `Net::HTTP` only, plus `lib/active_agent/evals/publisher.rb`, which is + standalone too. `test/cli/cli_test.rb` requires the CLI alone, so a + `require "active_support"` (or anything else from the framework) under + `lib/active_agent/cli/` fails that test. +- Every command answers through an API-key-authenticated endpoint, never a + session route: reads and runs through the facade's dashboard tools + (`Client#tool`, the dashboard API key); imports and the report export + through the endpoints the framework's own reporters post to + (`/api/evaluation_reports`, `/api/traces`; `Client#get` and + `Client#post`, the ingest key). A dashboard that offers neither exits + `EXIT_UNAVAILABLE` (5), never scrapes the UI's session API. +- A dashboard feature ships with its CLI command, its API-key endpoint and + its UI together (#619); the rake tasks in `actionagent/lib/tasks/` stay + for in-app and offline use. When a command needs a read the facade lacks, + add a dashboard tool (above) rather than a controller of its own. +- Adding a command: a private `_` method in the group's module + (`cli/eval_commands.rb`, `cli/resource_commands.rb`, + `cli/trace_commands.rb`, `cli/session_commands.rb`), named in the group's + list in `CLI#dispatch`. `parse` handles the options and `--help`, `emit` + prints `--json` or yields the table form, `write_output` writes a + document to `-o FILE` or standard output, and a failure is + `CLI::Error.new(message, exit_code:)`. Server messages reach the terminal + only through `Client#sanitize`, which filters both keys out; never print + a response body raw. +- Tests: `test/cli/cli_test.rb` stubs the dashboard with WebMock and runs + the CLI in process (`CLI.start(argv, out:, err:, env:, credentials_path:)` + returns the exit code); `actionagent/test/cli_contract_test.rb` hands the + same requests to the dummy app (`stub_request(...).to_rack(Rails.application)`), + so a change to the facade or the collector that breaks the CLI fails + there, not on a user's machine. +- Documentation: `docs/framework/cli.md` is the page (sidebar: Framework → + Command Line), with the facade's tools table in + `docs/framework/dashboard.md` and the run export in `docs/evals/publication.md`. ## Common Gotchas @@ -410,6 +481,11 @@ bin/test # Lint bin/rubocop +# The command line, from this checkout (`activeagent` once the gem is installed) +exe/activeagent --help +exe/activeagent login --url http://localhost:3000/activeagents --key aa_... +exe/activeagent evals run "Support orders" --min-pass-rate 0.8 + # Cross-repo: this checkout against a local solid_agent (strict = no skips) SOLID_AGENT_PATH=../solid_agent \ BUNDLE_GEMFILE=gemfiles/solid_agent_main.gemfile \ diff --git a/README.md b/README.md index fdb065dcc..9c4e4c86b 100644 --- a/README.md +++ b/README.md @@ -181,7 +181,12 @@ telemetry: The generator mounts the engine at `/activeagents` — open it and every generation appears as a trace. See [docs/framework/dashboard.md](docs/framework/dashboard.md) for -authentication, remote ingestion, and multi-tenant mode. The hosted +authentication, remote ingestion, and multi-tenant mode. The gem's +`activeagent` command line works with that dashboard from a terminal or a +CI job — `activeagent evals run "Support orders" --min-pass-rate 0.8` fails +a build below the threshold, and runs, scenario catalogs and traces move +between installs as JSON files; see +[docs/framework/cli.md](docs/framework/cli.md). The hosted platform at [activeagents.ai](https://activeagents.ai) runs this same engine multi-tenant, adding what a hosted product has to have — accounts, plans, billing, quotas and managed sandboxes; every workspace starts with @@ -198,6 +203,7 @@ a free low-volume trial. - **Agent-as-Tool Delegation**: Hand work to sub-agents with declared schemas, cost/latency budgets, and swappable backends - **Context Management**: Maintain conversation history across interactions - **Structured Output**: Define JSON schemas for predictable responses +- **Command Line**: `activeagent` runs evaluations and gates CI on a pass rate from a terminal, and moves runs, scenario catalogs and traces between dashboards as JSON ## Examples diff --git a/actionagent/README.md b/actionagent/README.md index bdb2d5805..428df5eda 100644 --- a/actionagent/README.md +++ b/actionagent/README.md @@ -112,7 +112,7 @@ and `?download=1` an attachment. A signed-in session reads it at `/api/evaluations/:id/runs/:run_id/report.json`, the HTML page at `report`, and GitHub-flavored Markdown at `report.md`, each downloadable with -`?download=1`. From the command line: +`?download=1`. Inside the host app: ```bash bin/rails action_agent:evaluations:export RUN= FORMAT=json|html|md [FILE=path] @@ -120,6 +120,14 @@ bin/rails action_agent:evaluations:export EVALUATION= RUN= FORMAT=md bin/rails action_agent:evaluations:import FILE=run.json [ACCOUNT_ID=] ``` +From any machine with the `activeagent` gem, over HTTP with the ingest key +(see [Command Line](https://docs.activeagents.ai/framework/cli)): + +```bash +activeagent evals report --format json -o run.json # or html, md +activeagent evals import run.json +``` + See [Publishing an externally executed evaluation](https://docs.activeagents.ai/evals/publication) for the envelope, the run_id a dashboard run exports under, and every response. diff --git a/docs/framework/dashboard.md b/docs/framework/dashboard.md index bc3e53fc2..410da58bc 100644 --- a/docs/framework/dashboard.md +++ b/docs/framework/dashboard.md @@ -651,7 +651,9 @@ the unframed page, which is the copy to archive next to a CI run, and page, the run as the evaluation report envelope another install imports (see [Exporting a stored run](/evals/publication#exporting-a-stored-run)), and the report as GitHub-flavored Markdown. The suite's run header carries -the same downloads. +the same downloads, and `activeagent evals report RUN --format html|json|md` +writes the same three from a terminal, by the run's `run_id` (see +[Command Line](/framework/cli#reports)). **Copy as Markdown** puts a piece of the run on the clipboard for an issue or a pull request: the whole report (the run report view and the run @@ -767,8 +769,10 @@ leaves archived evaluations out unless `?archived=1`, returns `archived_count`, and gives each evaluation its `standing`, `headline_run_id`, `archived_at` and `per_model` passes, plus the `headline_run` in full when a newer run is pending or failed; every run -carries its `agent_version` (`id`, `number`, `release_digest`, `revision`, -`release`) and `version_state` (`current`, `earlier` or `unrecorded`). +carries its `run_id` (the handle `GET /api/evaluation_reports/:run_id` +and the CLI's `evals report` take), its `agent_version` (`id`, `number`, +`release_digest`, `revision`, `release`) and `version_state` (`current`, +`earlier` or `unrecorded`). ## Scenario catalogs diff --git a/docs/framework/evaluations.md b/docs/framework/evaluations.md index 99e7e9a9a..534a7a869 100644 --- a/docs/framework/evaluations.md +++ b/docs/framework/evaluations.md @@ -747,7 +747,11 @@ two copies of a catalog can be compared without diffing them. The mounted dashboard stores catalogs in its own tables, writes the canonical document to Active Storage and runs one set at a time as an evaluation of the product's agent; see -[Scenario catalogs](/framework/dashboard#scenario-catalogs). +[Scenario catalogs](/framework/dashboard#scenario-catalogs). The +`activeagent` command line imports a catalog into that dashboard, previews +what an import would change, and writes the stored records back out as the +canonical YAML or as JSON (`catalogs import|export`; see +[Command Line](/framework/cli#scenario-catalogs)). ## Running a host application's agent from the mounted dashboard @@ -820,6 +824,9 @@ schema migration. Existing result and report URLs continue to work: `model=` or `scenario=`, for one piece), `report.json` as the [evaluation report envelope](/evals/publication#exporting-a-stored-run), and `?download=1` downloads any of them as `-run-.html|md|json`. +- From a terminal, `activeagent evals report RUN --format json|html|md` + writes the same three for a run's `run_id`, and `activeagent evals import` + posts the JSON to another install (see [Command Line](/framework/cli#reports)). - `/evaluations/:id/runs/:run_id/report` opens it within the dashboard. - `/evaluations/:id` opens an evaluation on its run history, and `/evaluations/:id/runs/:run_id` one of its runs — a page of its own @@ -828,7 +835,10 @@ schema migration. Existing result and report URLs continue to work: one outside the first index page. Add `&run=:run_id` to open its saved report. Every serialized run carries `number` (its position in the evaluation's -history, oldest first) and `usage`, which keeps the agent's spend apart from +history, oldest first), `run_id` (the handle +`GET /api/evaluation_reports/:run_id` and `activeagent evals report` +take: a published run's own, a dashboard run's +`dashboard-run--`) and `usage`, which keeps the agent's spend apart from the judge's: `replays` (or `samples`, for a sampling run), `cost`, `per_interaction`, tokens and timing on the agent's side — what the interactions cost to serve — and `judge` (`calls`, tokens, `cost`, `model`, diff --git a/docs/framework/self-hosted-observability.md b/docs/framework/self-hosted-observability.md index b722cbd19..7fd9b7547 100644 --- a/docs/framework/self-hosted-observability.md +++ b/docs/framework/self-hosted-observability.md @@ -108,6 +108,11 @@ capture writes through the trace model and issues no HTTP at all, so that path is a label, not a request; apps posting from elsewhere set `endpoint:` to the full URL, as shown below, and `resolved_endpoint` returns that.) +A stored trace moves between installs the way a run does: the `activeagent` +command line's `traces export TRACE` writes it as the payload this route +ingests, and `traces import FILE` posts it to another dashboard with that +dashboard's ingest key (see [Command Line](/framework/cli#traces)). + ```ruby # A path on your main app: mount ActionAgent::Engine => "/activeagents" @@ -236,10 +241,12 @@ Any complete run comes back out the same way: `GET /api/evaluation_reports/:run_id`, with the same key, returns the envelope that endpoint takes, so a run moves to another install with one POST, and posting it back here is an identical retry (200); `?format=html` -and `?format=md` return its report page or Markdown instead. From the command -line, `bin/rails action_agent:evaluations:export` and -`action_agent:evaluations:import` do the same (see -[Exporting a stored run](/evals/publication#exporting-a-stored-run)). +and `?format=md` return its report page or Markdown instead. Inside the host +app, `bin/rails action_agent:evaluations:export` and +`action_agent:evaluations:import` do the same, and from any machine with the +gem, `activeagent evals report` and `activeagent evals import` (see +[Exporting a stored run](/evals/publication#exporting-a-stored-run) and +[Command Line](/framework/cli#reports)). ## RubyLLM applications diff --git a/docs/framework/telemetry.md b/docs/framework/telemetry.md index d302e6528..a9f137a95 100644 --- a/docs/framework/telemetry.md +++ b/docs/framework/telemetry.md @@ -231,7 +231,10 @@ The `actionagent` dashboard engine already implements this endpoint spec: add the gem, mount the engine, and its ingest API (`/api/traces`) receives traces from any app in your fleet — see [Self-Hosted Observability](/framework/self-hosted-observability). The -requirements below are for building your own receiver instead. +`activeagent` command line reads a stored trace back out of that dashboard +as the same payload and posts it to another (`traces export`, `traces +import`; see [Command Line](/framework/cli#traces)). The requirements below +are for building your own receiver instead. ### Endpoint Requirements diff --git a/docs/index.md b/docs/index.md index e5a83d8de..f8106e7e2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -58,6 +58,10 @@ features: icon: 📈 link: /framework/dashboard details: Add the actionagent gem and mount its engine for traces, metrics, the agent builder, conversations and evaluations. + - title: Command Line + icon: ⌨️ + link: /framework/cli + details: Run evaluations and gate CI from a terminal; move runs, scenario catalogs and traces between dashboards as JSON. - title: Background Jobs link: /agents/generation icon: ⏳ From 3b4a4cd53635469d7b3993244696f17156a10640 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 15:17:08 +0000 Subject: [PATCH 4/4] Say what 1.9.4 releases, and which dashboard version each CLI command needs Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01RSuPEbVwjxwJub8eAmGM9N --- CHANGELOG.md | 11 +++++++++++ docs/framework/cli.md | 6 ++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ba403f44..de886acad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +The `activeagent` command line: the gem's third surface beside the dashboard +and its API, working with a dashboard over HTTP from a terminal or a CI job, +and the facade tools it needed. A run, a scenario catalog and a trace each +move between installs as one file. The framework and the dashboard engine +move to 1.9.4 together. + +Upgrading: no migration. The CLI needs `actionagent` 1.9.4 on the dashboard +for `catalogs export`, `catalogs import --preview`, `agents versions` and +`traces export` (the facade's new tools), and 1.9.3 for `evals report` (the +report export); an older dashboard answers those commands with exit code 5. + ### Added - **`activeagent` command line** (`activeagent`, `actionagent`; refs #619). diff --git a/docs/framework/cli.md b/docs/framework/cli.md index 22b2af5ba..82be52849 100644 --- a/docs/framework/cli.md +++ b/docs/framework/cli.md @@ -14,8 +14,10 @@ activeagent status activeagent evals list ``` -It needs `actionagent` 1.9.3 or later on the dashboard, with the facade's -dashboard tools on (the default). An older dashboard, or one that set +It needs `actionagent` 1.9.4 or later on the dashboard, with the facade's +dashboard tools on (the default): `catalogs export`, `catalogs import +--preview`, `agents versions` and `traces export` use tools 1.9.4 added, and +`evals report` the export 1.9.3 added. An older dashboard, or one that set `ActionAgent.mcp_dashboard_tools = false`, answers the commands that need them with exit code 5.