Export evaluation run reports (JSON, HTML, Markdown) and Copy as Markdown

CHANGELOG.md

@@ -21,6 +21,36 @@ engine: the file chooser relay and the per-turn browser tokens are in the
 
 ### Added
 
+- **Export evaluation run reports as JSON, HTML and Markdown** (`activeagent`,
+  `actionagent`). `GET <mount>/api/evaluation_reports/:run_id`, with the
+  ingest key and scoped to the tenant like the POST (and refused on a
+  single-tenant install that set no key), returns a complete run
+  as the version-1 envelope the collector takes, so a run moves to another
+  install with one POST and posting it back is an identical retry (200);
+  `?format=html` returns the run's report page and `?format=md` its
+  Markdown, and `?download=1` an attachment of any of them.
+  An imported run exports under its published `run_id`, a dashboard run
+  under `dashboard-run-<id>-<created_at>`. `ActiveAgent::Evals::Publisher.envelope`
+  builds the envelope for both the publisher and the export. The dashboard's
+  run report adds `report.json` and `report.md` (one piece with
+  `?part=verdict|fix_items|scenario`), and `?download=1` saves any format
+  as `<evaluation>-run-<n>.<ext>`; the run report view and the suite's run
+  header get *Download HTML / JSON / Markdown*. New rake tasks:
+  `action_agent:evaluations:export RUN=… [EVALUATION=…] FORMAT=json|html|md [FILE=…]`
+  and `action_agent:evaluations:import FILE=… [ACCOUNT_ID=…]`.
+
+- **Copy as Markdown** (`activeagent`, `actionagent`). `Report#to_markdown`
+  now has the HTML page's content in GitHub-flavored Markdown: the verdict's
+  rationale, What to fix from `Report#fix_items` (replacing the old
+  Recommendations list), and every scenario's answers in a `<details>` block
+  with a per-model table; table cells escape `|` and line breaks. Its pieces
+  are public — `verdict_markdown`, `fix_items_markdown(model:)`,
+  `fix_item_markdown(item)`, `scenario_markdown(key)` — and the dashboard
+  and the standalone HTML page copy each of them: the whole report, the
+  verdict with the comparison, What to fix for the filtered model, a fix
+  card, and a scenario. The HTML page now carries one inline script for
+  those buttons, with a clipboard fallback.
+
 - **Upload files to a page through the sandbox browser's live view**
   (`actionagent`, `@activeagents/browser-sidecar`). A file chooser the page
   opens while someone drives is answered from the driver's own machine

actionagent/README.md

@@ -100,6 +100,29 @@ the engine logs a warning at boot and the dashboard renders blank with
 (and, for an API-only app, re-enable the asset initializer and middleware) and
 the bundles are served with nothing else to configure.
 
+## Evaluation reports
+
+An application that runs its own evaluations publishes each report to
+`POST <mount>/api/evaluation_reports` with `ActiveAgent::Evals::Publisher`,
+authenticated with the ingest key. Any complete run comes back out as the
+same envelope from `GET <mount>/api/evaluation_reports/:run_id`, so it can be
+posted to another install; posting it back here is an identical retry.
+`?format=html` or `?format=md` returns its report page or Markdown instead,
+and `?download=1` an attachment. A
+signed-in session reads it at
+`<mount>/api/evaluations/:id/runs/:run_id/report.json`, the HTML page at
+`report`, and GitHub-flavored Markdown at `report.md`, each downloadable with
+`?download=1`. From the command line:
+
+```bash
+bin/rails action_agent:evaluations:export RUN=<run_id> FORMAT=json|html|md [FILE=path]
+bin/rails action_agent:evaluations:export EVALUATION=<id> RUN=<id> FORMAT=md
+bin/rails action_agent:evaluations:import FILE=run.json [ACCOUNT_ID=<id>]
+```
+
+See [Publishing an externally executed evaluation](https://docs.activeagents.ai/evals/publication)
+for the envelope, the run_id a dashboard run exports under, and every response.
+
 ## Upgrading from activeagent <= 1.1.0
 
 The dashboard used to live inside the framework gem as `ActiveAgent::Dashboard`.

actionagent/app/controllers/action_agent/api/evaluation_reports_controller.rb

@@ -25,15 +25,28 @@ module Api
     #   401 — a missing or unknown key
     #   501 — an install with no evaluation tables, or not yet migrated
     #   503 — another import for the same agent held the lock too long
+    #
+    # GET <mount>/api/evaluation_reports/:run_id returns a stored run as the
+    # same version-1 envelope (EvaluationReportExport), so it can be posted to
+    # another install, or back to this one as an identical retry. Responds
+    # 200 with the envelope (or, with `format=html` or `format=md`, the
+    # report page or its Markdown), 404 for a run_id the key's tenant holds
+    # no run under, 422 for a run that is not complete or an unknown format,
+    # 401 as above and also on a single-tenant install that has no
+    # ingest_api_key, and 501 as above.
     class EvaluationReportsController < ActionController::API
       include IngestAuthentication
 
+      # What #show serves for each `format`.
+      EXPORT_FORMATS = { "json" => "application/json", "html" => "text/html", "md" => "text/markdown" }.freeze
+
       # New reports one key may store per minute.
       RATE_LIMIT = 30
 
       wrap_parameters false
 
-      before_action :require_json!
+      before_action :require_json!, only: :create
+      before_action :require_export_key!, only: :show
       before_action :require_report_store!
 
       # POST <mount>/api/evaluation_reports
@@ -64,6 +77,41 @@ def create
         render json: { error: e.message }, status: :service_unavailable
       end
 
+      # GET <mount>/api/evaluation_reports/:run_id
+      #
+      # :run_id is an imported run's run_id as it was published, or the
+      # "dashboard-run-<id>-<stamp>" a run the dashboard executed exports
+      # under (EvaluationReportExport.run_id_for). Found within the key's
+      # tenant exactly as #create finds a stored run. `format=html` returns
+      # the run's self-contained report page (text/html) as run_report renders
+      # it, and `format=md` its Markdown (text/markdown), instead of the
+      # envelope; any other format is a 422. `download=1` sends any of them
+      # as an attachment named like the dashboard's downloads.
+      def show
+        run = EvaluationReportExport.find(params[:run_id], account: @account)
+        return render json: { error: "No evaluation report with that run_id" }, status: :not_found unless run
+
+        format = params[:format].presence || "json"
+        unless EXPORT_FORMATS.key?(format)
+          return render json: { error: "format is one of #{EXPORT_FORMATS.keys.join(', ')}" }, status: :unprocessable_entity
+        end
+
+        export = EvaluationReportExport.new(run)
+        export.exportable!
+        body =
+          case format
+          when "json" then export.to_json
+          when "md" then export.markdown
+          # The page run_report serves, its fix actions linking into this
+          # dashboard by absolute path; no theme, so it follows the viewer.
+          when "html" then run.to_report(links: run.report_links(mount: request.script_name)).to_html
+          end
+        send_data body, type: EXPORT_FORMATS.fetch(format), disposition: download? ? "attachment" : "inline",
+                        filename: export.filename(format)
+      rescue EvaluationReportExport::NotExportable => e
+        render json: { error: e.message }, status: :unprocessable_entity
+      end
+
       private
 
       # Rails reads and parses a JSON body into params before any callback
@@ -74,6 +122,18 @@ def process_action(*)
         super
       end
 
+      # A single-tenant install with no ingest key takes reports from anyone
+      # who reaches the mount, which a host keeps off the network. Reading a
+      # run back is different: every stored run, prompts and answers
+      # included, would be open under a guessable run_id. So the export
+      # needs the key to exist; a multi-tenant install authenticates the
+      # tenant's key already.
+      def require_export_key!
+        return if ActionAgent.multi_tenant? || ActionAgent.ingest_api_key.present?
+
+        render json: { error: "Set ActionAgent.ingest_api_key to export evaluation reports" }, status: :unauthorized
+      end
+
       # A cross-site page can send a text/plain or form POST without a CORS
       # preflight; it cannot send application/json.
       def require_json!
@@ -82,6 +142,10 @@ def require_json!
         render json: { error: "Content-Type must be application/json" }, status: :unsupported_media_type
       end
 
+      def download?
+        ActiveModel::Type::Boolean.new.cast(params[:download]) == true
+      end
+
       def require_report_store!
         reason = EvaluationReportImport.unavailable_reason
         render json: { error: reason }, status: :not_implemented if reason
@@ -110,7 +174,7 @@ def receipt(run, duplicate)
         {
           id: run.id,
           evaluation_id: run.evaluation_id,
-          run_id: run.external_run_id,
+          run_id: EvaluationReportExport.run_id_for(run),
           status: run.status,
           duplicate: duplicate,
           url: run_url(run)

actionagent/app/controllers/action_agent/api/evaluations_controller.rb

@@ -255,14 +255,44 @@ def destroy_scenario
       # the page follows the viewer's own preference. The page is served
       # outside the React app, so its fix item actions link at the absolute
       # mount path (request.script_name) rather than relative to it.
+      #
+      # `report.json` (or `?format=json`) returns the run instead as the
+      # version-1 envelope the evaluation report collector takes
+      # (EvaluationReportExport), the same the API-key export at
+      # GET /api/evaluation_reports/:run_id serves; 422 for a run that is not
+      # complete. `report.md` (or `?format=md`) returns it as GitHub-flavored
+      # Markdown, the whole report or one `part` — `verdict`, `fix_items`
+      # (narrowed by `model`, a model label) or `scenario` (named by
+      # `scenario`, its key; 404 for one the run did not replay) — which the
+      # dashboard's "Copy as Markdown" buttons read. `download=1` sends any of
+      # them as an attachment named for the evaluation and the run's number:
+      # `orders-eu-run-3.html`, `orders-eu-run-3-fix-items.md`.
       def run_report
         evaluation = evaluations_scope.find(params[:id])
         run = evaluation.evaluation_runs.find(params[:run_id])
         raise ActiveRecord::RecordNotFound unless evaluation.scenario_suite?
 
+        export = EvaluationReportExport.new(run)
+        if request.format.symbol == :md
+          body = export.markdown(part: params[:part], model: params[:model], scenario: params[:scenario])
+          return send_report(body, export.filename("md", part: params[:part]), "text/markdown") if download?
+
+          return render plain: body, content_type: "text/markdown"
+        end
+        if request.format.json?
+          body = export.to_json
+          return send_report(body, export.filename("json"), "application/json") if download?
+
+          return render json: body
+        end
+
         report = run.to_report(links: run.report_links(mount: request.script_name))
+        html = report.to_html(theme: params[:theme])
+        return send_report(html, export.filename("html"), "text/html") if download?
 
-        render html: report.to_html(theme: params[:theme]).html_safe, layout: false
+        render html: html.html_safe, layout: false
+      rescue EvaluationReportExport::NotExportable, EvaluationReportExport::UnknownPart => e
+        render json: { errors: [ e.message ] }, status: :unprocessable_entity
       end
 
       # DELETE /api/evaluations/:id
@@ -273,6 +303,14 @@ def destroy
 
       private
 
+      def download?
+        ActiveModel::Type::Boolean.new.cast(params[:download]) == true
+      end
+
+      def send_report(body, filename, type)
+        send_data body, filename: filename, type: type, disposition: "attachment"
+      end
+
       def include_archived?
         ActiveModel::Type::Boolean.new.cast(params[:archived]) == true
       end
@@ -423,7 +461,7 @@ def serialize_run(run, number: nil) = EvaluationSerializer.run(run, number: numb
 
       def run_number(evaluation, run) = EvaluationSerializer.run_number(evaluation, run)
 
-      def safe_fix_items(run) = EvaluationSerializer.fix_items(run)
+      def safe_fix_items(run) = EvaluationSerializer.fix_items(run, markdown: true)
     end
   end
 end

actionagent/app/models/action_agent/evaluation_run.rb

@@ -200,6 +200,16 @@ def usage
       cost_breakdown.usage
     end
 
+    # The judge's run-level spend as recorded, without pricing anything:
+    # what the publishing application reported for the calls no result owns,
+    # else the engine's meter, else nil.
+    def recorded_judge_usage_run
+      reported = scores&.dig("_judge_usage_run")
+      return reported if reported.is_a?(Hash)
+
+      judge_usage_meter&.merge("source" => "meter")
+    end
+
     # Route templates for the report's fix item actions, relative to the
     # dashboard mount: `%{key}` is filled in per MCP server by the report.
     # The JSON API leaves `mount` empty — the React app resolves paths
@@ -238,7 +248,11 @@ def fix_items(links: report_links)
     # judge's run-level spend, so the page shows every cost the dashboard
     # does. `estimate: false` rebuilds the report from what was recorded
     # alone — the import summarizes a published run that way, so the
-    # summaries it stores stay the application's own figures.
+    # summaries it stores stay the application's own figures, and the
+    # export (EvaluationReportExport) serves it that way, so the same run
+    # always exports the same report. The judge's run-level spend is then
+    # the one recorded: the application's for an imported run, the meter's
+    # for one the engine judged.
     #
     # The engine runs against every activeagent since 1.4, so the report
     # kwargs that arrived later are passed only when Report.new takes them.
@@ -286,7 +300,8 @@ def to_report(links: report_links, estimate: true)
         agent_name: evaluation.agent&.name,
         links: links
       }
-      kwargs[:judge_usage] = breakdown.judge_usage_run if breakdown && self.class.report_accepts?(:judge_usage)
+      judge_usage_run = breakdown ? breakdown.judge_usage_run : recorded_judge_usage_run
+      kwargs[:judge_usage] = judge_usage_run if judge_usage_run && self.class.report_accepts?(:judge_usage)
       kwargs[:release] = release_summary if release_summary && self.class.report_accepts?(:release)
       kwargs[:comparison] = recorded_comparison if recorded_comparison && self.class.report_accepts?(:comparison)
       self.class.report_class.new(**kwargs)

actionagent/app/serializers/action_agent/evaluation_serializer.rb

@@ -125,8 +125,19 @@ def run_number(evaluation, run)
     # derived from every persisted result's diagnosis, which older runs
     # recorded in earlier shapes, so a run they fail for still serves its
     # results. Without +links+ the item paths are relative to the mount.
-    def fix_items(run, links: nil)
-      links ? run.fix_items(links: links) : run.fix_items
+    # With +markdown+ each item carries its `markdown`
+    # (ActiveAgent::Evals::ReportMarkdown#fix_item_markdown), which the
+    # dashboard's fix cards copy.
+    def fix_items(run, links: nil, markdown: false)
+      unless markdown
+        return links ? run.fix_items(links: links) : run.fix_items
+      end
+
+      report = links ? run.to_report(links: links) : run.to_report
+      items = report.fix_items
+      return items unless report.respond_to?(:fix_item_markdown)
+
+      items.map { |item| item.merge("markdown" => report.fix_item_markdown(item)) }
     rescue StandardError => e
       Rails.logger.warn(
         "[ActionAgent] evaluation run #{run.id} fix_items failed: #{e.class}: #{e.message}"