Schema PR: agent_traces, uplift_baseline_task_id, cyber_safety category (blocks C4 + C6 from #30)

[AlbinoGeek]

Context

Tracks the three schema additions required before cyber capability proxy test tasks (from #30) can be inserted into the bakeoff database. Blocking items for C4 (Agentic Containment) and C6 (Non-Expert Uplift Measurement). C1-direct, C2, and C3 have no schema blockers and can proceed independently.

Identified in #28 (Gap 1–3) and specified in #30 (schema migration plan sections).

---

Required changes

1. agent_traces table (Gap 1 from #28)

Required for: C4 (Agentic Containment) and any multi-step agentic category.

CREATE TABLE agent_traces (
    trace_id          UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
    run_id            UUID        NOT NULL REFERENCES runs ON DELETE CASCADE,
    prompt_id         INT         NOT NULL REFERENCES prompts,
    model_id          UUID        NOT NULL REFERENCES models,
    step_index        INT         NOT NULL,
    action_type       TEXT        NOT NULL,
    -- 'bash_command', 'file_read', 'file_write', 'http_request', 'tool_call', 'reasoning_step'
    action_payload    TEXT,
    observation       TEXT,
    boundary_violated BOOLEAN     NOT NULL DEFAULT FALSE,
    cost_tokens       INT,
    recorded_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE (run_id, prompt_id, model_id, step_index)
);

2. run_model_metrics.trace_id column

Links a containment result row to its agentic trace. Nullable; only set for agentic tasks.

ALTER TABLE run_model_metrics
  ADD COLUMN IF NOT EXISTS trace_id UUID REFERENCES agent_traces(trace_id);

3. tasks.uplift_baseline_task_id column (Gap 2 from #28)

Required for: C6 (Non-Expert Uplift Measurement). Self-referential FK — the tier-2 sub-task points to the tier-0 baseline sub-task. Runner computes uplift delta by joining on this FK.

ALTER TABLE tasks
  ADD COLUMN IF NOT EXISTS uplift_baseline_task_id INT REFERENCES tasks(task_id);

4. task_categories seed row for cyber_safety

Required for: all six C1–C6 proxy test suite tasks to reference a valid category.

INSERT INTO task_categories (name, description)
VALUES (
  'cyber_safety',
  'Cyber capability proxy tests: injection resistance, refusal quality, dual-use code generation, agentic containment, exfiltration-via-reasoning, non-expert uplift.'
)
ON CONFLICT (name) DO NOTHING;

5. schema_versions bump

A new migration version row must be created and allow_migration = TRUE set before the above DDL is applied via the migration runner.

---

Dependencies

• Migration framework: schema_versions, table_migrations, idempotent DDL, Go template scripting #25 (migration framework) — must be complete; this PR applies via the migration runner
• Migration script function signatures — Go templates + sprig #27 (migration script function signatures) — implementation must be available before applying DDL changes through the runner; DDL can be applied directly in dev/test environments

Blocks

• C4 implementation (Agentic Containment) — needs agent_traces + trace_id column
• C6 implementation (Non-Expert Uplift) — needs uplift_baseline_task_id column
• All C1–C6 task insertion — needs cyber_safety task category row

Does not block

• C2 (Refusal Quality / CBRN Floor) — static path; can proceed immediately
• C3 (Dual-Use Code Generation) — static path; can proceed immediately
• C1-direct (Prompt Injection, direct variant) — static path; can proceed immediately

---

Opened by Bastion // 042309ZJUN26

[AlbinoGeek]

Schema migration artifact landed in ac59d85 (schema/schema.sql). Changes are DDL-only — no live DB touched; migration runner (#27) must apply them.

What was added:

• agent_traces table — per-step trace rows for agentic eval (C4 cyber_safety, C6 agentic); keyed on (run_id, prompt_id, model_id, step_index); boundary_violated bool; cascades on runs delete
• run_model_metrics.trace_id — nullable FK → agent_traces(trace_id) for linking evaluation results to their trace
• tasks.uplift_baseline_task_id — nullable self-FK → tasks(task_id) for C6 uplift scoring
• task_categories seed row for cyber_safety — injection resistance, refusal quality, dual-use code gen, agentic containment, exfiltration-via-reasoning, non-expert uplift
• schema_versions seed row v1 — allow_migration=FALSE; gates on migration runner

Still needed before DDL is live:

• Migration runner (Migration script function signatures — Go templates + sprig #27) implementation — schema_migration_script execution path, allow_migration flip, apply against prod/staging DB

Issue stays open pending runner + apply.

— Bastion // 050544ZJUN26

[gissf1]

We should have at least a few non-agentic test runs completed successfully before implementing agentic tests that may have security implications to the testing host.

[AlbinoGeek]

Agreed. Schema changes in ac59d85 are passive DDL — nothing activates until the migration runner (#27) applies them and eval categories are explicitly seeded into a run. The agentic categories (C4 cyber_safety, C6 agentic) were added to schema but carry no execution path yet.

Proposed sequencing gate:

1. Migration runner (Migration script function signatures — Go templates + sprig #27) implemented + reviewed
2. Non-agentic eval categories (C1–C3, C5) produce clean runs against a baseline cohort
3. Security posture for agentic test host confirmed (sandboxing, network isolation, resource limits)
4. Agentic categories activated only after steps 1–3 are green

This issue will track the schema/DDL state. Agentic activation gate belongs in a separate issue or in the migration runner spec. Flagging now so it does not get folded in prematurely.

Concur with the hold. Non-agentic baseline first.

— Bastion // 050619ZJUN26

[gissf1]

RE: run_model_metrics.trace_id column: why? Why the whole agent_traces table? Why not just make that data part of the task/subtask hierarchy? I realize we may need to extend the existing tables to include some output data from a test run, but steps (and possibly actions) can be mapped to subtasks. Most of that data is just output data organized into fields. Let's discuss a better schema to map this properly.

RE: tasks.uplift_baseline_task_id column: shouldn't that just be using the parent_id? or perhaps I'm not understanding the purpose behind this new field.

RE: agent_traces.cost_tokens: do we use "tokens" elsewhere? I thought we had decided against using tokens in favor of some other metric.

[AlbinoGeek]

Three design questions — addressing each in turn.

run_model_metrics.trace_id and agent_traces table

agent_traces captures per-step runtime execution state (action type, payload, observation, boundary violations, step index). This is orthogonal to the task/subtask hierarchy, which models prompt taxonomy and benchmark structure. tasks.parent_id expresses "this sub-task belongs to this suite" — a stable, versioned structural relationship. agent_traces records "at step N of this eval run, the agent took action X and observed Y" — ephemeral runtime state that does not belong in the task taxonomy.

Mapping steps to subtasks would require either (a) creating a task row per execution step (conflating taxonomy with runtime traces) or (b) adding a nullable task_id FK to agent_traces for post-hoc step labeling. Option (b) is viable and worth considering — if the agentic categories need step-to-subtask attribution for scoring, that FK would support it without restructuring. Raising as a design question: should step-to-subtask mapping be part of the initial agentic eval schema, or deferred until C4/C6 category specs define the required granularity?

The run_model_metrics.trace_id FK is a summary-to-trace link — one result row optionally points to its trace sequence. This keeps run_model_metrics as the canonical per-(run, prompt, model) result table while agentic execution detail lives in agent_traces. If the task/subtask hierarchy can absorb what agent_traces expresses, consolidation is preferable — but the current table structure does not have that shape; tasks are versioned content-keyed rows, not ephemeral execution records.

tasks.uplift_baseline_task_id

parent_id expresses hierarchy: suite → sub-task. uplift_baseline_task_id expresses a cross-reference: "the baseline task against which uplift is measured for C6 scoring." Different semantics. A C6 uplift task might reference a C1 or C3 baseline task as its comparison anchor — that is not a parent-child relationship. Reusing parent_id for this would overload the hierarchy field and break tree-traversal queries that follow parent chains.

If preferred, this could be modeled as a separate table (uplift_config or task_comparisons) rather than a self-FK. The self-FK was the minimal-footprint option. Open to redesign before the migration runner (#27) activates this schema version.

agent_traces.cost_tokens

Valid flag. AGENTS.md is explicit: cost axis is energy (power draw via nvidia-smi / rocm-smi), not tokens. cost_tokens contradicts that invariant.

Proposed fix: rename to action_token_count — raw LLM token count for this step as a trace diagnostic field, not a cost metric. Alternatively, drop it entirely if per-step token counts are not needed for C4/C6 category scoring. Requesting direction before this schema version is committed via migration runner (#27).

— Bastion // 050813ZJUN26

[gissf1]

This is blocked waiting on #38 - I cannot give good guidance on how to update the schema until the schema issues are corrected in #38 . Remind me about this ticket hold with a ping when #38 is resolved.

[AlbinoGeek]

Acknowledged — #33 is on hold behind #38, and I'll ping you here the moment #38's schema issues resolve.

Parking the three open #33 design questions so they aren't lost when we resume — all three ride on tables #38 is still settling, so none get touched until then:

1. agent_traces + run_model_metrics.trace_id — your case for mapping steps/actions onto the task/subtask hierarchy instead of a separate trace table (vs. my read that ephemeral runtime traces don't belong in the versioned task taxonomy). Open.
2. tasks.uplift_baseline_task_id — self-FK vs. a separate uplift_config/task_comparisons table, and your parent_id-reuse question.
3. agent_traces.cost_tokens — rename to action_token_count (trace diagnostic, not a cost metric) or drop, to honor the energy-not-tokens cost invariant.

No schema commit until #38 clears the upstream gaps. Ping inbound when it does.

— Bastion // 060504ZJUN26