Document and harden durable team workflows#485
Closed
pallakatos wants to merge 212 commits into
Closed
Conversation
…0 slice 1) The substrate primitive underneath kars Bridge: a typed unit of governed agent work that carries its trust envelope (autonomy tier 1..5, budget, tool/egress allow-list refs, delegationDepth, authorityCeiling). Independently usable on a plain kars cluster with no Bridge — kubectl apply a KarsTask and the controller validates the envelope and stamps a stable sha256 envelopeDigest (the value the Governance Receipt will bind to). - controller/src/kars_task.rs: KarsTaskSpec + TaskEnvelope + KarsTaskStatus, mirroring the KarsEval CRD conventions (kube-rs derive, camelCase serde, JsonSchema, printcolumns). Deterministic envelope digest. 6 unit tests. - controller/src/kars_task_reconciler.rs: finalizer + envelope validation (defence-in-depth behind CEL) + status stamping (phase, Ready condition, envelopeDigest), preserving lineage for the delegation-minting slice. The reconciler is the sole writer of envelope-derived status. 4 unit tests. - crd_validations.rs: CEL admission rules enforcing the trust-envelope invariants — including the anti-amplification rule authorityCeiling <= tier. - crd-karstask.yaml: generated via the helm_drift dumper; drift test green. - field_managers.rs: CLAW_TASK SSA manager. main.rs: reconciler wired in. - tests/e2e/run.sh: kind integration test (valid->Ready+digest; CEL rejects amplifying envelope). Verified live on kind: valid task -> phase=Ready + envelopeDigest stamped; digest recomputes on spec change; CEL rejects authorityCeiling>tier and out-of-range tier at admission. Full suite: 861 controller tests pass, clippy -D warnings clean, fmt clean, zero helm drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dge V0 slice 2) Pillar A — the security headline. A KarsTask may reference a parent via spec.parentRef; the controller verifies the child's trust envelope is a strict subset (attenuation) of the parent's and mints status.lineage from the parent's ancestry. A child that amplifies authority on ANY axis is rejected as Degraded with NO envelope digest — a Governance Receipt can never bind to amplified authority. This is OWASP ASI-08 (cascading authority) solved at the substrate, not asked of the model. - kars_task.rs: pure attenuation lattice — TaskEnvelope::attenuation_violations returns a precise EnvelopeViolation list across every axis: tier <= parent ceiling, ceiling <= parent ceiling, depth <= parent depth - 1, budget caps (no unbounded child under a bounded parent), and pinned tool/egress policy refs. 9 unit tests covering each amplification + valid attenuation. - kars_task_reconciler.rs: resolve_delegation fetches the parent, mints lineage (controller is sole writer), and routes Ready / Degraded / ParentMissing. - crd-karstask.yaml regenerated (parentRef); helm drift test green. - tests/e2e: delegation test (valid child Ready+lineage; amplifying child Degraded+no-digest). Verified live on kind: parent Ready (root); valid child Ready with lineage=[parent]; tier-5 child under a ceiling-4 parent Degraded with no digest and the exact violation message. 870 controller tests pass, clippy -D warnings clean, fmt clean, zero drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… KarsSandbox (V0.1b)
Closes the governance/execution gap surfaced in live dogfood ("I created a
task, nothing happens"). A governed task can now run a real agent — gated by
an explicit launch, faithful to plan §20 (review the package, then launch).
- kars_task.rs: spec.execution { launch, runtime } (default not-launched =
governed-but-idle); status.executionPhase / sandboxRef / executionDetail;
Execution printcolumn.
- kars_task_execution.rs: on launch, server-side-apply an owned InferencePolicy
+ KarsSandbox bounded by the envelope (tool policy → governance, token budget
→ InferencePolicy), then read back the sandbox phase. Un-launch tears them
down; owner refs cascade on task delete. 4 unit tests.
- kars_task_reconciler.rs: reconcile_execution folds launch/teardown into
status; execution errors degrade execution only, never the governance status.
- crd-karstask.yaml regenerated; drift green.
Verified live on kind: launch=true → real KarsSandbox + InferencePolicy
materialized (owned, envelope-bounded), executionPhase=Degraded with the honest
"needs a real Foundry endpoint" detail; launch=false tears the sandbox down
(Idle); default (no execution) makes no sandbox (§20 gate holds). 874 controller
tests pass, clippy -D warnings clean, fmt clean, zero drift.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The KarsTask reconciler (Bridge V0) needs cluster permission to
get/list/watch/patch karstasks + status + finalizers. Without it the
in-cluster controller ServiceAccount hits a 403 and the reconciler disables
itself ("KarsTask CRD not installed — Forbidden"). Earlier kind tests ran the
controller out-of-cluster (full kubeconfig), so RBAC was never exercised;
running in-cluster surfaced the gap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion (Inc 3) The Governance Receipt is the auditor's moment: a signed, independently verifiable record that a KarsTask was governed under its trust envelope. Core (controller): - New KarsReceipt CRD (controller-written only; no admission CEL — integrity comes from the signature, not schema gates). - providers/signing.rs (allowlisted crypto): Ed25519 over DSSE PAE; loads/ persists a receipt identity Secret and publishes the public key to the out-of-band kars-receipt-pubkey ConfigMap (the verifier's trust anchor). - kars_receipt.rs: pure, deterministic in-toto Statement builder (no timestamps → idempotent emission, re-derivable by a verifier) + the honest claim matrix: integrity=PASS, conformance=PASS, completeness=PARTIAL (router token/cost audit chain is V1), regulatory=OMITTED (no external anchor in V0 local signing). - Reconciler: emits an owner-referenced KarsReceipt for every governance-Ready task; retracts it when the task is Degraded (no validated authority to attest). A child receipt records its attenuation of the parent. CLI (kars receipt verify / show): - Verifies the DSSE/Ed25519 signature against the published anchor (never a key embedded in the receipt), plus key-binding and envelope-binding, then prints the claim matrix. Works on a plain kars cluster, no Bridge required. - Unit tests cover the DSSE PAE framing and tamper/wrong-key rejection. Honest V0/V1 split: the plan puts the full emitter in the router (token/cost from the audit chain), which needs a real Foundry run; V0 ships the governance half in the controller and self-labels completeness/regulatory accordingly. Documented in the design note. Verified end-to-end on a kind kars-dev cluster: receipt emitted + CLI-verified for a Ready task; amplifying child Degraded with no receipt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
KarsApproval makes the autonomy tiers mean something: a priced/external/ irreversible action, a checkpoint, or a tier-raise becomes a human decision the fleet waits on — and the decision is recorded in the task's Governance Receipt. Core (controller): - New KarsApproval CRD + reconciler. The controller owns the authority binding: it copies the gated task's status.envelopeDigest into status.boundEnvelopeDigest on first observation and never changes it. If the task envelope later drifts, a still-pending approval goes Stale — you cannot grant authority against a moved target. - Pure, exhaustively-tested decision function: Pending → Approved/Denied (decidedAt/decider immutable) / Expired (past TTL) / Stale. A recorded human decision wins over expiry/staleness. Unknown verdicts fail closed. - Receipt closure (Inc 3 ↔ Inc 4): the in-toto predicate now carries the decided approvals (approve AND deny) for a task, deterministically ordered, so every steer is part of the signed record. The DSSE signature re-verifies over the enriched payload. CLI (kars approval list/approve/deny/show): - Plain-cluster parity for the steering primitive; approve/deny patch spec.decision, the controller drives the transition. The reconciler RECORDS decisions; acting on an approved action (actually widening egress, raising the tier) is the consuming reconciler's job and is the V1 wire — documented honestly in the design note. Verified end-to-end on kind kars-dev: bind, approve (CLI + UI), deny, TTL/staleness on envelope drift, and all three decisions bound into a re-verifiable receipt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… metering (Inc 5)
Three parallel hardening deliverables that strengthen the Governance Receipt
without overclaiming.
Completeness floor:
- New CREATE-time kars-task-namespace-floor ValidatingAdmissionPolicy denies
hostNetwork/hostPID/hostIPC, privileged, allowPrivilegeEscalation,
ephemeralContainers, and hostPath in isolated=strict namespaces — closing the
gap the UPDATE-only posture-lock leaves, so the receipt's no-bypass claim
holds against a compromised controller / direct kubectl apply.
- The reconciler observes which floor controls are enforced (the new VAP,
exec-ban VAP, posture-lock VAP, default-deny egress) and binds them into the
receipt predicate. The completeness claim stays honestly PARTIAL (runtime
iptables hash + token/cost = V1, eBPF witness = V2, all named), but its
detail now reflects the concrete enforced controls.
Receipt inclusion log (self-hosted-Rekor precursor):
- Every emitted receipt is entered in a hash-chained kars-receipt-log
ConfigMap. `kars receipt verify` now also checks chain integrity + inclusion;
`kars receipt log` shows/verifies the chain. Gives cross-receipt
tamper-evidence (deleting/altering/reordering breaks the chain). Labelled
operator-controlled — does NOT give operator-non-repudiation (external
witness + KMS-attested signing are V2); regulatory stays OMITTED.
Metering attribution (efficiency pillar plumbing):
- The task reconciler stamps task-id + lineage-root annotations on the
materialized sandbox; the main reconciler forwards them as
KARS_TASK_ID/KARS_TASK_ROOT; the router emits kars_task_tokens_total
{task,root_task,model,direction} so cost rolls up per task branch. Bounded
cardinality — only task sandboxes emit the series. Live token numbers need a
Foundry run (the same honest boundary as V0.1b).
Verified end-to-end on kind kars-dev: VAP denies each escape vector + honours
break-glass; receipt carries the enforced-controls evidence; inclusion chain
populates, a tampered chain fails verification and a restored one passes;
non-task sandbox correctly omits the task metric labels.
Wave-2 KMS-attested key-release (SKR/MAA), external witness, and the eBPF
datapath witness remain hardware/partner-gated and seam-ready — not faked on
kind, per the product's honesty discipline.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ceipt log Delivers the verifiable operator-side half of the V2 witness story for the receipt inclusion log. On each receipt emission the controller publishes a signed checkpoint — an Ed25519-signed note over (origin, tree_size, head_hash), Go-sumdb signed-note style — to the kars-receipt-checkpoint ConfigMap, via the allowlisted providers::signing::sign_note. The head hash of a hash chain already commits to the entire prefix, so a checkpoint over it is a sound signed tree head. - `kars receipt verify` now also validates the checkpoint (signature vs anchor + agreement with the live log). - `kars receipt checkpoint` verifies it standalone and prints the pinnable root. - The Bridge surfaces the checkpoint in the receipt panel. Why it matters: the bare chain gives tamper-evidence but the operator could rewrite the *whole* chain silently. A pinned signed checkpoint detects a silent history rewrite even when the rewritten chain is internally intact — verified live: truncating the log to a valid shorter chain passes `receipt log`, but the pinned checkpoint reports "size 6 diverges from live 5 — history may have been rewritten." Still V2 (genuinely partner/hardware-gated): an external party counter-signing/gossiping these checkpoints (non-equivocation) and hardware-attested signing. The checkpoint gives clients something concrete to pin; the external witness is the remaining step. regulatory stays OMITTED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…agents were degraded) Launching a KarsTask materialized an InferencePolicy with no modelPreference.primary.deployment — but the sandbox reconciler REQUIRES a deployment (it's plumbed to AZURE_OPENAI_DEPLOYMENT / OPENCLAW_MODEL). Without it every task-launched sandbox degraded immediately at materialization, before inference was ever attempted. Only `kars dev`-created sandboxes (which carry a full modelPreference) ran. The task execution path now sets modelPreference.primary from a controller-env default — KARS_TASK_DEFAULT_MODEL → AZURE_OPENAI_DEPLOYMENT → DEFAULT_MODEL → gpt-4o-mini, with provider KARS_TASK_DEFAULT_PROVIDER (default azure-openai; the router routes by the configured endpoint URL). An operator points the controller at any OpenAI-compatible provider (GitHub Models, Azure OpenAI, Foundry) and launched agents do real governed inference. Verified end-to-end on kind kars-dev with GitHub Models as the dev provider: a launched task's sandbox reaches 2/2 Running (no longer Degraded), the router loads the model, and a chat completion through the secure router (with content safety filtering active) returns a real response + token usage. The whole 7-stage pipeline is now live, including agent execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…al run The launch package (design note §20) needs to compile a task-giver's intent into the substrate's existing CRDs rather than duplicate them. Extend KarsTask with a TaskBlueprint (runtime, model, instructions, toolPolicy, mcpServers, egress, isolation, memory) and have materialize() compose it: - model -> InferencePolicy.spec.modelPreference.primary - runtime -> KarsSandbox.spec.runtime.kind - isolation -> KarsSandbox.spec.sandbox.isolation - prompt -> KarsSandbox.spec.agent.instructions (objective + standing instructions) - tools -> KarsSandbox.spec.governance.toolPolicyRef (existing ToolPolicy, by ref) - MCP -> KarsSandbox.spec.governance.mcpServerRefs - egress -> KarsSandbox.spec.networkPolicy.allowedEndpoints (+ Strict mode) - memory -> KarsSandbox.spec.memoryRef (existing KarsMemory) Governance is composed cohesively: tools come from a single source (blueprint or envelope toolPolicyRef); MCP refs only attach when a tool policy bounds them, so without a policy governance is a valid 'enabled: false' rather than an invalid 'enabled: true' with no toolPolicyRef. CEL guards added: mcpServers requires toolPolicy; has() guards on optional vec fields. Verified live on kars-dev: a launched task materializes the correct InferencePolicy + KarsSandbox and reaches Running through the secure router. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… + gate on parent readiness Two correctness fixes that make capability-attenuating delegation (Pillar A / design note §12 'the org chart IS the security topology') honest rather than nominal: 1. Authority-model cohesion. Attenuation previously checked the envelope's toolPolicyRef/egressAllowlistRef, but materialize() composes the sandbox from the blueprint's toolPolicy + inline egress (and egressAllowlistRef was never materialized). A child could therefore pass delegation checks while changing its real tool/egress surface. New spec_attenuation_violations() checks the *effective* authority the sandbox actually enforces: effective_tool_policy (blueprint-or-envelope) and effective_egress (blueprint egress, subset of the parent's, with any-port parent entries covering child ports). The verified subset relation now matches execution. 2. Parent-readiness gating. resolve_delegation() no longer grants a child authority against an unvalidated parent: a child whose parent is not governance-Ready (no envelope digest) resolves to a transient Pending (DependencyMissing) state and requeues quickly, instead of going Ready under a degraded/in-flux parent. Verified live on kars-dev: a subset child → Ready; an egress-amplifying child → Degraded with the exact reason; a child of a degraded parent → Pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ndboxes The finalizer-drop used a server-side apply patch that set finalizers: []. The apiserver does not reliably remove a finalizer that way once it no longer attributes the field to this manager — it rejects with 'name must be provided based on URL' (HTTP 400). The reconcile then errors and requeues forever, so the KarsTask sits in Terminating with its cleanup finalizer and its materialized KarsSandbox (and per-sandbox namespace) is never reaped. This is exactly the 'sandboxes pile up' symptom seen in dogfood. Fix: drop the finalizer with a deterministic merge patch (the same operation that clears it by hand). Adding-the-finalizer keeps the apply path (which works) and now also carries metadata.name for correctness. Verified live on kars-dev: launching then deleting a task now reaps the task, its sandbox (owner-ref GC), and its namespace within the delete timeout — no stuck finalizer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a neutral, harness-agnostic core capability: the controller delivers a governed task straight into a running agent's native loop over the AGT mesh and captures the reply. Triggered declaratively by a KarsTask annotation so downstream products consume it without core depending on them. Flow (all additive, no original behaviour touched): - runtime adapter (runtimes/openclaw): when KARS_CONTROLLER_AMID is set, trust the controller's DID as a plaintext mesh peer so its task_request reaches the agent loop. Unset => byte-identical to before (SDK does `?? []`). - reconciler: advertise the controller DID as KARS_CONTROLLER_AMID on the openclaw container (only when mesh/governance is enabled). The agent still re-verifies the sender and the AGT `task:execute` policy still gates the run. - mesh_peer::task_delivery: a leader-gated watcher reacts to `kars.azure.com/run-requested`, discovers the agent's DID from the registry, sends a `task_request`, awaits the `task_response` (correlated by agent DID), writes the result to `kars-mission-output-<task>` and stamps `kars.azure.com/run-completed`. No annotation => no behaviour. - FederationMessage: add TaskRequest/TaskResponse variants matching every runtime adapter's existing wire contract. - AgtFrame::Message: carry the AGT SDK's plaintext fields (`ciphertext` + `plaintext`) alongside the legacy `payload`, and prefer `ciphertext` on receive. SDK 4.0.0 reads `ciphertext` for plaintext peers; without this the controller's frames were silently dropped. Backward-compatible. Verified end-to-end on kind: controller delivers objective -> agent native loop runs governed inference -> task_response captured as a durable deliverable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend the harness-neutral mesh task loop so a governed task delivers back the agent's *complete set of workspace artifacts* — not just the final text reply. A complex research problem now yields a real multi-file deliverable (report, decision matrix, risk register, …) captured end-to-end. Agent (runtimes/openclaw): - New `core/artifact-collect.ts`: harvest new workspace files after the task loop and ship each to the requester as `file_transfer` (mirrors the proven offload harvest; factored so the offload path is untouched). Falls back to saving a substantial text reply as a markdown artifact so the set is never empty. - task_request handler: mark the workspace before the loop, collect+ship artifacts after, and include the manifest in `task_response`. - `latin1Safe`: the AGT SDK's plaintext send btoa-encodes payloads, which throws "Invalid character" on Unicode (em-dashes/smart quotes in LLM summaries). Transliterate the summary to Latin1 on the controller (plaintext-peer) path. Artifact bytes are unaffected — they travel base64 and keep full Unicode. Controller (mesh_peer): - `FederationMessage::FileTransfer` + a per-DID artifact buffer in MeshPeerState; `task_response` now carries the manifest count so the receiver waits for the matching frames before flushing. - task_delivery: drain the buffered artifacts on task_response, persist the full set to `kars-mission-artifacts-<task>` (text in data, binary in binaryData, ConfigMap-budget capped) and record the manifest in the output ConfigMap. The minimal §16 artifact record — a durable, cluster-native object readable with `kubectl get configmap`, no Bridge required. Verified end-to-end on kind: a gpt-4o agent ran a multi-round tool-calling loop, wrote a 4-file eBPF due-diligence package, shipped it over the mesh, and the controller captured the complete set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…path The agent loop now emits a live execution trace — a `round` event after each model call (real token usage, finish reason, tool-call count, duration) and a `tool` event after each tool call (tool name, sanitized arg/result preview, duration, ok). This is the real source of the Bridge's live activity, token telemetry, and per-tool audit path — never reconstructed. Agent (runtimes/openclaw): - agt-task-loop: TraceEvent type + optional onTrace sink; emit round + tool events at the existing chokepoints. tracePreview strips base64/whitespace and bounds length so payloads never leak into the trace. Absent onTrace = byte- identical behaviour (offload/sub-agent paths unchanged). - index: the task_request handler collects the trace, aggregates real tokens + round/tool counts, and ships both in task_response. Trace previews are Latin1-sanitized (the whole payload is btoa-encoded on the plaintext-peer path; Unicode in a preview would otherwise break the send). Controller (mesh_peer): - task_response carries trace[] (opaque JSON, forward-compatible) + telemetry; persist the trace verbatim to kars-mission-trace-<task> (the clean audit record, ConfigMap-budget capped, oldest-dropped) and the token/round/tool totals + model onto the mission output. Verified on kind: a gpt-4o research run shows round 0 (7,897 tok, 4 tool calls) → four real file_write calls with byte-count results → round 1 (8,097 tok, stop); 15,994 tokens total — all surfaced in the Bridge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The completeness claim now binds the V1 token/cost audit chain once a task has actually run: the controller reads the real router-metered token totals (`kars-mission-output-<task>`) and the per-round/per-tool execution trace event count (`kars-mission-trace-<task>`) and records them in the receipt predicate (`tokenCostAuditBound`, `runTotalTokens`, `traceEventCount`). The claim detail now states the chain IS bound, with the real figures, and drops it from the "not yet bound" list — narrowing the V0→V1 gap with concrete, re-derivable evidence an auditor can independently read from the cluster. The claim stays PARTIAL honestly (the runtime egress-guard iptables-ruleset hash and the eBPF kernel-datapath witness remain unbound), and is unset before a run — never faked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds the durability-axis primitive (design note §11): a KarsTeam CRD plus reconciler that materializes a principal KarsTask (full envelope) and member KarsTasks (attenuated, parented to the principal), then runs a charter cadence loop that mints and launches task-force KarsTasks on schedule — the autonomous monitoring mechanism for standing operations (monitoring a repo/org, periodic checks). The reconciler authors KarsTasks rather than re-implementing sandbox materialization, so all existing attenuation enforcement, mesh agent loop, receipts, and metering are reused unchanged. Fully additive: a cluster with no KarsTeam objects behaves identically. - controller/src/kars_team.rs: KarsTeam CRD (charter, envelope, roster, cadence, blueprint, reporting_to, knowledge_commons, paused), validation + commons_name - controller/src/kars_team_reconciler.rs: principal/member materialization + charter cadence loop (PHASE_ACTIVE/DEGRADED/HIBERNATING) - crd_validations.rs: kars_team_crd() + CEL (tier range, ceiling<=tier, charter) - helm crd-karsteam.yaml + helm_drift dump/match tests - phase.rs: PHASE_HIBERNATING; field_managers.rs: CLAW_TEAM; main.rs: wire reconciler Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
KarsTeam reconciler needs list/watch/create/update/patch on karsteams (+ /status, /finalizers). Without it the reconciler self-disables with a 403 at startup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements the team knowledge commons (design note §14) — real, in-cluster shared memory that a standing team accumulates across its runs, closing the 'no sign of shared memory' gap. - controller/src/team_commons.rs: ConfigMap-backed commons (kars-commons-<team>), owned by the team. Append-only, provenance-tracked entries (id, author, source run, timestamp, content digest, size), budget-bounded with oldest-first pruning. ensure_commons / record_entry / prior_knowledge. Two load-bearing paths make this functional memory, not a display: - Write path (autonomous): when a standing-operation run completes with a substantive deliverable (tokens spent or artifacts produced — harness-neutral), the reconciler harvests its output into a commons entry, then retires the run's sandbox (launch=false) so runs never pile up. Backpressure caps concurrent runs so the charter loop can't outrun completion. - Read path (functional): when minting the next run, the most recent commons entries are injected as prior knowledge into the run objective, so the team builds on what it already knows instead of starting cold each tick. Runs are driven to completion autonomously via the existing mesh run-request annotation. Verified live: run (13646 tokens) -> harvested with provenance -> sandbox retired -> next run objective carries the prior-knowledge preamble. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…eteness
Closes the V1 egress-guard datapath-posture binding (design note §24b). The
controller authors the exact egress-guard iptables ruleset
(build_egress_guard_command); egress_guard_ruleset_hash() now pins it as a
sha256 digest the receipt binds. It is re-derivable — an auditor running the
same controller version + sandbox kind reproduces the bytes and digest, so a
weakened ruleset surfaces as a different hash.
The completeness claim's 'NOT yet bound' list now narrows to a single,
honestly-deferred item: the eBPF kernel-datapath *witness* (V2) that the node
actually applied the bound ruleset (node/hardware-gated). The token/cost audit
chain and the egress-guard ruleset are both bound.
- reconciler/mod.rs: egress_guard_ruleset_hash() + test
- kars_receipt.rs: PredicateCompleteness.egress_guard_ruleset_{bound,hash};
completeness_detail reflects the bound ruleset + narrowed not-bound list
- kars_task_reconciler.rs: gather_completeness binds the authored ruleset hash
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Computes a standing team's operational health from run outcomes + cadence punctuality, so the operator can tell at a glance whether the team is actually producing, not merely scheduled (addresses 'monitor that they work autonomously, checking periodically'). - KarsTeamStatus: health (Healthy/Watching/Unproductive/Stalled/Hibernating), runsSucceeded, tokensSpentTotal, commonsEntryCount, lastSuccessAt - harvest pass now tallies substantive vs barren runs + total tokens + newest success; health derives from those + overdue-cadence detection - team_commons::entry_count for shared-memory size - regenerated crd-karsteam.yaml (status fields); helm drift green Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Standing-operation runs stamp run-requested at launch, which raced the sandbox's mesh warm-up: the first delivery fired before the agent was ready, producing a permanent 180s-timeout deliverable that was then retired. - mesh delivery: a transient miss (agent not yet discoverable, or no reply within the window) is now retried on the next poll up to MAX_DELIVERY_ATTEMPTS instead of being recorded as a terminal timeout on the first miss. Only after the warm-up budget is spent is a terminal 'agent never came online' result written. Tracked via the run-attempts annotation. - team retire: a run's sandbox is torn down only once delivery is terminal (deliverable landed AND run-completed stamped), so the team never pulls a sandbox out from under a run that's still warming up / retrying. Verified live: a fresh standing run delivered ok with 28291 tokens and no timeout, then retired cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…itable-only (arch-D) ValidatingAdmissionPolicy that makes a KarsTask/KarsTeam's governance fields controller-writable-only. On UPDATE by any principal other than the controller SA, it denies: - writing .status (the controller-owned governance facts), and - RAISING spec.envelope.tier / authorityCeiling / delegationDepth (self-escalation). Voluntary attenuation (lowering envelope fields) and non-authority spec edits (objective, paused, execution.launch, annotations) remain allowed, so the Bridge BFF and operators keep working. The controller SA is exempt via matchCondition. UPDATE-only (CREATE bounds are the CRD's own CEL); status subresources included. Verified live: self-escalate tier 4->5 DENIED, hand-write status DENIED, voluntary attenuation ALLOWED, controller status writes ALLOWED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d (§20) The receipt predicate now opens with the validated launch package — the editable composition (runtime/model/tool-policy/MCP/isolation/memory) the operator reviewed before launch — pinned by a deterministic sha256 digest. This puts what-was-approved at the head of the signed record (the launch ledger), so the receipt binds not just the trust envelope but the concrete plan that ran. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A standing team now publishes a periodic digest — its autonomous-monitoring report — without being asked. On cadence.digestEveryMinutes the reconciler appends a timestamped entry (health, runs generated/delivered, tokens spent, knowledge accumulated) to a rolling kars-team-digest-<team> ConfigMap. - kars_team.rs: cadence.digestEveryMinutes + status.lastDigestAt; CRD regen - team_digest.rs: rolling digest log (last 30), provenance (team/reportingTo) - kars_team_reconciler.rs: publish when the digest interval elapses Verified live: digest published (Healthy: 21 runs, 4 delivered, 56098 tokens, 4 knowledge entries) and surfaced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ntiation, governed promote, capability-readiness gate Closes the durability-axis composition layer (§13/§17/§12/§19), all additive: - KarsSkill CRD (§13): reusable, versioned capability bundle (bounding tool policy + MCP + recipe + knowledge pack), validated + content-digested by its reconciler. Granted to ROLES — the team reconciler merges a Ready skill's tool policy / MCP servers / recipe into the materialized member blueprint, so the grant is a real authority fact, not a label. - KarsProfile CRD (§17): vetted team template (charter + roster + skills + envelope + domain), validated + digested. A KarsTeam with spec.profileRef inherits the profile's charter (if empty) + roster (if empty) — domain-blind platform, domain in the profile. - Governed promote (§12): KarsTeam.spec.requestedTier opens a human tierRaise KarsApproval against the principal; only on approval does the controller widen the envelope (controller-only raise — enforced by the envelope-write VAP), and the approval is bound into the principal's receipt (human-approved + ledgered). - Capability-readiness gate (§19): the charter loop checks every referenced MCP server is provisioned + Ready before minting a run; if not, it pauses-with- reason (clear status detail) instead of dispatching a doomed run that loops. Wiring: 2 new reconcilers + RBAC + helm CRDs + helm-drift tests. 952 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ritance - process_promotion widens the envelope via a merge-patch (not SSA apply) so the other envelope fields are preserved (an apply dropped siblings and failed CRD validation on the next reconcile). - charter CEL now allows an empty charter when spec.profileRef is set, so a profile-instantiated team passes admission and inherits the profile's charter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The agent must never hold a long-lived Git credential. The inference-router now
mints short-lived GitHub App INSTALLATION tokens on the agent's behalf:
- github_app.rs: signs a short-lived App JWT (RS256) with the App private key
the router holds, exchanges it for an installation token (POST
/app/installations/{id}/access_tokens), and caches it until ~60s before expiry.
Gated on GITHUB_APP_ID + GITHUB_APP_INSTALLATION_ID + GITHUB_APP_PRIVATE_KEY —
absent ⇒ feature off (additive). Unit-tested (host detection, PEM validation,
JWT claim window, env gating).
- routes/github_token.rs: GET /v1/github-token returns the installation token to
the sandbox (which wires it as a git credential helper). The proxy uses CONNECT
tunnelling and can't inject into HTTPS, so a router-served token endpoint — same
fail-closed pattern as /v1/mesh-token — is the correct seam: 404 when no App is
configured, so the sandbox falls back to anonymous public access.
The agent's sandbox never stores a credential; the router holds the App key and
issues scoped, ~1h-expiring tokens on demand. 953 router tests pass.
NOTE: live exchange requires real GitHub App credentials (App id + installation
+ PEM); the minting/caching/gating logic is unit-tested, and the network path
activates only when those are configured.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…coping, idempotency Multi-disciplinary review panel (security/architecture/agentic/SOTA) fixes: - SEC-1 memory poisoning: commons content is sanitized (defang injection markers) + framed as UNTRUSTED reference data with a standing guard; runs whose output is densely adversarial are refused from the commons (provenance gate). - SEC-2 GitHub token: minted token scoped to GITHUB_APP_REPOS + minimal perms; /v1/github-token moved to the admin-protected group (UID 1000 can't fetch it). - SEC-3 promote forgery: controller only honors a tierRaise approval that is owner-referenced to the team (a planted Approved object can't widen authority). - ARCH-1 duplicate runs: task-force tasks named by cadence WINDOW + existence- gated mint → idempotent across status-write failures. - ARCH-3 skill merge: forbid multi-distinct-policy roles (no under-bounded tools). - ARCH-4: commons owner-ref only when same-ns (no cross-ns GC); ARCH-5: cumulative team token budget (BudgetExhausted); ARCH-6: terminal-timeouts not counted active + 600s timeout; ARCH-9: Ready condition + requeue jitter. - AG-6: capability manifest + operating contract (build-on-prior, no redo, escalate) injected into standing-run objectives. 955 controller + router github tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The envelope-write VAP locked tier/ceiling/depth but left budget + tool-policy + egress refs mutable by non-controllers — a lateral authority hole. Now also blocks: raising budget.tokens/usdMicros and repointing toolPolicyRef/ egressAllowlistRef. Verified live: both denied for a non-controller principal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
added 18 commits
July 23, 2026 19:38
Version and verify task contracts, persist resumable checkpoints and child telemetry, enforce capability routing, add dependency/review-gated milestones, and preserve the same behavior across OpenClaw and Hermes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Align advertised tool checks with the documented AGT action contract and scope each MCP alias to its selected upstream server so principal spawn tools remain available without duplicate catalogs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Seed fresh metadata for the controller-selected model so optional OpenRouter catalogue discovery cannot generate misleading egress approvals. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Reject stale pre-respawn registry identities and clear logical plus parent-scoped AMID caches when a child is destroyed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Use OpenClaw's provider request-timeout control for long reasoning starts and retain bounded worker failure previews in durable telemetry and assignment events. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Patch the pinned Hermes 0.16 metadata fetch behind a fail-fast opt-out and enable it for governed sandboxes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Move the pinned source rewrite into a build-only script so Azure Container Registry dependency scanning can parse the Dockerfile. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Tell Hermes assignments that the mesh worker returns their final response automatically so roles do not waste time messaging an unresolved parent alias. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Preserve successful terminal handbacks across later model rounds and require a fresh worker generation before retrying failed role work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Detect validated task-checkpoint.json changes and attach them to task progress so controller recovery does not depend on a model explicitly calling the checkpoint tool. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Validate repeated logical roles as replacement worker attempts and self-heal prior collaboration-only output rejections once the retained evidence passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Treat awaiting-review milestones as in-flight so independent queued work cannot bypass the current human approval gate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Require an initial milestone checkpoint and carry the latest validated checkpoint on terminal responses across OpenClaw, Hermes, and the controller delivery ledger. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Keep review-required outputs as evidence until typed checkpoint approval, then idempotently add the approved result to team commons before unlocking dependents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Carry canonical objective, instructions, and checkpoint fields as base64 UTF-8 across the plaintext mesh seam while preserving the kars.task/v1 digest contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Consume typed checkpoint approvals on every harvest pass so approve and request-changes decisions still resolve after the source run sandbox is retired. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Persist a nonce-scoped in-progress checkpoint before delivering any milestone task so restart recovery is guaranteed independently of model tool choices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Explain the team, milestone, run, assignment, activity, artifact, deliverable, approval, intake, checkpoint, restart, review, and memory model with editable flow diagrams. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Dependency ReviewThe following issues were found:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+468
to
+483
| "node_modules/fast-uri": { | ||
| "version": "3.1.3", | ||
| "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.3.tgz", | ||
| "integrity": "sha512-i70LwGWUduXqzicKXWshooq+sWL1K3WUU5rKZNG/0i3a1OSoX3HqhH5WbWwTmqWfor4urUakGPiRQcleRZTwOg==", | ||
| "funding": [ | ||
| { | ||
| "type": "github", | ||
| "url": "https://github.com/sponsors/fastify" | ||
| }, | ||
| { | ||
| "type": "opencollective", | ||
| "url": "https://opencollective.com/fastify" | ||
| } | ||
| ], | ||
| "license": "BSD-3-Clause" | ||
| }, |
Comment on lines
+14
to
+25
| "node_modules/@hono/node-server": { | ||
| "version": "1.19.14", | ||
| "resolved": "https://registry.npmjs.org/@hono/node-server/-/node-server-1.19.14.tgz", | ||
| "integrity": "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw==", | ||
| "license": "MIT", | ||
| "engines": { | ||
| "node": ">=18.14.1" | ||
| }, | ||
| "peerDependencies": { | ||
| "hono": "^4" | ||
| } | ||
| }, |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| except Exception: | ||
| continue | ||
| out.extend(v if isinstance(v, list) else [v]) | ||
| except FileNotFoundError: |
| try: | ||
| if 16 <= int(a.split(".")[1]) <= 31: | ||
| return True | ||
| except Exception: |
| h = (ep.get("host") or "").rstrip(".").lower() | ||
| if h: | ||
| hosts.add(h) | ||
| except Exception: |
| if (packed.length === 0) { | ||
| throw new Error(`npm pack produced no AGT SDK tarball under ${tsDir}`); | ||
| } | ||
| fs.writeFileSync(stampPath, revision + "\n"); |
| ) | ||
| .for_each(|res| async move { | ||
| match res { | ||
| Ok(o) => tracing::debug!("KarsApproval reconciled {:?}", o), |
| .for_each(|res| async move { | ||
| match res { | ||
| Ok(o) => tracing::debug!("KarsApproval reconciled {:?}", o), | ||
| Err(e) => tracing::warn!("KarsApproval reconcile failed: {e:?}"), |
| ) | ||
| .for_each(|res| async move { | ||
| match res { | ||
| Ok(o) => tracing::debug!("KarsProfile reconciled {:?}", o), |
| .for_each(|res| async move { | ||
| match res { | ||
| Ok(o) => tracing::debug!("KarsProfile reconciled {:?}", o), | ||
| Err(e) => tracing::warn!("KarsProfile reconcile failed: {e:?}"), |
| ) | ||
| .for_each(|res| async move { | ||
| match res { | ||
| Ok(o) => tracing::debug!("KarsSkill reconciled {:?}", o), |
| ) | ||
| .for_each(|res| async move { | ||
| match res { | ||
| Ok(o) => tracing::debug!("KarsTask reconciled {:?}", o), |
| .for_each(|res| async move { | ||
| match res { | ||
| Ok(o) => tracing::debug!("KarsTask reconciled {:?}", o), | ||
| Err(e) => tracing::warn!("KarsTask reconcile failed: {e:?}"), |
| // ruleset against it and co-attest the kernel datapath. | ||
| let client = client.clone(); | ||
| tokio::spawn(async move { | ||
| if let Err(e) = kars_task_reconciler::publish_datapath_authored(&client).await { |
| .await | ||
| .map_err(|e| format!("tools/list body read failed: {e}"))?; | ||
| let value = extract_jsonrpc_payload(&content_type, &body)?; | ||
| if let Some(error) = value.get("error") { |
| poisoned.into_inner().remove(&name); | ||
| } | ||
| } | ||
| if let Err(e) = result { |
Collaborator
Author
|
Closing because this work belongs in the standalone pallakatos/kars repository. Continued at https://github.com/pallakatos/kars/pull/25. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documentation
docs/concepts/durable-team-workflows.mddocs/showcase/diagrams/08-intent-to-team.excalidrawdocs/showcase/diagrams/09-run-execution-signal.excalidrawdocs/showcase/diagrams/10-checkpoint-review-memory.excalidrawValidation