Sentinel

Purpose

Sentinel is an AI-native incident investigation and recovery platform for distributed systems. It correlates telemetry into incidents, sends specialist investigators after the evidence, challenges every hypothesis before it can lead, proposes remediation that policy must permit and a person must approve, and only calls an incident resolved when telemetry shows recovery.

Sentinel is built against a controlled reference system that fails on purpose. In the golden scenario a deployment leaks Redis connections: Sentinel must blame the deployment, reject the competing explanations, propose a rollback, wait for approval, execute it through a constrained adapter and verify recovery. In the confounder scenarios the most recent deployment is innocent; others plant an instruction to AI agents in the logs, leave a second cause behind the first remedy, set two equally plausible causes side by side, or run the failing service on AWS. A deterministic deploy-blame baseline runs beside Sentinel on every investigation, and Sentinel has to beat it, not just solve the golden case.

Non-goals for the first milestone are broad vendor coverage, unrestricted autonomous production access, a general-purpose ITSM tool, arbitrary command execution by a model, and custom infrastructure such as databases or message brokers.

Project type

microservices

Status

Sentinel runs end to end on a developer machine. There is no hosted deployment and no CI pipeline yet: the deployment definition is Phase 09's open exit criterion and the pipeline is Phase 00's.

Status by phase, with the evidence for each claim and the exit criteria still open, lives only in the phase status ledger; this README does not repeat it. In outline, Phases 01, 02, 03, 04, 05, 06 and 07 are complete, Phases 00, 01A, 08 and 09 are in progress, and Phase 10 waits for measured need. Of the enhancement phases, E02, E03 and E04 are complete, E01 and E05 are in progress, and E06 and E07 are deferred. Every scenario package, S0 to S11 and the AWS scenarios AWS-S1 and AWS-S7, runs end to end with the deterministic test double, and the evaluation harness scores Sentinel right on the cause in every one with no false remediation, where the deploy-blame baseline would have remediated falsely in more than half. A repeat of the leak is recognised as seen before, S10's first remedy is followed up when it does not end the incident, and when two explanations stand too close to decide between, Sentinel asks one discriminating question before it decides. Judge mode tells any incident as a guided story in which every claim opens its source. With Meta's models investigating, each role on its suggested profile, S1, S2 and S3 are diagnosed correctly as well: S1 is rolled back after approval and recovers, S2 escalates without a rollback, and S3's rollback of the right service awaits approval. What remains open is mostly decisions: the production deployment environment, service-to-service authentication and approver identity.

Models run through Sentinel's own model interface. The default provider is a deterministic test double, so every local run and test is reproducible; the Meta model adapter is used only when it is configured with credentials (see Getting started).

Architecture summary

Sentinel is designed as an autonomic loop split into a control plane, which holds configuration and governance, and a data plane, which runs incidents. Five coarse-grained backend services and one web app carry that design and establish its trust, scaling and failure boundaries from the start (ADR-001). A few principles shape every part of it:

• Vendor neutrality. Providers are translated into canonical observations at the edge; nothing downstream sees a vendor payload.
• Verifiable AI. Models propose evidence and hypotheses through typed tools and validated structured output. A conclusion counts only if it cites observations the tools actually returned, and it must survive an adversarial challenge.
• Human-controlled autonomy. Agents never hold credentials or execute anything. Every action is a typed plan that passes policy, approval when required, a single-use capability and recovery verification.
• Event-sourced truth. Each incident is an append-only event stream; every view, timeline, postmortem and replay is derived from it.
• Failure-aware by default. Outbox and inbox delivery, idempotent commands, fencing tokens, bounded queries and explicit partial-data handling are part of the design, not afterthoughts.

The full design is in the architecture document and the architecture overview. In the diagrams below, solid boxes are implemented and dashed boxes are planned.

The autonomic loop

Sentinel extends the classic monitor-analyze-plan-execute loop with explicit hypothesis, challenge, policy and verification steps. Knowledge spans the whole loop and grows with every incident.

flowchart LR
    monitor["Monitor"] --> correlate["Correlate"] --> investigate["Investigate"] --> hypothesize["Hypothesize"] --> challenge["Challenge"] --> decide["Decide"]
    decide --> plan["Plan"] --> policy["Policy"] --> approve["Approve"] --> execute["Execute"] --> verify["Verify"] --> learn["Learn"]

    subgraph knowledge["Knowledge"]
        catalog["Service catalog and topology"]
        memory["Incident memory"]
        runbooks["Runbooks"]
        policies["Policies"]
        reliability["Source reliability"]
    end

    learn --> memory
    knowledge -.-> investigate
    knowledge -.-> decide
    knowledge -.-> plan

Loading

Control plane and data plane

The control plane is low-volume and strongly consistent: who the tenants are, which services exist and who owns them, which providers and models are configured, and what policy permits. The data plane is high-volume and bursty: it turns telemetry into incidents and incidents into verified outcomes.

flowchart TB
    people(["Responders, approvers and operators"])

    subgraph edge["Edge"]
        web["Sentinel.Web<br/>incidents, timeline, evidence, approvals"]
        api["Sentinel.Gateway<br/>authorization, queries, approval signing"]
    end

    subgraph control["Control plane"]
        tenants["Tenants and environments"]
        catalogcfg["Service catalog and ownership"]
        policycfg["Policies, autonomy modes and operator controls"]
        providercfg["Provider, model and SLO configuration"]
        routing["Notification routing"]
    end

    subgraph data["Data plane"]
        ingest["Ingestion and normalization<br/>event time, watermarks, redaction"]
        bus["Event bus"]
        correlation["Incident correlation"]
        workflow["Durable incident workflow"]
        agents["Specialist investigators"]
        blackboard["Evidence blackboard and causal graph"]
        verifier["Adversarial verifier"]
        consensus["Consensus engine"]
        planner["Remediation planner"]
        gate{"Policy and safety gate"}
        approval["Human approval"]
        autonomous["Constrained autonomous execution"]
        blocked["Suggestion only or denied:<br/>escalate to responders"]
        adapters["Action adapters with capabilities"]
        recovery["Recovery verification"]
        memorystore["Incident memory"]
        replay["Replay and evaluation"]
    end

    subgraph outside["Observed environment and external systems"]
        telemetry[("Logs, metrics, traces,<br/>changes and runbooks")]
        models["Model API"]
        targets["Remediation targets"]
        channels["Notification channels"]
    end

    people --> web --> api
    api --> control
    api --> data
    telemetry --> ingest --> bus --> correlation --> workflow --> agents
    agents --> models
    agents --> blackboard --> verifier --> consensus --> planner --> gate
    gate --> approval --> adapters
    gate --> autonomous --> adapters
    gate --> blocked
    adapters --> targets
    adapters --> recovery --> memorystore
    memorystore -.-> agents
    memorystore --> replay
    routing --> channels

    classDef planned stroke-dasharray: 6 4
    class tenants,autonomous planned

Loading

Tenant and environment identity already scopes every observation, incident and query, and the Gateway routes live events to webhook and log channels by event, severity and tenant. Tenant administration and constrained autonomous execution are planned. The Gateway's control-plane API reads, updates and audits each service's configuration, including the service catalog, policy, operator controls, providers, models and SLO rules, with version history, and the web app has a page for each, plus a manual override that takes one kind of action out of service. Incident memory, replay and evaluation, with a baseline evaluation report, are in place from Phase 07.

Service topology and trust boundaries

Each service owns one concern, one database and one set of credentials. Services integrate only through versioned contracts: events over Redis Streams, written through a transactional outbox and deduplicated by an inbox, and bounded request/response calls. None reads another's tables (ADR-007, ADR-008, ADR-009).

flowchart LR
    web["Sentinel.Web"]

    subgraph edgezone["Edge: identity and approval signing"]
        api["Sentinel.Gateway"]
    end

    subgraph readzone["Read path: provider credentials, untrusted data"]
        ingestion["Sentinel.Ingestion"]
        agentworker["Sentinel.AgentWorker"]
    end

    subgraph corezone["Authoritative state"]
        runtime["Sentinel.Runtime"]
        temporal[("Temporal")]
    end

    subgraph actzone["Privileged path: action credentials"]
        executor["Sentinel.Executor"]
    end

    telemetry[("Prometheus, Loki, Jaeger,<br/>deploy controller, runbooks")]
    model["Meta Model API<br/>or deterministic test double"]
    targets["Deploy controller rollback,<br/>batch control"]

    web -->|"HTTPS"| api
    api -->|"incident queries"| runtime
    api -.->|"approval, rejection, revocation"| runtime
    telemetry -->|"pull, read-only"| ingestion
    ingestion -.->|"metrics, deployments, topology"| runtime
    runtime -.->|"investigate incident"| agentworker
    agentworker -->|"observation and knowledge queries"| ingestion
    agentworker -->|"model calls"| model
    agentworker -.->|"evidence, hypotheses, challenges"| runtime
    runtime -->|"workflow and activities"| temporal
    runtime -->|"recovery queries"| ingestion
    runtime -->|"plan-time policy evaluation"| executor
    runtime -.->|"execute plan with attestation and fencing token"| executor
    executor -->|"start confirmation"| runtime
    executor -->|"approval verification keys"| api
    executor -->|"single-use capability"| targets
    executor -.->|"execution result"| runtime

Loading

Solid arrows are synchronous calls; dotted arrows are events on Redis Streams.

Service	Owns	Holds credentials for	Publishes	Consumes
Sentinel.Gateway	Approvals, signing keys, the external API	Approval signing (local key today, managed key in production)	RecordApproval, RecordRejection, RecordRevocation	None
Sentinel.Ingestion	Source checkpoints, watermarks, canonical observations	Telemetry, change and knowledge providers	ObservationRecorded for metrics, deployments and topology (ADR-018)	None
Sentinel.Runtime	Incident streams, projections, correlation state, incident memory	None	InvestigateIncident, ExecuteRemediation, incident lifecycle events	Observations, findings, approvals, execution results
Sentinel.AgentWorker	Agent executions, tool calls, captured inputs, replays	The model API	InvestigationCompleted	InvestigateIncident
Sentinel.Executor	Policy, operator controls, capabilities, executions	Remediation targets	RemediationExecuted	ExecuteRemediation

The trust rules behind the zones:

• Telemetry, logs and runbooks are untrusted data, never instructions. They are redacted in Ingestion before they are stored or shown to a model, and a runbook that addresses "AI agents" is just text.
• Model output is an untrusted proposal until it passes schema validation, cites only observations its own tool calls returned, and is accepted by Runtime.
• Only Ingestion holds provider credentials and only Executor holds action credentials. AgentWorker reads through Ingestion's query contract and can change nothing.
• Api signs approvals; Executor verifies them; Runtime relays them but cannot create them. Approvals go to Runtime, never straight to Executor (ADR-006).
• Recovery is decided from the expected signals, not from an adapter's success response.

How an incident moves through Sentinel

Detection, investigation and decision, using the golden scenario:

sequenceDiagram
    autonumber
    participant Sys as Observed system
    participant Ing as Ingestion
    participant Rt as Runtime
    participant Wf as Temporal workflow
    participant Aw as AgentWorker
    participant Ex as Executor

    Sys->>Ing: checkout burns its SLO, logs Redis errors, deploys v2.4.1
    Ing->>Ing: pull by event time, redact, assign deterministic IDs
    Ing-->>Rt: ObservationRecorded for metrics, deployments, topology
    Rt->>Rt: correlate the burn with the deployment, open the incident
    Rt->>Wf: start the incident workflow for this cycle
    Wf->>Rt: request one investigation per specialist role
    Note over Rt: remembered similar incidents travel with the request as hints
    Rt-->>Aw: InvestigateIncident for metrics, logs, traces, changes, topology, knowledge
    Aw->>Ing: typed tool queries over the incident window
    Aw-->>Rt: InvestigationCompleted with evidence, hypotheses and unknowns
    Rt->>Rt: accept cited evidence, merge hypotheses on the blackboard
    Wf->>Rt: request the adversarial challenge
    Rt-->>Aw: InvestigateIncident for the verifier
    Aw-->>Rt: challenges and any alternative explanations
    Wf->>Rt: decide with consensus-v4
    Rt->>Ex: evaluate the rollback plan against policy
    Ex-->>Rt: approval required, with the rule trace
    Rt->>Rt: propose plan v1 and await approval

Loading

Approval, execution and verification follow the remediation control path (ADR-006):

sequenceDiagram
    autonumber
    actor Approver
    participant Web as Sentinel.Web
    participant Api as Sentinel.Gateway
    participant Rt as Runtime
    participant Ex as Executor
    participant Tgt as Deploy controller
    participant Ing as Ingestion

    Approver->>Web: review evidence, plan version and hash
    Web->>Api: approve plan v1 with its hash
    Api->>Api: sign an approval attestation
    Api-->>Rt: RecordApproval
    Rt->>Rt: check the hash against the current plan version
    Rt-->>Ex: ExecuteRemediation with plan, attestation, fencing token and idempotency key
    Ex->>Api: fetch verification keys and verify the attestation
    Ex->>Ex: re-evaluate policy and operator controls
    Ex->>Rt: confirm the start, the point of no return
    Ex->>Ex: issue a single-use capability for this action
    Ex->>Tgt: roll back checkout-service from v2.4.1 to v2.4.0
    Ex->>Tgt: verify the action took effect
    Ex-->>Rt: RemediationExecuted
    Rt->>Ing: measure recovery signals across the observation window
    alt signals recovered and the data covers the window
        Rt->>Rt: resolve the incident and record it in memory
    else not recovered, worse than before, or inconclusive
        Rt->>Rt: escalate, requesting compensation where needed
    end

Loading

Inside Runtime

Runtime keeps the incident's truth in its event stream (ADR-004). One Orleans grain per incident is the only writer, so commands for one incident never interleave; a Temporal workflow drives the incident's steps and sees only IDs and small summaries, never telemetry or model output (ADR-005, ADR-017).

flowchart LR
    inbox["Inbox handlers"] -->|"observations"| partitions["CorrelationGrain<br/>per tenant and environment"]
    partitions --> grain["IncidentGrain<br/>single writer"]
    inbox -->|"findings, approvals, execution results"| grain
    services["ServiceGrain<br/>catalog and topology cache"] -.-> partitions
    workflow["Incident workflow<br/>one run per cycle"] --> activities["Activities and incident steps"] --> grain

    subgraph tx["One transaction per command"]
        stream[("Incident event stream")]
        projection[("Incident projection")]
        outbox[("Outbox")]
        memory[("Incident memory and feature index")]
    end

    grain --> stream
    grain --> projection
    grain --> outbox
    grain --> memory
    projection --> queries["Queries for Api"]
    stream --> replay["Domain replay, postmortems, run records"]
    outbox --> bus["Redis Streams"]

Loading

Every command is idempotent and deterministic, so a retried workflow step either appends nothing or is refused by the aggregate's invariants. Projections, timelines, postmortems and replays are all rebuilt from the stream; a domain replay re-runs each recorded decision on the stream prefix it was made from and reports any difference.

Investigation and consensus

Investigation is a fan-out of specialists, each limited to its own tools, followed by a deliberate attempt to prove the leading explanations wrong.

flowchart TB
    command["InvestigateIncident"] --> roles

    subgraph roles["Specialist investigators"]
        direction LR
        metrics["Metrics"]
        logs["Logs"]
        traces["Traces"]
        changes["Changes"]
        topology["Topology"]
        knowledge["Knowledge"]
    end

    roles --> model["Model through IModelProvider<br/>typed tool calls only"]
    model --> validation["Findings validation<br/>schema, cited IDs, hypothesis keys, forbidden actions"]
    validation --> accept["Runtime acceptance<br/>evidence without observations is rejected"]
    accept --> board["Hypothesis blackboard<br/>merge by key, keep lineage"]
    board --> verifier["Adversarial verifier"]
    verifier --> falsifiers["Falsifiers<br/>change served before onset, mechanism follows change,<br/>culprit shift precedes onset, load rose before onset"]
    falsifiers --> consensus["consensus-v4<br/>source reliability, independence families,<br/>contradiction weight, mechanisms, dissent, completeness"]
    consensus --> diagnosed["Diagnosed: plan remediation"]
    consensus --> ambiguous["Two explanations too close:<br/>one discriminating query, once per cycle"]
    ambiguous --> verifier
    consensus --> abstained["Abstained: escalate with what is missing"]

    baseline["Deploy-blame baseline<br/>runs in shadow, recorded, never sent"] -.-> evaluation["Evaluation against ground truth"]
    diagnosed -.-> evaluation

Loading

A falsified hypothesis cannot lead, copies of one signal count once, partial data weighs less, and Sentinel abstains below the confidence threshold instead of guessing. When two explanations survive every check and stand too close, consensus asks the escalation investigator for the one read-only query that separates them, AgentWorker runs it, and the verifier challenges the revised evidence before consensus decides again (Phase E05). Every decision records its algorithm version, the settings it used and a hash of its inputs, so it can be replayed exactly. Every agent execution records its model, prompt version, tool calls and the provider responses it received, so it can be replayed without calling a live provider.

Incident lifecycle

stateDiagram-v2
    [*] --> Detected
    Detected --> Investigating: investigations requested
    Investigating --> AwaitingApproval: plan needs approval
    Investigating --> Escalated: abstained, suggestion only or denied
    AwaitingApproval --> Remediating: approved and execution requested
    AwaitingApproval --> Escalated: rejected or approval revoked
    Remediating --> Verifying: action verified, recovery window scheduled
    Remediating --> Escalated: action failed
    Verifying --> Resolved: recovery signals met
    Verifying --> Detected: unchanged or degraded after the action took effect, one follow-up cycle
    Verifying --> Escalated: worse, inconclusive or a second remedy not recovered
    AwaitingApproval --> Resolved: symptom cleared
    Escalated --> Resolved: symptom cleared
    Resolved --> Detected: symptom returned, new cycle
    Resolved --> [*]

Loading

A later observation can also supersede a decision and return the incident to Investigating; the superseded decision stays in the history.

Deployment topology

Locally, the .NET Aspire AppHost runs everything: the five services, the web app, PostgreSQL with one database per service, the messaging Redis, Temporal and the whole reference system with its observability stack (ADR-010). Tokens between the reference system and Sentinel are generated Aspire parameters kept in user secrets.

The production target keeps the same five deployables and adds what local development does not need. Only the local topology exists today.

flowchart TB
    subgraph local["Local: .NET Aspire AppHost"]
        direction LR
        svc["Five services and Sentinel.Web"]
        pg[("PostgreSQL<br/>one database per service")]
        redis[("Redis<br/>streams and Orleans membership")]
        temporal[("Temporal")]
        ref["Reference system<br/>Prometheus, Loki, Jaeger"]
    end

    subgraph prod["Production target"]
        direction LR
        containers["Service containers<br/>Runtime on a stable cluster for Orleans membership"]
        managed[("Managed PostgreSQL and Redis")]
        temporalprod[("Self-hosted Temporal")]
        secrets["Platform secret provider<br/>and managed signing key"]
    end

    subgraph cells["Long term: cells"]
        direction LR
        global["Global control plane"] --> cella["Cell A"]
        global --> cellb["Cell B"]
    end

    local -.->|"Phase 09"| prod -.->|"Phase 10, on measured need"| cells

    classDef planned stroke-dasharray: 6 4
    class containers,managed,temporalprod,secrets,global,cella,cellb planned

Loading

The reference system

Sentinel is exercised against a controlled distributed system of its own (reference system, ADR-011). A scenario runner drives seeded traffic and timed faults, a deploy controller switches and rolls back revisions, and Prometheus, Loki and Jaeger collect what happens. Sentinel reads that telemetry read-only and acts only through the deploy controller's token-protected API, after approval.

Cross-cutting concerns

• Reliability. Transactional outbox and inbox with dead-letter handling, idempotency keys, fencing tokens on execution, bounded and prioritized ingestion with backoff, a query bulkhead, and explicit completeness metadata so partial data is never mistaken for an absence of evidence.
• Observability. Every service emits OpenTelemetry traces, metrics and logs, with ingestion lag, watermark, model call, token, cost and schema-failure telemetry. The Aspire dashboard shows them locally.
• Auditability and replay. Incident streams, agent executions with captured inputs, policy rule traces and signed approvals make every conclusion and action traceable. Replay is designed at four levels: domain, investigation, model reevaluation and system simulation. The first three are built, and the reference system's scenario runner is the first simulation target.
• Privacy and retention. Redaction happens before storage, incident memory carries retention, legal-hold and privacy metadata, and purges are run by an operator, a dry run first.
• Multi-tenancy. Every observation, incident, memory record and query is scoped by tenant and environment; similar-incident retrieval never crosses tenants.

Technology choices

Concern	Choice	Decision
Backend platform	.NET 10 LTS, ASP.NET Core	ADR-003
Incident state	Event sourcing in PostgreSQL, Microsoft Orleans single writers	ADR-004, ADR-017
Durable workflows	Temporal	ADR-005
Event transport	Redis Streams with outbox and inbox	ADR-008, ADR-012
Persistence	PostgreSQL, one database per service	ADR-009
Local orchestration	.NET Aspire	ADR-010
Providers and models	Sentinel-owned provider and model interfaces; Meta Model API first, deterministic test double by default	ADR-014, ADR-015
Event time	Watermarks and a late-data policy per source	ADR-016
Frontend	Next.js, React, TypeScript, Tailwind CSS, shadcn/ui and Radix UI on a governed design system	ADR-002

All decisions are indexed in docs/decisions. Still open: the production deployment environment. Service-to-service authentication and real approver identity are deferred until production, which brings an identity provider; ADR-022 and ADR-023 are their starting points.

Repository map

• apps/Sentinel.Web: the Next.js frontend and its design system.
• services/: Sentinel.Gateway, Sentinel.Ingestion, Sentinel.Runtime, Sentinel.AgentWorker and Sentinel.Executor, each split into Api, Application, Domain and Infrastructure projects (composition guide); tests sit beside each service.
• packages/: the only shared code (ADR-007): CleanArchitecture.BuildingBlocks for results, messaging and persistence, Sentinel.Contracts for versioned messages, queries and views, and Sentinel.ServiceDefaults for hosting defaults. There is no shared domain package.
• reference-system/: the system Sentinel is tested against (ADR-011): checkout, order and reporting services, the queue consumer, the deploy controller, the observability stack configuration, the scenario manifests and their evaluation packages (S0 to S11, AWS-S0, AWS-S1 and AWS-S7), runbooks and the deploy-blame baseline.
• deploy/local/: the .NET Aspire AppHost that runs everything locally and the end-to-end suites.
• docs/: architecture, decisions, implementation phases, business and API contracts.
• artifacts/: generated diagrams and reports, with their sources.
• scripts/: repeatable build, test and lint automation.
• .agents/, .codex/, .claude/: project steering, commit guidelines and agent adapters.
• personal-docs/: personal-use documents in topic folders, excluded from version control.
• Sentinel.slnx, global.json, Directory.Build.props and Directory.Packages.props pin the SDK, analyzers and package versions for every project.

Getting started

Prerequisites:

• .NET SDK 10.0.302 or a later 10.0.3xx patch, as pinned in global.json.
• Node.js 24 LTS for Sentinel.Web, as pinned in apps/Sentinel.Web/.nvmrc.
• Docker with CLI and engine 25.0 or newer, running. Aspire 13.6 rejects older versions. The AppHost starts PostgreSQL, Redis, Temporal, Prometheus, Loki and Jaeger as containers.

From this directory:

1. Run setup once to restore .NET packages and install the web app's dependencies.
2. Run run to start all five services, the web app and the reference system. The Aspire dashboard opens with every resource's logs, traces, metrics and health.
3. Start a scenario from the scenario-runner resource's commands in the dashboard (Run S1 is the golden scenario), then follow the incident in Sentinel.Web. Judge mode, in the incident's header, tells the same incident as a guided story, live or from its run record.

Local sign-in uses synthetic development identities such as responder@example.test, which exist only in the development configuration. Tokens between the reference system and Sentinel are generated Aspire parameters kept in the AppHost's user secrets; none is stored in the repository.

To investigate with Meta models instead of the test double, put the API key in AgentWorker's user secrets as Model:Meta:ApiKey, set Model:Provider to a Meta profile, and route roles with Model:Routing:<role>. Never commit model credentials. AgentWorker's appsettings.json defines the profiles, each a model with its reasoning effort, reasoning-token allowance and list prices:

Profile	Model	Effort	Suggested roles
meta	muse-spark-1.3	medium	metrics, logs, knowledge
meta-deep	muse-spark-1.3	high	traces, topology, changes
meta-wonder	muse-wonder-1.0	high	escalation, which runs only when two explanations stand too close to decide between and makes one call, about $0.04. It costs about three to five times as much as Muse Spark, so route changes, where a misleading deployment is decided, to it only for demonstrations.
meta-contributor	muse-spark-1.3-contributor	medium	None by default. Meta may train on prompts and outputs in Contributor mode, so use it only with synthetic data.

Findings are requested with the findings JSON Schema alongside the tools, so the API shapes the answer, and reasoning gets its own share of the output limit because it counts against it. Every model call is costed at its profile's price and recorded with the execution. Model:Budget caps the estimated spend of priced models, by default at $100 in total and $15 a day, and muse-wonder-1.0 at $5 a day of its own. At a cap, investigations abstain and say why. An incident's six specialists run side by side (AgentWorker:InvestigationConcurrency, default 6).

Commands

Task	Command
Setup	powershell -NoProfile -File scripts/check.ps1 -Task setup
Run	dotnet run --project deploy/local/Sentinel.AppHost
Test	powershell -NoProfile -File scripts/check.ps1 -Task test
Lint	powershell -NoProfile -File scripts/check.ps1 -Task lint
Build	powershell -NoProfile -File scripts/check.ps1 -Task build

Each task covers the .NET solution and Sentinel.Web and stops at the first failure; on Linux or macOS run the same script with pwsh. build, test and lint need no containers, except for the integration tests that start their own with Testcontainers. test leaves out the local AWS conformance tests (ADR-021), which need its pinned image and the Docker socket; run them with dotnet test reference-system/tests/Reference.Tests --filter Category=Aws. It also leaves out the end-to-end suites below by their Category=EndToEnd trait, so every test class in deploy/local/Sentinel.EndToEnd.Tests must carry it; build still compiles them. The .NET build treats analyzer and code-style warnings as errors, so build also enforces C# style. lint checks C# whitespace, runs ESLint with the design-system rules and typechecks the web app. Sentinel.Web's README lists its own commands.

The end-to-end suites start the whole system and drive real scenarios, so they are slow. They run in tiers, each a filter on the same project; run the smallest tier that covers a change, and everything before a phase is closed:

Tier	Filter	What it runs
Smoke	Tier=Smoke	The healthy control S0, the golden S1 through approved rollback and verified recovery, and the poison message S6 through its quarantine
Reference	Suite=Reference	The reference system on its own, without Sentinel, for changes under reference-system/
AWS	Suite=Aws	AWS-S0, AWS-S1 and AWS-S7 in the AWS environment, for changes to its adapters or the environment; it needs the pinned local AWS image
Everything	none	Every suite, the restart, outage and failure scenarios included

dotnet test deploy/local/Sentinel.EndToEnd.Tests --filter "Tier=Smoke"

A scenario started right after the suites' quiet check has seen a healthy minute warms up for 20 seconds rather than its manifest's full warm-up, which repeats what the check has just seen.

The suites always investigate with the deterministic test double, even when Meta models are configured and routed in user secrets, so their results are reproducible and cost nothing. To run them live, set SENTINEL_E2E_MODEL_PROVIDER=configured to use AgentWorker's own configuration with its routing, or name one profile such as meta for every role. Live runs are slower, cost real money, and their assertions on the diagnosis depend on the model.

Keep this table synchronized with PROJECT.json and the scripts.

Documentation

• Documentation index
• Architecture: the full architecture, the overview, the system architecture and reference system diagrams, and the frontend design system
• Phase-wise implementation plan and the phase status ledger; enhancement phases extend the plan, starting with the AWS reference environment
• Architecture decisions
• Business and API contracts
• Runbooks: emergency stop, models and spend, dead letters, projection rebuild, signing key rotation, and backup and restore; their production versions arrive with the Phase 09 deployment definition.
• Generated diagrams and reports live under artifacts/ with their sources and provenance, for example the walking-skeleton comparison report.

Verification

Run build, test and lint before completing a change. Each service's architecture tests enforce layer direction and reject dependencies on other services, and the end-to-end suites keep the golden path passing across real process boundaries. Structural changes must also pass the workspace project validator. Tests stay with the service or package they verify.

Related learning

No learning-topic links have been established yet.