diff --git a/reference/implementation-specifications.md b/reference/implementation-specifications.md new file mode 100644 index 0000000..aa4d55e --- /dev/null +++ b/reference/implementation-specifications.md @@ -0,0 +1,728 @@ +# DCM Data Model β€” Implementation Specifications + +**Document Status:** πŸ“‹ Draft β€” Ready for Implementation Feedback +**Document Type:** Implementation Reference +**Related Documents:** [Control Plane Components](../architecture/control-plane/components.md) | [data stores](https://github.com/croadfeldt/udlm/blob/main/contracts/storage-providers.md) | [Universal Audit](https://github.com/croadfeldt/udlm/blob/main/observability/universal-audit.md) | [credential management service Model](https://github.com/croadfeldt/udlm/blob/main/governance/credentials.md) | [Deployment Redundancy](../architecture/runtime-features/deployment-redundancy.md) | [Session Revocation](../architecture/control-plane/session-revocation.md) + +> **AEP Alignment:** API endpoints follow [AEP](https://aep.dev) conventions. +> See `schemas/openapi/dcm-admin-api.yaml` and `dcm-consumer-api.yaml` for normative specs. + +--- + +## 1. Purpose + +This document specifies the implementation mechanics for capabilities that are architecturally defined elsewhere but whose runtime behavior β€” enforcement location, algorithm, data structure β€” has not been fully specified. It closes implementation gaps identified in the architecture gap analysis. + +--- + +## 2. Rate Limiting β€” Enforcement Implementation + +Rate limiting is defined at the interface level in the Consumer API Specification (Β§1.6) and the Admin API. This section specifies *how* it is enforced. + +### 2.1 Enforcement Location + +Rate limiting is enforced by the **API Gateway** component β€” the single ingress point for all consumer and admin API traffic. It is enforced before the request reaches any pipeline component. The Request Orchestrator never sees rate-limited requests. + +Rate limiting is **not** enforced at the network layer (load balancer) or application layer (Request Payload Processor). A single enforcement point at the API Gateway ensures: +- Consistent limits across all consumer paths (Web UI, direct API, CI/CD) +- No rate limit bypass via internal component calls +- Single source of rate limit state for accurate tracking + +### 2.2 Token Bucket Algorithm + +DCM uses the **token bucket** algorithm with a per-actor bucket: + +``` +Actor makes request: + β”‚ + β–Ό API Gateway looks up actor_uuid in rate limit store + β”‚ (in-memory cache backed by a fast PostgreSQL store contract) + β”‚ + β–Ό Current bucket state: + β”‚ tokens_remaining: + β”‚ last_refill_at: + β”‚ + β–Ό Refill calculation: + β”‚ elapsed_seconds = now - last_refill_at + β”‚ tokens_to_add = elapsed_seconds Γ— (rate_limit / 60) + β”‚ tokens_remaining = min(bucket_max, tokens_remaining + tokens_to_add) + β”‚ last_refill_at = now + β”‚ + β–Ό Token check: + β”œβ”€β”€ tokens_remaining >= 1: + β”‚ tokens_remaining -= 1 + β”‚ Request proceeds + β”‚ Response headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset + β”‚ + └── tokens_remaining < 1: + Request rejected: 429 Too Many Requests + Response headers: Retry-After, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset + Audit record written: rate_limit_exceeded +``` + +### 2.3 Bucket Parameters by Profile + +| Profile | Rate (req/min) | Burst Max | Bucket Max | +|---------|---------------|-----------|------------| +| `homelab` | 60 | 20 | 80 | +| `dev` | 120 | 40 | 160 | +| `standard` | 300 | 100 | 400 | +| `prod` | 600 | 200 | 800 | +| `fsi` | 600 | 200 | 800 | +| `sovereign` | 600 | 200 | 800 | + +System components (service accounts, provider callbacks) use `prod` bucket parameters regardless of profile. + +### 2.4 Rate Limit State Store + +The rate limit state is stored in a **dedicated in-memory cache** backed by a fast data store: +- Cache TTL: 2Γ— the rate limit window (120 seconds for 60 req/min rate) +- PostgreSQL store contract: key-value (Redis or equivalent) +- Consistency: eventual β€” brief over-counting tolerated to avoid distributed lock overhead +- Cross-replica sharing: rate limit state is shared across all API Gateway replicas via the backing store + +### 2.5 Exemptions + +The following are exempt from consumer rate limits: +- Admin API calls (separate rate limit bucket, 3Γ— profile limit) +- Provider callback endpoints (authenticated via provider callback credential; separate per-provider bucket) +- Internal DCM component calls (authenticated via mTLS + interaction credential; not rate limited) +- Health check endpoints (`/livez`, `/readyz`, `/metrics`) + +### 2.6 System Policies + +| Policy | Rule | +|--------|------| +| `RLM-001` | Rate limiting is enforced at the API Gateway. No other component enforces rate limits. | +| `RLM-002` | Rate limit buckets are per authenticated actor (actor_uuid). Unauthenticated requests are rejected at auth before reaching the rate limiter. | +| `RLM-003` | All rate limit rejections produce an audit record with actor_uuid, endpoint, and timestamp. | +| `RLM-004` | Rate limit parameters are governed by the active Profile. Operators may increase but not decrease profile-defined limits. | +| `RLM-005` | Rate limit state is not persisted across API Gateway restarts. Buckets refill from empty after restart β€” brief over-serving is acceptable. | + +--- + +## 3. Audit Log Hash Chain β€” Verification Schedule and Implementation + +The Merkle-tree structure is defined in [Universal Audit](https://github.com/croadfeldt/udlm/blob/main/observability/universal-audit.md) Β§8. This section specifies the verification schedule, triggering component, and response protocol. + +### 3.1 Hash Computation + +Each audit record's `record_hash` is computed as: + +``` +record_hash = SHA-256( + record_uuid || + record_timestamp || + entity_uuid || + action || + actor.immediate.uuid || + subject_handle || + chain_sequence || + previous_record_hash +) + +Where || denotes canonical concatenation with a field separator (0x1F β€” ASCII unit separator). +The hash is stored as a lowercase hex string. +``` + +The `previous_record_hash` for the first record in an entity's chain is `SHA-256("GENESIS")` β€” a known constant, not null. + +### 3.2 Verification Schedule + +Hash chain verification runs on two schedules: + +**Continuous verification (per-write):** +Every audit record write triggers an immediate verification of that record against its predecessor. This catches chain breaks at write time β€” before the record is committed. A write that would break the chain is rejected and triggers `audit.integrity_alert`. + +**Periodic batch verification (scheduled):** +The Audit component runs a full-chain verification sweep on a profile-governed schedule: + +| Profile | Sweep interval | Scope per sweep | +|---------|---------------|-----------------| +| `dev` | P7D | All entities modified in the last 7 days | +| `standard` | P1D | All entities modified in the last 24 hours | +| `prod` | PT12H | All entities modified in the last 12 hours | +| `fsi` | PT6H | All entities + random 5% sample of all-time records | +| `sovereign` | PT1H | All entities + random 10% sample of all-time records | + +### 3.3 Owning Component + +Hash chain verification is owned by the **Audit component** β€” the same component that writes audit records. It is not a separate service. The Audit component runs verification as a background goroutine with no external trigger required. + +For data store implementations: the Audit Store must support ordered range queries by `(entity_uuid, chain_sequence)` to enable efficient sweep verification. + +### 3.4 Breach Response Protocol + +``` +Chain break detected (during write-time or sweep verification): + β”‚ + β–Ό Affected records flagged: integrity_status = chain_break + β”‚ Break point: chain_sequence N where hash mismatch occurs + β”‚ All records with chain_sequence > N for this entity: integrity_status = unverified + β”‚ + β–Ό audit.integrity_alert event fired (urgency: critical, non-suppressable) + β”‚ payload: {entity_uuid, entity_type, break_at_sequence, break_detected_at, sweep_type} + β”‚ + β–Ό Notifications dispatched: + β”‚ β†’ Platform Admin (urgency: critical) + β”‚ β†’ Security team (if configured in notification routing) + β”‚ + β–Ό Affected entity flagged in audit dashboard + β”‚ Consumer-visible: "Audit integrity alert β€” contact platform admin" + β”‚ + └── Human investigation required: + Normal resolution paths: + - store failure caused write corruption β†’ data store replacement + - Clock skew between replicas caused ordering issue β†’ Non-malicious; document and reseal + - Administrative error (direct DB edit) β†’ Incident report, access review + - Malicious tampering β†’ Security incident declared +``` + +### 3.5 Chain Resealing + +After a chain break is investigated and root cause documented, a platform admin may reseal the chain: + +``` +POST /api/v1/admin/audit/entities/{entity_uuid}:reseal-chain + { + "investigation_reference": "INC-2026-042", + "root_cause": "storage_failure", + "resolution_notes": "PostgreSQL WAL corruption during storage migration" + } + +Response: + { + "entity_uuid": "", + "chain_resealed_at": "", + "records_affected": 7, + "new_chain_anchor": "", + "audit_record_uuid": "" + } +``` + +The reseal itself produces an audit record that references the investigation. Chain integrity is restored from the reseal point forward. + +--- + +## 4. Multi-Tenancy at the Storage Layer + +Tenant isolation in DCM is enforced at the data model level (every entity carries `tenant_uuid`) and at the API level (all consumer endpoints are scoped to the authenticated actor's tenant). This section specifies the storage-layer enforcement mechanisms. + +### 4.1 Isolation Strategy by Store Type + +| Store Type | Isolation Strategy | Implementation Notes | +|-----------|-------------------|---------------------| +| **DCM database** (Intent/Requested State) | Directory namespace per tenant | `/tenants/{tenant_uuid}/intents/`, `/tenants/{tenant_uuid}/requests/` β€” Git ACLs enforce read/write scope | +| **pipeline_events table** (Audit) | Separate stream per tenant | `dcm.audit.{tenant_uuid}` stream; Kafka topic ACLs restrict producer/consumer access | +| **realized data domain** (Realized State) | Row-level filter + column-level encryption | `tenant_uuid` column indexed; all queries mandatory-include `WHERE tenant_uuid = ?`; tenant-scoped encryption key | +| **Search Index** | Index namespace per tenant | Separate index prefix `tenant_{uuid}_*`; query routing enforces tenant scope | +| **Rate Limit Cache** | Key-namespaced per actor (actor carries tenant context) | `rl:{tenant_uuid}:{actor_uuid}` key structure | + +### 4.2 realized data domain β€” Row-Level Security Implementation + +The realized data domain (Realized State) uses row-level security as the primary isolation mechanism: + +```sql +-- PostgreSQL row-level security policy +CREATE POLICY tenant_isolation ON realized_state_records + USING (tenant_uuid = current_setting('dcm.current_tenant_uuid')::uuid); + +-- Every connection sets tenant context before any query: +SET LOCAL dcm.current_tenant_uuid = ''; + +-- This makes it impossible to query across tenant boundaries, +-- even with direct database access using the application credential. +-- Platform admin access uses a separate role without the RLS policy. +``` + +### 4.3 Tenant-Scoped Encryption + +For `fsi` and `sovereign` profiles, realized state records are encrypted at rest using a per-tenant encryption key: + +``` +Tenant provisioned: + β”‚ + β–Ό credential management service generates tenant encryption key (AES-256-GCM) + β”‚ Key stored in: credential management service (e.g., Vault) + β”‚ Key reference stored in: Tenant record as tenant_encryption_key_ref + β”‚ + β–Ό On write to realized data domain: + β”‚ API Gateway fetches tenant encryption key + β”‚ Payload encrypted with tenant key before storage + β”‚ data store stores ciphertext only + β”‚ + β–Ό On read from realized data domain: + β”‚ API Gateway fetches tenant encryption key + β”‚ Decrypts payload in memory + β”‚ Plaintext never written to data store logs + β”‚ + β–Ό Tenant decommission: + Tenant encryption key revoked in credential management service + All tenant data becomes unreadable without external recovery + This is the cryptographic equivalent of data deletion +``` + +### 4.4 Cross-Tenant Query Prevention + +Platform admin endpoints that query across tenants use a separate database role with explicit permission grants β€” they do not bypass RLS, they use a role that has cross-tenant read permission with full audit logging. The principle is: cross-tenant reads are possible only through intentional, audited, privileged operations. + +### 4.5 System Policies + +| Policy | Rule | +|--------|------| +| `STI-001` | Every query to a tenant-scoped store must include `tenant_uuid` as a mandatory predicate. Queries without tenant scope are rejected by the storage layer. | +| `STI-002` | Row-level security is enabled on all relational snapshot stores. Disabling RLS requires platform admin action and produces an audit record. | +| `STI-003` | For `fsi` and `sovereign` profiles, tenant-scoped encryption is mandatory. Key rotation is performed on a profile-governed schedule (P90D for fsi; P30D for sovereign). | +| `STI-004` | data store implementations must declare their tenant isolation strategy at registration. DCM validates the declared strategy against the active profile's isolation requirements during the registration approval pipeline. | + +--- + +## 5. Cross-Region Data Replication + +DCM's multi-region deployment model is specified in [Deployment Redundancy](../architecture/runtime-features/deployment-redundancy.md). This section specifies the replication mechanics and sovereignty enforcement at the replication layer. + +### 5.1 What Replicates Where + +| Store Type | Replication Model | Sovereignty Constraint | +|-----------|------------------|----------------------| +| **DCM database** (Intent State) | Git push/pull β€” upstream-downstream replication | Intent records tagged with `sovereignty_zone`; replicated only to stores within the declared zone | +| **pipeline_events table** (Audit) | Stream mirroring with lag monitoring | Audit records replicated to all authorized regions; cross-sovereignty replication requires explicit consent | +| **realized data domain** (Realized State) | Synchronous within-zone; async cross-zone with consent | `sovereignty_zone` on entity governs which regions may hold a copy | +| **Search Index** | Async replication; eventual consistency acceptable | Same sovereignty rules as realized data domain | + +### 5.2 Sovereignty-Aware Replication + +Every entity carries `sovereignty_zone` declarations that constrain which data store instances may hold copies: + +```yaml +entity: + entity_uuid: + sovereignty_zones: + - zone_id: EU-WEST + data_classifications: [restricted, phi] # these classifications must stay in EU-WEST + - zone_id: "*" + data_classifications: [internal] # internal data may replicate anywhere +``` + +The replication controller evaluates `sovereignty_zones` before routing any replication event. Replication to a non-authorized region for a given data classification is blocked at the replication layer β€” the storage provider receives a `SOVEREIGNTY_VIOLATION` rejection. + +### 5.3 Replication Lag Monitoring + +``` +data store declares: max_replication_lag: PT30S + +DCM monitoring: + Every PT10S: measure replication lag across all replica pairs + + If lag > max_replication_lag: + storage.replication_lag_exceeded event (urgency: medium) + + If lag > 5 Γ— max_replication_lag: + storage.replication_degraded event (urgency: high) + Affected region marked: capacity_status = degraded + New requests avoid degraded region for placement + + If replica unreachable: + storage.replica_unavailable event (urgency: critical) + Affected region marked: capacity_status = unavailable + Requests that require this region: held pending recovery +``` + +### 5.4 Conflict Resolution + +DCM uses a **last-write-wins with causality tracking** model for cross-region conflicts: + +- All writes carry a vector clock `{region_id: sequence_number}` +- Concurrent writes (same entity, different regions) are detected by vector clock comparison +- Resolution: the write with higher aggregate sequence number wins +- Losing write: preserved as a `conflict_record` in the Audit Store (never silently dropped) +- Platform admin notified of conflicts above a configurable threshold + +--- + +## 6. Secret Zero β€” Initial Credential Bootstrap + +The bootstrap sequence is specified in [Deployment Redundancy](../architecture/runtime-features/deployment-redundancy.md) Β§6. This section specifies the credential bootstrap specifically β€” how DCM components authenticate to each other before the credential management service is running. + +### 6.1 The Bootstrap Credential Problem + +At day-0, no credential management service exists. DCM components need credentials to communicate. The resolution is a **declarative bootstrap manifest** that contains one-time bootstrap credentials, plus a mandatory rotation on first successful startup. + +### 6.2 Bootstrap Sequence β€” Credential Perspective + +``` +1. Bootstrap manifest contains: + bootstrap_credentials: + internal_ca: + cert_pem: + key_pem: # sealed with bootstrap passphrase + + bootstrap_admin: + username: bootstrap-admin + password_hash: # operator sets this + + component_credentials: + # Pre-shared credentials for component-to-component auth + # until mTLS internal CA is operational + api_gateway: {shared_secret: } + orchestrator: {shared_secret: } + policy_engine: {shared_secret: } + audit: {shared_secret: } + +2. Bootstrap DCM starts: + - Internal CA initialized from bootstrap_credentials.internal_ca + - Components issued mTLS certificates from Internal CA + - Pre-shared secrets replaced by mTLS certificates on first successful CA handshake + - Pre-shared secrets deleted from memory and manifest after replacement + +3. credential management service starts: + - Bootstrapped with Internal CA certificate (trusts DCM's CA) + - Registered as the primary credential management service via bootstrap admin credential + - Takes ownership of internal CA key management + - Internal CA private key: transferred to credential management service, deleted from bootstrap manifest + +4. Bootstrap admin credential rotation (BOOT-002 β€” mandatory): + - Bootstrap admin password must be rotated on first login + - New credential issued by credential management service (not the bootstrap manifest) + - Old password hash deleted from manifest + - Manifest sealed: no more secrets, only configuration + +5. Bootstrap manifest after completion: + - Contains only: DCM deployment configuration, Git remote, profile + - No secrets remain in the manifest + - Manifest committed to Git (now safe, secret-free) +``` + +### 6.3 Air-Gapped Bootstrap + +For sovereign/air-gapped deployments where the credential management service requires network access to an external vault: + +``` +Option A β€” Embedded credential management service: + Use a locally-running credential management service (e.g., HashiCorp Vault in dev mode) + bootstrapped from the bootstrap manifest. + Upgrade to production Vault config post-bootstrap. + +Option B β€” Operator-held keys: + Bootstrap manifest contains encrypted key material. + Operator provides passphrase at bootstrap time via stdin or hardware token. + Keys are never stored unencrypted at rest. + +Option C β€” HSM-backed bootstrap: + Internal CA private key is generated inside an HSM. + Bootstrap manifest contains only the HSM endpoint and slot reference. + Requires HSM to be available before DCM bootstrap begins. +``` + +### 6.4 System Policies + +| Policy | Rule | +|--------|------| +| `BOOT-001` | The bootstrap manifest must not contain secrets after bootstrap completion. Any secret that persists in the manifest after first successful startup is a security violation. | +| `BOOT-002` | The bootstrap admin credential must be rotated on first login. DCM enforces this β€” the bootstrap admin account is locked from normal use until rotation is complete. | +| `BOOT-003` | Pre-shared component credentials must be replaced by mTLS certificates within PT5M of Internal CA startup. Any component still using pre-shared secrets after this window generates a security alert. | +| `BOOT-004` | The Internal CA private key must be transferred to the credential management service on credential management service registration. The key must not remain in any component's memory or storage after transfer is confirmed. | + +--- + +## 7. Ownership Ambiguities β€” Resolved + +### 7.1 Who Issues `operation_uuid`? + +**Decision: The API Gateway issues `operation_uuid` at request ingress.** + +Rationale: The API Gateway is the component that receives the POST request and must return the Operation response immediately. It assigns the UUID synchronously before any pipeline processing begins. The `operation_uuid` equals the `request_uuid` β€” they are the same UUID, assigned at ingress. + +``` +Consumer: POST /api/v1/requests {...} + +API Gateway: + 1. Authenticates consumer (checks session token) + 2. Assigns request_uuid = operation_uuid = UUID4() ← here + 3. Writes initial request record to Intent Store (status: INITIATED) + 4. Publishes request.initiated event to Request Orchestrator (with request_uuid) + 5. Returns Operation{name: /api/v1/operations/{request_uuid}, done: false} + +Request Orchestrator: + Receives request.initiated event with request_uuid + Uses the already-assigned request_uuid throughout the pipeline + Never assigns a new UUID +``` + +The Operation resource lives in a fast-queryable store owned by the API Gateway. When the pipeline progresses, the Request Orchestrator updates the Operation status by writing to this store (it has write access; the API Gateway reads from it for GET /api/v1/operations/{uuid} responses). + +### 7.2 Who Owns the Credential Revocation Registry? + +**Decision: The credential management service owns the Credential Revocation Registry.** + +Rationale: The credential management service is the authoritative source of credential lifecycle state. It issues credentials, rotates them, and revokes them. The revocation registry is a projection of that lifecycle state optimized for fast lookup. + +``` +Credential Revocation Registry: + Owner: credential management service + Storage: dedicated fast cache (Redis or equivalent) + Key structure: credential_uuid β†’ {revoked_at, revocation_reason, effective_at} + TTL: max(credential_ttl, P90D) β€” persists at minimum 90 days after revocation + +Access model: + Write: credential management service (on revocation event) + Read: All DCM components (via credential management service query API) + OR via local cache synced from credential management service push events + +Cache sync protocol: + credential management service publishes: credential.revoked event (Message Bus) + All subscribed components update local revocation cache + Cache TTL: PT1M standard; PT30S fsi/sovereign + On cache miss: component queries credential management service directly (not the cache) + +Session Revocation Registry: separate, owned by the Auth component + (Session revocation is distinct from credential revocation) +``` + +--- + +## 8. Security Posture Specifications + +### 8.1 Threat Model β€” Attack Surface Summary + +DCM's attack surface has five distinct boundaries. Each boundary has a specific trust model and mitigation set. + +**Boundary 1 β€” Consumer Ingress (Web UI, Consumer API)** +- Threat: Credential theft / session hijacking +- Mitigations: mTLS optional at consumer boundary; bearer tokens with short TTL (PT1H standard); session revocation registry checked on every request; rate limiting at API Gateway +- Threat: Tenant escape (accessing another tenant's data) +- Mitigations: All queries mandatory-include tenant_uuid; row-level security on storage; Governance Matrix enforced before any read + +**Boundary 2 β€” Provider Interface (Operator Interface, Callback API)** +- Threat: Provider impersonation (malicious actor claims to be a legitimate provider) +- Mitigations: mTLS required at provider boundary; provider callback credential required for callbacks; API Gateway validates dcm_entity_uuid in every callback against provider's registered entity scope +- Threat: Malicious provider payload (provider sends crafted Realized State payload) +- Mitigations: Realized State payloads validated against Resource Type Specification schema on receipt; compliance validation policies evaluate provider-supplied data before it enters DCM state + +**Boundary 3 β€” Admin Interface (Admin API)** +- Threat: Unauthorized platform admin action +- Mitigations: Authority Tier model enforces multi-tier approval for high-impact actions; all admin actions produce non-suppressable audit records; emergency admin access (break-glass) triggers immediate security notification +- Threat: Configuration injection via GitOps +- Mitigations: All GitOps PRs require domain-appropriate review before merge; policy contributions enter shadow mode before activation; compliance validation policies validate all contributions at submission + +**Boundary 4 β€” Internal Component Communication** +- Threat: Component impersonation (compromised component issues requests as another) +- Mitigations: mTLS with Internal CA for all component-to-component calls; interaction credentials checked on every call; Credential Revocation Registry queried on credential use +- Threat: Lateral movement after component compromise +- Mitigations: Each component holds minimum-scope interaction credentials; no component has write access to stores it does not own; audit records cannot be deleted by any component + +**Boundary 5 β€” Storage Layer** +- Threat: Direct database access bypassing application controls +- Mitigations: Row-level security enforces tenant isolation even with direct DB access using application credentials; platform admin credentials are separate, audited, and require MFA; data store provenance emission means all direct writes are detectable + +**Highest-risk paths (not mitigated by single control):** +1. credential management service compromise β†’ cascading trust failure. Mitigation: credential management service is air-gapped from consumer traffic; HSM-backed key storage for sovereign profiles; separate backup credential authority. +2. Internal CA compromise β†’ all component trust fails. Mitigation: CA private key held only in credential management service (HSM-backed for fsi/sovereign); CA certificate rotation procedure documented. + +### 8.2 Supply Chain Security + +**Provider OpenAPI Spec Signing:** +All Service Provider OpenAPI specifications submitted at registration must be signed using the provider's private key (corresponding to the public key in their mTLS certificate). DCM verifies the signature before the spec is processed. Unsigned specs are rejected with `SPEC_UNSIGNED` at GATE-SP-01. + +**Operator Container Image Provenance:** +The DCM reference implementation containers are signed using Sigstore (Cosign). Deployment manifests declare the expected image digest. Any container running a different digest triggers drift detection on DCM's own deployment. + +**DCM database Secrets Scanning:** +All content committed to DCM's GitOps stores passes through a secrets scanner before being accepted. The scanner checks for: +- High-entropy strings matching known secret patterns (API keys, tokens, private keys) +- Known credential formats (AWS access keys, GitHub PATs, JWT secrets) +- PEM-encoded private key blocks + +A commit containing detected secrets is rejected with `SECRETS_DETECTED` and an audit record is written. The committing actor is notified. + +**SBOM Declaration:** +Service Providers must declare a Software Bill of Materials reference at registration (optional for Tier 1 `dev` profiles; required for `fsi` and `sovereign`). The SBOM reference is stored in the provider record and included in accreditation evidence. + +### 8.3 System Policies + +| Policy | Rule | +|--------|------| +| `SEC-001` | All provider OpenAPI specs submitted at registration must be signed. Signature verification is performed at GATE-SP-01. | +| `SEC-002` | DCM GitOps stores enforce secrets scanning on all commits. Commits with detected secrets are rejected. | +| `SEC-003` | For `fsi` and `sovereign` profiles, SBOM declaration is mandatory for all Service Providers before activation. | +| `SEC-004` | The Internal CA private key must be stored in an HSM for `sovereign` profile deployments. Software-only key storage is not permitted at sovereign profile. | +| `SEC-005` | Any direct database write to a DCM store that bypasses the application layer is detectable via data store provenance emission. Detection triggers `audit.integrity_alert` for affected records. | + +--- + +## 9. Experience Gap Specifications + +### 9.1 New Tenant Onboarding Flow + +``` +Platform Admin initiates tenant creation: + POST /api/v1/admin/tenants + { + "display_name": "Payments Platform", + "handle": "payments-platform", + "group_class": "tenant_boundary", + "initial_quota_profile": "standard", + "billing_contact": "payments-ops@corp.example.com", + "data_classifications_permitted": ["internal", "restricted"], + "sovereignty_zones": ["EU-WEST"] + } + +DCM auto-provisions: + 1. Tenant entity created (tenant_uuid assigned) + 2. Default resource groups created: + - payments-platform/default (general resources) + - payments-platform/admins (tenant admin group) + 3. Initial quota applied per quota_profile declaration + 4. Tenant admin actor created (if initial_admin_email provided): + - Actor record created + - Welcome notification dispatched with first-login credential + 5. Tenant Git namespace provisioned in GitOps store: + - /tenants/payments-platform/ directory created + - Initial tenant-scope policy stubs committed (shadow mode) + 6. Search index namespace initialized + 7. Audit stream created: dcm.audit.{tenant_uuid} + +Tenant admin completes setup: + 1. First login β†’ mandatory credential rotation (BOOT-002 equivalent) + 2. Configure Auth Provider (or inherit platform default) + 3. Add tenant members (invite by email or LDAP group mapping) + 4. Review and activate initial policy stubs + 5. Submit first service request (onboarding validation complete) + +Onboarding event sequence: + tenant.created β†’ Platform Admin + tenant.member_added Γ— N β†’ new members (welcome email) + tenant.quota_configured β†’ Platform Admin + tenant.onboarding_complete β†’ Platform Admin + Tenant Admin + (fired when first OPERATIONAL entity exists in the tenant) +``` + +### 9.2 Pre-Request Cost Estimation UX + +The consumer experience for cost estimation before committing a request: + +``` +Step 1: Consumer browses catalog + GET /api/v1/catalog/{catalog_item_uuid} + Response includes: cost_estimate: {monthly_usd: 45.00, basis: "declared_static"} + +Step 2: Consumer configures request fields (e.g., selects VM size) + POST /api/v1/cost/estimate + { + "catalog_item_uuid": "", + "fields": {"cpu": 8, "ram_gb": 32, "storage_gb": 200, "environment": "prod"} + } + Response: + { + "estimated_monthly_usd": 187.50, + "cost_basis": "dynamic", + "cost_breakdown": [ + {"component": "compute", "monthly_usd": 120.00}, + {"component": "storage", "monthly_usd": 40.00}, + {"component": "network_egress", "monthly_usd": 27.50} + ], + "disclaimer": "Estimate based on declared provider rates. Actual costs may vary.", + "provider_uuid": null // not yet placed; estimate is across eligible providers + } + +Step 3: Consumer submits request with dry_run: true (optional pre-flight) + POST /api/v1/requests + { + "catalog_item_uuid": "", + "fields": {...}, + "dry_run": true // evaluate policy and placement; do not dispatch + } + Response: Operation with metadata.dry_run_result: + { + "policy_result": "PASS", + "placement_result": { + "selected_provider": "eu-west-prod-1", + "cost_at_selected_provider": 182.00 + }, + "gating_gates": [], + "warnings": ["Storage class 'premium' requested; 'standard' also eligible at $35.00/mo"] + } + +Step 4: Consumer submits without dry_run β†’ actual request +``` + +### 9.3 Provider Sandbox / Test Mode + +Providers can register in sandbox mode for development and certification testing without affecting production routing: + +```yaml +provider_registration: + # ...standard registration fields... + sandbox_mode: true # this provider never receives production requests + sandbox_profile: dev # sandbox providers only activated under dev profile + + # Sandbox providers: + # - Appear in the provider registry with status: sandbox + # - Can be explicitly targeted by test requests (fields.target_provider_uuid) + # - Never appear in placement engine candidate selection for non-test requests + # - Produce full audit records (useful for certification evidence) + # - Subject to same API validation as production providers + # - Can graduate to production via standard registration approval flow +``` + +Test request targeting a sandbox provider: +``` +POST /api/v1/requests +{ + "catalog_item_uuid": "", + "fields": { ... }, + "_test_context": { + "target_provider_uuid": "", + "suppress_billing": true, + "test_label": "certification-run-2026-04-01" + } +} +``` + +### 9.4 SLA/SLO Tracking + +DCM tracks service delivery against declared SLOs at the Resource Type level: + +```yaml +# Declared in Resource Type Specification ([resource-type-spec.schema.json](https://github.com/croadfeldt/udlm/blob/main/registry/resource-type-spec.schema.json)) +resource_type_slo: + resource_type: Compute.VirtualMachine + + slos: + - metric: time_to_operational + target_percentile: p95 + target_value: PT30M # 95% of VMs should be OPERATIONAL within 30 minutes + measurement_window: P7D # measured over trailing 7 days + + - metric: uptime + target_percentile: p99 + target_value: "99.5%" # 99.5% uptime over trailing 30 days + measurement_window: P30D + + - metric: drift_detection_latency + target_percentile: p90 + target_value: PT1H # drift detected within 1 hour of occurrence + measurement_window: P7D +``` + +SLO breach detection: +``` +DCM computes SLO metrics continuously from audit records and entity lifecycle events. + +When a metric crosses a threshold: + slo.breach_approaching (urgency: medium) β€” at 90% of SLO budget consumed + slo.breach_detected (urgency: high) β€” SLO violated + payload: {resource_type, slo_metric, target, actual, measurement_window} + +Consumer-facing: + GET /api/v1/resources/{entity_uuid}/slo-status + Returns: current SLO metrics for the entity's resource type + +Platform admin: + GET /api/v1/admin/slo/report?resource_type=Compute.VirtualMachine&window=P7D + Returns: aggregate SLO performance across all entities of this type +``` + +--- + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).* diff --git a/reference/implementation-standards.md b/reference/implementation-standards.md new file mode 100644 index 0000000..fd064a6 --- /dev/null +++ b/reference/implementation-standards.md @@ -0,0 +1,468 @@ +--- +Document Status: βœ… Stable β€” DCM implementation +Document Type: Reference β€” Implementation Standards +Established: 2026-05-26 +Maps to: udlm/reference/standards-catalog.md +--- + +# DCM Implementation Standards + +> **Selects implementations of the standards listed in UDLM**: +> [udlm/reference/standards-catalog.md](https://github.com/croadfeldt/udlm/blob/main/reference/standards-catalog.md). +> UDLM lists the normative external standards that any DCM-conformant +> realization must consume. This document records the specific +> implementation choices DCM makes: which algorithms, which protocol +> implementations, which OpenAPI conventions, which observability stack, +> which K8s integration, and which compliance configurations per profile. + +--- + +## 1. Cryptographic implementation details + +DCM selects specific algorithms from the UDLM-approved set, profile-governed. + +### 1.1 Algorithm choices by credential type + +| Credential type | Algorithm | Key size | Profile | +|---|---|---|---| +| `api_key` | Cryptographically random | 256 bits min | All | +| `x509_certificate` | Ed25519 (preferred) | 256-bit | standard/prod | +| `x509_certificate` | ECDSA P-384 | 384-bit | fsi/sovereign (FIPS) | +| `x509_certificate` | RSA-4096 | 4096-bit | Permitted for compatibility | +| `ssh_key` | Ed25519 (preferred) | 256-bit | standard/prod | +| `ssh_key` | ECDSA P-384 | 384-bit | fsi (FIPS) | +| `service_account_token` | ES256 (preferred) | P-256 | standard/prod | +| `service_account_token` | RS256 | RSA 4096 | Compatibility | +| `database_password` | Cryptographically random | 128-bit printable | standard | +| `database_password` | Cryptographically random | 256-bit | fsi/sovereign | +| `dcm_interaction` | HS256 or ES256 | 256-bit | All | +| `hsm_backed_key` | ECDSA P-384 | HSM-generated | fsi/sovereign | + +### 1.2 Forbidden algorithms (all profiles) + +DCM rejects credentials using these at issuance regardless of profile: + +- MD5, SHA-1 +- DES, 3DES +- RC4 +- RSA < 2048 +- ECDSA curves weaker than P-256 +- DSA-1024 + +The negative list applies even in `minimal`/`dev` β€” real attacks hit all +deployments. Homelab is not exempt. + +### 1.3 Hash and integrity + +| Use | Algorithm | Profile | +|---|---|---| +| Audit Merkle leaf/tree hash | SHA-256 | All (minimum) | +| Audit Merkle leaf/tree hash | SHA-384 / SHA-512 | fsi/sovereign (preferred) | +| TLS cipher suites | per RFC 8446 (TLS 1.3 mandatory) | All | +| Audit signing | Ed25519 or ECDSA P-384 | All | + +### 1.4 Symmetric encryption + +| Use | Algorithm | Profile | +|---|---|---| +| Credential at-rest | AES-256-GCM | standard+ | +| Credential at-rest | AES-128-GCM permitted | homelab/dev (performance trade) | +| Pipeline payload encryption | AES-256-GCM (envelope) | All | +| Audit record encryption | AES-256-GCM (sovereign) | sovereign | + +### 1.5 FIPS level per profile + +| Profile | FIPS 140 level | +|---|---| +| `homelab` | None required | +| `dev` | None required | +| `standard` | None required | +| `prod` | Level 1 (software-only acceptable) | +| `fsi` | Level 2 (role-based authentication) | +| `sovereign` | Level 3 (physical tamper evidence + identity-based auth) | + +--- + +## 2. Certificate and key management procedures + +### 2.1 Internal CA (built-in) + +DCM ships with an internal CA that issues mTLS certificates to control plane +components. Default configuration: + +```yaml +internal_ca: + algorithm: ECDSA-P-384 + certificate_lifetime: P90D # rotates quarterly + ca_root_lifetime: P10Y # long-lived; offline backup + ocsp_endpoint: /api/v1/internal/ocsp + crl_endpoint: /api/v1/internal/crl + bootstrap_token_lifetime: PT24H # for new component enrollment +``` + +The CA root is stored in DCM's internal secrets table with HSM-backed +encryption in `fsi`/`sovereign` profiles. + +### 2.2 External CA integration (recommended for fsi/sovereign) + +DCM supports replacing the internal CA with an external enterprise CA via +the Credential Provider model. Supported protocols: + +| Protocol | RFC | Implementations | +|---|---|---| +| ACME | RFC 8555 | Let's Encrypt, cert-manager, Venafi, DigiCert | +| EST | RFC 7030 | Cisco CA, Microsoft NDES, Venafi | +| SCEP | RFC 8894 | Microsoft NDES, Cisco iOS CA | +| CMP | RFC 4210 | EJBCA, OpenXPKI | +| Native API | β€” | HashiCorp Vault PKI, AWS ACM PCA, Azure Key Vault | + +This makes DCM's internal mTLS fully auditable through existing enterprise +PKI infrastructure β€” a key requirement for fsi/sovereign profiles. + +### 2.3 Certificate rotation + +- Internal CA-issued certs rotate at P90D (default); P30D in fsi/sovereign +- External CA-issued certs rotate per CA's protocol (ACME orders are + short-lived; EST/SCEP follow enterprise schedule) +- Rotation overlap window: P7D β€” both old and new certs accepted during + the window +- Pre-expiry warning: P14D before expiry + +### 2.4 OCSP and CRL + +- Internal CA exposes OCSP per RFC 6960 with `ocsp_stapling: true` +- CRL refresh: PT5M cache (standard); PT1M (fsi); PT30S (sovereign) +- External CA OCSP/CRL endpoints consumed via standard PKI library + +--- + +## 3. Authentication protocol integration + +### 3.1 OIDC implementation + +DCM uses standard OIDC discovery via the `/.well-known/openid-configuration` +endpoint. ID tokens validated per RFC 7519. Claims mapping configurable per +Auth Provider registration: + +```yaml +claims_mapping: + username: preferred_username + email: email + display_name: name + groups: groups # mapped to DCM roles per group_role_map + department: department # custom claim + cost_center: cost_center # custom claim +``` + +### 3.2 LDAP/AD implementation + +- RFC 4511 LDAP v3 +- StartTLS (RFC 4513) or LDAPS (port 636) β€” mandatory in standard+ +- Bind operation for authentication +- Group membership via `member` attribute or RFC 4510 `LDAP_MATCHING_RULE_IN_CHAIN` + (1.2.840.113556.1.4.1941) for AD nested group resolution +- Connection pooling; idle timeout PT5M; max idle conns: 10 per pool + +### 3.3 SCIM 2.0 implementation + +- RFC 7643 (Core Schema) + RFC 7644 (Protocol) +- Endpoint: `/scim/v2/Users` and `/scim/v2/Groups` +- Bearer-token authenticated; token rotation per AUTH-014 +- Deprovision (DELETE /Users/{id}) triggers AUTH-016 (session revoke) + + CPX-006 (credential revoke) in parallel +- Roles NOT SCIM-provisioned β€” explicit DCM policy authorization required + +### 3.4 SAML 2.0 (optional) + +- OASIS SAML 2.0 assertion parsing +- Signing cert + encryption cert via Credential Management Service +- AttributeStatement β†’ DCM claims mapping + +### 3.5 OAuth 2.0 (RFC 6749) + +- Authorization Code flow for OIDC Auth Providers +- Client Credentials flow for service account API keys +- Token introspection per RFC 7662 at `POST /api/v1/auth:introspect` + +--- + +## 4. OpenAPI implementation specifics + +DCM uses OpenAPI 3.1 (OpenAPI Initiative 3.1) for all REST APIs. + +### 4.1 API specification locations + +| API | Spec file | +|---|---| +| Consumer API | `schemas/openapi/dcm-consumer-api.yaml` | +| Admin API | `schemas/openapi/dcm-admin-api.yaml` | +| Provider Callback API | `schemas/openapi/dcm-provider-callback-api.yaml` | +| Operator Interface | `schemas/openapi/dcm-operator-api.yaml` | + +### 4.2 Endpoint conventions + +DCM follows [AEP (API Enhancement Proposals)](https://aep.dev) conventions: + +- Resource-oriented URLs: `/api/v1/{resource_collection}/{resource_id}` +- Custom methods use colon syntax: `/api/v1/requests/{uuid}:cancel` +- Async operations return Operation resources for long-running work +- Pagination: `page_size` + `page_token` (cursor-based) +- Filtering: `filter` query param with simple expression language +- Field masks: `fields` query param for partial response + +### 4.3 Versioning headers + +- `Sunset` header (RFC 8594) on deprecated endpoints +- `Deprecation` header (RFC 9745) paired with Sunset +- `Link: rel="successor-version"` on deprecated responses +- Version negotiation via `Accept: application/vnd.dcm.v2+json` + +### 4.4 Error format + +```json +{ + "error": { + "code": "VALIDATION_FAILED", + "message": "Resource type Compute.VirtualMachine version 2.0.0 not active", + "details": { + "resource_type": "Compute.VirtualMachine", + "version": "2.0.0", + "current_version": "1.5.3" + }, + "request_id": "", + "documentation_url": "https://docs.dcm-project/errors/VALIDATION_FAILED" + } +} +``` + +--- + +## 5. Observability implementation + +### 5.1 Metrics (Prometheus / OpenMetrics) + +DCM exposes Prometheus exposition format at `GET /metrics` on each control +plane component. Standard metric families: + +| Family | Description | +|---|---| +| `dcm_requests_total` | Counter β€” requests by tenant, resource_type, status | +| `dcm_requests_duration_seconds` | Histogram β€” request pipeline duration | +| `dcm_provider_health_status` | Gauge β€” 1=healthy, 0=unhealthy per provider | +| `dcm_policy_evaluation_seconds` | Histogram β€” policy evaluation duration | +| `dcm_placement_decisions_total` | Counter β€” placement outcomes by provider | +| `dcm_audit_records_total` | Counter β€” audit volume | +| `dcm_credential_operations_total` | Counter β€” by operation type, profile | +| `dcm_realized_entities` | Gauge β€” current entities by resource_type, tenant | + +Metrics are scraped by Prometheus; profile-governed visibility (some sensitive +metrics hidden in `sovereign` per HLT-005). + +### 5.2 Distributed tracing (OpenTelemetry) + +DCM emits OTel spans for the request pipeline: + +| Span | Parent | +|---|---| +| `dcm.request.submit` | β€” (root) | +| `dcm.request.assemble` | dcm.request.submit | +| `dcm.policy.evaluate` | dcm.request.assemble | +| `dcm.scoring.compute` | dcm.policy.evaluate | +| `dcm.placement.decide` | dcm.request.assemble | +| `dcm.provider.dispatch` | dcm.request.assemble | +| `dcm.provider.callback` | β€” (linked via X-DCM-Correlation-ID) | +| `dcm.realized.persist` | dcm.provider.callback | + +`X-DCM-Correlation-ID` header propagated across all service boundaries; +maps to the OTel trace ID for cross-service tracing. + +### 5.3 Logging + +- Structured JSON logs at all log levels +- `correlation_id`, `request_id`, `actor_uuid`, `tenant_uuid` in every log + entry +- Log levels: `error`, `warn`, `info`, `debug` +- Profile defaults: `info` for standard+, `debug` for dev/minimal +- Log aggregation: not prescribed (Loki, Elasticsearch, Splunk all supported) + +### 5.4 Health endpoints + +- `/livez` β€” process liveness; unauthenticated +- `/readyz` β€” ready to serve traffic; unauthenticated +- `/api/v1/admin/health` β€” detailed component health; authenticated +- `/api/v1/admin/health/dependencies` β€” external dependency status + +Format: IANA `application/health+json` per RFC 8615. + +--- + +## 6. Kubernetes integration + +DCM is designed for K8s deployment but does not require it. K8s-specific +features when deployed on K8s: + +### 6.1 K8s manifests + +- Deployment / StatefulSet per control plane component +- Service for internal service discovery +- HorizontalPodAutoscaler for stateless services +- PodDisruptionBudget for stateful components +- NetworkPolicy for component-to-component allowed paths +- ServiceAccount per component with minimum-necessary RBAC + +### 6.2 CRD-based DCM Operator + +Optional DCM Operator translates DCM artifacts to/from K8s CRDs: + +| CRD | DCM artifact | +|---|---| +| `DCMRequest` | Resource request (consumer-friendly K8s-native intent) | +| `DCMPolicy` | Policy artifact | +| `DCMProvider` | Provider registration | +| `DCMTenant` | Tenant configuration | + +CRDs enable `kubectl apply -f` workflow as an ingress alongside the API. + +### 6.3 Probes + +- Liveness probe: `GET /livez` every PT10S; failureThreshold 3 +- Readiness probe: `GET /readyz` every PT5S; failureThreshold 1 +- Startup probe: `GET /livez` every PT5S; failureThreshold 12 (PT60S total) + +### 6.4 Service mesh integration + +Optional service mesh (Istio, OpenShift Service Mesh, Linkerd) replaces +application-level TLS for internal component communication. DCM's +implementation-specifications.md documents the recommended mesh config. + +--- + +## 7. Compliance configuration + +DCM ships profile-specific compliance overlays. Per profile, the following +standards are enforced: + +### 7.1 Profile β†’ compliance mapping + +| Profile | Standards | +|---|---| +| `homelab` | None enforced | +| `dev` | None enforced | +| `standard` | ISO 27001 (advisory); SOC 2 (advisory) | +| `prod` | ISO 27001; SOC 2 Type II; NIST 800-53 (Moderate baseline advisory) | +| `fsi` | All of prod + PCI DSS overlay + HIPAA overlay (when active) + FedRAMP Moderate + DoD IL4 + FIPS 140-2 Level 2 | +| `sovereign` | All of fsi + sovereign-specific overlay + FIPS 140-3 Level 3 + air-gapped enforcement | + +### 7.2 Compliance overlay activation + +Compliance domains are activated additively: + +```yaml +active_profile: prod +active_compliance_domains: [hipaa, pci_dss] +``` + +DCM applies the union of profile defaults and all active compliance domain +overlays. Compliance overlays always tighten (never relax) the base profile +(`CPX-011`). + +### 7.3 Standard-to-DCM-feature mapping + +| Standard | DCM features | +|---|---| +| NIST SP 800-53 | Policy domains map to control families; access control, audit, configuration management | +| FedRAMP Moderate | Profile `fedramp_moderate`; NIST baseline; FIPS 140-2 Level 1+ | +| FedRAMP High | Profile `fedramp_high`; NIST baseline; FIPS 140-2 Level 2+; enhanced audit retention | +| DoD IL4 | Profile `dod_il4`; FIPS 140-2 Level 2; hardware attestation; enhanced logging | +| PCI DSS v4 | Req 8.3.9: P90D max credential rotation enforced; segmentation via sovereignty; cardholder data logging; 12-month audit retention | +| HIPAA | Profile `fsi` or `hipaa` overlay; PHI access logging; minimum necessary RBAC; transmission security TLS 1.2+; workforce MFA | +| SOC 2 | profile standard+; Type II audit trail; availability/security/confidentiality; GitOps change management | +| ISO 27001 | All profiles; risk-based approach; asset management; access control; cryptography | +| GDPR | Sovereignty constraints; data classification; right to erasure (entity decommission + audit retention) | +| Schrems II | Sovereignty + federation boundaries; data transfer restrictions | + +### 7.4 Authentication Assurance Levels (NIST SP 800-63B) + +DCM maps profiles to AAL: + +| Profile | AAL | Requirements | +|---|---|---| +| `homelab` | AAL1 | Single-factor; password or API key | +| `dev` | AAL1 | Single-factor | +| `standard` | AAL2 | MFA required for actor sessions | +| `prod` | AAL2 | MFA required; TOTP, FIDO2, or hardware token | +| `fsi` | AAL2+ | MFA required; phishing-resistant (FIDO2/hardware) | +| `sovereign` | AAL3 | Hardware-based authenticator; verifier impersonation resistance | + +--- + +## 8. ITSM integration standards + +| Standard | Use | Obligation | +|---|---|---| +| ServiceNow REST Table API | Primary ServiceNow integration | Normative for ServiceNow provider | +| Jira REST API v3 | Primary Jira Service Management | Normative for Jira provider | +| BMC AR REST API v1 | BMC Remedy/Helix ITSM | Normative for BMC provider | +| PagerDuty Events API v2 | Incident creation for alert-type integrations | Normative for PagerDuty | +| HMAC-SHA256 | Inbound webhook signature verification | Normative | +| ITIL v4 Change Management | DCM change record lifecycle mapping | Informative | +| JSON:API | Several ITSM REST APIs response formatting | Informative | +| JSONPath | Template expression for `generic_rest` action payloads | Normative for generic_rest | + +See [`../architecture/integrations/itsm.md`](../architecture/integrations/itsm.md) +for the ITSM integration implementation. + +--- + +## 9. CNCF ecosystem + +DCM is designed for CNCF ecosystem compatibility: + +| Project | CNCF Status | DCM Use | +|---|---|---| +| Kubernetes | Graduated | Deployment target; DCM Operator | +| Open Policy Agent (OPA) | Graduated | Policy evaluation engine; Rego for Validation | +| Prometheus | Graduated | Metrics exposition | +| OpenTelemetry | Graduated | Distributed tracing; correlation ID propagation | +| Istio | Graduated | Optional service mesh for internal mTLS | +| Argo CD / Flux | Graduated | GitOps delivery for DCM layers/policies | + +--- + +## 10a. Identifier implementation (UUID β€” RFC 9562) + +DCM realizes udlm `contracts/identifier-scheme.md` Β§2.1 (normative). Implementation requirements: + +- **Generation:** entity/artifact identity = **UUIDv4** from the platform CSPRNG + (`uuid.uuid4()`, `crypto.randomUUID()`, `gen_random_uuid()`); audit-chain leaves and event ids = + **UUIDv7** (RFC 9562 Β§5.7) for index locality + total order. No other versions are ever minted. +- **Validation at ingest (every API boundary):** reject malformed UUIDs and prohibited versions β€” + check the version nibble AND variant bits + (`^[0-9a-f]{8}-[0-9a-f]{4}-[47][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`, version per field + schema). Error: `validation.identifier_invalid`. +- **Canonical form everywhere:** lowercase hyphenated, no braces, no `urn:` prefix β€” normalize on + write, compare case-insensitively never (storage is already canonical). +- **Storage:** native `uuid` column type where the store supports it (Postgres `uuid`, + `gen_random_uuid()` default for v4); never store as free text. +- **Resolution:** cross-entity references resolve by `uuid` (authoritative); `handle` is advisory + display/lookup sugar and every handle change is audited (identifier-scheme Β§2.2). Resolution by + name alone is a conformance violation (udlm SPEC-DESIGN hard constraint 30). +- **Lifecycle:** uuids are minted exactly once, survive tenant/realization migration, remain + resolvable after `retired` (tombstone, DEP-007), and are NEVER reused (identifier-scheme Β§5). + +## 10. Policy-family-to-standard mapping (DCM realization) + +| Policy family | Standards basis | +|---|---| +| `AUTH-001-DCM` through `AUTH-015-DCM` | RFC 6749, RFC 7519, OIDC Core, NIST SP 800-63B, RFC 7643/7644 | +| `SES-001-DCM` through `SES-005-DCM` | RFC 7662, RFC 7009, OAuth 2.0 best practices | +| `CPX-001-DCM` through `CPX-012-DCM` | FIPS 140-2/3, RFC 5280, RFC 8555/7030/8894/4210, NIST SP 800-57 | +| `ATM-001-DCM` through `ATM-012-DCM` | ISO 27001 change management | +| `EVT-001-DCM` through `EVT-007-DCM` | OpenTelemetry, CNCF event-driven best practices | +| `VER-001-DCM` through `VER-009-DCM` | RFC 8594, RFC 9745, industry API lifecycle | +| `ICOM-001-DCM` through `ICOM-009-DCM` | RFC 8446, RFC 5280, SPIFFE conceptual, FIPS 140 | +| `HLT-001-DCM` through `HLT-006-DCM` | RFC 8615, K8s probe conventions, Prometheus OpenMetrics | +| `ZTS-001-DCM` through `ZTS-005-DCM` | Zero Trust Architecture (NIST SP 800-207), NIST SP 800-63B | + +The full list of policy IDs lives in each subsystem document under +`../architecture/`. diff --git a/reference/operational-reference.md b/reference/operational-reference.md new file mode 100644 index 0000000..2f210b5 --- /dev/null +++ b/reference/operational-reference.md @@ -0,0 +1,521 @@ +# DCM Data Model β€” Operational Reference + +**Document Status:** πŸ”„ In Progress +**Document Type:** SRE Reference β€” GitOps Scale, Store Migration, Disaster Recovery +**Related Documents:** [data stores](https://github.com/croadfeldt/udlm/blob/main/contracts/storage-providers.md) | [Deployment and Redundancy](../architecture/runtime-features/deployment-redundancy.md) | [Four States Model](https://github.com/croadfeldt/udlm/blob/main/foundations/four-states.md) | [Internal Component Authentication](../architecture/control-plane/internal-component-auth.md) | [DCM Self-Health Endpoints](../architecture/control-plane/self-health.md) + +> **Audience:** Platform engineers and SREs operating DCM in production. This document covers three operational concerns that require guidance beyond the architectural specifications: GitOps store partitioning at large scale, migrating between store implementations, and recovering from failure scenarios. + +--- + +## 1. DCM database Scale and Partitioning + +### 1.1 When a Repo Becomes Too Large + +The default DCM GitOps layout uses one repository per store type (Intent, Requested, Layer, Policy). For most deployments this is correct. At large scale β€” tens of thousands of active entities or hundreds of active tenants β€” a single repository can exhibit: + +- Git operation latency (clone, fetch, log) growing beyond SLA +- CI/CD pipeline fan-out delays as every write triggers the full repository +- Access control granularity limits (all tenants share one repo) +- Search index sync lag from large diffs + +**Thresholds that suggest partitioning:** + +| Signal | Threshold | Recommended action | +|--------|-----------|-------------------| +| Entities in Intent/Requested store | > 50,000 active | Consider tenant-shard partitioning | +| Git clone time | > PT30S | Add shallow-clone depth; consider partitioning | +| PR merge latency | > PT5M | Partition or add write buffer | +| Tenant count | > 500 | Consider per-tenant repositories | +| Repository size on disk | > 10 GB | Partition | + +These are guidelines, not hard limits. Profile, hardware, and Git host performance all affect the actual inflection point. + +### 1.2 Partitioning Strategies + +DCM supports three partitioning strategies. All are compatible with the storage contract β€” partitioning changes how stores are organized, not what the store contract requires. + +#### Strategy A β€” Tenant Shard Partitioning (recommended for most) + +Split each store type into N shard repositories, with tenants assigned to shards by a deterministic hash of `tenant_uuid`: + +``` +shard = hash(tenant_uuid) % N + +dcm-intent-shard-0/ ← tenants whose hash(uuid) % N == 0 +dcm-intent-shard-1/ ← tenants whose hash(uuid) % N == 1 + tenants/ + {tenant-uuid}/ + requests/ + {request-uuid}/ + intent.yaml +``` + +**DCM configuration:** +```yaml +gitops_store: + intent_store: + partitioning: tenant_shard + shard_count: 8 + shard_routing: hash_mod # deterministic; no routing table needed + repositories: + - shard: 0 + url: https://git.corp/dcm/dcm-intent-shard-0 + - shard: 1 + url: https://git.corp/dcm/dcm-intent-shard-1 + # ... +``` + +**Operational implications:** Adding shards requires re-hashing. Plan shard counts for 3–5 years of expected growth; use a power of 2 to simplify future doubling. + +#### Strategy B β€” Per-Tenant Repositories (for strict isolation) + +Each tenant has its own set of store repositories. Used when: +- Tenants are separate organizations (MSP model) +- Compliance requires complete data isolation per tenant +- Different retention policies per tenant + +``` +dcm-intent-{tenant-uuid}/ ← one repository per tenant + requests/ + {request-uuid}/ + intent.yaml +``` + +**Operational implications:** Repository count scales with tenant count. Requires automation for tenant onboarding (repository creation, access provisioning). Git host must support large numbers of repositories. + +#### Strategy C β€” Time-Based Archiving (for retention management) + +Active entities stay in the primary repository. Entities past a declared age threshold are archived to read-only archive repositories: + +``` +dcm-intent-active/ ← current entities (hot) +dcm-intent-archive-2025/ ← entities from 2025 (cold, read-only) +dcm-intent-archive-2024/ ← entities from 2024 (cold, read-only) +``` + +DCM's audit and search components are configured with both active and archive repository lists. Strategy C is typically combined with Strategy A or B. + +### 1.3 Large-Scale Layer Store Partitioning + +The Layer Store grows more slowly than the Intent/Requested stores (layers are reused across requests). Layer Store partitioning is by domain rather than by tenant: + +``` +dcm-layers-compute/ ← Compute.* resource type layers +dcm-layers-network/ ← Network.* resource type layers +dcm-layers-storage/ ← Storage.* resource type layers +dcm-layers-platform/ ← Platform.* and cross-cutting layers +dcm-policies-core/ ← System and core policies +dcm-policies-tenant/ ← Tenant-contributed policies (per tenant or sharded) +``` + +Domain-partitioned Layer stores are configured in DCM's layer assembly engine: + +```yaml +layer_store: + repositories: + - domain: Compute.* + url: https://git.corp/dcm/dcm-layers-compute + priority: provider_contribution # provider contributions go here + - domain: Network.* + url: https://git.corp/dcm/dcm-layers-network + - domain: "*" # catch-all for uncategorized + url: https://git.corp/dcm/dcm-layers-platform +``` + +### 1.4 Shallow Clones and Read-Only Mirrors + +For read-heavy operations (audit, search index rebuild, drift reconciliation) that do not need full Git history: + +```yaml +gitops_store: + read_operations: + clone_depth: 1 # shallow clone for read-only consumers + use_mirror: true # read from read-only mirror; writes go to primary + mirror_url: https://git-mirror.corp/dcm/ + mirror_sync_lag_max: PT5M # alert if mirror is more than 5 minutes behind +``` + +--- + +## 2. Store Migration + +### 2.1 Migration Principles + +DCM store migrations follow three invariants: + +1. **No data loss** β€” every record in the source store must exist in the target store after migration +2. **Audit continuity** β€” the audit Merkle tree must remain consistent across the migration; a consistency proof must hold between the pre- and post-migration signed tree heads, and `leaf_index` must continue monotonically +3. **Read availability during migration** β€” DCM continues serving read requests throughout; write availability may be briefly paused during cutover + +### 2.2 Migration Playbook Structure + +Every store migration follows this pattern regardless of source or target implementation: + +``` +Phase 1 β€” Prepare + β”‚ Provision target store alongside source + β”‚ Validate target store meets storage contract (health check, write test, read test) + β”‚ Configure DCM to write to BOTH source and target (dual-write mode) + β”‚ +Phase 2 β€” Backfill + β”‚ Export all existing records from source + β”‚ Import records to target in chronological order (preserving provenance timestamps) + β”‚ Verify record counts match; spot-check content hashes + β”‚ +Phase 3 β€” Validate + β”‚ Run DCM's store validation suite against target + β”‚ Verify audit chain integrity on target store + β”‚ Verify search index can be rebuilt from target store + β”‚ +Phase 4 β€” Cutover + β”‚ Brief write pause (PT30S–PT5M depending on profile) + β”‚ Disable dual-write; switch DCM to target as primary + β”‚ Verify /readyz returns healthy + β”‚ Resume writes to target only + β”‚ +Phase 5 β€” Decommission source (after burn-in period) + Default burn-in: P30D (standard), P90D (fsi/sovereign) + Keep source in read-only mode during burn-in for rollback +``` + +### 2.3 Common Migration Paths + +#### SQLite β†’ PostgreSQL (homelab/dev β†’ standard) + +Typical trigger: scaling beyond single-node evaluation environment. + +```bash +# Step 1: Export from SQLite +dcm-admin store export \ + --store realized \ + --format jsonl \ + --output realized-export.jsonl + +# Step 2: Import to PostgreSQL +dcm-admin store import \ + --store realized \ + --source realized-export.jsonl \ + --target postgres://pg-host:5432/dcm_realized \ + --validate-chain + +# Step 3: Enable dual-write +dcm-admin store dual-write enable \ + --store realized \ + --primary sqlite://dcm-realized.db \ + --secondary postgres://pg-host:5432/dcm_realized + +# Step 4: Validate +dcm-admin store validate \ + --store realized \ + --target postgres://pg-host:5432/dcm_realized \ + --check-count --check-chain --check-spot-sample 0.05 + +# Step 5: Cutover +dcm-admin store cutover \ + --store realized \ + --target postgres://pg-host:5432/dcm_realized +``` + +#### PostgreSQL single-instance β†’ CockroachDB / PostgreSQL HA + +Typical trigger: HA requirement for production; multi-region deployment. + +**Key difference from SQLite β†’ PostgreSQL:** CockroachDB uses serializable isolation and distributed transactions. Test write throughput under realistic load before cutover β€” CockroachDB's latency profile differs from single-node PostgreSQL. + +```yaml +# Pre-migration checklist +migration_checklist: + - Load test target under realistic DCM write volume (PT4H minimum) + - Verify CockroachDB schema compatibility (DCM uses standard PostgreSQL wire protocol) + - Configure connection pooler (PgBouncer or similar) β€” CockroachDB default connection count + - Verify time synchronization (CockroachDB requires NTP within PT500MS across nodes) + - Test audit chain write under partition scenario +``` + +#### DCM database β€” Repo Restructuring + +Restructuring a GitOps repository (e.g. monorepo to sharded) requires special handling because Git history must be preserved. + +``` +Step 1: Enable write buffer β€” all new writes queue while migration proceeds +Step 2: git filter-repo or git subtree to extract tenant directories to shard repos +Step 3: Validate file counts and content hashes in each shard +Step 4: Update DCM gitops_store configuration to point to shards +Step 5: Drain write buffer β€” queued writes replay to new shard repos +Step 6: Verify search index rebuild from shards +Step 7: Archive or delete monorepo after burn-in period +``` + +### 2.4 Rollback Procedure + +If migration fails before cutover: disable dual-write, discard target, no impact to production. + +If migration fails after cutover (during burn-in): + +``` +1. Alert: /readyz reports degraded or source store discrepancy detected +2. dcm-admin store rollback --store --to source + (requires source still in read-only mode β€” NOT decommissioned) +3. DCM restarts reads/writes from source +4. Export any writes that reached target but not source (if any, during dual-write gap) +5. Import gap records to source +6. Re-enable source as primary +``` + +**This is why burn-in period exists.** Do not decommission source stores until burn-in completes. + +### 2.5 Profile-Governed Migration Constraints + +| Profile | Min dual-write duration | Max cutover pause | Burn-in period | +|---------|------------------------|-------------------|----------------| +| `homelab` | P1D | PT5M | P7D | +| `dev` | P3D | PT5M | P14D | +| `standard` | P7D | PT2M | P30D | +| `prod` | P14D | PT1M | P30D | +| `fsi` | P30D | PT30S | P90D | +| `sovereign` | P60D | PT30S | P90D | + +--- + +## 3. Disaster Recovery Runbook + +### 3.1 DCM Recovery Architecture + +DCM's recovery model is built on a key property: **all durable state is in the stores, not in the control plane.** Control plane components (Policy Engine, Request Orchestrator, etc.) are stateless and can be restarted without data loss. Recovery from most failures is component restart, not data restoration. + +The five DCM stores and their recovery characteristics: + +| Store | Implementation | Data durability | Recovery method | +|-------|---------------|----------------|-----------------| +| Intent Store | GitOps (Git) | Git replication + remote | Re-clone from remote | +| Requested Store | GitOps or write-once | Git replication / DB replication | Re-clone or DB restore | +| Layer Store | GitOps (Git) | Git replication + remote | Re-clone from remote | +| Realized Store | Write-once (PostgreSQL/CockroachDB) | DB replication / WAL | DB failover or restore | +| Audit Store | Append-only (Kafka/PostgreSQL) | Replication / WAL | Kafka failover or restore | + +### 3.2 Recovery Scenarios and Procedures + +#### Scenario 1: Single Component Failure (Most Common) + +**Symptoms:** One DCM component (e.g. Policy Engine) is unhealthy. `/readyz` shows degraded. Requests may be delayed but not lost. + +**RTO:** PT5M +**RPO:** 0 (no data loss β€” components are stateless) + +``` +1. Identify failing component via GET /api/v1/admin/health +2. Check component logs for panic/OOM/deadlock +3. Kubernetes: pod restart is automatic (liveness probe) + Manual: kubectl rollout restart deployment/dcm-policy-engine +4. Monitor /readyz β€” should recover within PT2M of pod restart +5. If component repeatedly fails: check Internal CA cert expiry (ICOM-006) + dcm-admin component cert-status --component policy-engine +6. Write post-incident note to DCM audit store +``` + +#### Scenario 2: Store Failure (Database / Kafka) + +**Symptoms:** `/readyz` fails specific store check. Requests queue or fail depending on which store. + +**Realized Store failure (highest severity β€” blocks realization):** + +``` +RTO target: PT30M (standard), PT15M (prod), PT5M (fsi/sovereign) +RPO: 0 for PostgreSQL HA (synchronous replication); near-zero for async + +1. Confirm store failure: GET /api/v1/admin/health β†’ realized_store: fail +2. If HA: check if automatic failover occurred + kubectl get pods -n dcm-stores | grep postgres + Check PostgreSQL replication lag / CockroachDB node status +3. Manual failover if automatic did not trigger: + dcm-admin store failover --store realized --target replica-2 +4. Verify replication caught up: dcm-admin store lag --store realized +5. Verify /readyz recovers +6. Root cause analysis: WAL lag, disk full, network partition +``` + +**Audit Store failure:** + +``` +RTO: PT1H acceptable (audit trail can tolerate temporary buffering) +RPO: profile-governed β€” see Audit Store write buffer policy + +1. DCM buffers audit records locally (Commit Log) during store outage + Write buffer capacity: profile-governed (PT1H standard, PT15M sovereign) +2. Restore Kafka cluster from replica or snapshot +3. DCM drains buffer to restored store automatically on reconnection +4. Verify chain integrity: dcm-admin audit chain-verify --since +``` + +**DCM database failure (Intent/Requested/Layer):** + +``` +RTO: PT30M (stores are remountable from Git remote) +RPO: 0 (all writes go to Git remote; loss only if remote is also lost) + +1. Git remote unreachable: check network connectivity +2. If Git host is down: DCM switches to cached/buffered mode + New requests queue in write buffer; reads served from local clone +3. Write buffer capacity: PT4H (standard) β€” configure per deployment +4. When Git host recovers: buffer drains automatically +5. Force drain: dcm-admin store drain-buffer --store intent +``` + +#### Scenario 3: Full Control Plane Loss + +**Symptoms:** All DCM pods down. Stores intact. Users cannot submit requests. + +**RTO:** PT15M (kubernetes deployment restart) +**RPO:** 0 (stores are external β€” no data in pods) + +``` +1. Verify stores are healthy (connect directly): + dcm-admin store health-check --all --direct + +2. Verify Internal CA is available: + curl -k https://dcm-internal-ca.dcm-system.svc.cluster.local/health + +3. Restart DCM deployment (Kubernetes): + kubectl rollout restart deployment -n dcm-system + +4. Monitor /readyz β€” startup sequence should complete within PT3M: + watch -n 5 kubectl get pods -n dcm-system + +5. Verify session store recovers: + GET /api/v1/admin/health β†’ session_store: pass + +6. Alert consumers: any in-flight requests at time of failure + are in ACKNOWLEDGED/DISPATCHED state and may need status check + dcm-admin requests find --status in-flight --since +``` + +#### Scenario 4: Partial Region Loss (Multi-Region Deployments) + +**Symptoms:** One region's DCM instance degraded. Other regions operational. + +``` +1. DCM federation routes requests away from degraded region (automatic) + Verify: GET /api/v1/admin/health β†’ federation peer status + +2. If region is sovereign-scoped (data must not leave): + Alert: sovereignty.migration_required event fires + Consumers in that region may be blocked until region recovers + +3. For non-sovereign regions: traffic reroutes automatically + Monitor: dcm_requests_total{region} for traffic shift + +4. Region recovery: standard Scenario 3 procedure + After recovery: drift detection validates recovered state +``` + +#### Scenario 5: Complete Loss (Repave) + +The nuclear scenario: entire DCM installation destroyed. Git remote intact. + +**RTO:** PT4H–PT24H (depends on infrastructure provisioning speed) +**RPO:** 0 for GitOps stores; near-zero for Realized/Audit stores + +``` +1. Provision new Kubernetes cluster (or equivalent) + +2. Deploy DCM bootstrap installer: + helm install dcm-bootstrap dcm/dcm-bootstrap \ + --set gitops.manifest_url=https://git.corp/dcm/dcm-deployment \ + --set gitops.manifest_ref= + +3. DCM bootstrap reads dcm_deployment manifest from Git + Provisions itself: control plane, Internal CA, stores + +4. Restore Realized Store from backup: + dcm-admin store restore --store realized \ + --from s3://dcm-backups/realized/latest \ + --validate-chain + +5. Restore Audit Store from backup or Kafka snapshot: + dcm-admin store restore --store audit \ + --from s3://dcm-backups/audit/latest \ + --chain-verify + +6. Intent/Requested/Layer stores: re-clone from Git remote (already current) + +7. DCM rehydrates managed resources in dependency order: + dcm-admin rehydrate --all-tenants --dry-run # verify plan first + dcm-admin rehydrate --all-tenants + +8. Drift detection validates recovered state matches declared state: + dcm-admin drift scan --all --post-recovery + +9. Re-issue Internal CA certificates for all components: + (handled automatically by bootstrap β€” components acquire new certs) + +10. Notify consumers: recovery complete; request status available +``` + +### 3.3 Recovery Time Objectives by Profile + +| Profile | Scenario 1 (component) | Scenario 2 (store) | Scenario 3 (full CP) | Scenario 5 (repave) | +|---------|------------------------|--------------------|-----------------------|---------------------| +| `homelab` | PT15M | PT2H | PT30M | PT24H | +| `standard` | PT5M | PT30M | PT15M | PT8H | +| `prod` | PT2M | PT15M | PT10M | PT4H | +| `fsi` | PT2M | PT5M | PT5M | PT2H | +| `sovereign` | PT1M | PT5M | PT5M | PT2H | + +### 3.4 Recovery Point Objectives + +| Store | Standard RPO | fsi/sovereign RPO | Notes | +|-------|-------------|------------------|-------| +| Intent Store | 0 | 0 | Git remote is source of truth | +| Requested Store | 0 | 0 | Write-once; replicated | +| Layer Store | 0 | 0 | Git remote is source of truth | +| Realized Store | PT5M | PT1M | Async replication lag | +| Audit Store | PT15M | PT1M | Kafka replication + write buffer | + +### 3.5 Backup Schedule + +DCM does not manage backups of infrastructure stores directly β€” that responsibility belongs to the storage platform. Recommended schedules by store: + +| Store | Backup method | Frequency | Retention | +|-------|--------------|-----------|-----------| +| Intent / Requested / Layer | Git push to offsite remote | Continuous | Per Git host policy | +| Realized Store | PostgreSQL PITR + daily snapshot | Continuous WAL + P1D snapshot | P90D (standard), P365D (fsi/sovereign) | +| Audit Store | Kafka topic snapshot | P4H | P365D (all profiles β€” regulatory minimum) | +| Internal CA | Key material backup to HSM/Vault | On change | P7Y (key material outlives certs) | + +### 3.6 Post-Recovery Validation Checklist + +Run after any Scenario 3+ recovery: + +``` +β–‘ /livez returns pass on all control plane pods +β–‘ /readyz returns pass (all 5 core dependencies green) +β–‘ GET /api/v1/admin/health shows all components pass +β–‘ dcm-admin audit chain-verify --full returns no broken links +β–‘ dcm-admin store validate --all returns no discrepancies +β–‘ dcm-admin drift scan --all returns no unexpected drift +β–‘ Internal CA certificates valid for all components (ICOM-006) +β–‘ At least one Auth Provider healthy (GET /api/v1/admin/health β†’ auth_providers) +β–‘ Search index rebuild complete (if search index store was affected) +β–‘ Session Store empty (expected β€” all sessions expired during outage; users re-authenticate) +β–‘ Write post-incident note to audit store with recovery timeline +β–‘ Notify consumers of recovery completion +``` + +--- + +## 4. System Policies + +| Policy | Rule | +|--------|------| +| `OPS-001` | GitOps store partitioning strategy must be declared in the DCM deployment manifest. Changes to partitioning strategy require dual-write migration procedure (Section 2). | +| `OPS-002` | Store migrations must maintain audit chain continuity across cutover. Audit records written to the source store before cutover and to the target store after cutover must form an unbroken chain. | +| `OPS-003` | Source stores must remain accessible in read-only mode for the profile-governed burn-in period after cutover. Source stores must not be decommissioned until the burn-in period completes and rollback is confirmed unnecessary. | +| `OPS-004` | Recovery from Scenario 3 (full control plane loss) must complete within the profile-governed RTO. If RTO cannot be met, the incident must be escalated and root cause must address the recovery path. | +| `OPS-005` | The post-recovery validation checklist (Section 3.6) must be completed and its results written to the audit store before declaring an incident resolved. | +| `OPS-006` | Audit Store backups must be retained for a minimum of P365D in all profiles, regardless of other data retention policies, to satisfy regulatory audit trail requirements. | +| `OPS-007` | Git remote repositories serving as GitOps stores must be configured with push access from at least two geographically separated locations to prevent single-point-of-failure data loss. | + +--- + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).* diff --git a/taxonomy/DCM-Taxonomy.md b/taxonomy/DCM-Taxonomy.md new file mode 100644 index 0000000..4d63f4a --- /dev/null +++ b/taxonomy/DCM-Taxonomy.md @@ -0,0 +1,412 @@ +# DCM Taxonomy + +The DCM taxonomy defines the precise vocabulary used throughout the architecture. Every term used in the data model, specifications, and implementation should conform to these definitions. + +**Purpose:** Eliminate ambiguity. When two people use the same word to mean different things, architecture breaks down. This taxonomy prevents that. + +**Ownership:** DCM-specific vocabulary is normative here. Terms owned by the UDLM data model β€” +the entity families and shapes, the four states, relationship/edge vocabulary, rehydration β€” are +non-normative gists; the owner is the [udlm repo](https://github.com/croadfeldt/udlm) +(`foundations/`, `entities/`), and on any divergence the UDLM text wins. + +--- + +## Part 1 β€” Core Vocabulary + +### The Three Foundational Abstractions + +| Term | Definition | +|------|-----------| +| **Data** | Any structured artifact in DCM with a type, UUID, lifecycle state, fields, data classification, and provenance. Everything that exists, is stored, and has a lifecycle. Entities, layers, policies, accreditations, audit records, groups, relationships β€” all Data. | +| **Provider** | Any external component DCM calls or that calls DCM. Implements the unified Provider base contract (registration, health, sovereignty, accreditation, governance matrix enforcement, zero trust). What varies between provider types is the capability extension. | +| **Policy** | A rule artifact that fires when Data matches declared conditions, produces a typed output (decision, mutation, action, or directive), and is enforced at a declared level. Policies govern every transition, transformation, and constraint in DCM. | + +### Provider Types (11) + +| Term | Definition | +|------|-----------| +| **Service Provider** | Typed Provider. Capability: realize infrastructure resources. Implements naturalization, realization, denaturalization, and discovery. A Service Provider can additionally register Composite Services (catalog items composed of multiple constituent resource types) β€” see the Composite Service Composition Terms section below. | +| **Information Provider** | Typed Provider. Capability: serve authoritative external data (CMDB, HR, Finance, identity). | +| **Data Store Contract** | PostgreSQL store contract defining persistence requirements per data domain (Intent, Requested, Realized, Discovered, Audit). | +| **External Policy Evaluator** | Typed Provider. Capability: evaluate policies externally. Modes 1–4; Mode 3–4 for OPA/Rego sidecar and black-box query enrichment. | +| **credential management service** | Typed Provider. Capability: issue, rotate, and revoke secrets and credentials. | +| **Auth Provider** | Typed Provider. Capability: authenticate actor identities and resolve role/group memberships. | +| **notification service** | Typed Provider. Capability: deliver notification envelopes to configured channels (Slack, PagerDuty, email, webhook). | +| **event routing service** | Typed Provider. Capability: async event streaming between DCM and external systems. | +| **Resource Type Registry** | Typed Provider. Capability: serve the Resource Type Registry (core, community, organization tiers). | +| **Peer DCM** | Typed Provider. Another DCM instance participating in federation. Federation is the Provider abstraction applied across DCM instances. | + +### Policy Types (8) + +| Term | Definition | +|------|-----------| +| **Validation Policy** | Typed Policy. The unified request-evaluation policy type. Carries two orthogonal properties: `enforcement_class` β€” `compliance` (boolean deny β€” halts request) or `operational` (contributes `risk_score_contribution` to request risk score); `output_class` β€” `structural` (boolean pass/fail β€” halts on fail) or `advisory` (contributes completeness score + warning list without blocking). Compliance + structural is the default and fail-safe. A Validation Policy with `enforcement_class: compliance` is the hard gate (what was formerly "Gating Policy"). See [Scoring Model(`data-model/scoring-model/` β€” pre-split, superseded). | +| **Transformation Policy** | Typed Policy. Output: mutations[] β€” field additions, changes, locks. Fires on request payload. All mutations collected and applied with provenance. | +| **Recovery Policy** | Typed Policy. Output: action + parameters. Fires on failure/timeout trigger conditions. Governs what DCM does when things go wrong. | +| **Orchestration Flow Policy** | Typed Policy. Output: ordered step sequence. Fires on pipeline payload type events. Named workflow artifacts β€” the explicit, visible pipeline skeleton. | +| **Governance Matrix Rule** | Typed Policy. Output: ALLOW / DENY / ALLOW_WITH_CONDITIONS / STRIP_FIELD / REDACT / AUDIT_ONLY. Fires at every cross-boundary interaction. The single enforcement point for all data/capability boundary decisions. | +| **Lifecycle Policy** | Typed Policy. Output: action on related entity (cascade, protect, detach, notify). Fires on relationship events. | +| **Placement Policy** | Typed Policy (the 8th). Output: placement constraints β€” affinity / anti-affinity / spread / co-locate / prefer / pin (+ weights) over **abstract `Topology` kinds** (never provider-native ids). Declarative; the **Placement Engine** evaluates it. Consumes UDLM `Topology` + capacity/utilization + provider `topology_capability`. See [ADR-019](../architecture/adr/019-placement-policy.md). | + +### Data Model Terms + +| Term | Definition | +|------|-----------| +| **Entity** | A Resource Entity, Process Entity, or Composite Entity β€” the primary managed thing in DCM. Has a UUID that is stable across all four lifecycle states. | +| **Four States** | The UDLM lifecycle states β€” Intent β†’ Requested β†’ Realized β†’ Discovered β€” adopted by reference from udlm `foundations/four-states.md` (the authoritative definition); DCM realizes the same entity across them, one data domain per state. | +| **Data Layer** | A versioned data artifact that contributes fields to request payload assembly. Types: Base, Core, Intermediate, Service, Request. Each has a declared contributor type. | +| **Resource Type Specification** | The schema definition for a resource type. Declares fields, constraints, portability class, dependency graph, and field criticality. | +| **Provider Catalog Item** | A provider-specific instantiation of a Resource Type Specification. What consumers actually request from the service catalog. | +| **Artifact Metadata** | Standard metadata block on every DCM artifact: uuid, handle, version, status, owned_by, created_by, contributed_by. | +| **Provenance** | Field-level lineage metadata embedded in every payload field, recording origin and all modifications. | +| **Data Classification** | Field-level metadata: public / internal / confidential / restricted / phi / pci / sovereign / classified. Phi, sovereign, and classified are immutable once set. | +| **Sovereignty Zone** | A registered DCM artifact declaring a geopolitical/regulatory boundary. Rules reference zones by ID, not raw country codes. | +| **Accreditation** | A formal, versioned, time-bounded attestation that a DCM component satisfies a specific compliance framework. First-class Data artifact with its own lifecycle. | +| **DCMGroup** | Universal grouping artifact with typed group_class: tenant_boundary, resource_grouping, policy_collection, policy_profile, composite, federation. | +| **Drift Record** | A Data artifact produced by the Drift Reconciliation Component recording field-by-field comparison of Realized vs Discovered state with severity classification. | +| **Governance Matrix Rule** | A Data artifact (also a Policy type) β€” a rule artifact governing cross-boundary interactions using four-axis match conditions. | + +### Operational Terms + +| Term | Definition | +|------|-----------| +| **Request Orchestrator** | The event bus. Routes lifecycle events to **all subscribers** β€” the Policy Engine **and** any other component that consumes them (e.g. cost management, drift reconciliation). Consumer-neutral, never coupled to one engine. Contains no pipeline logic β€” Policies define all behavior. | +| **Policy Engine** | Evaluates all policy types using the same algorithm. The single policy evaluator β€” no component bypasses it. | +| **Placement Engine** | Six-step provider selection: sovereignty filter β†’ accreditation filter β†’ capability filter β†’ parallel reserve queries β†’ tie-breaking (policy/priority/affinity/cost/load/hash) β†’ confirm. | +| **Drift Reconciliation** | Control plane component. Compares Discovered State vs Realized State. Produces drift records and events. Never writes to the Realized Store. | +| **Shadow Mode** | A Policy in `proposed` status evaluates against real traffic; output is captured but never applied. The primary mechanism for safe policy change management. | +| **Naturalization** | Service Provider converts a DCM unified payload to provider-native format before execution. | +| **Denaturalization** | Service Provider converts provider-native result back to DCM unified format after execution. | +| **Rehydration** | Replaying a resource's intent state to a new provider or context. Produces a new Requested State from the existing Intent State. | +| **Contributor** | An actor type that authored a Data artifact. Recorded in artifact_metadata.contributed_by. Types: platform_admin, consumer, service_provider, peer_dcm. Determines review requirements. | +| **Two-Level Orchestration** | Level 1: Named Workflow Artifacts (Orchestration Flow Policy, ordered: true) β€” explicit sequence skeleton. Level 2: Dynamic Policies (Validation, Transformation, Recovery) β€” fire conditionally on same events without being declared in the workflow. | +| **Reserve Query** | A parallel capacity query sent to all eligible provider candidates. Providers confirm capacity and hold it for PT5M. The Placement Engine selects the winner and releases other holds. | + + + + +### credential management service Terms + +| Term | Definition | +|------|-----------| +| **Credential Record** | DCM Data artifact storing credential metadata (UUID, type, scope, expiry, status). Never contains the credential value β€” values are held by the credential management service. | +| **DCM Interaction Credential** | Short-lived (PT15M–PT1H profile-governed), scoped credential issued before every provider dispatch. Implements ZTS-002. Never stored beyond the interaction window. | +| **Credential Revocation Registry** | Fast-queryable store of revoked credential UUIDs. All components that receive interaction credentials must check this registry at each use. Cache TTL: PT1M standard; PT30S fsi/sovereign. | +| **Transition Window** | Period during rotation when both the old and new credential are valid. Prevents downtime. P1D for consumer credentials; PT5M for dcm_interaction; P7D for x509. | +| **Emergency Rotation** | Rotation triggered by a security event. No transition window β€” old credential revoked immediately. Fastest-channel notification delivery. | +| **CPX-001–CPX-012** | credential management service system policies. Key: CPX-001 (values never in DCM stores β€” every profile), CPX-002 (every dispatch must present scoped interaction credential), CPX-009 (algorithm and key_usage declared at issuance; validated at use), CPX-012 (CPX-001 applies in ALL profiles β€” no exceptions). | +| **credential_profile** | Profile-governed credential configuration block controlling: permitted credential types, max lifetime per type, rotation requirements, retrieval auth level (bearer/step-up-mfa/mtls), FIPS level enforcement, approved algorithms, revocation SLA, idle detection threshold, IP binding requirement. | +| **AAL (Authenticator Assurance Level)** | NIST 800-63B vocabulary: AAL1 (minimal/dev β€” single factor), AAL2 (standard/prod β€” MFA required for sensitive credentials), AAL2+ (fsi β€” hardware MFA, FIPS L2), AAL3 (sovereign β€” hardware-bound, FIPS L3, tamper evidence). | +| **Idle Credential** | A credential issued but not retrieved within the profile-governed threshold. Triggers notification but not automatic revocation. Auto-revocation after 2Γ— threshold is profile-configurable. | +| **key_usage** | Declared purpose of a credential: authentication, signing, or encryption. Non-overlapping β€” a credential issued for authentication cannot be used for signing even if the algorithm supports both. Validated at use time by credential management service. | + + +### Composite Service Composition Terms + +| Term | Definition | +|------|-----------| +| **Composite Service** | A catalog-level registration that declares a composite payload β€” multiple constituent resource types with declared dependencies and delivery requirements β€” fulfillable as a single request. Registered by an ordinary Service Provider; not a separate provider type. See [composite-service-model.md](https://github.com/croadfeldt/udlm/blob/main/entities/composite-service-model.md) (Composite Service Composition Model). | +| **Composite Entity** | A DCM entity produced by a Composite Service request. Exists across all four states as a single entity aggregating constituent Resource Entities. Has one entity UUID that links it through all states. | +| **Constituent** | A sub-resource within a Composite Service. Declared with a `component_id`, `resource_type`, `depends_on`, `provided_by` (`self` or `external`), and `required_for_delivery` classification. | +| **required_for_delivery** | Constituent delivery classification: `required` (failure halts the composite request and triggers compensation), `partial` (failure produces DEGRADED but not FAILED), `optional` (failure is noted but ignored). | +| **Composite Status** | Top-level outcome of a Composite Service request: `OPERATIONAL` (all required constituents succeeded), `DEGRADED` (required succeeded; partial(s) failed; accepted if profile permits), `FAILED` (required constituent(s) failed; triggers compensation). | +| **Compensation** | Ordered teardown of successfully realized constituents when a Composite Service cannot be delivered. Runs in dependency-reverse order. Best-effort; failures produce `PARTIALLY_COMPENSATED` with orphan detection. | +| **Composition Visibility** | How a Composite Service exposes its internal structure to consumers: `opaque` (top-level only), `transparent` (all constituents as DCM entities), `selective` (declared sub-set as DCM entities). | +| **Dependency Round** | A batch of constituents that can execute in parallel because all their `depends_on` constituents are complete. Multiple rounds execute sequentially; constituents within a round execute in parallel. | +| **CMP-001–CMP-008** | Composite Service system policies. Key: CMP-001 (`self` constituents dispatched via standard Services API), CMP-002 (constituent ordering derived from `depends_on` by DCM), CMP-004 (composite status determined by DCM), CMP-005 (Recovery Policy governs failure handling), CMP-007 (deterministic constituent UUIDs in transparent mode), CMP-008 (max nesting depth 3). | + + + + + + + + + +### External CA and Live Update Terms + +| Term | Definition | +|------|-----------| +| **External CA credential management service** | A credential management service backend that issues x509 certificates using standard protocols (ACME/RFC 8555, EST/RFC 7030, SCEP, CMP, or native API like HashiCorp Vault PKI). Recommended for fsi and sovereign profiles to maintain enterprise PKI chain. Registered trust anchor root cert must be installed in all component trust stores. | +| **Trust Anchor** | The root or intermediate CA certificate installed in all DCM component trust stores. May be the built-in Internal CA or an external CA registered as a credential management service. ICOM-009: components only accept certificates from registered trust anchors. | +| **Server-Sent Events (SSE)** | W3C standard HTTP/1.1 unidirectional event stream. DCM exposes `GET /api/v1/requests/{uuid}/stream` as an SSE endpoint for live request status updates without polling. Stream closes on terminal status. | +| **Interim Status** | Provider-sent progress update during a long-running operation, via `POST /api/v1/provider/entities/{uuid}/status`. Includes step_current/step_total, step_label, and constituent_status array for composite operations. Triggers `request.progress_updated` event. | +| **constituent_status** | Array of named component statuses in a Composite Service request (e.g. `[{ref: "vm", status: "OPERATIONAL"}, {ref: "dns", status: "PROVISIONING"}]`). Surfaced in SSE stream and polling response so consumers can track multi-part operations. | + + + + + +### Design System + +| Term | Definition | +|------|-----------| +| **PatternFly** | Red Hat's open-source design system and React component library β€” the design system for the DCM GUI surfaces. Key components: Nav (sidebar), NavGroup (non-clickable headers), NavItem with NotificationBadge (approvals count), Table, Toolbar, Gallery (catalog cards), Drawer (live status), Modal (step-up MFA). | + + + + + + +### ITSM Integration Terms + +| Term | Definition | +|------|-----------| +| **ITSM Integration** | Bidirectional integration with IT Service Management systems (ServiceNow, Jira Service Management, BMC Remedy/Helix, Freshservice, PagerDuty, etc.) via process_provider and service_provider types. Outbound: DCM lifecycle events β†’ ITSM records. Inbound: ITSM approvals β†’ DCM approval votes. | +| **ITSM Action Policy** | The 8th DCM Policy output type. Side-effect policy that triggers an action in a connected ITSM system when a DCM event matches. Non-blocking by default (`on_failure: log_and_continue`). May gate the pipeline via `block_until_created: true` with mandatory timeout (ITSM-005). | +| **ITSM Reference** | Metadata stored on a DCM entity linking it to an ITSM record: system, record_type (change_request/incident/cmdb_ci), record_id, record_url, status, last_synced_at. Preserved through entity lifecycle; included in audit records. | +| **block_until_created** | Optional ITSM Policy flag. When `true`, DCM waits for ITSM record creation confirmation before dispatching to Service Provider. Requires `block_timeout`. Timeout expiry triggers `on_failure` behavior β€” the pipeline never permanently stalls due to ITSM unavailability (ITSM-005). | +| **CMDB CI Mapping** | ITSM integration configuration mapping DCM resource type FQNs to ITSM CI class names (e.g. `Compute.VirtualMachine β†’ cmdb_ci_server` in ServiceNow). Used for `create_cmdb_ci`, `update_cmdb_ci`, and `retire_cmdb_ci` actions. | +| **recorded_via** | Field on DCM approval vote records identifying the system that submitted the vote (dcm_admin_ui / servicenow / jira / slack_bot / api_direct / other). Used by ITSM integration inbound approval routing and in audit records for compliance traceability. | +| **ITSM-001–007** | ITSM integration system policies. Key: ITSM-002 (DCM never requires ITSM β€” non-blocking default), ITSM-003 (inbound webhooks must be authenticated), ITSM-005 (block_until_created must have timeout β€” pipeline never permanently stalled). | +| **ITSM-POL-001–004** | ITSM Policy system policies. Key: ITSM-POL-002 (ITSM Policies are side-effect only β€” not Validation Policy substitutes), ITSM-POL-003 (full audit record per evaluation), ITSM-POL-004 (multiple ITSM Policies on same event fire independently). | +| **GUI-014** | ITSM Integration Bridge capability β€” ITSM references on entity pages, change records linked to requests, ITSM-sourced approval votes in request status, CMDB sync via notification service. | + + +### Web Interface Terms + +| Term | Definition | +|------|-----------| +| **Unified Shell** | One DCM web application with **role-gated surfaces** (views of one app, not separate apps): **Consumer Portal** (developers/owners/tenant-admins; wraps the Consumer API), **Admin Panel** (platform roles; wraps the Admin API), **Provider Management** (provider_owner), **Flow GUI** (policy_owner/sre). One login, one session; navigation adapts to the actor's privilege. *Per-surface features and the `GUI-001–GUI-010` capabilities are specified in the GUI specs (`docs/specifications/dcm-{consumer,admin,provider,flow}-gui-spec.md`), not enumerated in the taxonomy (right-altitude: the taxonomy names the concept; the spec carries the detail).* | + + +### Operational Reference Terms + +| Term | Definition | +|------|-----------| +| **GitOps Store Partitioning** | For deployments using Git as an ingress adapter (Git ingress adapter β€” optional): splitting the intent repository into multiple repositories to manage scale. Three strategies: tenant-shard (hash of tenant_uuid), per-tenant (one repo per tenant), and time-based archiving (active vs cold). Git is not a required DCM infrastructure component β€” it is one of several ingress paths (alongside API, CLI, and message bus). | +| **Dual-Write Mode** | Standard store-cutover safety pattern (adopt-by-reference, not redefined here) β€” DCM writes to both source and target store before cutover. DCM-specific: duration is profile-governed (P1D minimal β†’ P60D sovereign). | +| **Burn-In Period** | Post-cutover period during which the source store remains accessible in read-only mode for rollback. Profile-governed: P7D minimal β†’ P90D fsi/sovereign. Source must NOT be decommissioned before burn-in completes. | +| **Repave** | The complete recovery scenario: all DCM infrastructure lost but Git remotes intact. DCM bootstraps from Git, restores operational stores from backup, and rehydrates managed resources. OPS-005 requires post-recovery validation checklist completion. | +| **RTO / RPO** | **Standard DR objectives β€” adopted by reference, not redefined.** Only the DCM-specific *targets* are normative: **RTO** profile-governed PT1M–PT5M (component) / PT5M–PT30M (full control-plane loss); **RPO** GitOps stores 0, Realized Store PT1M (fsi/sovereign)–PT5M (standard), Audit Store PT1M–PT15M. | +| **OPS-001–007** | Operational Reference system policies. Key: OPS-002 (audit chain continuity across migration), OPS-003 (source read-only during burn-in; do not decommission early), OPS-006 (Audit Store retention minimum P365D all profiles). | + + +### Scheduling and Dependency Terms + +| Term | Definition | +|------|-----------| +| **Scheduled Request** | A DCM request with an explicit dispatch schedule (at a specific time, during a maintenance window, or recurring). Goes through the same pipeline as immediate requests; policy evaluates at declaration AND at dispatch time. | +| **PENDING_DEPENDENCY** | Intent State status for a request in a dependency group waiting for its declared dependency to reach the required wait_for state before dispatch. | +| **Request Dependency Group** | A consumer-declared set of requests with ordering constraints (depends_on) between them. Distinct from type-level dependencies ([service-dependencies.md](https://github.com/croadfeldt/udlm/blob/main/entities/service-dependencies.md)) and Composite Service composition ([composite-service-model.md](https://github.com/croadfeldt/udlm/blob/main/entities/composite-service-model.md)). | +| **Field Injection** | Mechanism for passing realized output fields from a dependency automatically into a dependent request's fields at dispatch time. Subject to Transformation policies. | +| **Maintenance Window** | A reusable, named recurrence artifact declaring approved change windows. Consumers reference window_uuid in scheduled requests to slot into the next matching window. | +| **SCH-001–SCH-006** | Scheduled requests system policies. Key: SCH-001 (dual policy evaluation: declaration + dispatch), SCH-003 (dispatch-time policy rejection β†’ FAILED), SCH-005 (not_after deadline miss β†’ FAILED, no retry). | +| **RDG-001–RDG-006** | Request dependency graph policies. Key: RDG-001 (circular deps rejected at submission), RDG-002 (max 50 requests per group), RDG-004 (PENDING_DEPENDENCY requests count against quota), RDG-006 (request may belong to one group only). | + +### Self-Health Terms + +| Term | Definition | +|------|-----------| +| **Liveness / Readiness (`/livez`, `/readyz`)** | The **standard Kubernetes probe convention β€” adopted by reference**, not redefined. DCM-specific only: `livez` PT5S max / no external calls / unauthenticated (failure β†’ restart); `readyz` validates Session Store, Audit Store, Policy Engine, Message Bus, Auth Provider (failure β†’ out of LB). | +| **HLT-001–HLT-006** | Self-health system policies. Key: HLT-001 (livez and readyz required, unauthenticated), HLT-002 (livez PT5S max, no external calls), HLT-003 (readyz fails if core dependencies unreachable), HLT-005 (Prometheus metrics required). | + + +### Session Revocation Terms + +| Term | Definition | +|------|-----------| +| **Session Record** | DCM Data artifact tracking an active actor session: session_uuid, actor_uuid, auth_provider_uuid, created_at, expires_at, status (active/refreshing/revoked/expired), revocation metadata. | +| **Session Revocation Registry** | Fast-queryable store of revoked-but-not-yet-expired session UUIDs. All components that accept bearer tokens must check this on every request. Cache age is profile-governed (PT5M minimal β†’ no cache sovereign). | +| **Session Store** | Operational store for active sessions (not GitOps-backed). Separate from Realized State Store. Backed by Redis or Postgres (standard+) or in-memory (homelab/dev). | +| **Token Introspection** | RFC 7662 endpoint (`POST /api/v1/auth/introspect`) for validating bearer tokens. Returns active/inactive plus session metadata. Requires `introspection` scope. | +| **AUTH-016–AUTH-022** | Session revocation system policies. Key: AUTH-016 (deprovisioning fires session + credential revocation in parallel), AUTH-017 (revocation SLA), AUTH-018 (all components check revocation registry), AUTH-019 (emergency revocation: critical urgency, non-suppressable). | + +### Internal Component Auth Terms + +| Term | Definition | +|------|-----------| +| **Internal CA** | The Certificate Authority operated by each DCM deployment for issuing mTLS certificates to internal components. Not exposed externally. Root cert installed in all component trust stores at deployment time. | +| **Component Identity** | Each DCM control plane component has a stable UUID, an mTLS certificate from the Internal CA, and a service account with declared allowed_sources and allowed_targets. | +| **Bootstrap Token** | A one-time-use credential (max PT1H lifetime) that enables a new component to acquire its first mTLS certificate from the Internal CA. Invalidated immediately after use. | +| **Component Communication Graph** | The declared graph of which components may call which others. Components may only call `allowed_targets`; endpoints only accept calls from `allowed_sources`. Violations are rejected and audited (ICOM-003, ICOM-004). | +| **ICOM-001–ICOM-009** | Internal Component Auth system policies. Key: ICOM-001 (mTLS required for all internal calls), ICOM-002 (interaction credential required in addition to mTLS), ICOM-007 (bootstrap tokens one-time-use, PT1H max), ICOM-008 (compromised certs β†’ CRL immediately). | + + +### API Versioning Terms + +> **Baseline adopted by reference:** DCM uses **semantic versioning** (semver) +> for version numbering and **RFC 8594** (`Deprecation` / `Sunset` HTTP headers) +> for the deprecation signal β€” neither is redefined here. Only the DCM-specific +> application below is normative: profile-governed support windows and the VER +> policies. + +| Term | Definition | +|------|-----------| +| **Breaking Change** | Any change that requires an existing client to modify code or configuration to continue working. Removing fields, changing types, removing endpoints, tightening validation, changing HTTP status semantics. See [34-api-versioning-strategy.md] Section 2 for the complete definition. | +| **Deprecation Period** | The window between when a version is announced as deprecated and when it reaches its sunset date. Deprecated versions continue to function; responses include `Deprecation` and `Sunset` headers. Profile-governed: prod=365 days notice, 2 years support; sovereign=2 years notice, 4 years support. | +| **Sunset Date** | The date after which a deprecated API version returns `410 Gone`. Clients must migrate before this date. | +| **Preview Endpoint** | An endpoint at `/api/v{N}/preview/` path with no stability commitment. May change or be removed without a major version increment. Not for production use. | +| **VER-001–VER-009** | API Versioning system policies. Key: VER-002 (breaking change definition β€” when in doubt, treat as breaking), VER-003 (deprecation headers required on all deprecated version responses), VER-005 (support windows are profile-governed), VER-008 (machine-readable migration guide required for each new major version). | + + +### Event Catalog Terms + +| Term | Definition | +|------|-----------| +| **Event Catalog** | The authoritative source for all DCM event types, their payload schemas, urgency levels, and trigger conditions. See [33-event-catalog.md]. 82 event types across 20 domains. | +| **Event Envelope** | The common wrapper all DCM events share: event_uuid (idempotency key), event_type, event_schema_version, timestamp (from Commit Log), dcm_version, dcm_instance_uuid, subject, urgency, payload, links. | +| **event_uuid** | Stable idempotency key assigned to each event. Consumers must treat duplicate event_uuid values as already-processed β€” DCM delivers at-least-once. | +| **event_schema_version** | Increments on breaking payload schema changes. Adding optional fields is not breaking. Removing fields, changing types, or changing semantics are breaking. | +| **EVT-001–EVT-007** | Event Catalog system policies. Key: EVT-001 (all events must include base envelope), EVT-002 (event_uuid is idempotency key), EVT-005 (critical urgency events delivered via push regardless of subscriptions), EVT-006 (non-standard events use reverse-DNS prefix), EVT-007 (audit.* critical events are non-suppressable). | + + +### Authority Tier Model Terms + +| Term | Definition | +|------|-----------| +| **Authority Tier Registry** | The ordered list of authority tiers that governs approval routing across all DCM pipelines. Stored as a versioned registry entry. Changes require impact detection before activation. | +| **Tier Impact Diff** | Computed before any tier registry change activates. Compares proposed ordered list to current list; classifies each changed tier as SECURITY_DEGRADATION, BROKEN_REFERENCE, PROFILE_GAP, SECURITY_UPGRADE, or NEW. | +| **SECURITY_DEGRADATION** | Impact classification for a tier whose gravity or position decreased after a registry change. Blocks registry activation until explicitly accepted by a verified-tier or above reviewer (ATM-009). | +| **BROKEN_REFERENCE** | Impact classification when a tier name referenced in active configuration no longer exists in the registry. Blocks activation until resolved (ATM-010). | +| **PROFILE_GAP** | Impact classification when a profile's threshold list is incomplete after new tier insertion. Warning only β€” does not block activation (ATM-012). | Stored as a versioned registry entry. Custom tiers are inserted into the list by position; existing tier names remain stable. | +| **decision_gravity** | Stable, position-independent severity classification on each tier: `none` (auto), `routine` (reviewed), `elevated` (verified), `critical` (authorized). Used by the scoring model and profile system to reason about tier severity independently of tier names. | +| **Tier Weight** | Numeric value derived from a tier's position in the ordered list. Never hardcoded β€” resolved at evaluation time. Stored in approval records for point-in-time audit (ATM-008). | +| **Custom Tier** | An organization-defined tier inserted between existing tiers. Must declare `decision_gravity` consistent with position. Requires `verified` tier approval to contribute (ATM-004). | +| **ATM-001–ATM-008** | Authority Tier Model system policies. Key: ATM-001 (tiers identified by name; weight derived from position), ATM-002 (auto tier max_score ≀ 50), ATM-003 (custom tier gravity must be consistent with position), ATM-008 (approval records store weight at creation for point-in-time audit). | + + +### Authority Tier Terms + +| Term | Definition | +|------|-----------| +| **Authority Tier** | The required organizational authority level for a decision, expressed as a named position in an ordered list. DCM defines four tiers; organizations define what constitutes sufficient authority at each level. | +| **`auto`** | No human judgment required. System confidence (scoring, validation) is sufficient to proceed. | +| **`reviewed`** | Standard authority required. One qualified reviewer in the relevant domain must record a decision via the DCM Admin API. Who constitutes a qualified reviewer is the organization's definition. | +| **`verified`** | Elevated authority required. Two independent, distinct reviewers must each record a decision. The same actor cannot satisfy both. Enforces separation of duties. | +| **`authorized`** | Highest authority level required. Most consequential decisions β€” policy governance changes, regulated actions, high-risk provider registrations. N members of a declared DCMGroup must record decisions via the Admin API. The authority group composition (CTO, CISO, security board, one person with delegated authority) is entirely the organization's definition. | +| **DCMGroup (authority context)** | A declared set of actors who constitute the required authority for `authorized` tier decisions. Platform admin declares membership; quorum threshold (N of M) is profile-governed. | +| **`recorded_via`** | Audit field on approval decisions capturing which system submitted the decision (dcm_admin_ui, servicenow, jira, slack_bot, api_direct). Informational for audit; not enforced. | + + +### Design Priority Terms + +| Term | Definition | +|------|-----------| +| **Design Priority Order** | The four-priority hierarchy governing all DCM design decisions: (1) Security β€” industry best practices are the baseline; (2) Ease of use β€” secure path must be easy path; (3) Extensibility β€” adaptable through configuration not code; (4) Fit for purpose β€” always required. | +| **DPO-001–006** | Design Priority system policies. Key: DPO-001 (security properties present in all profiles), DPO-002 (every security requirement needs ease-of-use mechanism), DPO-005 (`minimal` profile = minimal overhead not minimal security), DPO-006 (when security and ease conflict, redesign ease-of-use not security). | +| **Priority 1 (Security)** | Security properties are architecturally present in ALL profiles. Profiles control enforcement strictness and automation level β€” never whether the property exists. Non-negotiable across all profiles: CPX-001, SMX-004, SMX-008, CPX-003, CPX-005 first retrieval audit, forbidden algorithm baseline. | +| **Priority 2 (Ease of use)** | The secure path must also be the easy path. Every security requirement must be accompanied by an ease-of-use mechanism. The scoring model auto-approval threshold, profile defaults, and Flow GUI visual condition builder are all Priority 2 implementations. | + +### Scoring Model Terms + +| Term | Definition | +|------|-----------| +| **enforcement_class** | Required property of Validation policies. `compliance`: boolean deny gate β€” always halts on fire. `operational`: contributes `risk_score_contribution` to the request risk score. | +| **output_class** | Required property of Validation policies. `structural`: boolean pass/fail. `advisory`: contributes completeness score and warnings without blocking. | +| **request_risk_score** | Aggregate score (0–100) assembled from five weighted signals: operational Validation Policy contributions, completeness, actor risk history, quota pressure, provider accreditation richness. Drives approval routing. | +| **risk_score_contribution** | The weighted score a fired operational-class Validation Policy contributes to the request risk score. Declared as `scoring_weight` (1–100) in the policy. | +| **completeness_score** | Aggregate of advisory Validation contributions. Represents how incomplete or unusual the request is β€” higher = more warnings. Does not block requests. | +| **actor_risk_history_score** | Decay-weighted (Ξ»=0.1, half-life β‰ˆ7 days) history of an actor's previous request outcomes. Contributes to request risk score. Not exposed to other consumers. | +| **quota_pressure_score** | Continuous score representing how close a Tenant is to quota limits for the requested resource type. Zero below 75% utilization; 100 at full quota. | +| **accreditation_richness_score** | Weighted sum of a provider's accreditation portfolio normalized to 0–100. Influences placement preference and inversely contributes to provider risk signal. | +| **scoring_threshold** | Profile-governed boundary on the request risk score that maps to an approval routing tier. Four tiers: auto_approve, reviewed, verified, authorized. `auto_approve_below` may not exceed 50 (SMX-008). | +| **Risk Score Aggregator** | Sub-function of the Policy Engine. Assembles five scoring signals into the request risk score after all compliance-class and Governance Matrix evaluations complete. | +| **regulatory_mandate** | Policy metadata flag. When `true`, the policy's `enforcement_class: compliance` cannot be demoted to operational by any profile (SMX-003). Set by platform admins, audited. | +| **score_drivers** | Human-readable list of the top contributing factors to a request risk score. Exposed to consumers (top 3 only). Full breakdown in Score Record for platform admins. | +| **Score Record** | Immutable audit artifact recording the full signal breakdown, weights, routing decision, and threshold applied for a scored request evaluation. Written to Audit Store for every scored request. | + + +### Federation Topology + +| Term | Definition | +|------|-----------| +| **Peer DCM** | A federated DCM instance. Treated as a typed Provider. Trust postures: verified (manually approved), vouched (Hub-introduced), provisional (crypto-verified only). | +| **Hub DCM** | A DCM instance that coordinates Regional DCMs in hub-spoke topology. Policy distribution source. Cannot force-activate policies on Regional DCMs. | +| **Regional DCM** | A DCM instance in a specific sovereignty region, managed by a Hub DCM. | +| **Federation Tunnel** | Mutually authenticated, encrypted, scoped channel between DCM instances. Establishes secure transport β€” not implicit trust. | +| **Federated Contribution** | A Peer DCM contributing registry entries, policy templates, or service layers to a receiving DCM, scoped by the peer's federation trust posture. | + +--- + +## Part 2 β€” Anti-Vocabulary + +Terms to avoid because they introduce ambiguity. Use the precise alternatives instead. + +| Avoid | Because | Use Instead | +|-------|---------|-------------| +| **Widget** | Vague β€” what thing, exactly? | Name the specific resource type (VirtualMachine, IPAddress, VLAN, etc.) | +| **Realize** (standalone) | Ambiguous β€” "realize" can mean understand, achieve, or provision | **Provision** a VM, **fulfill** a request, **execute** a process. "Realized State" is accepted vocabulary. | +| **Data Center** (as architectural term) | A building β€” not architecturally meaningful | **Region** (large geographically distinct area) or **Zone** / **Availability Zone** (isolated group within a Region) | +| **Orchestrator** (as a standalone component) | Suggests a single sequencer; DCM orchestration is policy-driven, not procedural | **Request Orchestrator** (the event bus) + **Orchestration Flow Policy** (named workflow) + **Policy Engine** (evaluator) | +| **Tangible / Intangible** | Nothing in DCM architecture is intangible β€” these words add no precision | Describe what the thing actually is | +| **Workflow** (without qualification) | Ambiguous between Level 1 (named Orchestration Flow Policy) and general process | **Named Workflow** (Orchestration Flow Policy with `ordered: true`) or **dynamic policy** (conditional policy) | +| **Gatekeeper** | Collides with OPA Gatekeeper operator in OpenShift | **Validation Policy** with `enforcement_class: compliance` | +| **Gating Policy** | Merged into Validation Policy (2026-06-30). Gating is an `enforcement_class` property, not a separate policy type | **Validation Policy** with `enforcement_class: compliance` for hard gates, `enforcement_class: operational` for risk-score contributors | +| **Producer** | DCM uses "Provider" terminology with typed contracts | **Service Provider** (provisions resources) or the specific provider type | +| **Shore / Ship / Enclave** | Legacy terminology from defense IT contexts; replaced in DCM | **Hub DCM** (central/global) / **Regional DCM** (distributed regional) / **Sovereign DCM** (air-gapped/compliance-isolated) | +| **User** (generic) | Ambiguous across domains β€” humans "use" DCM in every domain | **Consumer**, **Developer**, **Application Owner**, **Platform Engineer**, **SRE**, **Tenant Admin**, **Policy Author** | +| **Service** (unqualified) | Means different things in different contexts β€” provision target, catalog entry, or provider | **Catalog Item** (what consumers browse), **Resource Type** (abstract classification), **Service Provider** (who provisions) | +| **Manage** (unqualified) | Too broad β€” what action specifically? | **Provision**, **Configure**, **Monitor**, **Decommission**, **Govern** β€” name the lifecycle operation | + +--- + +## Part 3 β€” Roles and Personas + +| Role | Scope | API surface | +|------|-------|-------------| +| **Consumer** | Requests services from the catalog; manages owned resources | Consumer API | +| **Tenant Admin** | Manages a Tenant; can author tenant-domain policies and groups | Consumer API + contribution endpoints | +| **Policy Author** | Authors policies within assigned domain scope | Consumer API contribution endpoints, Flow GUI | +| **Platform Admin** | Manages the DCM deployment; all artifact types; all domains | Admin API, Flow GUI (full) | +| **Platform Observer** | Read-only view across all platform operations | Flow GUI (read-only), Admin API (read) | +| **Service Provider Operator** | Manages a registered Service Provider | Operator Interface (provider side), Admin API (registration) | +| **Policy Reviewer** | Reviews and approves/rejects contributed policies | Admin API, Flow GUI | +| **Auditor** | Read-only access to audit records and compliance reports | Consumer API (audit), Admin API (audit) | + +--- + +## Part 4 β€” Capability Domain Prefixes + +| Prefix | Domain | +|--------|--------| +| IAM | Identity and Access Management | +| CAT | Service Catalog | +| REQ | Request Lifecycle Management | +| PRV | Provider Contract and Realization | +| LCM | Resource Lifecycle Management | +| DRF | Drift Detection and Remediation | +| POL | Policy Management | +| LAY | Data Layer Management | +| INF | Information and Data Integration | +| ING | Ingestion and Brownfield Management | +| AUD | Audit and Compliance | +| OBS | Observability and Operations | +| STO | Storage and State Management | +| FED | DCM Federation and Multi-Instance | +| GOV | Platform Governance and Administration | +| ACC | Accreditation Management | +| ZTS | Zero Trust and Security Posture | +| GMX | Unified Governance Matrix | +| DRC | Drift Reconciliation | +| FCM | Federated Contribution Model | +| SMX | Scoring Model | +| CMP | Composite Service Composition | +| CPX | credential management service Model | +| DPO | Design Priority Order | +| ATM | Authority Tier Model | +| EVT | Event Catalog | +| VER | API Versioning | +| SES | Session Revocation | +| ICOM | Internal Component Auth | +| SCH | Scheduled Requests | +| RDG | Request Dependency Graph | +| HLT | DCM Self-Health | +| OPS | Operational Reference | +| GUI | Web Interfaces | +| ITSM | ITSM Integration | +| AUTH | Auth Provider Capabilities | +| PCA | Provider Callback Authentication | +| PRR | Provider Readiness | +| WLA | Workload Analysis | +| ACM | Accreditation Monitoring | +| LOC | Location Topology Management | +| SUB | Subscription Management | + +--- + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).* diff --git a/taxonomy/README.md b/taxonomy/README.md index e09b3c6..5aa4aee 100644 --- a/taxonomy/README.md +++ b/taxonomy/README.md @@ -10,7 +10,7 @@ The top-level domains are defined as follows: 0. **Value** - The "zero domain" recognizes the business value achieved through the abstract capabilities enabled by the end-to-end architecture, for example, improved security and compliance, accelerated software deployments and cost saving thanks to reduced operational toil. 1. **Application** - This domain delivers the Unified interface that Developers and Application Owners use to request and manage Services from DCM. It includes the Service Catalog listing the available Services. Multi-tenancy is experienced at this level. -2. **Control Plane** - This is the central nervous system of DCM. It maintains the Unified API and Data Model including the inventory of Fulfilled Services. It also implements the authentication and role-based access control required to enforce multi-tenancy. +2. **Control Plane** - This is the central nervous system of DCM. It maintains the Unified API and Data Model including the inventory of Realized Services. It also implements the authentication and role-based access control required to enforce multi-tenancy. 3. **Resource** - This domain contains the code and workflows for provisioning and managing the resources required to Fulfill Service request. It provides the engines that execute requests from the Control Plane and translates those into API calls supported by specific Infrastructure Platforms hosted under the Data Center domain. 4. **Data Center** - This domain represents the physical and virtual composable infrastructure in the data center. This is where the basic compute, storage and networking resources are provisioned. 5. **Governance & FinOps** - This domain is the set of overarching requirements that ensure DCM operations are secure, compliant, and cost-effective. It encompasses policy management, real-time metrics, and cost allocation. @@ -41,9 +41,9 @@ An Application is a defined collection of Services configured to deploy a self-c ### Data Model -The Data Model defines the scope of Unified Data within the Control Plane. The Data Model includes static data as well as event data such as the history of Fulfilled Service requests and time series data such as Workload usage metrics. Data Model does not mean database, but rather is a broader term meaning the scope of any data supporting the DCM architecture. While Static Data may be stored using a traditional database service, Event Data from Service requests could be better sent to a logging service and Usage Metrics Data might be more suitably captured and stored using a time series data platform. +The Data Model defines the scope of Unified Data within the Control Plane. The Data Model includes static data as well as event data such as the history of Realized Service requests and time series data such as Workload usage metrics. Data Model does not mean database, but rather is a broader term meaning the scope of any data supporting the DCM architecture. While Static Data may be stored using a traditional database service, Event Data from Service requests could be better sent to a logging service and Usage Metrics Data might be more suitably captured and stored using a time series data platform. -Code may also be considered in the scope of the Data Model such as the Infrastructure-as-Code used to define Compound Services and should be managed using GitOps practices. +Code may also be considered in the scope of the Data Model such as the Infrastructure-as-Code used to define Composite Services and should be managed using GitOps practices. Dynamic Data also falls under the scope of the Data Model, for example, the current state of a provisioned resource. Dynamic Data can be directly queried using the Unified API and it may also be cached. @@ -73,7 +73,7 @@ A Region is a large, geographically distinct area that hosts resources. It can b A Service is the specific capability supported by a Service Catalog item within the Application domain. Services provide the underlying resources that Applications rely on. -Service Catalog items supporting Atomic Services are defined by Service Providers and tightly coupled to a provided resource. Compound Services can be defined to provides a configuration of any combination of Atomic Services making it possible for a single Service Catalog item to provide arbitrarily complex Application environments. +Service Catalog items supporting Atomic Services are defined by Service Providers and tightly coupled to a provided resource. Composite Services can be defined to provides a configuration of any combination of Atomic Services making it possible for a single Service Catalog item to provide arbitrarily complex Application environments. ### Service Provider @@ -91,7 +91,7 @@ To avoid confusion, refrain from saying User when referring to humans acting wit ### Workload -A Workload is a running instance of a Service that has been Fulfilled at the request of a User in the Application domain. A Workload is composed of the provisioned resources required by the Service. The State and usage of the Workload's resources are monitored by tools in the Governance & FinOps domain to enable observability and billing for Services. +A Workload is a running instance of a Service that has been Realized at the request of a User in the Application domain. A Workload is composed of the provisioned resources required by the Service. The State and usage of the Workload's resources are monitored by tools in the Governance & FinOps domain to enable observability and billing for Services. ### Zone