diff --git a/landing-zone-configurator/app/README.de.md b/landing-zone-configurator/app/README.de.md index 8d4f1eb..da6faf4 100644 --- a/landing-zone-configurator/app/README.de.md +++ b/landing-zone-configurator/app/README.de.md @@ -408,6 +408,33 @@ weder Runner-Stillstand noch Widerruf bereits ausgegebener Cloud-Credentials. Cloud-Abgleich, Runner-Stillstand, State-Wiederherstellung und Ausfuehrungsfreigabe bleiben separate Vorgänge mit eigenen Nachweisen und Freigaben. + + +### Application nach fehlgeschlagenem Apply neu planen + +Migration 064 ergänzt einen separaten unveränderlichen Abschlussbeleg, unabhängig +vom obigen Flag für beratende Prüfbelege. Der ursprüngliche Application Owner +kann einen abschließend als `apply_failed` gemeldeten Job mit verbrauchtem, +paketgebundenem Runner-Ticket, gebundenem Ausgabenachweis und ohne Recovery-Beleg +prüfen und für einen neuen Plan freigeben. Die Prüfung liest mit serverseitigen +Backend-Zugangsdaten ausschließlich den exakten Application-S3-Schlüssel, +kontrolliert Sperren vor und nach dem Lesen, prüft `terraform_data.contract` und +liefert Ressourcenanzahlen sowie Checkpoint-Metadaten. Rohe States oder +Zugangsdaten werden nicht ausgegeben. Das fünf Minuten gültige verschlüsselte +Prüftoken ist an Tenant, Sitzung, Job, Backend und beobachteten Checkpoint +gebunden; die Bestätigung prüft alle Bindungen und den S3-Inhalt erneut. + +Dies protokolliert das gemeldete Befehlsende, keinen unabhängig verifizierten +Runner-/Cloud-Abgleich. Ein frischer Plan mit normalem Refresh und die normale +Apply-Prüfung bleiben erforderlich. Bei unklarem Abschluss bleibt der separate +Operator-Abgleich durch den ursprünglich freigebenden PE nach Ablauf der +Ausführungsfenster nötig. State-Schreibfehler, Recovery-Belege, Sperren oder +abweichende States sind von diesem Weg ausgeschlossen. Der Beleg verändert +keinen historischen Dispatch oder State und startet keine Ausführung. SQL erzwingt +den frischen Plan bei der Artefaktabfrage und beim Anlegen eines Apply. Ein neuer +fehlgeschlagener Apply benötigt einen eigenen Beleg. Der ursprüngliche +Accelerator-Quellstand bleibt unverändert. + ### Lokale Plan-/Apply-Ausführung diff --git a/landing-zone-configurator/app/README.md b/landing-zone-configurator/app/README.md index f7e5752..89ddbfb 100644 --- a/landing-zone-configurator/app/README.md +++ b/landing-zone-configurator/app/README.md @@ -408,6 +408,27 @@ neither Runner termination nor revocation of cloud credentials already issued. Cloud reconciliation, Runner termination, state recovery and execution approval remain separate operations with their own evidence and approvals. +### Application replan after failed Apply + +Migration 064 adds a separate immutable completion-clearance receipt. It is +independent of the advisory review flag above. The original Application Owner +can check and clear a terminal `apply_failed` job with a consumed, package-bound +Runner ticket, bound output evidence and no recovery record. The check reads +only the exact Application S3 key using server-held backend credentials, checks +locks before and after reads, verifies `terraform_data.contract`, and returns +resource counts plus checkpoint metadata. It never returns raw State or secrets. +The five-minute encrypted review token is bound to the tenant, session, job, +backend and observed checkpoint; approval rechecks all bindings and S3 content. + +This records reported command completion, not independently verified Runner or +cloud reconciliation. A fresh, normally refreshed Plan and normal Apply review +remain mandatory. Unknown final completion retains the separate original-PE +operator-attestation path with expired execution windows; State write failures, +recovery records, locks or mismatched State cannot use this path. Clearance +changes no historical dispatch or State and starts no execution. SQL enforces +the fresh-Plan boundary at saved-plan lookup and Apply insertion, and a new failed +Apply requires its own clearance. The original Accelerator pin is unchanged. + ### Local Plan/Apply execution diff --git a/landing-zone-configurator/app/apps/api/db/064_application_recovery_replan.sql b/landing-zone-configurator/app/apps/api/db/064_application_recovery_replan.sql new file mode 100644 index 0000000..ded7562 --- /dev/null +++ b/landing-zone-configurator/app/apps/api/db/064_application_recovery_replan.sql @@ -0,0 +1,211 @@ +-- Historical completion clearance; never an Apply authorization or State mutation. +CREATE TABLE lzc.application_recovery_replans ( + job_id uuid PRIMARY KEY REFERENCES lzc.application_dispatches(job_id), + tenant_id uuid NOT NULL REFERENCES lzc.tenants(id), + owner_user_id uuid NOT NULL REFERENCES lzc_auth.users(id), + reviewed_by uuid NOT NULL REFERENCES lzc_auth.users(id), + review_session_id uuid NOT NULL, + mode text NOT NULL CHECK(mode IN ('reported-failure','operator-attested')), + reviewed_at timestamptz NOT NULL DEFAULT clock_timestamp(), + binding_sha256 text NOT NULL CHECK(binding_sha256 ~ '^[a-f0-9]{64}$'), + backend_id uuid NOT NULL REFERENCES lzc.state_backends(id), + backend_sha256 text NOT NULL CHECK(backend_sha256 ~ '^[a-f0-9]{64}$'), + state_key text NOT NULL, + state_sha256 text NOT NULL CHECK(state_sha256 ~ '^[a-f0-9]{64}$'), + state_serial bigint NOT NULL CHECK(state_serial>=0), + state_version text CHECK(state_version IS NULL OR length(state_version) BETWEEN 1 AND 1024), + attested_runner_stopped boolean NOT NULL, + attested_cloud_state_compared boolean NOT NULL, + CHECK(attested_runner_stopped=(mode='operator-attested') AND attested_cloud_state_compared=(mode='operator-attested')) +); +ALTER TABLE lzc.application_recovery_replans ENABLE ROW LEVEL SECURITY; +ALTER TABLE lzc.application_recovery_replans FORCE ROW LEVEL SECURITY; +CREATE POLICY application_recovery_replan_read ON lzc.application_recovery_replans FOR SELECT TO configurator_app + USING(tenant_id=lzc.current_tenant_id() AND ((owner_user_id=lzc.current_user_id() AND lzc.application_role('application-owner')) OR (reviewed_by=lzc.current_user_id() AND lzc.application_role('platform-engineer')))); +CREATE POLICY application_recovery_replan_migration ON lzc.application_recovery_replans TO configurator_migration USING(true) WITH CHECK(true); +GRANT SELECT ON lzc.application_recovery_replans TO configurator_app; +CREATE TRIGGER immutable_application_recovery_replan BEFORE UPDATE OR DELETE ON lzc.application_recovery_replans + FOR EACH ROW EXECUTE FUNCTION lzc.immutable_application_version(); + +CREATE FUNCTION lzc_auth.application_recovery_replan_context(p_session uuid,p_job uuid) +RETURNS jsonb LANGUAGE plpgsql SECURITY DEFINER SET search_path=pg_catalog AS $$ +DECLARE context jsonb; job lzc.application_jobs; dispatched lzc.application_dispatches; + grant_row lzc.application_job_grants; bound lzc.application_job_backends; registered lzc.state_backends; descriptor_hash text; review_mode text; +BEGIN + PERFORM 1 FROM lzc_auth.sessions WHERE id=p_session FOR SHARE; + PERFORM 1 FROM lzc.memberships WHERE tenant_id=lzc.current_tenant_id() AND user_id=lzc.current_user_id() FOR SHARE; + PERFORM lzc_auth.authorize_application(p_session,lzc.current_tenant_id(),'order'); + PERFORM 1 FROM lzc.stackit_identities human WHERE human.user_id=lzc.current_user_id() + AND human.issuer='https://accounts.stackit.cloud' AND human.revoked_at IS NULL AND human.valid_until>now() FOR SHARE; + IF NOT FOUND THEN RAISE EXCEPTION 'application_recovery_replan_access_denied' USING ERRCODE='42501'; END IF; + SELECT * INTO job FROM lzc.application_jobs WHERE id=p_job AND tenant_id=lzc.current_tenant_id(); + IF job.id IS NULL OR NOT ((job.owner_user_id=lzc.current_user_id() AND (lzc.application_role('application-owner') OR lzc.application_role('platform-engineer'))) + OR lzc.application_role('platform-engineer')) THEN + RAISE EXCEPTION 'application_recovery_replan_access_denied' USING ERRCODE='42501'; + END IF; + -- The same instance lock serializes Plan, archive and review. + PERFORM lzc_auth.require_active_application_order(p_session,job.instance_id); + SELECT * INTO dispatched FROM lzc.application_dispatches WHERE job_id=p_job FOR UPDATE; + SELECT * INTO grant_row FROM lzc.application_job_grants WHERE job_id=p_job FOR SHARE; + SELECT * INTO bound FROM lzc.application_job_backends WHERE job_id=p_job FOR SHARE; + PERFORM 1 FROM lzc.application_runner_tickets WHERE job_id=p_job FOR SHARE; + PERFORM lzc_auth.assert_application_job_group(p_job); + IF job.owner_user_id=lzc.current_user_id() AND dispatched.error_code='apply_failed' + AND EXISTS(SELECT 1 FROM lzc.application_runner_records WHERE job_id=p_job AND kind='output') THEN + review_mode:='reported-failure'; + IF NOT EXISTS(SELECT 1 FROM lzc.application_instances i JOIN lzc.stackit_identities human ON human.user_id=i.requested_by + AND human.issuer='https://accounts.stackit.cloud' AND human.revoked_at IS NULL AND human.valid_until>now() + AND human.email=i.resolved_settings->>'owner_email' WHERE i.id=job.instance_id AND i.tenant_id=job.tenant_id) THEN + RAISE EXCEPTION 'application_recovery_replan_access_denied' USING ERRCODE='42501'; + END IF; + ELSE + review_mode:='operator-attested'; + context:=lzc_auth.application_recovery_review_context(p_session,p_job); + IF NOT lzc.application_role('platform-engineer') OR NOT EXISTS( + SELECT 1 FROM lzc.application_instances i + JOIN lzc.application_template_versions v ON v.id=i.version_id AND v.tenant_id=i.tenant_id + JOIN lzc.application_platform_contracts c ON c.revision=v.platform_revision AND c.tenant_id=v.tenant_id + WHERE i.id=job.instance_id AND i.tenant_id=job.tenant_id AND c.approved_by=lzc.current_user_id() + ) THEN RAISE EXCEPTION 'application_recovery_replan_access_denied' USING ERRCODE='42501'; END IF; + END IF; + SELECT * INTO registered FROM lzc.state_backends b WHERE b.id=bound.backend_id AND b.tenant_id=job.tenant_id; + IF job.operation IS DISTINCT FROM 'apply' OR dispatched.status IS DISTINCT FROM 'reconciliation_required' + OR dispatched.finished_at IS NULL OR dispatched.runner_app_id IS NULL OR dispatched.error_code IS NOT DISTINCT FROM 'state_failed' + OR grant_row.job_id IS NULL OR bound.job_id IS NULL OR registered.id IS NULL + OR grant_row.binding_sha256 IS DISTINCT FROM job.binding_sha256 + OR bound.tenant_id IS DISTINCT FROM job.tenant_id OR bound.owner_user_id IS DISTINCT FROM job.owner_user_id + OR bound.approved_by IS DISTINCT FROM grant_row.approved_by + OR bound.descriptor->>'key' IS DISTINCT FROM grant_row.state_key + OR grant_row.state_key IS DISTINCT FROM 'applications/'||job.tenant_id::text||'/'||job.instance_id::text||'/terraform.tfstate' + OR bound.descriptor->>'useLockfile' IS DISTINCT FROM 'true' + OR (review_mode='operator-attested' AND ((grant_row.revoked_at IS NULL AND grant_row.expires_at>now()) + OR EXISTS(SELECT 1 FROM lzc.application_runner_tickets WHERE job_id=p_job AND expires_at>now()))) + OR (review_mode='reported-failure' AND (NOT EXISTS(SELECT 1 FROM lzc.application_runner_tickets ticket + WHERE ticket.job_id=p_job AND ticket.tenant_id=job.tenant_id AND ticket.consumed_at IS NOT NULL AND ticket.runner_package_id=dispatched.runner_package_id) + OR NOT EXISTS(SELECT 1 FROM lzc.application_runner_records output WHERE output.job_id=p_job AND output.kind='output' + AND output.tenant_id=job.tenant_id AND output.owner_user_id=job.owner_user_id AND output.binding_sha256=job.binding_sha256 + AND output.state_key=grant_row.state_key AND output.runner_package_id=dispatched.runner_package_id))) + OR EXISTS(SELECT 1 FROM lzc.application_runner_records WHERE job_id=p_job AND kind='recovery') + OR EXISTS(SELECT 1 FROM lzc.application_dispatches other WHERE other.instance_id=job.instance_id AND other.tenant_id=job.tenant_id + AND other.job_id<>p_job AND other.status NOT IN ('succeeded','failed','reconciliation_required')) + OR (registered.descriptor - 'key' - 'useLockfile') IS DISTINCT FROM (bound.descriptor - 'key' - 'useLockfile') + OR NOT EXISTS(SELECT 1 FROM lzc.tenants t WHERE t.id=job.tenant_id AND t.organization_id=grant_row.organization_id AND t.organization_verified AND t.archived_at IS NULL) + THEN RAISE EXCEPTION 'application_recovery_replan_unavailable' USING ERRCODE='40001'; END IF; + descriptor_hash:=encode(sha256(convert_to(bound.descriptor::text,'UTF8')),'hex'); + -- Ciphertext is internal broker material; never serialize it into UI or proof tokens. + RETURN jsonb_build_object('jobId',job.id,'instanceId',job.instance_id,'bindingSha256',job.binding_sha256, + 'status',dispatched.status,'operation',job.operation,'finishedAt',dispatched.finished_at,'recoverySha256',NULL, + 'finding','completion-review-required','mode',review_mode, + 'backendId',bound.backend_id,'backendDescriptor',bound.descriptor,'runnerAppId',dispatched.runner_app_id, + 'registeredBackendDescriptor',registered.descriptor,'backendCiphertext',encode(registered.credentials_ciphertext,'hex'), + 'stateKey',grant_row.state_key,'backendSha256',descriptor_hash, + 'existingReceipt',(SELECT to_jsonb(r) FROM lzc.application_recovery_replans r WHERE r.job_id=p_job)); +END $$; +REVOKE ALL ON FUNCTION lzc_auth.application_recovery_replan_context(uuid,uuid) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION lzc_auth.application_recovery_replan_context(uuid,uuid) TO configurator_app; + +CREATE FUNCTION lzc_auth.record_application_recovery_replan(p_session uuid,p_job uuid,p_binding text,p_backend_sha text,p_state_sha text,p_state_serial bigint,p_state_version text,p_attested boolean) +RETURNS lzc.application_recovery_replans LANGUAGE plpgsql SECURITY DEFINER SET search_path=pg_catalog AS $$ +DECLARE context jsonb; job lzc.application_jobs; receipt lzc.application_recovery_replans; +BEGIN + context:=lzc_auth.application_recovery_replan_context(p_session,p_job); + IF p_attested IS DISTINCT FROM (context->>'mode'='operator-attested') OR p_state_sha IS NULL OR p_state_sha !~ '^[a-f0-9]{64}$' + OR p_state_serial IS NULL OR p_state_serial<0 OR (p_state_version IS NOT NULL AND length(p_state_version) NOT BETWEEN 1 AND 1024) + THEN RAISE EXCEPTION 'application_recovery_replan_unavailable' USING ERRCODE='40001'; END IF; + IF context->>'bindingSha256' IS DISTINCT FROM p_binding OR context->>'backendSha256' IS DISTINCT FROM p_backend_sha THEN + RAISE EXCEPTION 'application_recovery_replan_changed' USING ERRCODE='40001'; + END IF; + SELECT * INTO job FROM lzc.application_jobs WHERE id=p_job; + SELECT * INTO receipt FROM lzc.application_recovery_replans WHERE job_id=p_job; + IF FOUND THEN + IF receipt.reviewed_by<>lzc.current_user_id() OR receipt.mode IS DISTINCT FROM context->>'mode' OR receipt.binding_sha256 IS DISTINCT FROM p_binding + OR receipt.backend_sha256 IS DISTINCT FROM p_backend_sha OR receipt.state_sha256 IS DISTINCT FROM p_state_sha + OR receipt.state_serial IS DISTINCT FROM p_state_serial OR receipt.state_version IS DISTINCT FROM p_state_version THEN + RAISE EXCEPTION 'application_recovery_replan_changed' USING ERRCODE='40001'; + END IF; + RETURN receipt; + END IF; + INSERT INTO lzc.application_recovery_replans(job_id,tenant_id,owner_user_id,reviewed_by,review_session_id,mode,binding_sha256,backend_id,backend_sha256,state_key,state_sha256,state_serial,state_version,attested_runner_stopped,attested_cloud_state_compared) + VALUES(p_job,job.tenant_id,job.owner_user_id,lzc.current_user_id(),p_session,context->>'mode',p_binding,(context->>'backendId')::uuid,p_backend_sha,context->>'stateKey',p_state_sha,p_state_serial,p_state_version,p_attested,p_attested) + RETURNING * INTO receipt; + RETURN receipt; +END $$; +REVOKE ALL ON FUNCTION lzc_auth.record_application_recovery_replan(uuid,uuid,text,text,text,bigint,text,boolean) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION lzc_auth.record_application_recovery_replan(uuid,uuid,text,text,text,bigint,text,boolean) TO configurator_app; + +-- Revalidate every historical clearance at every execution gate. New recovery +-- evidence, renewed capabilities or changed bindings close the gate again. +CREATE FUNCTION lzc_auth.application_recovery_replan_valid(p_job uuid) RETURNS boolean +LANGUAGE sql SECURITY DEFINER SET search_path=pg_catalog AS $$ + SELECT EXISTS(SELECT 1 FROM lzc.application_recovery_replans r + JOIN lzc.application_jobs j ON j.id=r.job_id AND j.tenant_id=r.tenant_id AND j.owner_user_id=r.owner_user_id AND j.operation='apply' AND j.binding_sha256=r.binding_sha256 + JOIN lzc.application_dispatches d ON d.job_id=j.id AND d.status='reconciliation_required' AND d.finished_at IS NOT NULL AND d.error_code IS DISTINCT FROM 'state_failed' + JOIN lzc.application_job_grants g ON g.job_id=j.id AND g.binding_sha256=r.binding_sha256 AND g.state_key=r.state_key + JOIN lzc.application_job_backends b ON b.job_id=j.id AND b.backend_id=r.backend_id AND b.tenant_id=r.tenant_id + AND encode(sha256(convert_to(b.descriptor::text,'UTF8')),'hex')=r.backend_sha256 + WHERE r.job_id=p_job AND ( + (r.mode='operator-attested' AND (g.revoked_at IS NOT NULL OR g.expires_at<=now()) + AND NOT EXISTS(SELECT 1 FROM lzc.application_runner_tickets WHERE job_id=p_job AND expires_at>now())) + OR (r.mode='reported-failure' AND d.error_code='apply_failed' + AND EXISTS(SELECT 1 FROM lzc.application_runner_tickets ticket WHERE ticket.job_id=p_job AND ticket.tenant_id=j.tenant_id + AND ticket.consumed_at IS NOT NULL AND ticket.runner_package_id=d.runner_package_id) + AND EXISTS(SELECT 1 FROM lzc.application_runner_records output WHERE output.job_id=p_job AND output.kind='output' + AND output.tenant_id=j.tenant_id AND output.owner_user_id=j.owner_user_id AND output.binding_sha256=j.binding_sha256 + AND output.state_key=g.state_key AND output.runner_package_id=d.runner_package_id))) + AND NOT EXISTS(SELECT 1 FROM lzc.application_runner_records WHERE job_id=p_job AND kind='recovery')); +$$; +REVOKE ALL ON FUNCTION lzc_auth.application_recovery_replan_valid(uuid) FROM PUBLIC; + +-- Public listing exposes only a caller-visible historical clearance boolean. +CREATE FUNCTION lzc_auth.application_recovery_replan_approved(p_job uuid) RETURNS boolean +LANGUAGE sql SECURITY DEFINER SET search_path=pg_catalog AS $$ + SELECT EXISTS(SELECT 1 FROM lzc.application_jobs j + JOIN lzc.application_job_grants g ON g.job_id=j.id AND g.tenant_id=j.tenant_id + WHERE j.id=p_job AND j.tenant_id=lzc.current_tenant_id() + AND ((j.owner_user_id=lzc.current_user_id() AND (lzc.application_role('application-owner') OR lzc.application_role('platform-engineer'))) + OR (g.approved_by=lzc.current_user_id() AND lzc.application_role('platform-engineer'))) + AND lzc_auth.application_recovery_replan_valid(j.id)); +$$; +REVOKE ALL ON FUNCTION lzc_auth.application_recovery_replan_approved(uuid) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION lzc_auth.application_recovery_replan_approved(uuid) TO configurator_app; + +CREATE OR REPLACE FUNCTION lzc_auth.application_execution_blocked(p_job uuid,p_instance uuid,p_tenant uuid,p_purpose text,p_apply_only boolean DEFAULT false) RETURNS boolean +LANGUAGE sql SECURITY DEFINER SET search_path=pg_catalog AS $$ + SELECT EXISTS( + SELECT 1 FROM lzc.application_dispatches d WHERE d.tenant_id=p_tenant AND d.instance_id=p_instance AND d.job_id<>p_job + AND (NOT p_apply_only OR EXISTS(SELECT 1 FROM lzc.application_jobs job WHERE job.id=d.job_id AND job.operation='apply')) + AND (d.status NOT IN ('succeeded','failed','reconciliation_required') OR (d.status='reconciliation_required' AND NOT ( + (p_purpose='standard' AND lzc_auth.application_recovery_replan_valid(d.job_id)) OR + (p_purpose IN ('destroy','drift') AND d.finished_at IS NOT NULL + AND NOT EXISTS(SELECT 1 FROM lzc.application_runner_tickets t WHERE t.job_id=d.job_id AND t.expires_at>now()) + AND NOT EXISTS(SELECT 1 FROM lzc.application_runner_records r WHERE r.job_id=d.job_id AND r.kind='recovery')) + ))) + ); +$$; + +CREATE FUNCTION lzc_auth.assert_application_replan_fresh(p_plan uuid) RETURNS void +LANGUAGE plpgsql SECURITY DEFINER SET search_path=pg_catalog AS $$ +BEGIN + IF EXISTS(SELECT 1 FROM lzc.application_jobs planned + JOIN lzc.application_recovery_replans r ON r.tenant_id=planned.tenant_id + JOIN lzc.application_dispatches d ON d.job_id=r.job_id AND d.instance_id=planned.instance_id + WHERE planned.id=p_plan AND planned.created_at<=greatest(r.reviewed_at,d.finished_at)) THEN + RAISE EXCEPTION 'application_recovery_fresh_plan_required' USING ERRCODE='40001'; + END IF; +END $$; +REVOKE ALL ON FUNCTION lzc_auth.assert_application_replan_fresh(uuid) FROM PUBLIC; + +DO $$ +DECLARE definition text; signature text; anchor text; +BEGIN + FOREACH signature IN ARRAY ARRAY['lzc_auth.application_apply_plan(uuid,uuid,text)','lzc_auth.application_running_apply_plan(uuid,uuid,text,uuid)'] LOOP + definition:=pg_get_functiondef(signature::regprocedure); + anchor:=' RETURN jsonb_build_object(''planId'',p_plan'; + IF strpos(definition,anchor)=0 THEN RAISE EXCEPTION 'application recovery saved plan source mismatch'; END IF; + EXECUTE replace(definition,anchor,' PERFORM lzc_auth.assert_application_replan_fresh(p_plan);'||chr(10)||anchor); + END LOOP; + definition:=pg_get_functiondef('lzc.validate_application_apply()'::regprocedure); + anchor:=' context:=lzc_auth.application_apply_plan(NEW.issuer_session_id,NEW.plan_id,NEW.artifact_sha256);'; + IF strpos(definition,anchor)=0 THEN RAISE EXCEPTION 'application recovery apply source mismatch'; END IF; + EXECUTE replace(definition,anchor,' PERFORM lzc_auth.assert_application_replan_fresh(NEW.plan_id);'||chr(10)||anchor); +END $$; diff --git a/landing-zone-configurator/app/apps/api/src/applications/recovery-replan.ts b/landing-zone-configurator/app/apps/api/src/applications/recovery-replan.ts new file mode 100644 index 0000000..a113a24 --- /dev/null +++ b/landing-zone-configurator/app/apps/api/src/applications/recovery-replan.ts @@ -0,0 +1,329 @@ +import { + applicationRecoveryReplanCheckSchema, + applicationRecoveryReplanReceiptSchema, + applicationRecoveryReplanRequestSchema, + s3BackendDescriptorSchema, + s3RunnerBackendSchema, +} from "@lzc/contracts"; +import type pg from "pg"; +import { z } from "zod"; +import type { Session } from "../auth/store.js"; +import { CredentialError } from "../credentials/profiles.js"; +import type { Backends } from "../deployments/backends.js"; +import { type ArtifactCrypto, canonicalBase64 } from "../plans/crypto.js"; +import { canonicalJson, sha256 } from "../plans/execution.js"; + +// A separate error type avoids a circular import of Applications. +export class RecoveryReplanError extends Error { + constructor( + readonly status: number, + readonly code: string, + ) { + super(code); + } +} +const hash = z.string().regex(/^[a-f0-9]{64}$/); +const contextSchema = z.object({ + mode: z.enum(["reported-failure", "operator-attested"]), + jobId: z.uuid(), + runnerAppId: z.uuid(), + instanceId: z.uuid(), + bindingSha256: hash, + backendId: z.uuid(), + backendSha256: hash, + backendDescriptor: s3BackendDescriptorSchema, + stateKey: z.string(), +}); +const proofSchema = z.strictObject({ + context: contextSchema.strict(), + stateSha256: hash, + stateSerial: applicationRecoveryReplanCheckSchema.shape.stateSerial, + stateVersion: z.string().min(1).max(1024), + expiresAt: z.number().int(), +}); +export type RecoveryReplanBackends = Pick & + Partial>; +type Dependencies = { + backends: RecoveryReplanBackends; + crypto: ArtifactCrypto; +}; +const changed = () => + new RecoveryReplanError(409, "application_recovery_replan_changed"); +async function context(client: pg.PoolClient, session: Session, jobId: string) { + const row = ( + await client.query<{ context: unknown }>( + "SELECT lzc_auth.application_recovery_replan_context($1,$2) AS context", + [session.id, jobId], + ) + ).rows[0]; + const bound = contextSchema.parse(row?.context); + if ( + bound.jobId !== jobId || + bound.stateKey !== + `applications/${session.tenantId}/${bound.instanceId}/terraform.tfstate` || + bound.backendDescriptor.key !== bound.stateKey || + !bound.backendDescriptor.useLockfile + ) + throw changed(); + const backendCiphertext = z + .object({ + backendCiphertext: z + .string() + .regex(/^[a-f0-9]+$/) + .optional(), + }) + .parse(row?.context).backendCiphertext; + return { bound, backendCiphertext }; +} +async function snapshot( + client: pg.PoolClient, + session: Session, + bound: z.infer, + deps: Dependencies, + backendCiphertext?: string, +) { + if (!deps.backends.current) + throw new RecoveryReplanError( + 503, + "application_recovery_replan_unavailable", + ); + // The authorized SQL context supplies ciphertext for delegated owners whose + // tenant role deliberately cannot enumerate backend credentials or objects. + const registered = backendCiphertext + ? s3RunnerBackendSchema.parse({ + kind: "s3", + descriptor: bound.backendDescriptor, + credentials: JSON.parse( + deps.crypto + .decrypt( + Buffer.from(backendCiphertext, "hex"), + session.tenantId, + bound.backendId, + `backend:${bound.backendId}`, + ) + .toString("utf8"), + ), + }) + : await deps.backends.runner(client, session, bound.backendId); + const { + key: _registeredKey, + useLockfile: _registeredLock, + ...registeredLocation + } = registered.descriptor; + const { + key: _boundKey, + useLockfile: _boundLock, + ...boundLocation + } = bound.backendDescriptor; + if (canonicalJson(registeredLocation) !== canonicalJson(boundLocation)) + throw changed(); + const backend = s3RunnerBackendSchema.parse({ + ...registered, + descriptor: bound.backendDescriptor, + }); + try { + // current() checks the lock before reading; a second read also checks for a + // newly appeared lock and a changed object. This is observation, no unlock. + const first = await deps.backends.current(backend, true); + const last = await deps.backends.current(backend, true); + const stateSha256 = sha256(first.bytes); + if (first.identity !== last.identity || stateSha256 !== sha256(last.bytes)) + throw changed(); + const roots = first.document.resources.filter( + (r) => + !r.module && + r.mode === "managed" && + r.type === "terraform_data" && + r.name === "contract", + ); + const instances = roots[0]?.instances; + const input = instances?.[0]?.attributes?.input; + const instanceId = + typeof input === "string" + ? input + : input && typeof input === "object" && "value" in input + ? input.value + : undefined; + if ( + roots.length !== 1 || + instances?.length !== 1 || + instanceId !== bound.instanceId || + first.document.resources.some((r) => r.instances?.some((i) => i.deposed)) + ) + throw new RecoveryReplanError( + 409, + "application_recovery_replan_state_invalid", + ); + const counts = new Map< + string, + { type: string; mode: "managed" | "data"; count: number } + >(); + for (const r of first.document.resources) { + if ( + !r.type || + !/^[a-zA-Z0-9_]+$/.test(r.type) || + (r.mode !== "managed" && r.mode !== "data") + ) + throw new RecoveryReplanError( + 409, + "application_recovery_replan_state_invalid", + ); + const key = `${r.mode}:${r.type}`; + const item = counts.get(key) ?? { type: r.type, mode: r.mode, count: 0 }; + item.count += r.instances?.length ?? 0; + counts.set(key, item); + } + return { + stateSha256, + stateSerial: first.document.serial, + stateVersion: first.identity, + resources: [...counts.values()].sort((a, b) => + `${a.mode}:${a.type}`.localeCompare(`${b.mode}:${b.type}`), + ), + }; + } catch (error) { + if (error instanceof RecoveryReplanError) throw error; + if (error instanceof CredentialError && error.code === "state_locked") + throw new RecoveryReplanError( + 423, + "application_recovery_replan_state_locked", + ); + throw new RecoveryReplanError( + 409, + "application_recovery_replan_state_unavailable", + ); + } +} +export async function checkRecoveryReplan( + client: pg.PoolClient, + session: Session, + jobId: string, + deps: Dependencies, +) { + const { bound, backendCiphertext } = await context(client, session, jobId); + const state = await snapshot(client, session, bound, deps, backendCiphertext); + const observedAt = new Date(); + const expiresAt = observedAt.getTime() + 5 * 60_000; + const proof = proofSchema.parse({ + context: bound, + stateSha256: state.stateSha256, + stateSerial: state.stateSerial, + stateVersion: state.stateVersion, + expiresAt, + }); + return applicationRecoveryReplanCheckSchema.parse({ + jobId, + runnerAppId: bound.runnerAppId, + instanceId: bound.instanceId, + observedAt: observedAt.toISOString(), + expiresAt: new Date(expiresAt).toISOString(), + stateSha256: state.stateSha256, + stateSerial: state.stateSerial, + resources: state.resources, + stateLock: "free", + reviewToken: deps.crypto + .encrypt( + Buffer.from(JSON.stringify(proof)), + session.tenantId, + session.id, + `application-replan:${jobId}`, + ) + .toString("base64"), + mode: bound.mode, + runnerStop: + bound.mode === "reported-failure" + ? "reported-command-completed" + : "operator-attestation-required", + cloudState: + bound.mode === "reported-failure" + ? "requires-fresh-plan-review" + : "operator-attestation-required", + startsExecution: false, + }); +} +export async function approveRecoveryReplan( + client: pg.PoolClient, + session: Session, + jobId: string, + input: unknown, + deps: Dependencies, +) { + const request = applicationRecoveryReplanRequestSchema.parse(input); + let proof: z.infer; + try { + proof = proofSchema.parse( + JSON.parse( + deps.crypto + .decrypt( + canonicalBase64(request.reviewToken, 6144), + session.tenantId, + session.id, + `application-replan:${jobId}`, + ) + .toString("utf8"), + ), + ); + } catch { + throw changed(); + } + if (proof.expiresAt <= Date.now()) throw changed(); + const { bound, backendCiphertext } = await context(client, session, jobId); + if ( + bound.mode === "operator-attested" && + (!request.confirmRunnerStopped || !request.confirmCloudStateCompared) + ) + throw changed(); + if (canonicalJson(bound) !== canonicalJson(proof.context)) throw changed(); + const state = await snapshot(client, session, bound, deps, backendCiphertext); + if ( + state.stateSha256 !== proof.stateSha256 || + state.stateSerial !== proof.stateSerial || + state.stateVersion !== proof.stateVersion || + proof.expiresAt <= Date.now() + ) + throw changed(); + const row = ( + await client.query<{ + job_id: string; + reviewed_by: string; + reviewed_at: Date; + state_sha256: string; + state_serial: string; + }>( + "SELECT * FROM lzc_auth.record_application_recovery_replan($1,$2,$3,$4,$5,$6,$7,$8)", + [ + session.id, + jobId, + bound.bindingSha256, + bound.backendSha256, + state.stateSha256, + state.stateSerial, + state.stateVersion, + bound.mode === "operator-attested", + ], + ) + ).rows[0]; + if (!row) + throw new RecoveryReplanError( + 503, + "application_recovery_replan_unavailable", + ); + return applicationRecoveryReplanReceiptSchema.parse({ + jobId: row.job_id, + reviewedBy: row.reviewed_by, + reviewedAt: row.reviewed_at.toISOString(), + stateSha256: row.state_sha256, + stateSerial: Number(row.state_serial), + mode: bound.mode, + runnerStop: + bound.mode === "reported-failure" + ? "reported-command-completed" + : "operator-attested", + cloudState: + bound.mode === "reported-failure" + ? "requires-fresh-plan-review" + : "operator-attested", + requiresFreshPlan: true, + startsExecution: false, + }); +} diff --git a/landing-zone-configurator/app/apps/api/src/applications/routes.ts b/landing-zone-configurator/app/apps/api/src/applications/routes.ts index 8747a33..114dd5b 100644 --- a/landing-zone-configurator/app/apps/api/src/applications/routes.ts +++ b/landing-zone-configurator/app/apps/api/src/applications/routes.ts @@ -1,6 +1,7 @@ import { applicationOrderDecisionSchema, applicationPlanPurposeSchema, + applicationRecoveryReplanRequestSchema, applicationRecoveryReviewRequestSchema, } from "@lzc/contracts"; import type { FastifyInstance, FastifyRequest } from "fastify"; @@ -11,6 +12,7 @@ import { validMutation, } from "../auth/routes.js"; import type { Session } from "../auth/store.js"; +import { RecoveryReplanError } from "./recovery-replan.js"; import { ApplicationError, type Applications, @@ -151,6 +153,8 @@ export function registerApplications( | "diagnoseRecovery" | "recoveryReviewContext" | "recordRecoveryReview" + | "checkRecoveryReplan" + | "approveRecoveryReplan" | "startApply" | "listGroups" | "createGroup" @@ -167,7 +171,10 @@ export function registerApplications( const sessions = new WeakMap(); app.register(async (routes) => { routes.setErrorHandler((error, _request, reply) => { - if (error instanceof ApplicationError) + if ( + error instanceof ApplicationError || + error instanceof RecoveryReplanError + ) return reply.code(error.status).send({ error: error.code }); if (error instanceof z.ZodError) return reply.code(400).send({ error: "invalid_application_request" }); @@ -185,6 +192,9 @@ export function registerApplications( "application_execution_binding_conflict", "application_apply_plan_unavailable", "application_apply_package_changed", + "application_recovery_fresh_plan_required", + "application_recovery_replan_changed", + "application_recovery_replan_unavailable", "application_recovery_review_changed", "application_recovery_review_unavailable", ].includes(error.message) @@ -552,6 +562,39 @@ export function registerApplications( ); }, ); + routes.post( + "/api/v1/applications/jobs/:id/recovery-replan/check", + { bodyLimit: 1024 }, + async (request, reply) => { + const session = sessions.get(request); + if (!session) return reply.code(401).send(); + if (!applications.checkRecoveryReplan) + return reply + .code(503) + .send({ error: "application_recovery_replan_unavailable" }); + const { id } = z.strictObject({ id: z.uuid() }).parse(request.params); + z.strictObject({}).parse(request.body); + return applications.checkRecoveryReplan(session, id); + }, + ); + routes.post( + "/api/v1/applications/jobs/:id/recovery-replan", + { bodyLimit: 16384 }, + async (request, reply) => { + const session = sessions.get(request); + if (!session) return reply.code(401).send(); + if (!applications.approveRecoveryReplan) + return reply + .code(503) + .send({ error: "application_recovery_replan_unavailable" }); + const { id } = z.strictObject({ id: z.uuid() }).parse(request.params); + return applications.approveRecoveryReplan( + session, + id, + applicationRecoveryReplanRequestSchema.parse(request.body), + ); + }, + ); routes.post( "/api/v1/applications/jobs/:id/apply", { bodyLimit: 1024 }, diff --git a/landing-zone-configurator/app/apps/api/src/applications/service.ts b/landing-zone-configurator/app/apps/api/src/applications/service.ts index ed458eb..8dddba2 100644 --- a/landing-zone-configurator/app/apps/api/src/applications/service.ts +++ b/landing-zone-configurator/app/apps/api/src/applications/service.ts @@ -39,13 +39,18 @@ import type { Session } from "../auth/store.js"; import { tokenHash } from "../auth/store.js"; import type { PostgresCredentialProfiles } from "../credentials/profiles.js"; import type { CredentialSecrets } from "../credentials/secrets.js"; -import type { Backends } from "../deployments/backends.js"; import type { PlanRunner } from "../plans/cloud-foundry.js"; import { type ArtifactCrypto, canonicalBase64 } from "../plans/crypto.js"; import { canonicalJson, sha256 } from "../plans/execution.js"; import type { Plans } from "../plans/service.js"; import { withTenant } from "../storage/database.js"; +import { + approveRecoveryReplan, + checkRecoveryReplan, + type RecoveryReplanBackends, +} from "./recovery-replan.js"; + export class ApplicationError extends Error { constructor( readonly status: number, @@ -195,7 +200,7 @@ export class Applications { > & Partial>, private readonly secrets?: Pick, - private readonly backends?: Pick, + private readonly backends?: RecoveryReplanBackends, private readonly dispatch?: { runner: PlanRunner; origin: string }, private readonly artifactCrypto?: ArtifactCrypto, private readonly platforms?: Pick, @@ -1156,6 +1161,7 @@ export class Applications { plan_id: string | null; artifact_sha256: string | null; can_apply: boolean; + recovery_replan_approved: boolean; }>( `SELECT j.id,j.instance_id,j.owner_user_id,g.approved_by,j.created_at,j.expires_at, coalesce(d.status,j.status) AS status,g.revoked_at IS NULL AND g.expires_at>now() AS grant_active, @@ -1166,6 +1172,7 @@ export class Applications { jd.job_id IS NOT NULL AS delegated_execution, j.operation,coalesce(j.inputs->>'purpose','standard') AS purpose,j.plan_id,a.sha256 AS artifact_sha256, CASE WHEN j.operation='plan' AND a.sha256 IS NOT NULL AND j.owner_user_id=$3 THEN lzc_auth.application_apply_available($4,j.id,a.sha256) ELSE false END AS can_apply, + lzc_auth.application_recovery_replan_approved(j.id) AS recovery_replan_approved, d.summary,d.error_code FROM lzc.application_jobs j JOIN lzc.application_job_grants g ON g.job_id=j.id AND g.tenant_id=j.tenant_id LEFT JOIN lzc.application_job_backends b ON b.job_id=j.id AND b.tenant_id=j.tenant_id @@ -1189,6 +1196,7 @@ export class Applications { planId: row.plan_id, artifactSha256: row.artifact_sha256, canApply: this.executionCapabilities().applyEnabled && row.can_apply, + recoveryReplanApproved: row.recovery_replan_approved ?? false, status: row.status, grantActive: row.grant_active, backendId: row.backend_id, @@ -1318,6 +1326,34 @@ export class Applications { }); } + async checkRecoveryReplan(session: Session, jobId: string) { + const id = z.uuid().parse(jobId); + const backends = this.backends; + const crypto = this.artifactCrypto; + if (!backends?.current || !crypto) + throw new ApplicationError( + 503, + "application_recovery_replan_unavailable", + ); + return this.work(session, "read", (client) => + checkRecoveryReplan(client, session, id, { backends, crypto }), + ); + } + + async approveRecoveryReplan(session: Session, jobId: string, input: unknown) { + const id = z.uuid().parse(jobId); + const backends = this.backends; + const crypto = this.artifactCrypto; + if (!backends?.current || !crypto) + throw new ApplicationError( + 503, + "application_recovery_replan_unavailable", + ); + return this.work(session, "read", (client) => + approveRecoveryReplan(client, session, id, input, { backends, crypto }), + ); + } + async recoveryReviewContext(session: Session, jobId: string) { const id = z.uuid().parse(jobId); if (!this.options.recoveryReviewsEnabled) diff --git a/landing-zone-configurator/app/apps/api/src/deployments/backends.ts b/landing-zone-configurator/app/apps/api/src/deployments/backends.ts index c35d939..9a79e62 100644 --- a/landing-zone-configurator/app/apps/api/src/deployments/backends.ts +++ b/landing-zone-configurator/app/apps/api/src/deployments/backends.ts @@ -343,12 +343,17 @@ export class Backends { throw new CredentialError(409, "backend_credentials_invalid"); } } - async current(backend: S3RunnerBackend) { + async current(backend: S3RunnerBackend, verifyUnlockedAfterRead = false) { if (await this.read(backend, `${backend.descriptor.key}.tflock`)) throw new CredentialError(423, "state_locked"); const remote = await this.read(backend, backend.descriptor.key); if (!remote) throw new CredentialError(409, "backend_state_missing"); const document = stateDocument(remote.bytes); + if ( + verifyUnlockedAfterRead && + (await this.read(backend, `${backend.descriptor.key}.tflock`)) + ) + throw new CredentialError(423, "state_locked"); return { ...remote, document }; } } diff --git a/landing-zone-configurator/app/apps/web/src/components/Applications.tsx b/landing-zone-configurator/app/apps/web/src/components/Applications.tsx index b65aca6..3e27024 100644 --- a/landing-zone-configurator/app/apps/web/src/components/Applications.tsx +++ b/landing-zone-configurator/app/apps/web/src/components/Applications.tsx @@ -5,6 +5,8 @@ import { applicationPlanJobSchema, applicationPlanPreviewSchema, applicationRecoveryDiagnosisSchema, + applicationRecoveryReplanCheckSchema, + applicationRecoveryReplanReceiptSchema, applicationRecoveryReviewSchema, applicationRecoveryReviewViewSchema, appliedPlatformSourceSchema, @@ -43,6 +45,18 @@ import { OrganizationBinding } from "./OrganizationBinding"; import { templateObservabilityConfigurable } from "./TemplateParameters"; const errors: Record = { + application_recovery_fresh_plan_required: + "Nach der Checkpoint-Prüfung ist ein neuer Plan erforderlich. Bitte dieselbe Bestellung neu planen und den neuen Plan prüfen.", + application_recovery_replan_changed: + "Der Checkpoint oder Prüfkontext hat sich geändert oder ist abgelaufen. Bitte erneut prüfen.", + application_recovery_replan_unavailable: + "Dieser Apply ist noch nicht für einen neuen Plan freigebbar. Ausführung, Berechtigungen und Recovery-State müssen vom Platform Owner geprüft werden.", + application_recovery_replan_state_locked: + "Der Application-State ist gesperrt. Keine Freigabe möglich; die Sperre wird nicht verändert.", + application_recovery_replan_state_invalid: + "Der State passt nicht eindeutig zu dieser Bestellung oder enthält ersetzte Instanzen. Operator-Abgleich erforderlich.", + application_recovery_replan_state_unavailable: + "Der gebundene Application-State konnte nicht sicher gelesen werden. Bitte den Platform Owner kontaktieren.", application_recovery_review_changed: "Der Prüfstand hat sich geändert. Bitte den Prüfkontext neu laden.", application_recovery_review_unavailable: @@ -742,6 +756,187 @@ function RecoveryReview({ ); } +function RecoveryReplan({ + session, + jobId, + onApproved, +}: { + session: Session; + jobId: string; + onApproved: () => Promise; +}) { + const [checkpoint, setCheckpoint] = useState | null>(null); + const [busy, setBusy] = useState(false); + const [error, setError] = useState(""); + const [runnerStopped, setRunnerStopped] = useState(false); + const [cloudCompared, setCloudCompared] = useState(false); + const [freshPlan, setFreshPlan] = useState(false); + async function check() { + setBusy(true); + setError(""); + setCheckpoint(null); + setRunnerStopped(false); + setCloudCompared(false); + setFreshPlan(false); + try { + setCheckpoint( + applicationRecoveryReplanCheckSchema.parse( + await request(`jobs/${jobId}/recovery-replan/check`, session, {}), + ), + ); + } catch (cause) { + setError( + cause instanceof Error + ? cause.message + : "Die Recovery-Prüfung ist derzeit nicht verfügbar.", + ); + } finally { + setBusy(false); + } + } + async function approve() { + if ( + !checkpoint || + !freshPlan || + (checkpoint.mode === "operator-attested" && + (!runnerStopped || !cloudCompared)) + ) + return; + setBusy(true); + setError(""); + try { + applicationRecoveryReplanReceiptSchema.parse( + await request(`jobs/${jobId}/recovery-replan`, session, { + reviewToken: checkpoint.reviewToken, + confirmFreshPlan: true, + ...(checkpoint.mode === "operator-attested" + ? { confirmRunnerStopped: true, confirmCloudStateCompared: true } + : {}), + }), + ); + await onApproved(); + } catch (cause) { + setCheckpoint(null); + setError( + cause instanceof Error + ? cause.message + : "Die Recovery-Prüfung ist derzeit nicht verfügbar.", + ); + } finally { + setBusy(false); + } + } + return ( +
+

{t("Neuen Plan nach Apply-Fehler ermöglichen")}

+

+ {t( + "Bei regulär gemeldetem Apply-Fehler prüft der Server den gespeicherten State für einen neuen Plan. Nur unklare Abschlüsse benötigen einen externen Operator-Abgleich.", + )} +

+ {error &&

{t(error)}

} + + {checkpoint && ( + <> +
+
{t("Runner-App")}
+
{checkpoint.runnerAppId}
+
{t("State-Serial")}
+
{checkpoint.stateSerial}
+
{t("Checkpoint-SHA256")}
+
{checkpoint.stateSha256}
+
{t("State-Lock")}
+
{t("Frei")}
+
{t("Prüfung gültig bis")}
+
+ {new Date(checkpoint.expiresAt).toLocaleString(currentLanguage())} +
+
+
    + {checkpoint.resources.map((r) => ( +
  • + {r.type} ({r.mode}): {r.count} +
  • + ))} +
+
+ + {t( + checkpoint.mode === "reported-failure" + ? "Neuen Plan prüfen" + : "Externer Operator-Abgleich", + )} + + {checkpoint.mode === "reported-failure" && ( +

+ {t( + "Der Runner hat das Ende des Apply-Befehls gemeldet. Der gebundene State ist lesbar und entsperrt; ein separater Recovery-State liegt nicht vor. Der neue Plan aktualisiert den Cloud-Zustand. Prüfe ihn vor einem erneuten Apply vollständig.", + )} +

+ )} + {checkpoint.mode === "operator-attested" && ( + <> + + + + )} + + +
+ + )} +
+ ); +} + function ApplicationPlanControls({ instance, session, @@ -1060,9 +1255,10 @@ function ApplicationPlanControls({ (job.status === "prepared" && job.grantActive) || (!["prepared", "succeeded", "failed"].includes(job.status) && !( - purpose !== "standard" && job.status === "reconciliation_required" && - new Date(job.expiresAt).getTime() <= Date.now() + (job.recoveryReplanApproved || + (purpose !== "standard" && + new Date(job.expiresAt).getTime() <= Date.now())) )), ); return ( @@ -1103,6 +1299,18 @@ function ApplicationPlanControls({ {!planEnabled && (

{t("Der Application-Plan-Runner ist nicht aktiviert.")}

)} + {planView && + jobs.some( + (job) => + job.status === "reconciliation_required" && + !job.recoveryReplanApproved, + ) && ( +

+ {t( + "Ein vorheriger Apply ist fehlgeschlagen. Prüfe in der Apply-Ansicht den Application-Checkpoint, um dieselbe Bestellung neu zu planen. Nur bei unklarem Abschluss ist ein Operator-Abgleich nötig.", + )} +

+ )} {planView && instance.requestedBy === session.user.id && approved && (
)} + {job.recoveryReplanApproved && ( +

+ {t( + "Der Checkpoint wurde geprüft. Ein neuer Plan derselben Bestellung ist möglich; der alte Plan bleibt für Apply gesperrt.", + )} +

+ )} + {((job.requestedBy === session.user.id && + job.errorCode === "apply_failed") || + (isEngineer && job.approvedBy === session.user.id)) && + job.operation === "apply" && + job.status === "reconciliation_required" && + !job.recoveryReplanApproved && ( + refresh()} + /> + )} {recoveryReviewsEnabled && isEngineer && job.approvedBy === session.user.id && diff --git a/landing-zone-configurator/app/apps/web/src/locales/en.ts b/landing-zone-configurator/app/apps/web/src/locales/en.ts index b8a571f..e68d598 100644 --- a/landing-zone-configurator/app/apps/web/src/locales/en.ts +++ b/landing-zone-configurator/app/apps/web/src/locales/en.ts @@ -1,4 +1,40 @@ export const englishMessages: Record = { + "Nach der Checkpoint-Prüfung ist ein neuer Plan erforderlich. Bitte dieselbe Bestellung neu planen und den neuen Plan prüfen.": + "A new Plan is required after checkpoint review. Replan the same order and review the new Plan.", + "Bei regulär gemeldetem Apply-Fehler prüft der Server den gespeicherten State für einen neuen Plan. Nur unklare Abschlüsse benötigen einen externen Operator-Abgleich.": + "For a normally reported Apply failure, the server checks saved state for a new Plan. Only unresolved completion requires external operator reconciliation.", + "Neuen Plan prüfen": "Review new Plan", + "Der Runner hat das Ende des Apply-Befehls gemeldet. Der gebundene State ist lesbar und entsperrt; ein separater Recovery-State liegt nicht vor. Der neue Plan aktualisiert den Cloud-Zustand. Prüfe ihn vor einem erneuten Apply vollständig.": + "The runner reported completion of the Apply command. The bound state is readable and unlocked; no separate recovery state exists. The new Plan refreshes cloud state. Review it fully before another Apply.", + "Ich möchte dieselbe Bestellung mit dem vorhandenen State neu planen. Den neuen Plan prüfe ich vor Apply; ein Apply wird jetzt nicht gestartet oder freigegeben.": + "I want to replan the same order using existing state. I will review the new Plan before Apply; no Apply is started or authorized now.", + "Neuen Plan ermöglichen": "Enable new Plan", + "Ein vorheriger Apply ist fehlgeschlagen. Prüfe in der Apply-Ansicht den Application-Checkpoint, um dieselbe Bestellung neu zu planen. Nur bei unklarem Abschluss ist ein Operator-Abgleich nötig.": + "A previous Apply failed. In the Apply view, check the Application checkpoint to replan the same order. Only unresolved completion requires operator reconciliation.", + "Der Checkpoint wurde geprüft. Ein neuer Plan derselben Bestellung ist möglich; der alte Plan bleibt für Apply gesperrt.": + "The checkpoint was checked. A new Plan for the same order is available; the old Plan remains blocked for Apply.", + + "Der Checkpoint oder Prüfkontext hat sich geändert oder ist abgelaufen. Bitte erneut prüfen.": + "The checkpoint or review context changed or expired. Check again.", + "Dieser Apply ist noch nicht für einen neuen Plan freigebbar. Ausführung, Berechtigungen und Recovery-State müssen vom Platform Owner geprüft werden.": + "This Apply cannot yet be cleared for a new Plan. The Platform Owner must review execution, permissions and recovery state.", + "Der Application-State ist gesperrt. Keine Freigabe möglich; die Sperre wird nicht verändert.": + "The Application state is locked. Clearance is unavailable; the lock is not changed.", + "Der State passt nicht eindeutig zu dieser Bestellung oder enthält ersetzte Instanzen. Operator-Abgleich erforderlich.": + "The state cannot be bound unambiguously to this order or contains deposed instances. Operator reconciliation is required.", + "Der gebundene Application-State konnte nicht sicher gelesen werden. Bitte den Platform Owner kontaktieren.": + "The bound Application state could not be read safely. Contact the Platform Owner.", + "Neuen Plan nach Apply-Fehler ermöglichen": + "Enable a new Plan after a failed Apply", + "Application-Checkpoint prüfen": "Check Application checkpoint", + "Runner-App": "Runner app", + "Prüfung gültig bis": "Review valid until", + "Externer Operator-Abgleich": "External operator reconciliation", + "Ich habe extern geprüft, dass der angezeigte Runner keine Ausführung mehr ausführt und nicht erneut starten kann.": + "I checked externally that the displayed runner has stopped executing and cannot restart execution.", + "Ich habe Cloud-Ressourcen und Checkpoint abgeglichen; es gibt keine ungeklärten, nicht im State erfassten Ressourcen dieses Apply.": + "I compared cloud resources with the checkpoint; this Apply has no unresolved resources missing from state.", + "{{value0}} · Version {{value1}} veröffentlicht. Es wurde kein Plan oder Apply gestartet.": "{{value0}} · Version {{value1}} published. No Plan or Apply was started.", "Templates stammen aus der geöffneten Konfiguration. Plattformvertrag und State-Backend werden unabhängig davon geladen.": diff --git a/landing-zone-configurator/app/e2e/organisation.spec.ts b/landing-zone-configurator/app/e2e/organisation.spec.ts index c088497..ca6db7b 100644 --- a/landing-zone-configurator/app/e2e/organisation.spec.ts +++ b/landing-zone-configurator/app/e2e/organisation.spec.ts @@ -27,6 +27,9 @@ for (const scenario of [ "delegated-missing", "apply-running", "apply-failed", + "replan-owner", + "replan-conflict", + "replan-operator", "apply-report-missing", "recovery-review", "recovery-review-conflict", @@ -48,6 +51,8 @@ for (const scenario of [ const instanceId = "44444444-4444-4444-8444-444444444444"; const jobId = "77777777-7777-4777-8777-777777777777"; const backendId = "88888888-8888-4888-8888-888888888888"; + const replan = scenario.startsWith("replan-"); + let replanApproved = false; const maintenance = scenario.startsWith("maintenance-"); const recoveryReview = scenario.startsWith("recovery-review"); const purpose = @@ -59,6 +64,7 @@ for (const scenario of [ scenario === "plan-failed" || scenario === "maintenance-replan-failed"; let maintenanceStarted = scenario === "maintenance-replan-failed"; const initialApply = + replan || recoveryReview || scenario.startsWith("apply-") || scenario.startsWith("archive-") || @@ -67,6 +73,7 @@ for (const scenario of [ scenario === "maintenance-running" || scenario === "maintenance-replan-failed"; const engineer = + scenario === "replan-operator" || recoveryReview || scenario === "engineer" || scenario === "dispatch-failed" || @@ -402,9 +409,10 @@ for (const scenario of [ id: "aaaaaaa1-aaaa-4aaa-8aaa-aaaaaaaaaaaa", instanceId, requestedBy: requesterId, - approvedBy: recoveryReview - ? userId - : "99999999-9999-4999-8999-999999999999", + approvedBy: + recoveryReview || scenario === "replan-operator" + ? userId + : "99999999-9999-4999-8999-999999999999", createdAt: new Date().toISOString(), expiresAt: new Date( Date.now() + @@ -413,6 +421,7 @@ for (const scenario of [ : 120000), ).toISOString(), operation: "apply", + recoveryReplanApproved: replanApproved, purpose: scenario === "maintenance-replan-failed" ? "destroy" @@ -629,6 +638,65 @@ for (const scenario of [ expect(request.headers()["x-lzc-tenant"]).toBe(tenantId); expect(request.headers()["x-lzc-csrf"]).toBe("csrf-plan"); mutations.push(path); + if (path === `jobs/${reviewJobId}/recovery-replan/check`) { + expect(replan).toBe(true); + expect(request.postDataJSON()).toEqual({}); + return route.fulfill({ + json: { + jobId: reviewJobId, + instanceId, + runnerAppId: backendId, + observedAt: new Date().toISOString(), + expiresAt: new Date(Date.now() + 300000).toISOString(), + stateSha256: "d".repeat(64), + stateSerial: 3, + stateLock: "free", + resources: [{ type: "terraform_data", mode: "managed", count: 1 }], + reviewToken: "opaque-proof", + mode: engineer ? "operator-attested" : "reported-failure", + runnerStop: engineer + ? "operator-attestation-required" + : "reported-command-completed", + cloudState: engineer + ? "operator-attestation-required" + : "requires-fresh-plan-review", + startsExecution: false, + }, + }); + } + if (path === `jobs/${reviewJobId}/recovery-replan`) { + expect(request.postDataJSON()).toEqual({ + reviewToken: "opaque-proof", + confirmFreshPlan: true, + ...(engineer + ? { confirmRunnerStopped: true, confirmCloudStateCompared: true } + : {}), + }); + if (scenario === "replan-conflict") + return route.fulfill({ + status: 409, + json: { error: "application_recovery_replan_changed" }, + }); + replanApproved = true; + return route.fulfill({ + json: { + jobId: reviewJobId, + reviewedBy: userId, + reviewedAt: new Date().toISOString(), + stateSha256: "d".repeat(64), + stateSerial: 3, + mode: engineer ? "operator-attested" : "reported-failure", + runnerStop: engineer + ? "operator-attested" + : "reported-command-completed", + cloudState: engineer + ? "operator-attested" + : "requires-fresh-plan-review", + requiresFreshPlan: true, + startsExecution: false, + }, + }); + } if (path === `jobs/${reviewJobId}/recovery-review`) { expect(recoveryReview).toBe(true); expect(request.postDataJSON()).toEqual({ @@ -1060,6 +1128,95 @@ for (const scenario of [ }), ).toBeVisible(); await expect(plan.locator(".application-plan-job")).toHaveCount(1); + if (replan) { + const check = plan.getByRole("region", { + name: "Neuen Plan nach Apply-Fehler ermöglichen", + exact: true, + }); + await expect(check).toBeVisible(); + if (!engineer) { + await tabs.getByRole("tab", { name: "Plan", exact: true }).click(); + await expect( + plan.getByRole("button", { + name: "Cloud-Plan starten", + exact: true, + }), + ).toBeDisabled(); + await expect(plan).toContainText( + "Prüfe in der Apply-Ansicht den Application-Checkpoint", + ); + await tabs.getByRole("tab", { name: "Apply", exact: true }).click(); + } + await check + .getByRole("button", { + name: "Application-Checkpoint prüfen", + exact: true, + }) + .click(); + await expect(check).toContainText("d".repeat(64)); + const enable = check.getByRole("button", { + name: "Neuen Plan ermöglichen", + exact: true, + }); + await expect(enable).toBeDisabled(); + await expect(check.getByRole("checkbox")).toHaveCount(engineer ? 3 : 1); + if (engineer) { + await check + .getByRole("checkbox", { name: /Ich habe extern geprüft/ }) + .check(); + await check + .getByRole("checkbox", { name: /Ich habe Cloud-Ressourcen/ }) + .check(); + } else { + await expect(check).toContainText( + "Der Runner hat das Ende des Apply-Befehls gemeldet.", + ); + } + await check + .getByRole("checkbox", { name: /Ich möchte dieselbe Bestellung/ }) + .check(); + await expect(enable).toBeEnabled(); + await enable.click(); + if (scenario === "replan-conflict") { + await expect(check.getByRole("alert")).toContainText("erneut prüfen"); + await expect(enable).toHaveCount(0); + await expect(check).not.toContainText("d".repeat(64)); + } else { + await expect(check).toHaveCount(0); + await expect(plan).toContainText( + "Ein neuer Plan derselben Bestellung ist möglich", + ); + if (!engineer) { + await tabs.getByRole("tab", { name: "Plan", exact: true }).click(); + await expect( + plan.getByRole("button", { + name: "Cloud-Plan starten", + exact: true, + }), + ).toBeEnabled(); + } + } + expect(mutations).toEqual([ + `jobs/${reviewJobId}/recovery-replan/check`, + `jobs/${reviewJobId}/recovery-replan`, + ]); + await tabs.getByRole("tab", { name: "Apply", exact: true }).click(); + await page.getByLabel("Sprache").selectOption("en"); + if (scenario !== "replan-conflict") + await expect( + page.getByText(/A new Plan for the same order is available/), + ).toBeVisible(); + expect( + await page.evaluate( + () => document.documentElement.scrollWidth <= innerWidth, + ), + ).toBe(true); + await page.screenshot({ + path: testInfo.outputPath("application-replan.png"), + fullPage: true, + }); + return; + } if (recoveryReview) { const review = plan.getByRole("region", { name: "Recovery-Prüfung", diff --git a/landing-zone-configurator/app/integration/identity.test.ts b/landing-zone-configurator/app/integration/identity.test.ts index 19507a0..48a27be 100644 --- a/landing-zone-configurator/app/integration/identity.test.ts +++ b/landing-zone-configurator/app/integration/identity.test.ts @@ -6427,6 +6427,399 @@ describe("real PostgreSQL session and tenant boundaries", () => { }); return { ordered, id: started.jobId, sha: receipt.sha256 }; }; + // A reconciled partial Apply keeps its original order and all evidence. + // An operator clearance opens only a fresh standard Plan, never the old artifact. + for (const selfService of [false, true]) { + const reviewer = selfService ? owner : engineer; + const resumable = await freshPlan("Application checkpoint replan"); + const partialApply = await delegatedApplications.startApply( + owner, + resumable.id, + { + artifactSha256: resumable.sha, + }, + ); + const partialTicket = delegatedTicket; + await delegatedApplications.runnerInput(partialTicket, ticketPackage); + for (const stage of ["initializing", "validating", "applying"]) + await delegatedApplications.runnerStage( + partialTicket, + ticketPackage, + { stage }, + ); + if (selfService) + await delegatedApplications.runnerOutput( + partialTicket, + ticketPackage, + { text: "Synthetic partial Apply failed", truncated: false }, + ); + await delegatedApplications.runnerResult( + partialTicket, + ticketPackage, + { + status: "failed", + errorCode: "apply_failed", + }, + ); + const originalOrder = ( + await migration.query( + "SELECT to_jsonb(i) AS value FROM lzc.application_instances i WHERE id=$1", + [resumable.ordered.id], + ) + ).rows[0].value; + const originalDispatch = ( + await migration.query( + "SELECT to_jsonb(d) AS value FROM lzc.application_dispatches d WHERE job_id=$1", + [partialApply.jobId], + ) + ).rows[0].value; + await migration.query( + "UPDATE lzc_auth.sessions SET expires_at=$2 WHERE id=$1", + [engineer.id, engineerExpiry], + ); + const context = (session: Session) => + withTenant( + pool, + session, + async (client) => + ( + await client.query( + "SELECT lzc_auth.application_recovery_replan_context($1,$2) AS value", + [session.id, partialApply.jobId], + ) + ).rows[0].value, + ); + const approved = (session: Session) => + withTenant( + pool, + session, + async (client) => + ( + await client.query( + "SELECT lzc_auth.application_recovery_replan_approved($1) AS approved", + [partialApply.jobId], + ) + ).rows[0].approved, + ); + const clear = ( + session: Session, + input: { + binding?: string; + backend?: string; + sha?: string; + attested?: boolean; + } = {}, + database: pg.Pool = pool, + ) => + withTenant( + database, + session, + async (client) => + ( + await client.query( + "SELECT receipt.* FROM lzc_auth.record_application_recovery_replan($1,$2,$3,$4,$5,$6,$7,$8) receipt", + [ + session.id, + partialApply.jobId, + input.binding ?? reviewed.bindingSha256, + input.backend ?? reviewed.backendSha256, + input.sha ?? "a".repeat(64), + 3, + "synthetic-state-version", + input.attested ?? !selfService, + ], + ) + ).rows[0], + ); + let reviewed: { + bindingSha256: string; + backendSha256: string; + backendId: string; + stateKey: string; + backendDescriptor: unknown; + existingReceipt: unknown; + }; + try { + await expect(context(engineer)).rejects.toMatchObject({ + code: "40001", + message: "application_recovery_replan_unavailable", + }); + if (selfService) { + expect(await context(owner)).toMatchObject({ + mode: "reported-failure", + }); + await expect( + delegatedApplications.runnerInput(partialTicket, ticketPackage), + ).rejects.toBeDefined(); + await expect( + delegatedApplications.releaseRunnerCredential( + partialTicket, + ticketPackage, + ), + ).rejects.toBeDefined(); + await expect( + delegatedApplications.runnerStage( + partialTicket, + ticketPackage, + { stage: "applying" }, + ), + ).rejects.toBeDefined(); + await expect( + delegatedApplications.runnerResult( + partialTicket, + ticketPackage, + { status: "succeeded" }, + ), + ).rejects.toBeDefined(); + } else + await expect(context(owner)).rejects.toMatchObject({ + code: "42501", + }); + await expect(context(bob)).rejects.toBeDefined(); + if (!selfService) { + await admin.query( + "BEGIN; SET LOCAL session_replication_role=replica", + ); + await admin.query( + "UPDATE lzc.application_runner_tickets SET expires_at=clock_timestamp()-interval '1 microsecond' WHERE job_id=$1", + [partialApply.jobId], + ); + await admin.query( + "UPDATE lzc.application_job_grants SET expires_at=clock_timestamp()-interval '1 microsecond' WHERE job_id=$1", + [partialApply.jobId], + ); + await admin.query("COMMIT"); + } + reviewed = await context(reviewer); + expect(await approved(owner)).toBe(false); + expect(await approved(bob)).toBe(false); + expect(reviewed).toMatchObject({ + stateKey: resumable.ordered.stateKey, + backendId: delegatedBackendId, + existingReceipt: null, + }); + expect(reviewed.backendDescriptor).toMatchObject({ + key: resumable.ordered.stateKey, + useLockfile: true, + }); + await expect( + clear(reviewer, { binding: "f".repeat(64) }), + ).rejects.toMatchObject({ + code: "40001", + message: "application_recovery_replan_changed", + }); + await expect( + clear(reviewer, { backend: "f".repeat(64) }), + ).rejects.toMatchObject({ + code: "40001", + message: "application_recovery_replan_changed", + }); + await expect( + clear(reviewer, { attested: selfService }), + ).rejects.toMatchObject({ code: "40001" }); + await expect( + clear(reviewer, { sha: "invalid" }), + ).rejects.toMatchObject({ code: "40001" }); + await expect( + withTenant(pool, engineer, (client) => + client.query( + "INSERT INTO lzc.application_recovery_replans(job_id) VALUES($1)", + [partialApply.jobId], + ), + ), + ).rejects.toMatchObject({ code: "42501" }); + const concurrent = new pg.Pool({ + ...migrationConfig, + user: "configurator_app", + password: "runtime-test-only", + max: 2, + }); + let receipt: Record; + try { + const [first, second] = await Promise.all([ + clear(reviewer, {}, concurrent), + clear(reviewer, {}, concurrent), + ]); + expect(second).toEqual(first); + receipt = first; + } finally { + await concurrent.end(); + } + expect(receipt).toMatchObject({ + job_id: partialApply.jobId, + state_key: resumable.ordered.stateKey, + state_sha256: "a".repeat(64), + state_serial: "3", + mode: selfService ? "reported-failure" : "operator-attested", + attested_runner_stopped: !selfService, + attested_cloud_state_compared: !selfService, + }); + expect(await clear(reviewer)).toEqual(receipt); + expect(await approved(owner)).toBe(true); + expect(await approved(engineer)).toBe(true); + expect(await approved(bob)).toBe(false); + await expect( + clear(reviewer, { sha: "b".repeat(64) }), + ).rejects.toMatchObject({ + code: "40001", + message: "application_recovery_replan_changed", + }); + await expect( + migration.query( + "DELETE FROM lzc.application_recovery_replans WHERE job_id=$1", + [partialApply.jobId], + ), + ).rejects.toMatchObject({ code: "55000" }); + await expect( + migration.query( + "UPDATE lzc.application_recovery_replans SET state_serial=4 WHERE job_id=$1", + [partialApply.jobId], + ), + ).rejects.toMatchObject({ code: "55000" }); + await expect( + migration.query( + "SELECT lzc_auth.assert_application_replan_fresh($1)", + [resumable.id], + ), + ).rejects.toMatchObject({ + code: "40001", + message: "application_recovery_fresh_plan_required", + }); + // A later local recovery record invalidates even a previously recorded clearance. + await admin.query( + "BEGIN; SET LOCAL session_replication_role=replica", + ); + await admin.query( + "INSERT INTO lzc.application_runner_records(job_id,kind,tenant_id,owner_user_id,ciphertext,sha256,binding_sha256,state_key,runner_package_id) SELECT job_id,'recovery',tenant_id,owner_user_id,$2,$3,binding_sha256,state_key,$4 FROM lzc.application_job_grants WHERE job_id=$1", + [ + partialApply.jobId, + Buffer.alloc(40), + "c".repeat(64), + ticketPackage.runnerPackageId, + ], + ); + await admin.query("COMMIT"); + expect(await approved(owner)).toBe(false); + await expect(context(reviewer)).rejects.toBeDefined(); + await expect( + delegatedApplications.startPlan(owner, resumable.ordered.id, { + idempotencyKey: randomUUID(), + }), + ).rejects.toMatchObject({ code: "40001" }); + await admin.query( + "BEGIN; SET LOCAL session_replication_role=replica", + ); + await admin.query( + "DELETE FROM lzc.application_runner_records WHERE job_id=$1 AND kind='recovery'", + [partialApply.jobId], + ); + await admin.query( + "UPDATE lzc.application_dispatches SET error_code='state_failed' WHERE job_id=$1", + [partialApply.jobId], + ); + await admin.query("COMMIT"); + expect(await approved(owner)).toBe(false); + await expect(context(reviewer)).rejects.toBeDefined(); + await expect( + delegatedApplications.startPlan(owner, resumable.ordered.id, { + idempotencyKey: randomUUID(), + }), + ).rejects.toMatchObject({ code: "40001" }); + await admin.query( + "BEGIN; SET LOCAL session_replication_role=replica", + ); + await admin.query( + "UPDATE lzc.application_dispatches SET error_code='apply_failed' WHERE job_id=$1", + [partialApply.jobId], + ); + await admin.query("COMMIT"); + const nextPlan = await delegatedApplications.startPlan( + owner, + resumable.ordered.id, + { idempotencyKey: randomUUID() }, + ); + expect(nextPlan.jobId).not.toBe(resumable.id); + const nextTicket = delegatedTicket; + await delegatedApplications.runnerInput(nextTicket, ticketPackage); + for (const stage of ["initializing", "validating", "planning"]) + await delegatedApplications.runnerStage( + nextTicket, + ticketPackage, + { stage }, + ); + const summary = summarizePlan( + await delegatedRunner.preview(), + 2, + "opentofu-1.12.6", + ); + const artifact = await delegatedApplications.runnerArtifact( + nextTicket, + ticketPackage, + { + data: Buffer.from("fresh-recovery-plan").toString("base64"), + summary, + }, + ); + await delegatedApplications.runnerResult( + nextTicket, + ticketPackage, + { status: "succeeded", summary, artifactSha256: artifact.sha256 }, + ); + await migration.query( + "SELECT lzc_auth.assert_application_replan_fresh($1)", + [nextPlan.jobId], + ); + const nextApply = await delegatedApplications.startApply( + owner, + nextPlan.jobId, + { artifactSha256: artifact.sha256 }, + ); + expect(nextApply.jobId).not.toBe(partialApply.jobId); + const nextApplyTicket = delegatedTicket; + await delegatedApplications.runnerInput( + nextApplyTicket, + ticketPackage, + ); + for (const stage of ["initializing", "validating", "applying"]) + await delegatedApplications.runnerStage( + nextApplyTicket, + ticketPackage, + { stage }, + ); + await delegatedApplications.runnerResult( + nextApplyTicket, + ticketPackage, + { status: "failed", errorCode: "apply_failed" }, + ); + await expect( + delegatedApplications.startPlan(owner, resumable.ordered.id, { + idempotencyKey: randomUUID(), + }), + ).rejects.toMatchObject({ code: "40001" }); + expect( + ( + await migration.query( + "SELECT to_jsonb(i) AS value FROM lzc.application_instances i WHERE id=$1", + [resumable.ordered.id], + ) + ).rows[0].value, + ).toEqual(originalOrder); + expect( + ( + await migration.query( + "SELECT to_jsonb(d) AS value FROM lzc.application_dispatches d WHERE job_id=$1", + [partialApply.jobId], + ) + ).rows[0].value, + ).toEqual(originalDispatch); + } finally { + await admin.query("ROLLBACK"); + await migration.query( + "UPDATE lzc_auth.sessions SET expires_at=now()-interval '1 second' WHERE id=$1", + [engineer.id], + ); + } + } const maintenance = await freshPlan("Application maintenance"); const failedApply = await delegatedApplications.startApply( owner, diff --git a/landing-zone-configurator/app/packages/contracts/src/index.ts b/landing-zone-configurator/app/packages/contracts/src/index.ts index edb9d4d..e2c729c 100644 --- a/landing-zone-configurator/app/packages/contracts/src/index.ts +++ b/landing-zone-configurator/app/packages/contracts/src/index.ts @@ -226,6 +226,7 @@ export const applicationPlanJobSchema = z.strictObject({ .nullable() .optional(), canApply: z.boolean().default(false), + recoveryReplanApproved: z.boolean().default(false), status: z.enum([ "prepared", "reserved", @@ -363,6 +364,56 @@ export const applicationRecoveryReviewViewSchema = review: applicationRecoveryReviewSchema.nullable(), }); +// Checkpoint contents and credentials never cross this API boundary. +export const applicationRecoveryReplanCheckSchema = z.strictObject({ + jobId: z.uuid(), + runnerAppId: z.uuid(), + instanceId: z.uuid(), + observedAt: z.iso.datetime(), + expiresAt: z.iso.datetime(), + stateSha256: z.string().regex(/^[a-f0-9]{64}$/), + stateSerial: z.number().int().nonnegative().refine(Number.isSafeInteger), + stateLock: z.literal("free"), + resources: z + .array( + z.strictObject({ + type: z.string(), + mode: z.enum(["managed", "data"]), + count: z.number().int().nonnegative(), + }), + ) + .max(1024), + reviewToken: z.string().min(1).max(8192), + mode: z.enum(["reported-failure", "operator-attested"]), + runnerStop: z.enum([ + "reported-command-completed", + "operator-attestation-required", + ]), + cloudState: z.enum([ + "requires-fresh-plan-review", + "operator-attestation-required", + ]), + startsExecution: z.literal(false), +}); +export const applicationRecoveryReplanRequestSchema = z.strictObject({ + reviewToken: applicationRecoveryReplanCheckSchema.shape.reviewToken, + confirmRunnerStopped: z.literal(true).optional(), + confirmCloudStateCompared: z.literal(true).optional(), + confirmFreshPlan: z.literal(true), +}); +export const applicationRecoveryReplanReceiptSchema = z.strictObject({ + jobId: z.uuid(), + reviewedBy: z.uuid(), + reviewedAt: z.iso.datetime(), + stateSha256: applicationRecoveryReplanCheckSchema.shape.stateSha256, + stateSerial: applicationRecoveryReplanCheckSchema.shape.stateSerial, + mode: z.enum(["reported-failure", "operator-attested"]), + runnerStop: z.enum(["reported-command-completed", "operator-attested"]), + cloudState: z.enum(["requires-fresh-plan-review", "operator-attested"]), + requiresFreshPlan: z.literal(true), + startsExecution: z.literal(false), +}); + export const planResultSchema = z.discriminatedUnion("status", [ z .object({ diff --git a/landing-zone-configurator/app/tests/api-backends.test.ts b/landing-zone-configurator/app/tests/api-backends.test.ts index 2cdc41c..3fa53a4 100644 --- a/landing-zone-configurator/app/tests/api-backends.test.ts +++ b/landing-zone-configurator/app/tests/api-backends.test.ts @@ -301,6 +301,31 @@ const credentials = { const backend: S3RunnerBackend = { kind: "s3", descriptor, credentials }; const session = { tenantId: randomUUID(), userId: randomUUID() } as Session; const crypto = new ArtifactCrypto(randomBytes(32).toString("base64")); +it("checks a newly acquired lock after reading an Application checkpoint without changing it", async () => { + const bytes = Buffer.from( + JSON.stringify({ + version: 4, + lineage: randomUUID(), + serial: 3, + resources: [], + }), + ); + const read = vi + .fn() + .mockResolvedValueOnce(null) + .mockResolvedValueOnce({ bytes, identity: "snapshot" }) + .mockResolvedValueOnce({ bytes: Buffer.from("lock"), identity: "lock" }); + const backends = new Backends({} as pg.Pool, crypto, read); + await expect(backends.current(backend, true)).rejects.toMatchObject({ + code: "state_locked", + }); + expect(read.mock.calls.map((call) => call[1])).toEqual([ + `${descriptor.key}.tflock`, + descriptor.key, + `${descriptor.key}.tflock`, + ]); +}); + function fixture() { const source = { version: 4 as const, diff --git a/landing-zone-configurator/app/tests/application-recovery-replan.test.ts b/landing-zone-configurator/app/tests/application-recovery-replan.test.ts new file mode 100644 index 0000000..f9b8b4f --- /dev/null +++ b/landing-zone-configurator/app/tests/application-recovery-replan.test.ts @@ -0,0 +1,498 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import type pg from "pg"; +import { afterEach, expect, it, vi } from "vitest"; +import { + approveRecoveryReplan, + checkRecoveryReplan, +} from "../apps/api/src/applications/recovery-replan.js"; +import type { Session } from "../apps/api/src/auth/store.js"; +import { CredentialError } from "../apps/api/src/credentials/profiles.js"; +import { stateDocument } from "../apps/api/src/deployments/backends.js"; +import { ArtifactCrypto } from "../apps/api/src/plans/crypto.js"; +import { sha256 } from "../apps/api/src/plans/execution.js"; + +const secret = "private-state-and-credential-canary"; +afterEach(() => vi.useRealTimers()); +function fixture() { + const session: Session = { + id: randomUUID(), + userId: randomUUID(), + tenantId: randomUUID(), + githubId: "synthetic", + login: "synthetic", + csrfToken: "synthetic", + expiresAt: new Date(Date.now() + 60_000), + }; + const instanceId = randomUUID(); + const bound = { + mode: "operator-attested" as "operator-attested" | "reported-failure", + jobId: randomUUID(), + instanceId, + runnerAppId: randomUUID(), + bindingSha256: "a".repeat(64), + backendId: randomUUID(), + backendSha256: "b".repeat(64), + stateKey: `applications/${session.tenantId}/${instanceId}/terraform.tfstate`, + backendDescriptor: { + bucket: "synthetic-bucket", + endpoint: "https://object.storage.eu01.onstackit.cloud" as const, + region: "eu01" as const, + key: `applications/${session.tenantId}/${instanceId}/terraform.tfstate`, + useLockfile: true as const, + }, + }; + const contractInstance = { + attributes: { input: instanceId as string, private: secret }, + }; + const projectInstance = { attributes: { password: secret } }; + const contract = { + mode: "managed", + type: "terraform_data", + name: "contract", + instances: [contractInstance], + }; + const project = { + module: "module.application", + mode: "managed", + type: "stackit_resourcemanager_project", + name: "this", + instances: [projectInstance], + }; + const raw = { + version: 4, + lineage: randomUUID(), + serial: 3, + outputs: { private: { value: secret, sensitive: true } }, + resources: [contract, project], + }; + const snapshot = () => { + const bytes = Buffer.from(JSON.stringify(raw)); + return { + bytes, + identity: "synthetic-version", + document: stateDocument(bytes), + }; + }; + const current = vi.fn( + async (_backend: unknown, _verifyUnlockedAfterRead?: boolean) => snapshot(), + ); + const runner = vi.fn(async () => ({ + kind: "s3" as const, + descriptor: { + ...bound.backendDescriptor, + key: "platform/terraform.tfstate", + }, + credentials: { accessKeyId: secret, secretAccessKey: secret }, + })); + const query = vi.fn( + async ( + sql: string, + args: unknown[], + ): Promise<{ rows: Record[] }> => { + if (sql.includes("application_recovery_replan_context")) + return { rows: [{ context: bound }] }; + if (sql.includes("record_application_recovery_replan")) + return { + rows: [ + { + job_id: bound.jobId, + reviewed_by: session.userId, + reviewed_at: new Date(), + state_sha256: args[4], + state_serial: String(args[5]), + }, + ], + }; + throw new Error("Unexpected SQL operation"); + }, + ); + const client = { query } as unknown as pg.PoolClient; + const deps = { + backends: { runner, current }, + crypto: new ArtifactCrypto(randomBytes(32).toString("base64")), + }; + const check = () => checkRecoveryReplan(client, session, bound.jobId, deps); + const request = (reviewToken: string) => ({ + reviewToken, + confirmRunnerStopped: true, + confirmCloudStateCompared: true, + confirmFreshPlan: true, + }); + const approve = (input: unknown) => + approveRecoveryReplan(client, session, bound.jobId, input, deps); + return { + session, + bound, + contract, + project, + contractInstance, + projectInstance, + raw, + snapshot, + current, + runner, + query, + client, + deps, + check, + request, + approve, + }; +} +it("observes only the exact Application key and returns counts without raw State or credentials", async () => { + const f = fixture(); + const result = await f.check(); + expect(f.current).toHaveBeenCalledTimes(2); + expect(f.runner).toHaveBeenCalledExactlyOnceWith( + f.client, + f.session, + f.bound.backendId, + ); + for (const [backend, verifyUnlockedAfterRead] of f.current.mock.calls) { + expect(backend).toMatchObject({ + descriptor: { key: f.bound.stateKey, useLockfile: true }, + }); + expect(verifyUnlockedAfterRead).toBe(true); + } + expect(result).toMatchObject({ + jobId: f.bound.jobId, + instanceId: f.bound.instanceId, + stateSerial: 3, + stateSha256: sha256(f.snapshot().bytes), + stateLock: "free", + runnerStop: "operator-attestation-required", + cloudState: "operator-attestation-required", + startsExecution: false, + resources: [ + { mode: "managed", type: "stackit_resourcemanager_project", count: 1 }, + { mode: "managed", type: "terraform_data", count: 1 }, + ], + }); + expect(JSON.stringify(result)).not.toContain(secret); + expect(JSON.stringify(result)).not.toContain(f.bound.stateKey); + expect(f.query).toHaveBeenCalledTimes(1); +}); +it("requires a fresh observation again before an immutable historical clearance", async () => { + const f = fixture(); + const checked = await f.check(); + const receipt = await f.approve(f.request(checked.reviewToken)); + expect(receipt).toMatchObject({ + jobId: f.bound.jobId, + stateSerial: 3, + requiresFreshPlan: true, + startsExecution: false, + runnerStop: "operator-attested", + cloudState: "operator-attested", + }); + expect(JSON.stringify(receipt)).not.toContain(secret); + expect(f.current).toHaveBeenCalledTimes(4); + const [sql, args] = f.query.mock.calls.at(-1) ?? ["", []]; + expect(sql).toContain("record_application_recovery_replan"); + expect(args).toEqual([ + f.session.id, + f.bound.jobId, + f.bound.bindingSha256, + f.bound.backendSha256, + checked.stateSha256, + 3, + "synthetic-version", + true, + ]); +}); +it.each(["instance", "key", "job", "lockfile", "backend"])( + "rejects an incorrect %s binding before reading customer State", + async (kind) => { + const f = fixture(); + if (kind === "instance") f.bound.instanceId = randomUUID(); + if (kind === "key") + f.bound.backendDescriptor.key = "platform/terraform.tfstate"; + if (kind === "job") + f.query.mockResolvedValueOnce({ + rows: [{ context: { ...f.bound, jobId: randomUUID() } }], + }); + if (kind === "lockfile") + Object.assign(f.bound.backendDescriptor, { useLockfile: false }); + if (kind === "backend") + f.runner.mockResolvedValueOnce({ + kind: "s3", + descriptor: { + ...f.bound.backendDescriptor, + bucket: "different-bucket", + }, + credentials: { accessKeyId: secret, secretAccessKey: secret }, + }); + await expect(f.check()).rejects.toBeDefined(); + expect(f.current).not.toHaveBeenCalled(); + }, +); +it.each(["contract", "duplicate", "deposed", "mode", "resource-type"])( + "rejects invalid State %s without recording a clearance", + async (kind) => { + const f = fixture(); + if (kind === "contract") f.contractInstance.attributes.input = randomUUID(); + if (kind === "duplicate") f.raw.resources.push(structuredClone(f.contract)); + if (kind === "deposed") + Object.assign(f.projectInstance, { + deposed: "synthetic", + }); + if (kind === "mode") f.project.mode = "unknown"; + if (kind === "resource-type") + f.project.type = "private-state-and-credential-canary!"; + await expect(f.check()).rejects.toMatchObject({ + status: 409, + code: "application_recovery_replan_state_invalid", + }); + expect(f.query).toHaveBeenCalledTimes(1); + }, +); +it("accepts the typed OpenTofu terraform_data input representation", async () => { + const f = fixture(); + Object.assign(f.contractInstance.attributes, { + input: { value: f.bound.instanceId, type: "string" }, + }); + expect((await f.check()).stateSerial).toBe(3); +}); +it.each([ + "first-lock", + "second-lock", + "missing", + "changed-version", + "changed-bytes", +])("fails closed for %s during snapshot observation", async (kind) => { + const f = fixture(); + if (kind === "first-lock") + f.current.mockRejectedValueOnce(new CredentialError(423, "state_locked")); + if (kind === "second-lock") + f.current + .mockResolvedValueOnce(f.snapshot()) + .mockRejectedValueOnce(new CredentialError(423, "state_locked")); + if (kind === "missing") f.current.mockRejectedValueOnce(new Error(secret)); + if (kind === "changed-version") + f.current + .mockResolvedValueOnce(f.snapshot()) + .mockResolvedValueOnce({ ...f.snapshot(), identity: "changed" }); + if (kind === "changed-bytes") + f.current + .mockResolvedValueOnce(f.snapshot()) + .mockResolvedValueOnce({ ...f.snapshot(), bytes: Buffer.from(secret) }); + await expect(f.check()).rejects.toMatchObject({ + status: kind.includes("lock") ? 423 : 409, + code: kind.includes("lock") + ? "application_recovery_replan_state_locked" + : kind === "missing" + ? "application_recovery_replan_state_unavailable" + : "application_recovery_replan_changed", + }); + expect(f.query).toHaveBeenCalledTimes(1); +}); +it.each(["session", "tenant", "job", "tamper", "expired"])( + "rejects a %s review token before further reads or writes", + async (kind) => { + const f = fixture(); + const checked = await f.check(); + let token = checked.reviewToken; + if (kind === "session") f.session.id = randomUUID(); + if (kind === "tenant") f.session.tenantId = randomUUID(); + if (kind === "job") f.bound.jobId = randomUUID(); + if (kind === "tamper") { + const bytes = Buffer.from(token, "base64"); + bytes[bytes.length - 1] = (bytes.at(-1) ?? 0) ^ 1; + token = bytes.toString("base64"); + } + if (kind === "expired") { + vi.useFakeTimers(); + vi.setSystemTime(new Date(Date.parse(checked.expiresAt) + 1)); + } + f.query.mockClear(); + f.current.mockClear(); + await expect(f.approve(f.request(token))).rejects.toMatchObject({ + status: 409, + code: "application_recovery_replan_changed", + }); + expect(f.query).not.toHaveBeenCalled(); + expect(f.current).not.toHaveBeenCalled(); + }, +); +it.each(["binding", "backend", "runner", "state"])( + "rejects changed %s between check and approval without a receipt", + async (kind) => { + const f = fixture(); + const checked = await f.check(); + if (kind === "binding") f.bound.bindingSha256 = "c".repeat(64); + if (kind === "backend") f.bound.backendSha256 = "c".repeat(64); + if (kind === "runner") f.bound.runnerAppId = randomUUID(); + if (kind === "state") f.raw.serial = 4; + await expect( + f.approve(f.request(checked.reviewToken)), + ).rejects.toMatchObject({ + status: 409, + code: "application_recovery_replan_changed", + }); + expect( + f.query.mock.calls.every( + ([sql]) => !sql.includes("record_application_recovery_replan"), + ), + ).toBe(true); + }, +); +it.each([ + "confirmRunnerStopped", + "confirmCloudStateCompared", + "confirmFreshPlan", +])("requires explicit %s and rejects arbitrary proof fields", async (flag) => { + const f = fixture(); + const checked = await f.check(); + f.query.mockClear(); + f.current.mockClear(); + await expect( + f.approve({ ...f.request(checked.reviewToken), [flag]: false }), + ).rejects.toBeDefined(); + await expect( + f.approve({ + ...f.request(checked.reviewToken), + stateSha256: "f".repeat(64), + }), + ).rejects.toBeDefined(); + expect(f.query).not.toHaveBeenCalled(); + expect(f.current).not.toHaveBeenCalled(); +}); +it("fails closed when checkpoint observation is unavailable", async () => { + const f = fixture(); + await expect( + checkRecoveryReplan(f.client, f.session, f.bound.jobId, { + ...f.deps, + backends: { runner: f.runner }, + }), + ).rejects.toMatchObject({ + status: 503, + code: "application_recovery_replan_unavailable", + }); + expect(f.runner).not.toHaveBeenCalled(); +}); + +it("supports the original owner's reported failure without external operator attestations", async () => { + const f = fixture(); + f.bound.mode = "reported-failure"; + const checked = await f.check(); + expect(checked).toMatchObject({ + mode: "reported-failure", + runnerStop: "reported-command-completed", + cloudState: "requires-fresh-plan-review", + }); + const receipt = await f.approve({ + reviewToken: checked.reviewToken, + confirmFreshPlan: true, + }); + expect(receipt).toMatchObject({ + mode: "reported-failure", + runnerStop: "reported-command-completed", + cloudState: "requires-fresh-plan-review", + startsExecution: false, + requiresFreshPlan: true, + }); + expect(f.query.mock.calls.at(-1)?.[1].at(-1)).toBe(false); +}); +it.each(["confirmRunnerStopped", "confirmCloudStateCompared"])( + "requires %s in operator mode even when the optional flag is omitted", + async (flag) => { + const f = fixture(); + const checked = await f.check(); + const input: Record = f.request(checked.reviewToken); + delete input[flag]; + await expect(f.approve(input)).rejects.toMatchObject({ + code: "application_recovery_replan_changed", + }); + expect( + f.query.mock.calls.every( + ([sql]) => !sql.includes("record_application_recovery_replan"), + ), + ).toBe(true); + }, +); +it("decrypts only the SQL-authorized backend ciphertext and keeps it outside public results and proof plaintext", async () => { + const f = fixture(); + f.bound.mode = "reported-failure"; + const ciphertext = f.deps.crypto + .encrypt( + Buffer.from( + JSON.stringify({ accessKeyId: secret, secretAccessKey: secret }), + ), + f.session.tenantId, + f.bound.backendId, + `backend:${f.bound.backendId}`, + ) + .toString("hex"); + f.query.mockResolvedValueOnce({ + rows: [ + { + context: { + ...f.bound, + backendCiphertext: ciphertext, + registeredBackendDescriptor: { + ...f.bound.backendDescriptor, + key: "platform-state", + }, + }, + }, + ], + }); + const checked = await f.check(); + expect(f.runner).not.toHaveBeenCalled(); + expect(f.current.mock.calls[0]?.[0]).toMatchObject({ + descriptor: { key: f.bound.stateKey }, + credentials: { accessKeyId: secret, secretAccessKey: secret }, + }); + const plaintext = f.deps.crypto + .decrypt( + Buffer.from(checked.reviewToken, "base64"), + f.session.tenantId, + f.session.id, + `application-replan:${f.bound.jobId}`, + ) + .toString("utf8"); + for (const value of [ + secret, + ciphertext, + "backendCiphertext", + "registeredBackendDescriptor", + ]) { + expect(JSON.stringify(checked)).not.toContain(value); + expect(plaintext).not.toContain(value); + } +}); +it("rejects ciphertext encrypted for another backend before any S3 call", async () => { + const f = fixture(); + const ciphertext = f.deps.crypto + .encrypt( + Buffer.from( + JSON.stringify({ accessKeyId: secret, secretAccessKey: secret }), + ), + f.session.tenantId, + randomUUID(), + `backend:${f.bound.backendId}`, + ) + .toString("hex"); + f.query.mockResolvedValueOnce({ + rows: [{ context: { ...f.bound, backendCiphertext: ciphertext } }], + }); + await expect(f.check()).rejects.toBeDefined(); + expect(f.runner).not.toHaveBeenCalled(); + expect(f.current).not.toHaveBeenCalled(); +}); +it("rejects expiration occurring during the approval's S3 observations without recording a receipt", async () => { + const f = fixture(); + const checked = await f.check(); + vi.useFakeTimers(); + f.current.mockImplementationOnce(async () => { + vi.setSystemTime(new Date(Date.parse(checked.expiresAt) + 1)); + return f.snapshot(); + }); + await expect(f.approve(f.request(checked.reviewToken))).rejects.toMatchObject( + { code: "application_recovery_replan_changed" }, + ); + expect( + f.query.mock.calls.every( + ([sql]) => !sql.includes("record_application_recovery_replan"), + ), + ).toBe(true); +}); diff --git a/landing-zone-configurator/app/tests/organisation.test.ts b/landing-zone-configurator/app/tests/organisation.test.ts index 1bb2a14..14c8a66 100644 --- a/landing-zone-configurator/app/tests/organisation.test.ts +++ b/landing-zone-configurator/app/tests/organisation.test.ts @@ -300,6 +300,12 @@ function setup(dispatchEnabled = false, groupsEnabled = false) { retiredBy: session.userId, })), listInstances: vi.fn(async () => []), + checkRecoveryReplan: vi.fn(async (_session: Session, jobId: string) => ({ + jobId, + })), + approveRecoveryReplan: vi.fn( + async (_session: Session, jobId: string, _input: unknown) => ({ jobId }), + ), recoveryReviewContext: vi.fn(async (_session: Session, jobId: string) => ({ reconciliation: reconciliationFixture, jobId, @@ -648,6 +654,84 @@ it("requires explicit snapshot-bound recovery review with tenant, origin and CSR ); }); +it("protects checkpoint checks and replan clearance with tenant, origin, CSRF and strict requests", async () => { + const { app, headers, applications } = setup(); + const id = randomUUID(); + const url = `/api/v1/applications/jobs/${id}/recovery-replan`; + const payload = { reviewToken: "opaque-proof", confirmFreshPlan: true }; + for (const path of [url, `${url}/check`]) { + const body = path.endsWith("/check") ? {} : payload; + for (const rejected of [ + { ...headers, "x-lzc-tenant": randomUUID() }, + { ...headers, "x-lzc-csrf": "" }, + { ...headers, origin: "https://untrusted.example" }, + ]) + expect( + ( + await app.inject({ + method: "POST", + url: path, + headers: rejected, + payload: body, + }) + ).statusCode, + ).toBe(403); + expect( + ( + await app.inject({ + method: "POST", + url: path, + headers: {}, + payload: body, + }) + ).statusCode, + ).toBe(401); + expect((await app.inject({ url: path, headers })).statusCode).toBe(404); + } + for (const invalid of [ + { ...payload, confirmFreshPlan: false }, + { ...payload, stateSha256: "a".repeat(64) }, + { ...payload, reviewedBy: randomUUID() }, + { confirmFreshPlan: true }, + ]) + expect( + (await app.inject({ method: "POST", url, headers, payload: invalid })) + .statusCode, + ).toBe(400); + expect( + ( + await app.inject({ + method: "POST", + url: `${url}/check`, + headers, + payload: { backendId: randomUUID() }, + }) + ).statusCode, + ).toBe(400); + expect(applications.checkRecoveryReplan).not.toHaveBeenCalled(); + expect(applications.approveRecoveryReplan).not.toHaveBeenCalled(); + const check = await app.inject({ + method: "POST", + url: `${url}/check`, + headers, + payload: {}, + }); + expect(check.statusCode).toBe(200); + expect(check.headers["cache-control"]).toBe("no-store"); + expect(applications.checkRecoveryReplan).toHaveBeenCalledExactlyOnceWith( + session, + id, + ); + const approve = await app.inject({ method: "POST", url, headers, payload }); + expect(approve.statusCode).toBe(200); + expect(approve.headers["cache-control"]).toBe("no-store"); + expect(applications.approveRecoveryReplan).toHaveBeenCalledExactlyOnceWith( + session, + id, + payload, + ); +}); + it("protects application order decisions with current tenant, origin, CSRF and explicit confirmation", async () => { const { app, headers, applications } = setup(); const id = randomUUID(); diff --git a/landing-zone-configurator/docs/user-guide.de.md b/landing-zone-configurator/docs/user-guide.de.md index a508d45..988683b 100644 --- a/landing-zone-configurator/docs/user-guide.de.md +++ b/landing-zone-configurator/docs/user-guide.de.md @@ -122,6 +122,28 @@ fehlgeschlagene Phase; zur Ursachenklärung dient der OpenTofu-Fehler in den Log Die Logs erteilen keine Plan-/Apply-Berechtigung und werden keinem anderen Auftraggeber offengelegt. +Behalte nach einem regulär gemeldeten Application-Apply-Fehler (`apply_failed`) +die bestehende Bestellung. Wähle in deren Schritt **Apply** die Aktion +**Application-Checkpoint prüfen**. Der Server prüft den exakt zur Bestellung +gehörenden S3-State, eine mögliche Sperre, separate Recovery-Nachweise und die +Abschlussmeldung des Runners. Bestätige, dass du dieselbe Bestellung neu planen +möchtest, wechsle zu **Plan** und starte einen neuen Cloud-Plan. Das kann der +Application Owner ohne erneute Freigabe des Platform Owners und ohne Warten auf +den Ablauf der alten Freigabe durchführen. Bestehende Zugangsdaten sowie +ursprüngliche Quelle, Bestellung und State-Bindung bleiben erhalten. Diese +Aktion startet keinen Apply; der alte gespeicherte Plan kann nicht erneut +angewendet werden. + +Ein neuer Plan aktualisiert die Cloud-Ressourcen; prüfe seine vollständigen +Änderungen vor einem weiteren Apply. Nach einem Service-Timeout kann eine +asynchrone Cloud-Operation ungeklärt sein. Ein lesbarer Checkpoint beweist daher +allein keine vollständige Übereinstimmung von Cloud und State. Fehlende +Abschlussmeldungen, State-Schreibfehler, separate Recovery-States, Sperren oder +abweichende Checkpoints bleiben blockiert. Bei dafür geeigneten unklaren +Abschlüssen kann der ursprünglich freigebende Platform Owner nach Ablauf der +alten Ausführungsberechtigungen einen externen Runner-/Cloud-Abgleich gesondert +bestätigen. Das ist keine automatische Cloud-Prüfung oder State-Reparatur. + Wähle die richtige Umgebung und das richtige Ziel, binde die autorisierten technischen Zugangsdaten, erstelle eine Vorbereitung und fordere anschließend einen Plan an. Verfolge die Job-Ausgabe und prüfe die resultierenden Änderungen, Quell-/Konfigurationsversion diff --git a/landing-zone-configurator/docs/user-guide.md b/landing-zone-configurator/docs/user-guide.md index 356cd85..53c87c7 100644 --- a/landing-zone-configurator/docs/user-guide.md +++ b/landing-zone-configurator/docs/user-guide.md @@ -108,6 +108,25 @@ another execution. A code such as `plan_failed` identifies the failed phase; the OpenTofu error in the logs is needed to diagnose its cause. Logs do not grant Plan/Apply permission and are not shared with another requester. +After a normally reported Application Apply failure (`apply_failed`), keep the +existing order. In its **Apply** step, choose **Check Application checkpoint**. +The server verifies the exact order-bound S3 state, checks for a lock and separate +recovery evidence, and verifies the Runner's terminal report. Confirm that you +want to replan the same order, then switch to **Plan** and start a new Cloud Plan. +The Application Owner can do this without another Platform Owner approval and +without waiting for the old grant to expire. Existing credentials and the +original source, order and State binding are preserved. No Apply starts from +this action, and the old saved Plan cannot be reused. + +A new Plan refreshes cloud resources; review its complete changes before another +Apply. A service timeout can leave an asynchronous cloud operation unresolved, +so a readable checkpoint alone does not prove that every cloud resource matches +State. Missing final reports, State write failures, separate recovery State, +locks or mismatched checkpoints remain blocked. For eligible uncertain +completion, the originally approving Platform Owner can separately attest an +external Runner/cloud reconciliation after the old execution credentials expire; +this is not an automatic cloud verification or State repair. + Choose the correct environment and target, bind the authorized technical credentials, then create a preparation and request a Plan. Follow the job output and review the resulting changes, source/configuration version and state binding.