feat(temporal): add worker-side payload encryption

[daryllimyt]

Checklist

• Read CONTRIBUTING.md.
• PR title is short and non-generic (see previously merged PRs for examples).
• PR only implements a single feature or fixes a single bug.
• Tests passing (uv run pytest tests)?
• Lint / pre-commits passing (pre-commit run --all-files)?

Description

This PR adds worker-side Temporal payload encryption without relying on an external codec server. When enabled, the Temporal data converter composes compression with AES-GCM encryption, scopes keys by workspace, and keeps decode marker-driven so historical payloads remain readable across configuration changes.

Today's updates refine the rollout shape:

• Remove the Temporal codec server API route and router tests.
• Add versioned payload keyring support from environment JSON or AWS Secrets Manager, with cached keyring retrieval and memoized codec construction.
• Scope encryption configuration and keyring secret access to Temporal services in Docker Compose and Fargate.
• Harden history, agent memo, and child workflow memo decode paths so failures are logged and tolerated instead of breaking workflow inspection.
• Update unit coverage for keyring versions, codec caching, converter decode behavior, and memo decode failure handling.

Related Issues

N/A

Screenshots / Recordings

N/A. Backend, Temporal, and infrastructure change only.

Steps to QA

1. Run uv run pytest tests/unit/test_temporal_codec.py tests/unit/test_dsl_converter.py tests/temporal/test_workflow_timers.py.
2. Run uv run ruff check tracecat/temporal/codec.py tracecat/dsl/common.py tracecat/config.py tracecat/api/app.py tests/unit/test_temporal_codec.py tests/unit/test_dsl_converter.py tests/temporal/test_workflow_timers.py.
3. Run uv run basedpyright tracecat/temporal/codec.py tracecat/dsl/common.py tracecat/config.py tracecat/api/app.py.

---

Summary by cubic

Adds worker-side, fail-open AES‑GCM encryption for Temporal payloads with per-workspace keys and marker-driven decode; no codec server required. Off by default; enable with TEMPORAL__PAYLOAD_ENCRYPTION_ENABLED=true and provide a keyring via TEMPORAL__PAYLOAD_ENCRYPTION_KEYRING or TEMPORAL__PAYLOAD_ENCRYPTION_KEYRING_ARN.

• New Features

  • Composite codec: optional compression (zstd/gzip/brotli) then AES‑GCM; decode is marker‑driven and always tries both codecs so old payloads stay readable; fail‑open passthrough with minimal error logging; validates workspace scope and nonce; workspace derived from ctx_role.workspace_id (falls back to __global__).
  • Key management: async, versioned keyring from env JSON or AWS Secrets Manager (via asyncio.to_thread); cachetools TTL cache with configurable TTL/max items; memoized codec factory with test reset helpers.
  • Data converter/decoding: shared codec on all decode paths via decode_payloads; uses encoded‑attributes failure converter when encryption is enabled; agent/child memo parsing is async and tolerant.
  • Infra: Terraform/IAM policy for temporal_payload_encryption_keyring_arn; scope Temporal secrets to worker/agent tasks; Docker Compose exposes encryption env vars; new Fargate vars for cache TTL/max items; removed codec server endpoint.

• Bug Fixes

  • Historical decode reliability: always include both codecs on the decode path regardless of current config; removed init‑time compression algorithm validation and reject invalid algorithms on encode only.
  • Keyring robustness: wrap AWS fetch and binary keyring decode errors, preserve last‑known‑good cache on refresh failures, revalidate derived keys, back off failed refreshes, and add double‑checked locking for safe concurrent cache access.
  • Encode stability: derive the encryption key for encode from a single keyring snapshot to avoid mixed versions during refresh.
  • Workflow failures: decode encoded workflow failure messages and cause chains in history/compact views; switch to async failure parsing.
  • Unreadable payloads: surface a structured UnreadableTemporalPayload placeholder (includes encoding and size) in histories and compact views instead of raising; skip trigger context resolution for these; update client schemas/types to accept the placeholder.
  • Misc: fixed ARN handling in Alembic env secret retrieval; timer tests ignore cooperative yields; platform registry tests use UUID‑based origins.

^{Written for commit 7b894af. Summary will update on new commits.}

[daryllimyt]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat(temporal): add worker-side payload encryption #2297 👈 (View in Graphite)
• feat(temporal): encrypt durable agent histories #2296
• feat(temporal): encrypt core workflow histories #2295
• feat(temporal): add payload codec foundation #2294
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to 7b894af.

Security Overview

• 🔎 Scanned files: 31 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/00dffd6c-8834-4dc9-b6d8-b44cd1622986?scanId=41d89715-c6de-4209-99d9-171c595cc9e4&codeScanTypes=PrScan&tab=issues

Detected Code Changes

The diff is too large to display a summary of code changes.

[jordan-umusu]

LGTM with a few nits and two concerns:

1. K8s coming separately?
2. What does the rollout for this look like?

[daryllimyt]

K8s coming separately?

yes, internal PR

What does the rollout for this look like?

merge first with encryption disabled, then we enable it afterwards. this is safe because the codec can handle both encrypted and non-encrypted Payloads simultaneously

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c49c5348d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c760b22af

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
test_large_collection_regressions/test_scatter_gather_massive_payload_50x2mb_e2e	View Logs

[jordan-umusu]

LGTM once CI passing