CloudLedger

An audit-grade, event-sourced financial ledger API on AWS — the primitive layer payment systems are built on.

What it is

A financial ledger must be tamper-proof, auditable, and correct under concurrent writes. CloudLedger solves this with an append-only Aurora event store — balances are never updated in place; every deposit, withdrawal, and transfer is an immutable fact (MoneyDeposited, TransferDebited, …) replayed on read, so every state an account was ever in is reconstructable. Concurrent writes are coordinated by an optimistic-lock fence (WHERE version = :expected) that surfaces contention as a 409 Conflict and a retry — never silent data loss.

On right-sized dev infrastructure it sustains p99 < 200 ms up to ~150 transfers/s before the tail knee, with reads served sub-10 ms from a Redis write-through cache (DynamoDB fallback). Full latency curve and load evidence: ADR-004. All infrastructure is Terraform — S3 backend + native lock, per-service KMS CMKs, local/prod environments.

Architecture

Commands append events to Aurora (atomic with the optimistic-lock fence) → the new balance is write-through to Redis → events publish to SQS post-commit (transactional-outbox fallback if SQS is down) → a Lambda projector builds the DynamoDB read model. Reads hit Redis (sub-ms) with DynamoDB fallback; Aurora is never on the read path. Collector-less OpenTelemetry → X-Ray traces every hop, propagated across the SQS boundary.

Aurora + Redis in private subnets; ECS in public subnets reaching AWS via VPC endpoints (no NAT).

Demo

Recorded on live AWS (a temporary deploy, since torn down for cost). Videos in progress — links coming soon:

• ▶️ Concurrent conflict — two simultaneous transfers → one 409 → retry succeeds, no double-spend
• ▶️ Projection rebuild — delete the DynamoDB read model, watch it reconstruct 0→N from the Aurora event log
• ▶️ Breakpoint under load — the CloudWatch dashboard saturating as k6 climbs past the knee

Transfer latency vs. arrival rate (prod, 2 vCPU) — flat ~87 ms to 150 rps, p99 knee ~200 rps, collapse ~300 rps (ADR-004).

Demo scenarios

Assumes the local stack from Local development is already up (docker compose up -d + local-bootstrap.sh + set variables, API healthy on http://localhost). Get a token and open an account once, then run any of the three scenarios below against it:

TOKEN=$(curl -s -X POST "$TOKEN_URL" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&scope=https://api.getcloudledger.com/write%20https://api.getcloudledger.com/read" \
  | jq -r '.access_token')

ACCOUNT_ID=$(uuidgen)
curl -s -X POST http://localhost/v1/accounts -H "Authorization: Bearer $TOKEN" \
  -H "Idempotency-Key: $(uuidgen)" -H "Content-Type: application/json" \
  -d "{\"accountId\":\"$ACCOUNT_ID\",\"currency\":\"USD\"}"

1. Idempotency — replay a write with the same Idempotency-Key → the original 201, no second event:

KEY=$(uuidgen)
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost/v1/accounts/$ACCOUNT_ID/deposits \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $KEY" -H "Content-Type: application/json" \
  -d '{"amount":100}'
# replay with the SAME key → identical 201, exactly one event in Aurora
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost/v1/accounts/$ACCOUNT_ID/deposits \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $KEY" -H "Content-Type: application/json" \
  -d '{"amount":100}'

2. Concurrent conflict — two same-version writes race on the same account → one 201, one 409 {"error":"version_conflict"} → re-read + retry succeeds. Money neither lost nor created. Plain curl & backgrounding races the two requests but doesn't guarantee they land on the exact same in-memory version — for a real, repeatable proof of the 409 (fired at true concurrency via http.batch) run k6/scenarios/conflict.js instead:

curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost/v1/accounts/$ACCOUNT_ID/withdrawals \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" -H "Content-Type: application/json" \
  -d '{"amount":10}' &
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost/v1/accounts/$ACCOUNT_ID/withdrawals \
  -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" -H "Content-Type: application/json" \
  -d '{"amount":10}' &
wait

3. CQRS projection — the command returns on Aurora commit + Redis write-through (read-your-writes immediately); the projector catches DynamoDB up within ~seconds. GET /transactions reflects it once projected — no Aurora replay on the read path:

curl -s http://localhost/v1/accounts/$ACCOUNT_ID/balance -H "Authorization: Bearer $TOKEN" | jq       # Redis write-through — fresh immediately
curl -s http://localhost/v1/accounts/$ACCOUNT_ID/transactions -H "Authorization: Bearer $TOKEN" | jq  # DynamoDB TXNS# — populated once the projector catches up

Tech stack

Layer	Technology
API language / framework	Java 21 · Spring Boot 4.1 (hexagonal ports & adapters)
Lambda language	Python 3.12 (uv-managed) — outbox-poller, projector, cleanup
Event store	Aurora PostgreSQL — ACID: the optimistic-lock fence + event insert commit in one txn
Async fan-out	SQS-first (KMS-encrypted, DLQ) + transactional-outbox fallback → Python projector λ
Read model	DynamoDB single-table + GSI · ElastiCache Redis (write-through, 30-min TTL, circuit breaker)
Auth	Cognito M2M (Client Credentials JWT); a separate api/admin scope for operator endpoints
IaC	Terraform — S3 backend + native lock, per-service KMS CMKs, local/prod envs
Observability	Collector-less OpenTelemetry (ADOT, SigV4) → X-Ray Transaction Search · CloudWatch dashboard
Load & verification	k6 — smoke/conflict CI gates + load/read-balance/breakpoint sandbox runs
CI/CD	GitHub Actions → ECR → rolling ECS deploy → k6 smoke + conflict gates
Local dev	Docker Compose + Floci (LocalStack-compatible) · Testcontainers · pytest e2e

Key engineering decisions

• Optimistic locking via a DB unique constraint — not SERIALIZABLE, not row locks — a version race loses atomically on UNIQUE (aggregate_id, version) and surfaces as a retryable 409 instead of opaque serialization retries or hot-account queuing; proven under a real race by the k6 conflict gate (ADR-001)
• SQS-first publish, transactional outbox demoted to fallback — the classic outbox makes every event pay the relay hop; publishing post-commit and writing outbox rows only when SQS fails keeps projection lag at publish latency while the event store remains the recovery path (ADR-002)
• Command-path Redis write-through over cache-aside — the write path already holds the authoritative balance at commit, so writing the snapshot through gives read-your-writes without waiting on the async projector (ADR-003)
• Right-size before tuning — an X-Ray trace showed 2 ms queries inside a 3.5 s p99: contention (10-connection pool on 0.5 vCPU), not slow work; 4× CPU + a fixed pool of 20 tripled the knee, with virtual threads deliberately held as a separate experiment (ADR-004)
• Collector-less OpenTelemetry → X-Ray — ADOT SigV4-signs OTLP itself, so no collector sidecar runs anywhere; trace context is persisted on outbox rows so the projector joins the original trace across the SQS hop (ADR-005)

For the longer-form walkthrough — how these five decisions connect, including the trace that led from ADR-005 to the ADR-004 performance fix — see docs/architecture-narrative.md.

Local development

Prerequisites: Java 21 · Docker · Terraform · Python 3.12 + uv · k6

git clone https://github.com/rafaesc/cloud-ledger && cd cloud-ledger
docker compose up -d                        # Floci: SQS, DynamoDB, Lambda, RDS, Cognito
bash terraform/scripts/local-bootstrap.sh   # apply Terraform + migrations + build/push + boot API + Lambda images

# Pull M2M creds from the local Terraform state (-chdir works from the repo root)
export CLIENT_ID=$(terraform -chdir=terraform/envs/local output -raw cognito_client_id)
export CLIENT_SECRET=$(terraform -chdir=terraform/envs/local output -json cognito_client_secret | jq -r .)
export BASE_URL=http://localhost            # Floci ALB on :80 — not the .elb.localhost output
export TOKEN_URL="http://localhost:4566/cognito-idp/oauth2/token"

cd api && ./gradlew test                    # unit + Testcontainers
cd lambdas && uv run pytest                 # Lambda tests (moto)
cd e2e && uv run pytest -v                  # full-stack e2e against the running API (http://localhost)

local-bootstrap.sh also builds and pushes the API image, and Floci runs it as an ECS task behind an ALB — no separate ./gradlew bootRun needed. The API is live at http://localhost (port 80, not :8080); confirm with curl http://localhost/actuator/health, the same check e2e.yml polls before running the e2e suite against it.

Running the k6 suite

Requires the stack from the steps above already running (docker compose up -d + local-bootstrap.sh) — CLIENT_ID/CLIENT_SECRET come from the Cognito pool Terraform just created, so nothing here works standalone:

curl -s -o /dev/null -w '%{http_code}\n' http://localhost/actuator/health   # expect 200 before continuing
BASE_URL=$BASE_URL PROJECTION_LAG_MS=60000 k6 run k6/scenarios/smoke.js     # CI gate: 1 VU x 10 transfers E2E
BASE_URL=$BASE_URL PROJECTION_LAG_MS=60000 k6 run k6/scenarios/conflict.js  # correctness demo: one 201 + N-1 409s, no double-spend

More details: k6/README.md.