diff --git a/docs/adr/121-policy-rename-clarity.md b/docs/adr/124-policy-rename-clarity.md similarity index 98% rename from docs/adr/121-policy-rename-clarity.md rename to docs/adr/124-policy-rename-clarity.md index 1f4f6482..1c8d077a 100644 --- a/docs/adr/121-policy-rename-clarity.md +++ b/docs/adr/124-policy-rename-clarity.md @@ -1,4 +1,4 @@ -# ADR-121: Rename IAM Policies for Clarity +# ADR-124: Rename IAM Policies for Clarity **Status:** Accepted **Date:** 2026-02-02 diff --git a/docs/adr/126-multi-region-independent-tables.md b/docs/adr/126-multi-region-independent-tables.md new file mode 100644 index 00000000..d1135d09 --- /dev/null +++ b/docs/adr/126-multi-region-independent-tables.md @@ -0,0 +1,58 @@ +# ADR-126: Multi-Region via Independent Regional Tables + +**Status:** Proposed +**Date:** 2026-02-14 + +## Context + +Users need to enforce rate limits across multiple AWS regions (e.g., an organization's +global RPM limit must be shared by clients in us-east-1 and eu-west-1). DynamoDB Global +Tables is the obvious candidate, but it has fundamental conflicts with zae-limiter's +write patterns: + +- **ADD counter loss:** Global Tables uses last-writer-wins at the item level. Concurrent + `ADD tk -consumed` from two regions results in one write overwriting the other, silently + losing consumption data and causing over-admission. +- **Transaction non-atomicity:** `TransactWriteItems` is ACID only in the originating + region. Cascade child+parent writes appear as partial updates in other regions. +- **Double refill:** Each region's aggregator processes its own stream. Replicated writes + appear in all streams, requiring filtering to avoid double-counting and double-refilling. + +The namespace feature (issue #376) already provides write isolation: each namespace has +its own partition key prefix, so records in different namespaces never collide. + +## Decision + +Multi-region must use **independent DynamoDB tables per region**, one per deployed stack. +Each region must use a dedicated namespace for its rate-limiting data. Cross-region +coordination must be handled by a periodic sync mechanism (see ADR-127, ADR-130), not by +DynamoDB replication. Global Tables must not be used for the rate-limiting table. + +## Consequences + +**Positive:** +- Write cost stays at 1x (no replicated WCU tax) +- All existing write patterns (speculative, optimistic lock, transactions) work unchanged +- Aggregator Lambda processes only local events, no stream filtering needed +- Each region is fully independent; one region's failure does not affect others + +**Negative:** +- No automatic data replication; regional data is lost if a region fails permanently +- Cross-region coordination requires a new sync component (ADR-127, ADR-130) +- Rate limiter state is ephemeral; region loss causes temporary over-admission until + sync catches up, bounded by one sync window + +## Alternatives Considered + +### DynamoDB Global Tables with namespace-per-region isolation +Rejected because: replicated WCUs double write cost, ADD operations lose data under +concurrent cross-region writes, and transactions are not atomic across regions. + +### DynamoDB Global Tables with counter sharding (per-region SET attributes) +Rejected because: requires reworking the composite bucket schema (ADR-114), breaks +speculative writes, and the 2x write cost is not justified when a sync mechanism is +needed regardless. + +### Centralized single-region table with cross-region API calls +Rejected because: adds 50-150ms latency to every acquire() call for remote-region +clients, creating a single point of failure with no local fallback. diff --git a/docs/adr/127-s3-sync-exchange.md b/docs/adr/127-s3-sync-exchange.md new file mode 100644 index 00000000..6d190974 --- /dev/null +++ b/docs/adr/127-s3-sync-exchange.md @@ -0,0 +1,59 @@ +# ADR-127: S3-Based Cross-Region Sync Exchange + +**Status:** Proposed +**Date:** 2026-02-14 + +## Context + +With independent DynamoDB tables per region (ADR-126), a sync mechanism must exchange +consumption data between regions. The exchange payload is a snapshot of all active +entities' bucket states: `total_consumed_milli`, `tokens_milli`, and `capacity_milli` +per entity, resource, and limit. + +Using DynamoDB as the exchange medium means writing one item per active (entity, resource) +pair per sync cycle. At 2,000 active entities with 10 resources and a 10-second sync +window, this costs ~$3,900/month in WCU alone — roughly 3x the entire acquire budget. + +The sync payload is a batch snapshot: all active entities' state at a point in time. +This is a bulk data transfer problem, not an item-level access problem. + +## Decision + +Each region's sync Lambda must write its consumption snapshot as a single S3 object +(JSON) to a shared sync bucket, keyed by `{region}/snapshot.json`. Remote regions must +read these objects via cross-region S3 GET. DynamoDB must not be used for publishing +sync reports. + +The snapshot must include, per active (entity, resource) pair: the per-limit +`total_consumed_milli` counter, the current `tokens_milli`, and the configured +`capacity_milli`. Snapshot objects must have a TTL (S3 lifecycle) of 5 minutes. + +## Consequences + +**Positive:** +- Publishing cost drops to ~$1.30/month regardless of entity count (1 S3 PUT per cycle) +- Reading cost is ~$0.10/month per remote region (1 S3 GET per cycle) +- Snapshot size is bounded: 2,000 entities x 10 resources x 60 bytes = ~1.2 MB per PUT +- S3 is highly available and durable; no capacity planning needed + +**Negative:** +- Introduces S3 dependency for cross-region coordination (new failure mode) +- S3 eventual consistency means a GET may return a slightly stale snapshot (~1s) +- Requires a shared S3 bucket accessible from all regions (cross-region GET latency + ~100ms, acceptable for background sync) +- Snapshot format must be versioned to handle schema evolution + +## Alternatives Considered + +### DynamoDB items for sync reports (one per entity per resource) +Rejected because: WCU cost scales linearly with entity count, reaching $3,900/month +at 2,000 active entities with 10-second sync — 3x the acquire budget. + +### DynamoDB items for sync reports (one batch item per resource) +Rejected because: 400KB item size limit caps at ~4,000 entities per item, and large +item writes consume proportionally more WCUs, offering no cost advantage over +individual items. + +### SQS/SNS for event-driven sync +Rejected because: requires per-event cross-region message delivery, adding complexity +and cost proportional to acquire volume rather than sync frequency. diff --git a/docs/adr/128-quota-enforcement-via-config.md b/docs/adr/128-quota-enforcement-via-config.md new file mode 100644 index 00000000..0254597a --- /dev/null +++ b/docs/adr/128-quota-enforcement-via-config.md @@ -0,0 +1,63 @@ +# ADR-128: Quota Enforcement via Entity Config Overrides + +**Status:** Proposed +**Date:** 2026-02-14 + +## Context + +With independent tables per region (ADR-126) and S3-based sync (ADR-127), each region's +sync Lambda computes a regional quota for each entity. This quota must be enforced by +the rate limiter's hot path without modifying the acquire flow. + +Three enforcement mechanisms were evaluated: + +1. **Entity config overrides:** Write adjusted limits via `set_limits()`, picked up by + the existing config cache on the next resolve. +2. **Shadow counter on bucket item:** Write a `remote_tc` attribute on the bucket and + add a condition to speculative writes. +3. **Direct token deduction:** `ADD b_rpm_tk -remote_delta` on the bucket item. + +The shadow counter approach has a semantic mismatch: `total_consumed_milli` is a lifetime +monotonic counter incompatible with the per-window token bucket model. Direct token +deduction does not adjust the refill ceiling — each region's bucket refills at the full +global rate, causing N regions to provide Nx the intended refill. + +## Decision + +Regional quotas must be enforced by writing entity-level config overrides via the +existing `set_limits(entity_id, resource, limits)` API. The sync Lambda must compute +the allocated capacity per entity and write it as an entity config record. The rate +limiter's existing config resolution hierarchy (Entity > Resource > System) must be +the sole mechanism for quota enforcement. + +## Consequences + +**Positive:** +- Zero changes to the acquire hot path (speculative writes, optimistic lock, bucket math) +- Uses the existing config hierarchy; no new DynamoDB schema or access patterns +- Capacity adjustment naturally controls refill ceiling via token bucket math +- Config cache TTL provides built-in staleness tolerance (already accepted in ADR-105) + +**Negative:** +- Token drain lag: if current tokens exceed the new reduced capacity, the entity can + consume excess tokens until they drain naturally (bounded by consumption rate) +- Config writes are the dominant sync cost (~$40/month at 2,000 active entities with + trigger-based filtering per ADR-129) +- Config cache TTL (default 60s) delays quota enforcement after a config write; the + sync Lambda and application use separate Repository instances + +## Alternatives Considered + +### Shadow counter attribute on bucket item (remote_tc) +Rejected because: `total_consumed_milli` is a monotonic lifetime counter that cannot +be compared against a per-window capacity limit, and modifying the speculative write +condition changes the hot path for all users. + +### Direct token deduction (ADD tk -remote_delta) +Rejected because: each region's bucket still refills at the full global rate, so N +regions produce Nx total refill — the deduction fights the bucket math without +correcting the underlying refill ceiling. + +### In-memory client-side consumption map (no DynamoDB writes) +Rejected because: requires background polling threads and in-memory state, which works +for long-running services but not for Lambda-based rate limiting. diff --git a/docs/adr/129-trigger-based-sync-writes.md b/docs/adr/129-trigger-based-sync-writes.md new file mode 100644 index 00000000..89f1afb3 --- /dev/null +++ b/docs/adr/129-trigger-based-sync-writes.md @@ -0,0 +1,61 @@ +# ADR-129: Trigger-Based Sync Config Writes + +**Status:** Proposed +**Date:** 2026-02-14 + +## Context + +With quota enforcement via entity config overrides (ADR-128), the sync Lambda writes +a `set_limits()` call for every active (entity, resource) pair each cycle. At 2,000 +active entities with 10 resources and a 10-second sync window, this produces 20,000 +WCU per cycle — ~$486/month. Most of these writes are wasted: 80% of entities are well +under their limits with stable allocations. + +The sync window already defines the worst-case over-admission bound (entities can +double-consume for one sync window regardless of write frequency). Config writes do +not improve the worst case; they only tighten steady-state accuracy. + +## Decision + +The sync Lambda must only write entity config overrides when one of two triggers fires: + +1. **Exhaustion trigger:** The entity's projected time-to-exhaustion (remaining tokens + divided by recent consumption rate) is less than twice the sync window. This prevents + the entity from running out of tokens before the next sync cycle can react. + +2. **Drift trigger:** The computed allocation differs from the currently configured + capacity by more than 15%. This corrects stale quotas for entities whose traffic + pattern has shifted significantly. + +All other entities must be skipped (no config write). Trigger evaluation must be +computed from S3 snapshot data (ADR-127) without additional DynamoDB reads. + +## Consequences + +**Positive:** +- Config writes drop from ~20,000 to ~600 per cycle at steady state (~97% reduction) +- Monthly sync cost drops from ~$486 to ~$40 at 2,000 active entities +- DynamoDB write throughput spikes are smoothed (fewer concurrent writes) +- Worst-case over-admission is unchanged (bounded by sync window, not write frequency) + +**Negative:** +- Entities with slowly drifting traffic (<15% per cycle) may have stale quotas for + multiple sync cycles before the drift threshold triggers +- Exhaustion prediction depends on consumption rate estimation, which may be noisy for + bursty workloads +- Two tunable parameters (exhaustion horizon = 2x sync window, drift threshold = 15%) + require validation under production traffic patterns + +## Alternatives Considered + +### Write every entity every cycle (no filtering) +Rejected because: 97% of writes are redundant, costing ~$450/month in unnecessary WCU +without improving the over-admission bound set by the sync window. + +### Write only on exhaustion (drop drift trigger) +Rejected because: entities with shifting traffic patterns would keep stale allocations +indefinitely, wasting regional quota until they approach exhaustion. + +### Event-driven writes via DynamoDB Streams (write on every bucket change) +Rejected because: couples sync frequency to acquire volume rather than a fixed window, +producing more writes than periodic polling for high-throughput entities. diff --git a/docs/adr/130-per-region-sync-lambda.md b/docs/adr/130-per-region-sync-lambda.md new file mode 100644 index 00000000..fe6ba184 --- /dev/null +++ b/docs/adr/130-per-region-sync-lambda.md @@ -0,0 +1,56 @@ +# ADR-130: Per-Region Sync Lambda + +**Status:** Proposed +**Date:** 2026-02-14 + +## Context + +Cross-region sync (ADR-126, ADR-127) requires a Lambda function that reads consumption +snapshots, computes quota allocations, and writes config overrides. Two topologies were +evaluated: + +- **Single coordinator:** One Lambda in a designated region reads all snapshots, computes + all quotas, and writes configs to all regions. Total cross-region calls: `2(N-1)` per + cycle (reads + writes). Single point of failure. +- **Per-region:** Each region runs its own Lambda that reads remote snapshots, computes + its own quota locally, and writes only to its local table. Total cross-region calls + per Lambda: `N-1` reads, 0 writes. + +Both topologies produce the same quota allocation when given the same inputs. The +per-region Lambda runs a deterministic function: given the same S3 snapshots, every +region independently computes the same allocation. No distributed consensus is needed. + +## Decision + +Each region must run its own sync Lambda, triggered by EventBridge on a fixed schedule +(configurable sync window). Each Lambda must read its local bucket states, write its +snapshot to S3 (ADR-127), read all remote snapshots from S3, compute quotas using a +deterministic allocation function, and write triggered config overrides (ADR-129) to +its local DynamoDB table only. No Lambda may write to a remote region's DynamoDB table. + +## Consequences + +**Positive:** +- Symmetric architecture: every region deploys the same CloudFormation stack +- No single point of failure: one region's Lambda failure does not affect other regions +- Zero cross-region DynamoDB writes (only cross-region S3 reads, ~100ms latency) +- Scales naturally: adding a region means deploying the same stack, no coordinator changes +- Each region can independently tune its sync window + +**Negative:** +- N Lambdas compute the same allocation independently (redundant CPU, negligible cost) +- Slight snapshot staleness between Lambdas reading at different moments within a cycle + (sub-second divergence, converges on next cycle) +- More infrastructure per region (EventBridge rule + Lambda + IAM), though identical + across regions and part of the standard stack deployment + +## Alternatives Considered + +### Single coordinator Lambda in a designated region +Rejected because: introduces an asymmetric "special" region, creates a single point of +failure for all global quota allocation, and requires cross-region DynamoDB writes for +config overrides in remote regions. + +### Peer-to-peer gossip between regional Lambdas +Rejected because: adds network coordination complexity (discovery, message ordering) +without improving on the deterministic-computation-from-shared-S3 approach. diff --git a/docs/adr/131-sync-config-ttl.md b/docs/adr/131-sync-config-ttl.md new file mode 100644 index 00000000..cc813896 --- /dev/null +++ b/docs/adr/131-sync-config-ttl.md @@ -0,0 +1,66 @@ +# ADR-131: TTL on Sync-Written Entity Config Records + +**Status:** Proposed +**Date:** 2026-02-14 + +## Context + +The sync Lambda enforces regional quotas by writing entity-level config overrides via +`set_limits()` (ADR-128). Per ADR-119, buckets using entity custom limits persist +indefinitely (no TTL), while buckets using resource/system defaults have TTL and +auto-expire. + +When the sync Lambda writes an entity config, the bucket transitions from +"default-limit" (has TTL) to "custom-limit" (no TTL). If the entity later goes idle +and the sync Lambda stops writing configs (no trigger fires per ADR-129), both the +config record and its bucket persist indefinitely. For high-churn entity populations +(anonymous users, ephemeral API keys), this causes unbounded storage growth. + +The fix must not affect operator-written entity configs, which intentionally persist +indefinitely per ADR-119. + +## Decision + +Sync-written entity config records must include a DynamoDB TTL attribute set to +`now + 3 × sync_window`. The sync Lambda must refresh the TTL on each config write. + +This extends ADR-119's bucket TTL rule. The updated bucket TTL logic is: + +- Bucket has **no TTL** if: entity config exists **without** a `ttl` attribute + (operator-written, persists indefinitely — unchanged from ADR-119) +- Bucket **has TTL** if: entity config exists **with** a `ttl` attribute + (sync-written, treated as default-like for TTL calculation) +- Bucket **has TTL** if: no entity config exists + (using resource/system defaults — unchanged from ADR-119) + +When the entity goes idle (no trigger fires for 3 sync windows), the config record +auto-expires via DynamoDB TTL. The entity reverts to resource/system defaults, and +bucket TTL behavior per ADR-119 resumes. + +## Consequences + +**Positive:** +- Idle entities auto-cleanup: sync config expires, bucket regains TTL, storage bounded +- No new attributes needed: the DynamoDB `ttl` attribute already exists in the table schema +- Self-healing: if a sync Lambda fails permanently, all its configs expire within 3 windows + +**Negative:** +- Bucket TTL logic (ADR-119) must check whether the entity config has a `ttl` attribute + to distinguish sync-written from operator-written configs +- Config records gain a new write pattern: conditional refresh of TTL alongside limits +- DynamoDB TTL deletion is asynchronous (up to 48 hours), so expired configs may linger + in scans; queries using strong conditions are unaffected + +## Alternatives Considered + +### Explicit cleanup pass in the sync Lambda (delete stale configs) +Rejected because: requires the sync Lambda to maintain a "previously synced" entity set +across invocations, adding state management complexity to a stateless Lambda function. + +### Separate DynamoDB sort key for sync configs (#SYNC_CONFIG#{resource}) +Rejected because: adds a new config level to the resolution hierarchy (ADR-118), breaking +the existing four-level precedence model and requiring changes to `resolve_limits()`. + +### No TTL on sync configs (rely on operator cleanup) +Rejected because: operators should not need to manually clean up configs created by an +automated sync process, especially for ephemeral entities at scale. diff --git a/docs/adr/132-sync-config-ownership.md b/docs/adr/132-sync-config-ownership.md new file mode 100644 index 00000000..452b7a17 --- /dev/null +++ b/docs/adr/132-sync-config-ownership.md @@ -0,0 +1,64 @@ +# ADR-132: Sync Config Ownership via TTL Presence + +**Status:** Proposed +**Date:** 2026-02-14 + +## Context + +Both operators and the sync Lambda write entity-level config records to the same +DynamoDB item (`PK=ENTITY#{id}, SK=#CONFIG#{resource}`) via `set_limits()`. Without +an ownership mechanism, the sync Lambda overwrites operator-set limits on its next +cycle, and operators overwrite sync-computed quotas on manual updates. + +The sync Lambda writes configs with a TTL attribute (ADR-131). Operator-written configs +have no TTL (they persist indefinitely per ADR-119). This difference in TTL presence +is a natural discriminator for config ownership. + +## Decision + +The sync Lambda must only write to an entity config record when the record is absent or +the existing record has a `ttl` attribute (indicating it was previously sync-written). +The sync Lambda's write must use the condition +`attribute_not_exists(PK) OR attribute_exists(ttl)`. This ownership check must be +implemented in the Repository layer (not in RateLimiter or the sync Lambda itself), +consistent with ADR-122's requirement that data access logic lives in the repository. + +Operator-written configs (no `ttl` attribute) must never be overwritten by the sync +Lambda. When an operator writes entity config via `set_limits()`, the record must not +include a `ttl` attribute, signaling operator ownership. The sync Lambda must skip +that entity for all subsequent cycles. + +To return an entity to sync-managed quotas, the operator must delete the entity config +via `delete_limits()`. The sync Lambda will then recreate it with TTL on the next +triggered cycle (ADR-129). + +## Consequences + +**Positive:** +- Operator configs always win: manual overrides are never clobbered by automated sync +- No new attributes: TTL presence is a sufficient ownership discriminator +- Reversible: `delete_limits()` returns the entity to sync management +- Condition check costs 0 extra RCU (evaluated server-side in the UpdateItem condition) + +**Negative:** +- Operators must delete (not overwrite) entity configs to return to sync management; + overwriting with `set_limits()` produces a record without TTL, taking operator ownership +- The sync Lambda's `ConditionalCheckFailedException` for operator-owned entities is + silent (expected), but increases CloudWatch error metrics unless filtered +- If an operator accidentally creates entity config, the entity silently leaves sync + management with no warning; observability tooling must surface this + +## Alternatives Considered + +### Explicit `origin` attribute ("sync" vs "operator") on config records +Rejected because: adds a new attribute that must be threaded through all config read/write +paths, requires migration for existing records, and provides no benefit over the TTL +presence check that ADR-131 already establishes. + +### Sync Lambda always wins (overwrite operator configs) +Rejected because: operators set entity limits for business reasons (premium tiers, custom +SLAs); automated sync should not override intentional business decisions. + +### Separate config level for sync (Entity > Sync > Resource) +Rejected because: breaks the four-level config hierarchy (ADR-118) and requires changes +to `resolve_limits()` in every backend implementation.