Skip to content

Commit 82f3d51

Browse files
authored
Merge pull request #12 from 0-sayed/feat/improvement
feat: scale improvements — O(1) reporting, Redis idempotency cache, centi-cent accrual
2 parents 253c77b + f6d8ee4 commit 82f3d51

21 files changed

Lines changed: 1806 additions & 259 deletions

docs/adr/ADR-003-integer-cents.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,7 @@ The only special case is royalty splits where division produces remainders (e.g.
2424
- All monetary values are integers in the codebase — no decimal types anywhere
2525
- API accepts and returns cent values (documented in API contract)
2626
- Division for royalty splits requires explicit rounding strategy (see ADR-005)
27+
28+
## Addendum (2026-04-12)
29+
30+
Centi-cents (1/100 of a cent, stored as `wallets.fractional_balance`) extend this principle for sub-cent royalty accrual. See ADR-010.

docs/adr/ADR-005-platform-royalty-remainder.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
# ADR-005: Platform Receives Royalty Remainder
22

3-
**Status:** Accepted
3+
**Status:** Superseded by ADR-010
44
**Date:** 2026-02-26
55

6+
> **Superseded:** This ADR describes the original floor-only rounding strategy. It has been replaced by ADR-010 (centi-cent accrual), which eliminates the systematic under-payment of authors. The platform still receives the remainder after sweep, but over time the amounts converge to the nominal 30%.
7+
68
## Context
79

810
Royalty splits (70% author, 30% platform) applied to integer cent values can produce remainders. For example: 70% of 99 cents = 69.3 cents — not a valid integer. A rounding strategy must be defined.
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# ADR-008: Running Totals for Reporting
2+
3+
**Status:** Accepted
4+
**Date:** 2026-04-12
5+
6+
## Context
7+
8+
`ReportsProcessor.aggregate()` ran a `GROUP BY` over the entire `ledger` table inside a `REPEATABLE READ` transaction. At 100M+ rows this causes a full table scan, consumes memory proportional to result size, and holds a long-running transaction on the operational database.
9+
10+
## Decision
11+
12+
A new `ledger_totals` table maintains pre-aggregated running sums per ledger type:
13+
14+
```sql
15+
type ledger_type PRIMARY KEY
16+
total BIGINT NOT NULL DEFAULT 0
17+
```
18+
19+
The table is updated inside the same transaction as every ledger insert using `INSERT ... ON CONFLICT DO UPDATE`. `ReportsProcessor` replaces the `GROUP BY` with a single `SELECT * FROM ledger_totals` (4 rows, no scan).
20+
21+
## Reasoning
22+
23+
- The running total is always consistent with the ledger because it is updated in the same transaction. No eventual consistency.
24+
- O(1) read path replaces O(N) scan.
25+
- `BIGINT` is chosen because the sum of 100M+ cent-denominated values can exceed INT4's ~2.1 billion max.
26+
- Upsert (`INSERT ... ON CONFLICT DO UPDATE`) is used rather than bare `UPDATE` to handle fresh databases where the row may not yet exist.
27+
28+
## Consequences
29+
30+
- Reporting is O(1) regardless of ledger size.
31+
- Write path has a small additional upsert per transaction (negligible vs. ledger insert cost).
32+
- Migration includes a backfill from existing ledger data.
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# ADR-009: Redis Idempotency Cache
2+
3+
**Status:** Accepted
4+
**Date:** 2026-04-12
5+
6+
## Context
7+
8+
Every purchase request hit Postgres with a `SELECT WHERE idempotency_key = ?` before the transaction. Under high retry volume this is unnecessary latency and DB load for requests that already have a known result.
9+
10+
## Decision
11+
12+
Redis SETNX edge cache. Before the DB check:
13+
14+
```
15+
SET idempotency:<key> 'processing' EX 86400 NX
16+
```
17+
18+
- If NX returns `null` (key exists): read the cached value and return or `409`.
19+
- If NX returns `'OK'` (new request): proceed to DB.
20+
- Cache the result after commit.
21+
- Delete sentinel on transaction failure so clients can retry.
22+
23+
## Reasoning
24+
25+
- Redis is a latency shortcut — the DB remains the correctness backstop. The `UNIQUE` constraint on `purchases.idempotency_key` and the existing DB SELECT are untouched.
26+
- Any request that slips past Redis (eviction, cold start, Redis unavailable) falls through to the DB path.
27+
- The `'processing'` sentinel must be deleted on transaction failure — otherwise recoverable failures (e.g., 402 insufficient funds) would block client retries for 24 hours.
28+
- TTL of 86400s (24 hours) covers any reasonable client retry window.
29+
30+
## Consequences
31+
32+
- Repeated requests with a completed purchase are served from Redis without hitting Postgres.
33+
- Redis unavailability degrades gracefully (falls through to DB).
34+
- A separate `RedisModule` is required because BullMQ does not expose its internal ioredis instance via NestJS DI.
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# ADR-010: Centi-cent Fractional Accrual
2+
3+
**Status:** Accepted
4+
**Date:** 2026-04-12
5+
**Supersedes:** ADR-005
6+
7+
## Context
8+
9+
`Math.floor(price * 70 / 100)` always floors the author royalty, giving every sub-cent remainder to the platform. Over millions of microtransactions this is a systematic wealth transfer away from authors.
10+
11+
Example over three 99-cent transactions: author should receive 3 × 69.3 = 207.9 cents but actually receives 3 × 69 = 207 cents; platform over-receives ~1 cent per 3 transactions.
12+
13+
## Decision
14+
15+
Per-author centi-cent accrual. Track each author wallet's fractional royalty remainder at centi-cent precision (1/100 of a cent, stored as integer 0–99 in `wallets.fractional_balance`). When accumulated centi-cents reach 100, sweep one whole cent into the author's balance within the same transaction.
16+
17+
```typescript
18+
const exactNumerator = itemPrice * AUTHOR_ROYALTY_PERCENT; // e.g. 99*70 = 6930
19+
const authorFloorCents = Math.floor(exactNumerator / 100); // 69
20+
const remainderCentiCents = exactNumerator % 100; // 30
21+
const newFractional = authorWallet.fractionalBalance + remainderCentiCents;
22+
const sweepCents = Math.floor(newFractional / 100); // whole cents to sweep
23+
const leftoverCenti = newFractional % 100; // stored back
24+
const totalAuthorCents = authorFloorCents + sweepCents; // actual credit
25+
const platformCut = itemPrice - totalAuthorCents; // always = itemPrice
26+
```
27+
28+
## Reasoning
29+
30+
- Money conservation holds — `totalAuthorCents + platformCut === itemPrice` always.
31+
- Stays within ADR-003's integer-only philosophy — centi-cents are a finer-grained integer unit.
32+
- The `SELECT FOR UPDATE` on the author wallet row means `fractionalBalance` can be set directly (no SQL arithmetic needed — we hold the lock).
33+
- No background job required.
34+
35+
## Consequences
36+
37+
- Author wallets gain a `fractional_balance` column (0–99).
38+
- Authors receive the correct economic share over time rather than systematically losing sub-cent remainders.
39+
- Platform receives slightly less on average (closer to the nominal 30%).
40+
- Ledger entries record `totalAuthorCents` and `platformCut` (actual money moved, not nominal percentages).

docs/architecture.md

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ Wallet is a digital wallet API that manages integer-cent balances, purchase tran
1010
graph LR
1111
Client -->|HTTP| API[NestJS API :3000]
1212
API -->|Drizzle ORM| PG[(PostgreSQL 16)]
13-
API -->|BullMQ| Redis[(Redis 7)]
13+
API -->|ioredis - idempotency cache| Redis[(Redis 7)]
14+
API -->|BullMQ| Redis
1415
Redis -->|Job| Worker[Report Processor]
1516
Worker -->|Drizzle ORM| PG
1617
```
@@ -84,6 +85,13 @@ erDiagram
8485
jsonb result
8586
timestamp completed_at
8687
}
88+
89+
ledger_totals {
90+
enum type PK "ledger_type"
91+
bigint total "running sum"
92+
}
93+
94+
ledger ||--o{ ledger_totals : "aggregated into"
8795
```
8896

8997
## Concurrency & Transactions
@@ -97,17 +105,26 @@ Each deposit runs inside a transaction that locks the wallet row with `SELECT FO
97105
A purchase involves three wallets: buyer, author, and platform. All three are locked in a single query ordered by `id ASC` (`FOR UPDATE`) to enforce consistent lock acquisition and prevent deadlocks. Within the same transaction:
98106

99107
1. Buyer balance is decremented by the item price
100-
2. Author receives `floor(price * 70 / 100)` (author royalty)
101-
3. Platform receives the remainder (`price - authorCut`)
108+
2. Author receives floor royalty plus any accrued centi-cents that crossed 100 (see ADR-010)
109+
3. Platform receives the remainder (`price - totalAuthorCents`)
102110
4. A purchase record and three ledger entries are inserted
103111

104112
If PostgreSQL detects a deadlock (`40P01`), the service catches it and returns `409 Conflict` with a retry hint.
105113

106114
### Idempotency
107115

108-
Each purchase carries a client-owned `Idempotency-Key` header (UUID). Before entering the transaction, the service checks for an existing purchase with that key:
116+
Each purchase carries a client-owned `Idempotency-Key` header (UUID). Before entering the transaction, the service checks a Redis edge cache then the database:
117+
118+
**Redis SETNX (fast path):** `SET idempotency:<key> 'processing' EX 86400 NX`
119+
120+
- NX returns `null` (key exists): read cached value — return completed purchase or `409`
121+
- NX returns `'OK'` (new request): proceed to DB check below
122+
- Redis unavailable: log warning, fall through to DB (degraded but not broken)
123+
- Transaction failure: sentinel is deleted so clients can retry
124+
125+
**DB check (cold-start / Redis miss):**
109126

110-
- **Completed + same payload**returns the cached result (safe replay)
127+
- **Completed + same payload**populate Redis, return cached result (safe replay)
111128
- **Completed + different payload**`409 Conflict` (payload drift)
112129
- **Still in flight**`409 Conflict`
113130

@@ -118,7 +135,7 @@ The `idempotency_key` column has a unique constraint — concurrent inserts with
118135
Financial reports are generated asynchronously:
119136

120137
1. `POST /reports/financial` — inserts a report row with status `queued`, enqueues a BullMQ job, returns `{ jobId, status }`
121-
2. **ReportsProcessor** (BullMQ worker) picks up the job, sets status to `processing`, runs an aggregation query inside a `REPEATABLE READ` transaction, and stores the JSONB result
138+
2. **ReportsProcessor** (BullMQ worker) picks up the job, sets status to `processing`, reads pre-aggregated totals from `ledger_totals` (O(1), no table scan), and stores the JSONB result
122139
3. `GET /reports/financial/:jobId` — polls the report status and result, scoped to the requesting user
123140

124141
If Redis is down when enqueuing, the report is immediately marked `failed` rather than left orphaned in `queued`.
@@ -134,6 +151,9 @@ Architectural decisions are recorded as ADRs:
134151
- [ADR-005: Platform Receives Royalty Remainder](adr/ADR-005-platform-royalty-remainder.md)
135152
- [ADR-006: BullMQ for Async Report Generation](adr/ADR-006-bullmq-async-reports.md)
136153
- [ADR-007: Idempotency Key Owned by Client](adr/ADR-007-client-owned-idempotency-key.md)
154+
- [ADR-008: Running Totals for Reporting](adr/ADR-008-running-totals-for-reporting.md)
155+
- [ADR-009: Redis Idempotency Cache](adr/ADR-009-redis-idempotency-cache.md)
156+
- [ADR-010: Centi-cent Fractional Accrual](adr/ADR-010-centi-cent-fractional-accrual.md)
137157

138158
## Security
139159

drizzle/0001_modern_rogue.sql

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
CREATE TABLE "ledger_totals" (
2+
"type" "ledger_type" PRIMARY KEY NOT NULL,
3+
"total" bigint DEFAULT 0 NOT NULL
4+
);
5+
--> statement-breakpoint
6+
ALTER TABLE "wallets" ADD COLUMN "fractional_balance" integer DEFAULT 0 NOT NULL;--> statement-breakpoint
7+
ALTER TABLE "wallets" ADD CONSTRAINT "wallets_fractional_balance_non_negative" CHECK ("wallets"."fractional_balance" >= 0);--> statement-breakpoint
8+
ALTER TABLE "wallets" ADD CONSTRAINT "wallets_fractional_balance_lt_100" CHECK ("wallets"."fractional_balance" < 100);
9+
10+
-- Backfill running totals from existing ledger data
11+
INSERT INTO ledger_totals (type, total)
12+
SELECT type, COALESCE(SUM(amount), 0) FROM ledger GROUP BY type
13+
ON CONFLICT (type) DO NOTHING;

0 commit comments

Comments
 (0)