A hand-built API gateway sitting in front of two mock backend services, used to learn the real infrastructure patterns (auth boundary, rate limiting, circuit breaking, distributed tracing, load balancing) that production gateways implement — each piece built from scratch rather than reached for as a library, for the understanding.
┌──────────────┐
│ client │
└──────┬───────┘
│
▼
┌─────────────────┐
│ gateway │ :8080
│ ─────────────── │
│ auth (JWT) │──┐
│ rate limiting │ │ traces
│ circuit breaker │ │
│ load balancing │ │
│ tracing │ │
└───┬─────────┬─────┘ │
│ │ │
┌────────────┘ └──────┐ │
▼ ▼ │
┌─────────────────┐ ┌─────────────────────┐
│ users-service │ │ orders-service-1 │
│ :4001 │ │ :4002 │
└─────────────────┘ └─────────────────────┘
┌─────────────────────┐
│ orders-service-2 │
│ :4002 (host :4003) │
└─────────────────────┘
supporting infra:
┌──────────┐ ┌──────────┐
│ redis │ │ jaeger │ UI :16686, OTLP :4318
│ (rate │ │ (trace │
│ limits, │ │ storage │
│ refresh │ │ + query) │
│ tokens) │ └──────────┘
└──────────┘
The gateway is the only component clients talk to directly. users-service runs
as a single instance; orders-service runs as two instances behind the gateway's
own load balancer (Phase 6). Redis backs both the rate limiters (Phase 3) and
refresh token storage (Phase 2's follow-up). Every service exports trace spans to
Jaeger via OTLP (Phase 5).
The gateway is the single auth boundary. POST /auth/login issues a JWT (15m expiry,
signed with JWT_SECRET). Protected routes (/api/users/*, /api/orders/*) run
authenticateJWT before the request reaches the proxy — it verifies the token and
attaches req.user = { id, role }.
The gateway then forwards the decoded identity to backend services via plain headers
(X-User-Id, X-User-Role) and never forwards the raw token. Backend services trust
these headers and do not re-verify JWTs themselves.
Why gateway-validates-once instead of backend re-validation:
- Single point of truth for auth logic — one JWT secret, one verification path, one place to rotate keys or change the auth scheme later.
- Backend services stay simple: they read a header instead of carrying a JWT library, secret, and verification logic each.
- Removing the raw token at the edge means it never propagates deeper into the network than necessary.
This assumes backend services are only reachable through the gateway (enforced at the Docker network / infra level) — a header-based identity claim is only trustworthy if nothing else can reach those services directly to forge it.
POST /auth/login returns both an access token (15m JWT) and a refresh token (opaque
random value, 7d TTL, stored server-side in Redis as refresh:<token> -> {userId, role}).
POST /auth/refresh— exchanges a valid refresh token for a new access token. The refresh token is rotated on every use: the old one is deleted from Redis and a new one issued in the same response. Replaying an old (already-rotated) refresh token fails with401, which is the detection signal for a stolen/replayed token.POST /auth/logout— deletes the refresh token from Redis, revoking it immediately.
Refresh tokens are opaque (not JWTs) and stored server-side specifically so they can be revoked on demand — a self-contained JWT refresh token can't be invalidated before its expiry without a separate blocklist, which is effectively the same Redis-backed approach anyway.
Two rate-limiter-middleware instances are wired in, deliberately using different
algorithms because the two endpoints have different abuse profiles:
| Endpoint | Key | Algorithm | Limit |
|---|---|---|---|
/api/users/*, /api/orders/* (post-auth) |
per user ID | token bucket | 20 req / 60s, shared budget across both routes |
POST /auth/login |
per IP | fixed window | 5 req / 60s |
- Token bucket on authenticated API routes — normal usage is bursty (a client opening several tabs, a page firing a handful of requests at once), and token bucket is the only one of the three algorithms that tolerates a burst after idle time while still capping sustained throughput to the refill rate. Punishing every legitimate burst the way a fixed window would creates a worse experience for real users without meaningfully slowing down an attacker.
- Fixed window on
/auth/login— login isn't a usage pattern to accommodate, it's a brute-force target. The bar is "how many password guesses can this IP make per minute," not "how should we shape bursty legitimate traffic." Fixed window gives a hard, simple, cheap-to-compute cap, which matters because login is hit by unauthenticated traffic where there's no per-user identity to rate limit on yet — IP is the only signal available, and it's checked on every single request.
Both limiters return 429 with Retry-After, X-RateLimit-Limit, and
X-RateLimit-Remaining headers (where applicable) computed from the limiter's
result — reset (seconds left in the window) for fixed window, and the time until
the next token refills for token bucket.
Each proxied target (users-service, orders-service) gets its own hand-rolled
circuit breaker (apps/gateway/circuitBreaker.js), wrapping the proxy call
(apps/gateway/proxyWithBreaker.js). Three states:
| State | Behavior |
|---|---|
| CLOSED | Normal operation. Every 5xx response or connection error increments a failure counter. Reaching failureThreshold (3) trips the breaker to OPEN. |
| OPEN | Every request is rejected immediately with 503 — no network call is made to the downstream service. After resetTimeoutMs (10s) elapses, the next request is let through as a probe and the breaker moves to HALF_OPEN. |
| HALF_OPEN | The probe request is allowed through. Success resets the failure count and closes the breaker. Failure reopens it and restarts the 10s timer. |
GET /health/circuit-breakers exposes live state for both breakers (state +
ms until the next retry attempt is allowed), used to observe transitions during
testing.
Verified by killing orders-service: the first 3 requests took the real
connection-failure latency and returned 502; the 3rd failure flipped the breaker
to OPEN; the 4th request returned 503 immediately. After the container was
restarted and the reset timeout elapsed, the next request was let through as a
HALF_OPEN probe, succeeded, and closed the breaker.
Why these thresholds: 3 failures is aggressive enough to stop hammering a clearly-dead service quickly without tripping on a single transient blip. 10 seconds is short enough to notice recovery promptly in a demo/dev setting; in production this would typically be tuned higher and informed by the downstream service's actual recovery time.
Tracing has to do two things: emit spans, and propagate trace context across the gateway → backend-service proxy hop so both sides' spans land in one connected trace instead of two disconnected ones. Three approaches were considered:
| Option | Description | Tradeoff |
|---|---|---|
| A — Full auto-instrumentation | @opentelemetry/auto-instrumentations-node patches everything (HTTP, Express, DNS, etc.) with no manual code. |
Least code, but it's a black box — hard to point at something and explain what's happening, and pulls in instrumentation for things this system doesn't even use. |
| B — Targeted auto-instrumentation | Manually register only HttpInstrumentation + ExpressInstrumentation; context propagation across the proxy hop happens automatically because both rely on the same patched http module. |
Same automatic propagation as A with less noise, but propagation itself is still something a library does for you, not something you implement. |
| C — Manual spans + manual propagation (chosen) | Hand-write the span creation, context extraction/injection, and the proxy-hop header propagation using @opentelemetry/api directly. |
Roughly double the code of B, and the failure mode is unforgiving: get context propagation wrong and spans don't error, they silently start a new disconnected trace. |
Why C: this project's whole premise is building infra pieces by hand for the
understanding (same reasoning as hand-rolling the rate limiter and circuit breaker
instead of using a library). Auto-instrumentation makes the context-propagation
problem disappear by solving it for you; writing it manually means actually
implementing W3C Trace Context propagation — extracting the incoming traceparent
header, keeping the span active via AsyncLocalStorage-backed context across the
whole request lifecycle, and injecting traceparent into the outgoing proxied
request — which is the part of this phase the roadmap calls "the actual hard part."
Side benefit: auto-instrumentation patches modules at import time, which under
ESM requires launching with node --import ./tracing.js instead of a normal
top-of-file import (static imports are hoisted, so a plain import can run too late
to patch anything). Manual instrumentation doesn't monkey-patch anything, so this
project's services start normally with node index.js — no special flag needed.
Implementation (packages/otel-tracing): initTracing(serviceName) sets up a
NodeTracerProvider with an OTLP HTTP exporter to Jaeger and an
AsyncLocalStorageContextManager. tracingMiddleware(tracer) (mounted globally
in all three services) extracts any incoming trace context, starts a SERVER span
as its child, makes it the active context for the rest of the request via
context.with(), and ends it on res.on("finish") — which fires regardless of
which middleware ultimately produced the response, so circuit-breaker fast-fails
and rate-limit rejections still produce a clean, correctly-closed span.
injectTraceContext(proxyReq) is called from the gateway's existing proxyReq
hook to inject the active trace context into the outgoing request to the backend
service. Verified end-to-end via Jaeger: a request through the gateway produces a
single trace ID with two spans, the backend span correctly referencing the gateway
span as its parent — including when the circuit breaker fast-fails a request.
Catching silent propagation failures: because a dropped context doesn't throw —
it just silently starts a new root trace instead of continuing the existing one —
verifying this by eye in the Jaeger UI doesn't scale. At the end of Phase 5 this was
deferred with a list of options to revisit; in Phase 7,
tests/integration/tracing.test.js implements the one that was always the real
answer: scripting against Jaeger's query API (GET /api/traces?...) to assert a
request through the gateway produces a trace with the expected span count and the
backend span referencing the gateway span as its parent. The other deferred options
(X-Trace-Id response header diffing, a dev-only ConsoleSpanExporter, api.diag.setLogger()
for SDK-level export errors) remain useful but unimplemented ideas for deeper
observability, not required once an automated assertion exists.
orders-service runs as two separate instances (orders-service-1,
orders-service-2 in docker-compose.yml, each its own container/port — not
Docker Compose --scale, which would let Docker's embedded DNS round-robin
automatically and leave nothing for the gateway to do). users-service stays a
single instance; only orders-service was duplicated, per the brief.
Selection: round-robin. LoadBalancer (apps/gateway/loadBalancer.js) holds
a fixed list of instances and a cursor; pickInstance() walks forward from the
cursor and returns the first instance that is both healthy and not breaker-OPEN,
wrapping back to the start if needed. Chosen over random selection because it's
deterministic and easy to verify — alternation shows up cleanly as 1, 2, 1, 2, ...
in a request log, where random selection would need many more samples to
convincingly demonstrate even distribution.
Health checking: active polling, not reactive. startHealthChecks
(apps/gateway/healthCheck.js) hits GET /health on every instance every 5s
(2s timeout) and sets instance.healthy directly — independent of whether any
request traffic is flowing. This is deliberately separate from the per-instance
circuit breaker carried over from Phase 4: the breaker reacts to real request
failures during traffic, while the poller notices an instance is down (or has
recovered) even with zero requests in flight. Together they answer the same
question — "is this instance safe to send traffic to right now" — from two
different signals, and an instance is only selected if both agree yes.
Why this composes with the circuit breaker instead of replacing it: with a
single instance (Phase 4), a tripped breaker had nowhere else to send traffic, so
it had to fast-fail. With multiple instances, a tripped breaker on instance A just
means instance B (if healthy) absorbs the traffic instead — the gateway only
returns 503 when every instance is unhealthy or breaker-open. loadBalancedProxy
(apps/gateway/loadBalancedProxy.js) picks an instance per-request via the load
balancer and records the outcome (5xx / connection error) against that specific
instance's breaker, not a single shared one.
GET /health/load-balancer exposes live per-instance state (health + breaker)
for observability.
Verified end-to-end: added an X-Instance-Id response header to
orders-service (env-configured per container) purely to make alternation
observable in tests. Six consecutive requests alternated 1, 2, 1, 2, 1, 2.
Killing orders-service-1 caused the next poll cycle to mark it unhealthy and
all subsequent requests landed on instance 2 with zero client-visible errors.
Restarting it and waiting one more poll cycle brought it back into rotation,
and alternation resumed.
This is a deliberately simplified stand-in for what NGINX, HAProxy, or a cloud load balancer/service mesh would do: real implementations add weighted/ least-connections strategies, TLS termination, much richer health-check configurability (path, expected status, consecutive-failure thresholds before marking down), and operate outside the request path entirely rather than as in-process gateway logic.
tests/integration/ holds black-box integration tests using Node's built-in test
runner (node:test + node:assert/strict — no extra test framework dependency)
against the live docker-compose stack, the same way every phase in this README
was manually verified during development. Run with:
docker compose up -d
pnpm test| File | Covers |
|---|---|
auth.test.js |
No-token rejection, bad credentials, successful login + protected route access, malformed token, refresh rotation + replay rejection, logout revocation |
routing.test.js |
Status/body passthrough for both proxied routes, backend 404 passthrough |
rateLimit.test.js |
Per-IP login limiter trips on the 6th attempt with Retry-After; per-user token bucket trips after the burst limit |
circuitBreaker.test.js |
Kills both orders-service instances via docker kill, drives enough failures to trip both per-instance breakers, asserts 503, restarts the containers, and asserts recovery once health checks + breakers reset |
tracing.test.js |
Asserts a request through the gateway produces one Jaeger trace with both spans present and the backend span referencing the gateway span as parent — the automated version of the check deferred in Phase 5 |
Test isolation: Redis (rate-limiter counters, refresh tokens) is flushed before
each file via a before hook, and rateLimit.test.js flushes before each
individual test (beforeEach) since both its tests deliberately exhaust a budget
that would otherwise bleed into the next test. --test-concurrency=1 runs test
files sequentially rather than in parallel — these tests share live, stateful
infrastructure (Redis, circuit breakers, real containers), so running files
concurrently would make them interfere with each other.
Why integration tests against the real stack instead of unit tests with mocks: nearly everything interesting in this project is the interaction between components — JWT validation feeding into per-user rate-limit keys, the circuit breaker's behavior depending on real connection failures, trace context surviving an actual proxy hop. Mocking any of those out would mean testing something other than the thing that actually breaks in production.
scripts/loadtest.js sends the same number of sequential, single-connection
requests to users-service directly and through the gateway, and reports latency
percentiles for both. Run with the stack up: node scripts/loadtest.js.
This is deliberately not a max-throughput benchmark through the gateway: the per-user rate limiter caps sustained throughput to 20 req/60s by design (see "Rate limiting" above), so a high-concurrency load test through the gateway would just measure the rate limiter tripping, which is already documented behavior, not new information. What's actually useful to know is the per-request latency cost of everything the gateway does on top of a plain proxy pass-through.
Two representative runs:
Direct to users-service |
Through gateway | Overhead | |
|---|---|---|---|
| Run 1 — avg | 2.5ms | 12.2ms | +9.7ms |
| Run 1 — p50 | 1.8ms | 11.8ms | +10.1ms |
| Run 2 — avg | 2.9ms | 9.6ms | +6.7ms |
| Run 2 — p50 | 1.9ms | 9.5ms | +7.6ms |
Where the ~7-10ms goes (in request order): JWT verification (jsonwebtoken,
synchronous HMAC), a Redis round-trip for the per-user token-bucket check (Lua
script execution), trace span creation, load-balancer instance selection +
breaker state check, then the actual proxied HTTP call to the backend (itself
included in the "direct" baseline, so the overhead figure isolates everything
before that hop). The Redis round-trip is the single largest contributor of
these — everything else is in-process and sub-millisecond.
Is this overhead acceptable? For an API gateway doing auth, rate limiting, circuit breaking, and tracing on every request, single-digit milliseconds is reasonable and in line with what equivalent functionality costs in production gateways (most of which also hit Redis or an equivalent store for distributed rate limiting). It would matter more for latency-sensitive paths than for typical CRUD APIs — worth knowing, not necessarily worth optimizing away here.
No browser/screenshot tooling was available in this environment to capture the
Jaeger UI directly, so here's the equivalent evidence pulled from Jaeger's query
API for a real request through the gateway (GET /api/orders/101) — the same
data the UI renders, showing one connected trace across the proxy hop:
traceID: 10ce88af0c48f25cdd0aeb8333a477c5
gateway | GET /api/orders/101 | duration: 8582us | spanID: 75a1ad6632ebbd47 | parent: none
orders-service | GET /orders/101 | duration: 2482us | spanID: d52971f0286e2036 | parent: 75a1ad6632ebbd47
The orders-service span's parent is the gateway span's ID — exactly what
manual context propagation (Phase 5, Option C) is supposed to produce, and what
tests/integration/tracing.test.js asserts automatically on every test run. To
see this rendered visually: docker compose up -d, make a request through the
gateway, then open http://localhost:16686, select service gateway, and view
the trace — it renders as two stacked spans (gateway wrapping the backend call)
with a visible parent/child connector.
This project intentionally hand-rolls infrastructure that production systems buy or adopt as battle-tested libraries — the value here was in building each piece to understand the mechanism, not in producing something production-ready as-is. At real scale:
- Proxy logic — replace
http-proxy-middleware+ hand-rolled load balancing with NGINX, Envoy, or a managed API gateway (Kong, AWS API Gateway). These handle connection pooling, HTTP/2, TLS termination, and edge cases this project doesn't attempt. - Load balancing — weighted/least-connections strategies instead of plain round-robin, and richer health checks (configurable path, expected status, consecutive-failure-before-marking-down thresholds) instead of a flat healthy/unhealthy boolean.
- Circuit breaker —
opossumor a service mesh's built-in breaker (Istio, Linkerd) instead of a hand-rolled state machine, for battle-tested edge-case handling (concurrent half-open probes, richer event hooks) — see the Phase 4 discussion on the hand-roll-vs-library tradeoff. - Rate limiting — this gateway already uses the right pattern (Redis-backed, Lua-script atomic, algorithm-per-endpoint); at scale this would mostly need tuning (limits, window sizes) and possibly a managed Redis cluster for HA, not a redesign.
- Tracing — auto-instrumentation (Option B from Phase 5) over hand-rolled spans for anything beyond a learning project — manual propagation's silent failure mode (documented above) is a real operational risk at scale, where "did context propagate" needs to be boringly reliable, not hand-verified.
- Auth — the gateway-validates-once pattern holds up at scale, but JWT secret rotation, refresh-token storage sharding, and possibly moving to asymmetric signing (RS256) so backend services could verify independently if ever split from the gateway, would all need addressing.
- Service mesh — at a large enough number of services, much of what this gateway does (mTLS, retries, circuit breaking, tracing) moves to a sidecar mesh layer instead of living in gateway code, with the gateway itself shrinking to routing + edge auth.