gRPC load balancing on Kubernetes — a lab

Demonstrates why gRPC traffic collapses onto a single pod behind a normal Kubernetes Service, and compares three fixes. Everything runs on a throwaway local kind cluster.

Why the problem exists

gRPC runs over HTTP/2 and multiplexes every call onto one long-lived TCP connection. A ClusterIP Service load-balances at L4, per connection — so the client connects once, lands on one pod, and every subsequent request follows that same connection to that same pod. Balancing has to happen either in the client (which needs to see all pod IPs) or in an L7 proxy that understands HTTP/2 streams.

Layout

app/         echo server (reports its pod name) + tallying client
  echo.proto   Say() -> { pod, message }
  server.js    grpc-js server, gRPC health service, optional max_connection_age
  client.js    fires N calls, tallies which pod answered, prints a histogram
k8s/
  00-namespace.yaml
  10-backend.yaml   echo Deployment + ClusterIP Service + headless Service
  20-envoy.yaml     Envoy ConfigMap + Deployment + Service
  30-client.yaml    a sleep-forever pod to exec into
  40-haproxy.yaml   HAProxy ConfigMap + Deployment + Service
scripts/
  env.sh            shared config; pins the kube context (see Safety)
  setup.sh          create cluster, build image, deploy
  run-scenarios.sh  run all four scenarios
  scale-test.sh     scale UP mid-run and see if new pods get traffic
  churn-test.sh     scale DOWN mid-run; should produce zero client errors
  envoy-stats.sh    Envoy's own per-endpoint counters
  teardown.sh       delete the cluster

Safety

The lab is pinned to the local kind context kind-grpc-lab, set in one place (scripts/env.sh) and passed explicitly on every kubectl call. It cannot touch whatever kubectl config current-context happens to point at.

Run it

scripts/setup.sh          # ~1 min: creates cluster, builds image, deploys
scripts/run-scenarios.sh  # the five-way comparison
scripts/scale-test.sh envoy      # scale UP mid-run;  or: headless | clusterip
scripts/churn-test.sh haproxy    # scale DOWN mid-run; or: envoy
scripts/teardown.sh       # deletes the cluster

The scenarios

#	Setup	Client config	Result
A	ClusterIP	pick_first (default)	1 of 4 pods, 100% on one
B	ClusterIP	8-channel pool	3–4 of 4 pods, evenness 0.00–0.75 (varies per run)
C	Headless	round_robin	4 of 4 pods, evenness 0.98
D	Envoy	pick_first (default)	4 of 4 pods, evenness 1.00
E	HAProxy	pick_first (default)	4 of 4 pods, evenness 1.00

Measured on this lab, 200 calls, 4 backend pods:

A. ClusterIP + pick_first  (the broken default)
  echo-...-nv8x7    200  100.0%  ####################################

B. ClusterIP + 8-channel pool  (the 80% hack)
  echo-...-mjp6k     75   37.5%  ####################################
  echo-...-rkwvf     75   37.5%  ####################################
  echo-...-rvmk5     50   25.0%  ########################
  evenness: 0.75

C. Headless + round_robin  (client-side LB)
  echo-...-nv8x7     51   25.5%
  echo-...-mjp6k     50   25.0%
  echo-...-rkwvf     50   25.0%
  echo-...-rvmk5     49   24.5%
  evenness: 0.98

D. Via Envoy + pick_first client  (L7 proxy)
  all four pods      50   25.0%  each
  evenness: 1.00

Note what scenario B shows: 8 channels over 4 pods still missed one pod entirely and put 75% on two of them. Connection spreading is statistical, not uniform — that is the cost of the no-infra-changes hack. Re-run it and you get a different answer every time: a later run of the identical command hit all 4 pods but put 50% on one and 12.5% on two others (evenness 0.00). Scenarios C and D reproduce to within one call, run after run. That reproducibility gap is the argument against the hack — you cannot capacity-plan against a distribution that reshuffles on every deploy.

Scenarios D and E are the notable ones: the client is completely unconfigured, a plain pick_first dial. All the balancing intelligence lives in the proxy.

Envoy vs HAProxy

Both work, and both produced an identical perfect 50/50/50/50 split. HAProxy has had end-to-end HTTP/2 since 1.9, and in mode http it treats each HTTP/2 stream as its own request — so balance roundrobin balances per RPC, not per connection. That is the entire trick.

The two configs map onto each other closely:

Concern	Envoy	HAProxy
Accept h2c from clients	http2_protocol_options: {}	bind :50051 proto h2
Speak h2c upstream	explicit_http_config.http2_protocol_options	proto h2 on the server line
Discover pods	type: STRICT_DNS + headless A records	server-template + SRV via resolvers
Per-request balancing	lb_policy: ROUND_ROBIN	balance roundrobin
Retry on a dead pod	retry_on + previous_hosts predicate	retry-on + option redispatch
Fail fast on connect	connect_timeout: 1s	timeout connect 1s

Both survived the scale-down churn test with zero client-visible errors and discovered all 6 pods after a scale-up.

Where they differ in practice:

• Endpoint count is preallocated in HAProxy. server-template echo 10 reserves exactly 10 slots. Scale past 10 pods and the extras get no traffic until you edit the config. Envoy's endpoint list is unbounded.
• Health checking is richer in Envoy. Envoy has native grpc_health_check that speaks the standard gRPC health protocol, plus outlier_detection for passive ejection. The HAProxy config here uses a TCP-level check, which catches a dead pod but not one that is up and failing health internally.
• HAProxy is simpler to operate. One config file, a built-in stats page at :8404/stats, far less YAML. If you do not need gRPC-aware health checks or the xDS story, it is less machinery for the same result.
• Envoy's config is more verbose but more explicit. The typed-config protobuf style is painful to write by hand and much easier to generate, which is why every mesh and Gateway API implementation builds on it.

Rough guidance: if you already run HAProxy, use HAProxy. If you expect to grow into a mesh, Gateway API, or per-service routing policy later, start with Envoy.

One gotcha found while writing this

HAProxy has no line-continuation syntax. A trailing \ on a long server-template line is parsed as a literal keyword and aborts startup with a confusing cascade:

[ALERT] config : 'server grpc_back/(null)' : unknown keyword '\'.
[ALERT] config : resolvers 'kubedns' has same name as another resolvers
[ALERT] config : parsing : out of memory.

None of those messages point at the real problem. Long directives must be on one line.

The scale-up test

This is where the approaches really diverge. scale-test.sh starts a slow 300-call run against 2 pods, scales to 6 mid-run, and tallies.

Through Envoy — all 6 pods picked up:

echo-...-nv8x7   68  22.8%      # the 2 original pods lead only because
echo-...-mjp6k   67  22.5%      # they served the first ~25s alone
echo-...-hf94v   41  13.8%
echo-...-nfmgs   41  13.8%
echo-...-2g4dr   41  13.8%
echo-...-q2784   40  13.4%

Envoy's dns_refresh_rate: 5s finds the new pods and starts using them within seconds. The four new pods get near-identical shares of the post-scale traffic.

Through ClusterIP — only 3 of 6 pods ever saw traffic:

echo-...-fjchh  191  63.7%
echo-...-nv8x7  101  33.7%
echo-...-mjp6k    8   2.7%
evenness: 0.08

And it only moved at all because the server sets grpc.max_connection_age_ms: 30000, which sends a graceful GOAWAY and forces a reconnect that re-rolls the dice. Set MAX_CONNECTION_AGE_MS=0 in k8s/10-backend.yaml and it pins to one pod for the entire run.

The scale-DOWN test — where the config actually got fixed

Scale-up is the easy direction. Scale-down is where a proxy config gets tested, because pods disappear while the proxy may still be routing to them. churn-test.sh runs 200 calls while scaling 6 -> 3.

The first version of the Envoy config in this lab leaked errors to the caller here. Two separate bugs, both worth knowing about:

1. The retry policy didn't cover connection-level failures. The original retry_on was "cancelled,deadline-exceeded,unavailable" — all gRPC status conditions, which only fire when the upstream actually returns a grpc-status trailer. A pod that vanished mid-drain fails before any headers, which is only covered by the HTTP-level conditions. Symptom:

ERROR x1: 14 upstream connect error ... transport failure reason: delayed connect error: Connection refused
ERROR x2: 14 upstream connect error ... reset reason: connection timeout

Fix — you need both families:

retry_on: "connect-failure,refused-stream,reset,unavailable,cancelled,deadline-exceeded"

2. The retry budget didn't fit inside the caller's deadline. With per_try_timeout: 2s, num_retries: 3 and the cluster's default connect_timeout: 5s, a call could burn 8s of retries against a 5s client deadline — so the client gave up before the retries that would have saved it ever ran. Symptom: retries "working" but the caller still sees 4 Deadline exceeded after 5.00s. Fix:

connect_timeout: 1s      # on the cluster; default 5s
per_try_timeout: 1s      # 4 attempts x 1s = 4s, fits under a 5s deadline

Plus a preStop drain hook on the backend so pods keep serving for 15s after they leave Endpoints, giving Envoy's 5s DNS refresh time to drop them.

Measured progression over the same 6 -> 3 scale-down, 200 calls:

Config	Failed calls
original retry policy	4 (2 connect-level, 2 deadline)
+ connection-level retry conditions	2 (deadline only)
+ tightened connect/per-try timeouts	0

The general lesson: a retry policy is only as good as its budget. Retries that can't finish inside the caller's deadline are indistinguishable from no retries at all.

Things worth poking at

• envoy-stats.sh prints Envoy's own per-endpoint rq_total and health state — useful to confirm the proxy agrees with what the client observed, and that grpc_health_check marks all endpoints healthy.
• Kill a backend mid-run (kubectl --context kind-grpc-lab -n grpc-lab delete pod <name>) and watch Envoy's retry policy with the previous_hosts predicate absorb it — the retry lands on a different pod.
• Set MAX_CONNECTION_AGE_MS=0 to see the pure pinning failure mode.
• Scale Envoy itself — it's a single replica here, which is fine for a lab but a single point of failure in production.

Caveats about the lab vs. production

• Everything is a single-node kind cluster, so "which pod" is the only real variable — there's no topology, zone, or network latency to consider.
• The retry timeouts here (per_try_timeout: 1s, connect_timeout: 1s) are sized for this lab's 5s client deadline and a single-node cluster with sub-millisecond latency. Size yours from your real p99 plus retry budget — copying 1s blindly onto a slow upstream will turn healthy-but-slow calls into retry storms.
• Steady state is clean: 400 calls through Envoy with no pod churn gave a perfect 100/100/100/100 split and zero errors. All the errors documented above occur only during deliberate scale events.
• Envoy runs 1 replica with no PDB or HPA. In production the proxy tier needs enough replicas to carry aggregate traffic and survive a node loss.