Skip to content

Add opt-in reject_throttle: pace connection opens after establishment failures#1180

Open
fabio-vapi wants to merge 1 commit into
porsager:masterfrom
fabio-vapi:reject-throttle
Open

Add opt-in reject_throttle: pace connection opens after establishment failures#1180
fabio-vapi wants to merge 1 commit into
porsager:masterfrom
fabio-vapi:reject-throttle

Conversation

@fabio-vapi

Copy link
Copy Markdown

Fixes #1179.

Adds an opt-in reject_throttle (default off; when off, behaviour is identical to today) that fixes the establishment-failure herd described in the issue.

What it does

  • Backoff on establishment. Establishment retries now use the existing backoff curve instead of retrying immediately. (Today closed()'s if (initial) return reconnect() returns before closedTime/delay are assigned, so reconnect() computes setTimeout(connect, 0) — a tight loop.)
  • Coordinate the pool. On an establishment failure, one failed connection becomes the lone prober (retrying with backoff) while every other connection's open is blocked. On each prober handshake-success an exponentially-growing batch of blocked connections is admitted (slow-start, 1 → 2 → 4 → … → max); a failed admission collapses the ramp back to 1. Once the backlog drains, the breaker disengages back to baseline. Liveness is the prober's own retry — no timers.

It complements the existing backoff rather than duplicating it: backoff only delays each retry, and because its counter lives on the shared pool object, every connection waits the same delay and then fires together — a delayed herd. reject_throttle keeps exactly one attempt in flight during an outage and ramps back up on recovery.

It governs only the clean-close establishment-reconnect path; error-close (RST) and connect-timeout still reject exactly as before, so it never masks a hard failure. With it enabled, the issue's reproduction drops from ~12.5k attempts/sec to single digits and recovers via slow-start once the endpoint accepts again.

Changes

  • src/index.js — the reject_throttle option and shared.throttle state; the gate (throttleAllows) at the three closed → connecting promotion sites; throttleAdmit (slow-start admission + disengage) in onopen; requeue.
  • src/connection.js — establishment-failure routing (reconnect_with_reject_throttleprober_reconnect | stand_down), backOffOnError, and throttle_on_success at ReadyForQuery.

Tests

tests/reject_throttle.js — 21 cases: option plumbing, throttle-vs-herd, single prober, slow-start recovery and disengagement (including large backlogs and max: 1), the drop path staying intact via max_lifetime, and the boundaries (RST reject, connect_timeout, off == vanilla). Wired into test:esm; kept as a separate file for reviewability — happy to fold into tests/index.js if you'd prefer.

@fabio-vapi
fabio-vapi marked this pull request as ready for review July 16, 2026 15:32
…lures

Opt-in (default off; off is behaviourally identical to today). When a connection
fails to ESTABLISH, the pool otherwise retries every connection immediately and in
lockstep — stampeding a server that is already rejecting connections. With
reject_throttle enabled the pool keeps a single failed connection as the lone
prober (retrying with backoff), blocks the rest, and on each prober handshake
success admits an exponentially growing batch of waiters (slow-start, capped at
`max`), disengaging once the backlog drains.

This complements the existing per-connection `backoff`: backoff only delays each
retry, and since its counter is pool-shared every connection waits the same delay
and then fires together — a delayed herd. reject_throttle instead keeps exactly one
attempt in flight during an outage.

It governs only the clean-close establishment-reconnect path
(`if (initial) return reconnect()`); error-close (RST) and connect-timeout still
reject as before.

Tests in tests/reject_throttle.js (21 cases): option plumbing, throttle-vs-herd,
single prober, ramp collapse, recovery and disengagement (including large backlogs
and max:1), drop-path intact via max_lifetime, and boundaries (RST reject,
connect_timeout, off == vanilla).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failed connection establishment retries with no backoff and stampedes the whole pool (pathological against proxied/serverless Postgres)

1 participant