Skip to content

Harden API input validation and bound rate-limiter memory#4

Merged
edaywalid merged 5 commits into
mainfrom
harden/input-and-resource-limits
Jul 10, 2026
Merged

Harden API input validation and bound rate-limiter memory#4
edaywalid merged 5 commits into
mainfrom
harden/input-and-resource-limits

Conversation

@edaywalid

Copy link
Copy Markdown
Owner

This picks up the two open PRs (#2 destination URL validation, #3 rate-limiter stale eviction), lands them correctly, and applies the same two ideas to the other places in the code where they were missing.

The two themes:

  1. Validate and reject bad input at the API boundary instead of letting it fail deep in the delivery worker or silently produce wrong results.
  2. Bound in-memory structures keyed by external input, and actually wire up the maintenance that prunes them.

Rate-limiter eviction (supersedes #3)

  • Track lastSeen per bucket and add EvictStale plus a RunEviction ticker loop.
  • Wire RunEviction into the app lifecycle next to the worker and pruner goroutines. Fix/ratelimit stale eviction #3 added the eviction method but nothing ever called it, so the leak it described stayed open. This starts the sweep so the map is actually bounded.
  • Add WHOOK_INGEST_EVICT_INTERVAL (default 10m) and WHOOK_INGEST_IDLE_TTL (default 1h).

The bucket map is keyed by source name and is populated before the source is validated, so hitting POST /ingest/{random} with many names grows it without bound even though every request 404s. Eviction bounds it to the idle window.

API boundary validation (supersedes #2)

  • Destination URL must be an absolute http/https URL, on both create and update. api: reject invalid destination URLs at the API boundary #2 only guarded create; the update path had no validation.
  • Per-destination backoff must have sane bounds. A zero max_attempts dead-letters on the first failure and a zero delay causes a tight redelivery loop, both straight from unvalidated JSON.
  • Filter clauses that can never match (empty field paths, empty header names, empty body_in lists) are rejected instead of silently dropping every event for that destination.
  • Unknown event status filters return 400 instead of an empty list.
  • List pagination is clamped: limit is capped at 500 so a huge value cannot force an unbounded query, and a negative offset is floored to zero (it is a hard error on Postgres).

Source names

  • Source names are validated on register and update. The name is the ingest URL path segment and the rate-limiter key, but only non-emptiness was checked on register and nothing on update. Empty, over-long, and names with spaces, control characters, or slashes are now rejected.

All new logic has tests (pure-function unit tests plus handler-level integration tests over a real server and store). go vet and go test -race ./... pass.

The per-key bucket map grew without bound because entries were never
removed, so requests under many distinct keys could exhaust memory.
Track lastSeen per bucket and add EvictStale plus a RunEviction ticker
loop that prunes idle buckets. Cutoff is passed in so the logic stays
pure and testable without sleeps.
Wire the limiter's RunEviction loop into the app lifecycle next to the
worker and pruner goroutines, so the eviction added in the previous
commit actually runs. Add WHOOK_INGEST_EVICT_INTERVAL and
WHOOK_INGEST_IDLE_TTL to tune the sweep, defaulting to a 10m sweep and a
1h idle TTL, and document them.
A source name becomes the ingest URL path segment and the rate-limiter
key, but only non-emptiness was checked on register and nothing on
update. Reject empty, over-long, and names containing spaces, control
characters, or slashes so bad names are caught at registration instead
of producing awkward ingest paths later.
Bad input was accepted and stored, then failed deep in the delivery
worker or silently produced wrong results. Reject it with a 400 up front:

- destination URL must be an absolute http/https URL, on create and update
- per-destination backoff must have sane bounds (a zero max_attempts
  dead-letters on the first failure; zero delays cause a tight retry loop)
- filter clauses that can never match (empty paths, empty header names,
  empty allow-lists) are rejected instead of silently dropping every event
- unknown event status filters return 400 rather than an empty list
- list pagination is clamped: limit is capped so a huge value cannot force
  an unbounded query, and a negative offset is floored to zero (it is a
  hard error on Postgres)
It is imported directly by the ratelimit package; go mod tidy moves it
out of the indirect block.
@edaywalid edaywalid merged commit a7d54e4 into main Jul 10, 2026
3 checks passed
@edaywalid edaywalid deleted the harden/input-and-resource-limits branch July 10, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant