From 3b1a406807b0ef0f7fba0135781e81599e603c15 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Tue, 14 Jul 2026 17:03:01 +0300 Subject: [PATCH] test: gate healthy-gating on TCP, not the postgres unix socket The scenario's healthcheck was `pg_isready -U postgres` -- no -h, so it probes the unix socket -- while the dependent runs `pg_isready -h 127.0.0.1`, i.e. TCP. The postgres image runs initdb, then a temporary server that listens on the unix socket ONLY and announces "database system is ready to accept connections", then shuts it down and restarts on TCP. So the healthcheck can pass while no IPv4 listener exists at all, opening the gate before the dependent can connect. Proven from the image's own logs; the window is ~220ms locally and wider on a slower CI runner, which is why it flaked only there. Probing TCP makes the gate assert exactly what the dependent needs: the temp server cannot satisfy it, because nothing is listening on TCP to satisfy it against. Deterministic, not a widened timeout. compose2pod itself is unchanged -- it gated faithfully on the healthcheck it was handed. The defect was in the scenario's compose document. --- .../2026-07-14.04-healthy-gating-flake.md | 71 +++++++++++++++++++ tests/integration/test_healthy_gating.py | 7 +- 2 files changed, 77 insertions(+), 1 deletion(-) create mode 100644 planning/changes/2026-07-14.04-healthy-gating-flake.md diff --git a/planning/changes/2026-07-14.04-healthy-gating-flake.md b/planning/changes/2026-07-14.04-healthy-gating-flake.md new file mode 100644 index 0000000..db0a05e --- /dev/null +++ b/planning/changes/2026-07-14.04-healthy-gating-flake.md @@ -0,0 +1,71 @@ +--- +summary: The healthy-gating scenario's postgres healthcheck probes TCP rather than the Unix socket, so the gate now opens only once the dependent's actual transport is accepting connections. +--- + +# Change: Fix the healthy-gating scenario's socket-vs-TCP race + +**Lane:** lightweight — one line in one integration scenario. + +## Goal + +`tests/integration/test_healthy_gating.py` fails intermittently in CI (twice so +far, each time passing on rerun). It is not flaky infrastructure: the scenario +gates on a healthcheck that probes a *different transport* from the one the +dependent service uses. + +- `db`'s healthcheck is `pg_isready -U postgres` — no `-h`, so it probes the + **Unix socket**. +- `app` runs `pg_isready -h 127.0.0.1 -p 5432` — **TCP**. + +The official postgres image runs `initdb`, then starts a **temporary** server to +run its init scripts. That server listens on the Unix socket *only* — and it +announces itself as ready. From its own logs: + +``` +[36] listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" <- temp server, socket ONLY +[36] database system is ready to accept connections <- `pg_isready -U postgres` returns 0 HERE +[37] shutting down <- temp server stops + PostgreSQL init process complete; ready for start up. +[1] listening on IPv4 address "0.0.0.0", port 5432 <- TCP only appears NOW +[1] database system is ready to accept connections <- for real +``` + +There is no IPv4 listener during the init phase, so `pg_isready -U postgres` +(Unix socket) reports ready while `app`'s TCP probe cannot possibly succeed — +and the temp server then shuts down. Measured at 220ms on a developer machine +(21.697 -> 21.917); wider on a slower, contended CI runner, which is why it +fails there and not locally. + +A TCP healthcheck cannot pass during that phase, because there is nothing +listening on TCP for it to pass against. That is what makes the fix +deterministic rather than a widened timeout. + +The 120-second wait budget is untouched and irrelevant — the gate is not timing +out, it is opening on the wrong signal. + +## Approach + +Probe the transport the dependent actually uses: + +```python +"test": ["CMD-SHELL", "pg_isready -U postgres -h 127.0.0.1 -p 5432"] +``` + +This is what the scenario meant to assert all along: `service_healthy` should +mean "the thing `app` depends on is ready", and `app` depends on TCP. It also +makes the scenario a stricter test of compose2pod — the pod's shared-namespace +`127.0.0.1` now has to work for the healthcheck as well as for the target. + +No change to `compose2pod` itself: the tool gated faithfully on the healthcheck +it was handed. The defect is in the scenario's compose document. + +## Files + +- `tests/integration/test_healthy_gating.py` — healthcheck probes TCP + +## Verification + +- [ ] Reproduce the window (socket ready before TCP) against real postgres. +- [ ] Apply the change. +- [ ] Run the scenario repeatedly against real podman — must pass every time. +- [ ] `just test-integration` green; `just test-ci` and `just lint-ci` clean. diff --git a/tests/integration/test_healthy_gating.py b/tests/integration/test_healthy_gating.py index 509c6ab..9ab4f97 100644 --- a/tests/integration/test_healthy_gating.py +++ b/tests/integration/test_healthy_gating.py @@ -12,7 +12,12 @@ def test_healthy_gating_reaches_postgres(run_pod: Callable[..., PodRun]) -> None "image": "postgres:16-alpine", "environment": ["POSTGRES_PASSWORD=pw"], "healthcheck": { - "test": ["CMD-SHELL", "pg_isready -U postgres"], + # Probe TCP, not the Unix socket. The postgres image runs initdb, then a + # temporary server with listen_addresses='' (socket only) for its init + # scripts, and only then restarts on TCP -- so a bare `pg_isready -U postgres` + # reports ready for ~0.5s before 127.0.0.1:5432 accepts anything, opening the + # gate before `app` (which connects over TCP) can reach it. + "test": ["CMD-SHELL", "pg_isready -U postgres -h 127.0.0.1 -p 5432"], "interval": "1s", "timeout": "5s", "retries": 30,