api: make readiness reflect database availability without restarting on database outages #727

After a database outage, the API continues to report healthy readiness while protected requests fail session lookup. Kubernetes can keep routing traffic to an unusable API.

Priority: High. Evidence: static inspection of upstream 04e41075cafc4c1a99bcb3723b0a8745ee810957 (2026-10-16). The failure scenarios below have not been reproduced in a running cluster.

Evidence and scenario

• api/cmd/main.go:316 returns ok from /healthz without consulting the store.
• charts/gameplane/templates/api.yaml:501 points both readiness and liveness at that endpoint.
• api/internal/auth/sessions.go:127 performs SQL-backed session and permission lookup for protected requests; store errors become an unauthenticated response.
• api/internal/db/db_postgres.go:15 checks connectivity at startup, but that does not establish ongoing readiness.

Start the API with PostgreSQL, interrupt database connectivity, and compare readiness with authenticated API requests. Existing game pods and their Kubernetes controllers are a separate availability domain; this report concerns the panel API.

Recommended fix

Add a separate bounded /readyz check. A minimal connectivity check in the existing router is:

r.Get("/readyz", func(w http.ResponseWriter, req *http.Request) {
    ctx, cancel := context.WithTimeout(req.Context(), 2*time.Second)
    defer cancel()
    if err := store.DB.PingContext(ctx); err != nil {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

Keep process liveness on /healthz and move the readiness probe to /readyz with a probe timeout longer than the DB-check deadline. Do not restart otherwise healthy API processes during an external database outage. Consider a bounded health-check cache or dedicated connection budget if probe traffic competes with the single SQLite connection.

Distinguish transient store failures from invalid sessions internally and return a generic service-unavailable response where appropriate, without exposing credentials, database addresses, or account existence. Preserve fail-closed authorization. Export dependency availability and recovery signals.

Acceptance criteria

• A database outage makes readiness fail within a documented bounded interval while liveness remains healthy.
• Recovery restores readiness and authenticated access without requiring an API restart.
• A slow connection attempt cannot leave health handlers or probe work accumulating.
• Invalid sessions remain unauthorized; DB outages do not grant access or reveal sensitive error details.
• CI exercises the PostgreSQL outage/recovery path and the normal SQLite path.

Related: #707 adds PostgreSQL-backed API tests and provides a fixture for this failure-mode coverage.