Skip to content

fix(helm): align Hindsight probes with DB pressure remediation#20

Open
allyblockcast[bot] wants to merge 2 commits into
mainfrom
fix/BLO-14004-hindsight-probe-contract
Open

fix(helm): align Hindsight probes with DB pressure remediation#20
allyblockcast[bot] wants to merge 2 commits into
mainfrom
fix/BLO-14004-hindsight-probe-contract

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Jul 9, 2026

Copy link
Copy Markdown

Summary

  • Align Helm defaults with the live Hindsight remediation contract: API liveness/readiness on /version and worker liveness/readiness on /metrics.
  • Set worker metrics probe timeout to 30s after live /metrics probes timed out at the prior 5s setting.
  • Keep worker batch size 1, worker max slots 2, consolidation reservation 1, retain slots 1, and retain max concurrency 1 in chart defaults.

Live rollout evidence

  • Patched live namespace objects directly for BLO-14004: deployment/hindsight-api and statefulset/hindsight-worker.
  • Final live state after timeout rollout: api 1/1 ready; worker 4/4 ready; all fresh api/worker pods restart count 0 after the short soak.
  • Final live settings verified with kubectl: API /version liveness+readiness; worker /metrics liveness+readiness; worker retain slots 1; worker probe timeout 30s.

Verification

  • helm lint helm/hindsight
  • helm template hindsight helm/hindsight
  • helm template hindsight helm/hindsight --set worker.enabled=true

Known limitations

  • ./scripts/hooks/lint.sh could not run in the Paperclip runtime because uv is not installed: ./scripts/hooks/lint.sh: line 39: uv: command not found.
  • This runtime lacks pods/log permission in the hindsight namespace, so API/worker log verification needs QA/operator re-run.
  • Existing namespace events include rollout startup readiness misses and one PostgreSQL pg_isready timeout after rollout; no fresh api/worker pod restarts were observed.

@allyblockcast

allyblockcast Bot commented Jul 9, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-14004

1 similar comment
@allyblockcast

allyblockcast Bot commented Jul 9, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-14004

@allyblockcast

allyblockcast Bot commented Jul 9, 2026

Copy link
Copy Markdown
Author

@ally please review this replacement clean PR for BLO-14004/BLO-13560. Focus: whether the Helm defaults now match the live remediation contract (/version API probes, /metrics worker probes, 30s worker metrics probe timeout, worker max slots 2 with consolidation/retain slots 1 each) and whether any Helm upgrade would still reintroduce the live drift QA found.

@allyblockcast

allyblockcast Bot commented Jul 12, 2026

Copy link
Copy Markdown
Author

@ally review requested for BLO-14004 before merge. Please focus on the Helm chart probe contracts (/version for API, /metrics for worker), worker DB-pressure concurrency values (batch size 1, max slots 2, consolidation reservation 1, retain concurrency 1), and StatefulSet-to-Deployment migration safety. CI is green at c81ae00c.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants