Repository navigation
fix(health): stop liveness probes from restarting every replica during dependency outages - #2896
Closed
rohithb-hub wants to merge 1 commit into
Closed
rohithb-hub wants to merge 1 commit into
rohithb-hub wants to merge 1 commit into
Conversation
…g dependency outages Signed-off-by: rohithb <rohithb@nvidia.com>
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Cassandra session liveness src/libraries/java/nv-boot-parent/nv-boot-starter-cassandra/... |
The Cassandra starter adds a configurable liveness indicator. It reports DOWN when the application session remains broken after a fresh session succeeds for the configured grace period. Tests cover registration, disabled configuration, and session health transitions. |
Control-plane health groups src/control-plane-services/api-keys/..., src/control-plane-services/cloud-functions/nvcf-service/..., src/control-plane-services/cloud-tasks/nvct-service/..., src/control-plane-services/encrypted-secret-store/ess-service/..., src/control-plane-services/instance-cluster-management/icms-service/... |
Liveness groups use cassandraSession instead of Cassandra checks. NVCF service liveness also excludes NATS. Tests cover liveness and readiness results across the affected service configurations and profiles. |
HTTP invocation readiness src/invocation-plane-services/http-invocation/crates/server/... |
The health route checks NATS without checking NVCF API health. The NVCF health client and cache configuration are removed. Tests cover NVCF API failure and NATS disconnection. |
gRPC proxy readiness src/invocation-plane-services/grpc-proxy/proxy/... |
The health manager checks NATS and no longer checks NVCF API health. Its call sites and tests use the updated interface. |
Chart probe configuration deploy/helm/grpc-proxy/..., deploy/helm/http-invocation/..., deploy/helm/vanity-gateway/..., src/invocation-plane-services/vanity-gateway/README.md, src/invocation-plane-services/vanity-gateway/gateway/health.go |
The charts use process-focused liveness probes and retain dependency checks for readiness. Render tests verify probe settings. Vanity gateway documentation and comments describe its readiness and liveness behavior. |
Priority:
Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Bug fix
Suggested reviewers: balajinvda, borao
Merge Risk: 
ade34
-
No identified issue prevents merging after the planned checks.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 4.17% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 72 functions across 25 files. (10 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title uses the required Conventional Commits format, includes the required scope for a customer-impacting fix, and accurately describes the primary change to liveness probe behavior. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 4.17% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 72 functions across 25 files. (10 skipped: 10 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Contributor
🛡️ CodeQL Analysis✅ No security issues found! 🔗 View full details in Security tab 🕐 Last updated: 2026-10-26 13:32:42 UTC | Commit: ade349b |
Contributor
Author
This was referenced Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Liveness probes checked dependencies shared by every replica (Cassandra, NATS,
nvcf-api). One outage restarted all replicas, and back-off delayed recovery.
Liveness now checks the process only; dependencies gate readiness.
Additional Details
livenessStateplus newcassandraSession. Readiness is unchanged.cassandraSessioncontributor. It reportsDOWN only when the app's Cassandra session fails for 3 min while a new
session succeeds. This restarts a pod whose session stays broken after
Cassandra recovers, and never restarts pods during an outage. Settings:
app.cassandra.session-liveness.{enabled,grace-period,check-interval}./health(readiness) checks NATS only.Liveness uses a process-only endpoint (
:10083/health,/info)./health.Measured on a k3d stack with live traffic:
For the Reviewer
CassandraSessionLivenessIndicator.javagrpc-proxy/proxy/health.go,http-invocation/.../routes/health.rsFor QA
HealthProbeGroupsTest(5 services), nv-boot indicator tests,grpc-proxy and invocation
/healthtests, chart probe tests.test_healthand full Java suites run in CI only.both nvcf-api replicas.
Issues
Relates to #1524
Checklist
Generated with Claude Code (https://claude.com/claude-code)