[shobham-nv]

TL;DR

Backports two fixes to the self-managed 1.1 stack in one patch: the NATS reconnect fix for function workers (worker-utils 1.4.2) and the Cassandra replica-down fixes (#2918). Supersedes #3003.

Additional Details (optional for docs, build, test, refactor, ci, chore, style, and revert PRs)

• helmfile.d/01-dependencies.yaml.gotmpl: cassandra 0.23.0 -> 0.23.2.
  • Sets range_request_timeout: 5000ms. A secondary-index read that picked a crashed replica now times out at 5 s instead of 10 s, leaving the driver time to retry it on another coordinator.
  • Also moves the migrations image 0.18.0 -> 0.18.1 (fix(cassandra): patch migration image runtime dependencies #2740, runtime dependency patch, no schema change).
• helmfile.d/02-core.yaml.gotmpl: api 1.33.0 -> 1.33.2, the chart release cut from fix(cloud-functions): default nvcf-api to 1.28.2 for the Cassandra replica-down fixes #3012.
  • The chart already defaults worker-utils to 1.4.2 (fix(cloud-functions): default worker-utils to 1.4.2 for the NATS reconnect fixes #3002): after a NATS reconnect the worker re-pulls right away instead of hanging until the heartbeat timeout (fix(worker): re-pull JetStream work right after a NATS reconnect #2895), and keeps retrying consumer setup while NATS is unreachable (fix(worker): keep retrying consumer setup while NATS is unreachable #2833). It also includes feat(invocation-service): serve worker callbacks over TLS #2829 (worker callbacks over TLS, opt-in) and fix(security): rebuild worker and Cassandra images with patched Go #2880 (patched Go).
  • fix(cloud-functions): default nvcf-api to 1.28.2 for the Cassandra replica-down fixes #3012 adds nvcf-api 1.28.2: range read timeouts retry once on another coordinator (RangeReadRetryPolicy in the ncp profile), and retriable Cassandra failures return 503 with Retry-After instead of a 500 with CQL.
• No compute-plane change is needed. Workers get their image from the control plane.
• Merging cuts the next self-managed 1.1 patch (version: TODO, to be updated once released).

For the Reviewer

• Running functions keep their current worker-utils image. They pick up 1.4.2 when redeployed or when their instances are recreated. Functions created with a custom utilsContainerImage keep it.

For QA (optional for docs, build, test, refactor, ci, chore, style, and revert PRs)

• NATS: kill the NATS member the HTTP worker is connected to while invoking continuously. Expect 0 invocations hung to the client timeout, and no nats: no heartbeat received after reconnected to nats.
• Cassandra: nodetool gettimeout range reports 5000 ms on every node. On a 3-node HA install, run function lookup traffic and kill -9 the Cassandra process on one node. Expect no stall longer than about 5 s, and no 500s from read timeouts.

Issues

Relates to #2832 and #2893.

Checklist

• I am familiar with the Contributing Guidelines.
• I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: NVIDIA/nvcf
• Branch: shobham/v1.1-cassandra-0.23.2
• File: .coderabbit.yaml
• Commit: 9cd6846
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 2c35e058-e271-4105-bb63-d518f96e7a3b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[balajinvda]

This PR is included in version 1.1.1.

The release is available on GitHub release.