feat(tonic-xds): gRFC A29 data-plane TLS connector wiring by YutaoMa · Pull Request #2640 · grpc/grpc-rust

YutaoMa · 2026-05-12T21:46:49Z

Motivation

Closes out gRFC A29 (xDS-Based TLS Security) in tonic-xds. The cert provider foundation merged in #2593 and #2616 left the connector integration deferred — its pre-req Endpoint::tls_config_with_verifier landed in #2612.

With that hook available, this PR wires up the per-cluster TLS connector.

Solution

Five commits, each independently reviewable:

file_watcher background refresh.
Eager cert parsing in cert provider, verifier access it per handshake.
Atomically swap Connector on CDS update. Existing endpoint channels keep their connector; new connections pick up the latest. Invalid CDS updates are logged and the previous-good connector is kept.
Data-plane TlsConnector — for clusters with Some(security), the connector holds the A29 verifier plus an optional identity provider (mTLS). connect() fetches identity per call, assembles a ClientTlsConfig, and builds the endpoint.
Docs + dead_code cleanup

Tests

Sync refresh_once tests in file_watcher, synthetic-cert chain tests for SAN matching, build_connector dispatch tests, and a counter-based test asserting identity is fetched per connect(). Existing channel.rs integration tests exercise the cluster-aware discovery path end-to-end on the plaintext.

Interop validation

End-to-end mTLS against Istio (Kind + grpc-agent template + SPIFFE creds)
in YutaoMa/tonic-xds-istio-interop@tonic-pr-2640:

git clone https://github.com/YutaoMa/tonic-xds-istio-interop.git
cd tonic-xds-istio-interop && git checkout tonic-pr-2640
./setup.sh && ./run-test.sh

Confirmed successful run with logs request ok request_num=1 … N shown below. The harness uncovered the
two parser-interop bugs fixed in 66d77c5a.

yutma@yutma-mn3741 tonic-xds-istio-interop % ./run-test.sh
==> Applying tonic-xds-client deployment...
serviceaccount/tonic-xds-client unchanged
deployment.apps/tonic-xds-client unchanged
==> Restarting tonic-xds-client to pick up latest image...
deployment.apps/tonic-xds-client restarted
Waiting for deployment "tonic-xds-client" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "tonic-xds-client" rollout to finish: 1 old replicas are pending termination...
deployment "tonic-xds-client" successfully rolled out
==> Streaming tonic-xds-client logs (Ctrl-C to stop)...

Found 2 pods, using pod/tonic-xds-client-fc69984b-mjwlz
2026-05-12T23:32:39.301779Z  INFO tonic_xds_client: building xDS channel target=xds:///greeter.xds-test.svc.cluster.local:50051
2026-05-12T23:32:39.302790Z  INFO tonic_xds_client: channel built; sending requests every 5s
2026-05-12T23:32:39.302844Z  INFO tonic_xds_client: sending unary request request_num=1
2026-05-12T23:32:39.329498Z  INFO tonic_xds_client: request ok request_num=1 response=Hello request-1 from 0.0.0.0:50051
2026-05-12T23:32:44.331071Z  INFO tonic_xds_client: sending unary request request_num=2
2026-05-12T23:32:44.333457Z  INFO tonic_xds_client: request ok request_num=2 response=Hello request-2 from 0.0.0.0:50051
2026-05-12T23:32:49.336730Z  INFO tonic_xds_client: sending unary request request_num=3
2026-05-12T23:32:49.338296Z  INFO tonic_xds_client: request ok request_num=3 response=Hello request-3 from 0.0.0.0:50051
2026-05-12T23:32:54.340085Z  INFO tonic_xds_client: sending unary request request_num=4
2026-05-12T23:32:54.344715Z  INFO tonic_xds_client: request ok request_num=4 response=Hello request-4 from 0.0.0.0:50051
2026-05-12T23:32:59.346421Z  INFO tonic_xds_client: sending unary request request_num=5
2026-05-12T23:32:59.348229Z  INFO tonic_xds_client: request ok request_num=5 response=Hello request-5 from 0.0.0.0:50051
2026-05-12T23:33:04.350184Z  INFO tonic_xds_client: sending unary request request_num=6
2026-05-12T23:33:04.351546Z  INFO tonic_xds_client: request ok request_num=6 response=Hello request-6 from 0.0.0.0:50051
2026-05-12T23:33:09.353003Z  INFO tonic_xds_client: sending unary request request_num=7
2026-05-12T23:33:09.354252Z  INFO tonic_xds_client: request ok request_num=7 response=Hello request-7 from 0.0.0.0:50051
2026-05-12T23:33:14.356055Z  INFO tonic_xds_client: sending unary request request_num=8
2026-05-12T23:33:14.357055Z  INFO tonic_xds_client: request ok request_num=8 response=Hello request-8 from 0.0.0.0:50051
2026-05-12T23:33:19.359671Z  INFO tonic_xds_client: sending unary request request_num=9
2026-05-12T23:33:19.361201Z  INFO tonic_xds_client: request ok request_num=9 response=Hello request-9 from 0.0.0.0:50051
2026-05-12T23:33:24.364393Z  INFO tonic_xds_client: sending unary request request_num=10

…ds provider Arc

…lsContext

gu0keno0

I'll continue to look at the other files.

gu0keno0 · 2026-05-22T14:57:12Z

+/// so each connection picks up the latest snapshot. Existing endpoint
+/// channels keep their `EndpointChannel` instance (and any in-flight TLS
+/// session) — only freshly-discovered endpoints see the swapped value.
+pub(crate) type ConnectorSwap<S> = Arc<ArcSwap<Arc<dyn Connector<Service = S> + Send + Sync>>>;


Currently loadbalancer just take a Connector object, would it be possible to create a wrapper connector that provide atomic method to swap the inner one? Then we can reuse the same wrapper in LoadBalancer.

Or the other way is like you commented there, make LoadBalancer call loadfull everytime, I just feel a wrapper might be simpler for callers. Yet it's up to you. If you want to do one way versus the other, maybe change it for LoadBalancer as well.

yeah wrapping the load call makes sense, I'll do that.

gu0keno0 · 2026-05-22T23:40:07Z

+
+            let connector_swap: ConnectorSwap<EndpointChannel<Channel>> = loop {
+                let Some(cluster) = cluster_watch.next().await else {
+                    return;


Add a info log?

gu0keno0 · 2026-05-23T00:20:10Z

+                tokio::select! {
+                    Some(change) = endpoints.next() => {
+                        if tx.send(change).await.is_err() {
+                            return;


log here too

gu0keno0 · 2026-05-23T00:28:02Z

+            let manager = EndpointManager::new(Arc::clone(&connector_swap));
+            let mut endpoints = manager.discover_endpoints(cache.watch_endpoints(&cluster_name));
+
+            loop {
+                tokio::select! {
+                    Some(change) = endpoints.next() => {


Note: there's no guarantee that EDS resource name == CDS cluster name, so I think we should change the logic there to allow changing of EDS resource after CDS updates. Usually people don't configure xds to do this, but the protocol totally allows it.

So the current code actually handles EDS resource name correctly: in resource_manager.rs:228 the call to xds-client::watch is using eds_name. The confusion here is because we store EDS watches by cluster_name as the key (designed this way because of the ClusterDiscovery abstraction: cache updates is tied to a cluster's lifecycle).

here is my analysis of whether we should change to keying the EDS watch by EDS resource name:
pros:

readability

sharing of the watch channel if same EDS referenced by many CDS. Diffing is still owned by different tasks so not much save on compute just some storage.
cons:

need another layer of lifecycle management for eds updates, especially for the case "one eds reference is removed but one more cds reference it", so we'll end up having to refcount or reverse mapping. for context grpc-go/grpc-java basically does the refcount design through its gc-based lifecycle management of eds.

let me know what you think, I prefer keep the design as is now, possibly with more documentation to clarify this design.

Sounds good to me, let's add a comment here about the EDS key name.

gu0keno0 · 2026-05-23T00:28:50Z

+                            return;
+                        }
+                    }
+                    Some(cluster) = cluster_watch.next() => {


I think we should use biased to allow CDS to be handled first? It should be slightly better.

YutaoMa · 2026-05-26T20:36:12Z

Filing the remaining review feedback as a follow-up PR for a cleaner review as they are all style / debug changes:

Wrapper type on Connector to encapsulate the atomic operations.
comment for cluster-name cache keying
Debug logs on early-exit paths in cluster_discovery.rs
biased; in tokio::select! to prioritize CDS update over EDS

YutaoMa added 6 commits May 12, 2026 10:22

feat(tonic-xds): background cert refresh in file_watcher provider

eca0752

refactor(tonic-xds): cert provider returns parsed roots, verifier hol…

59ef1ed

…ds provider Arc

refactor(tonic-xds): per-cluster Connector swappable on CDS update

f0ce90d

feat(tonic-xds): data-plane TLS connector for clusters with UpstreamT…

bc45eaf

…lsContext

docs(tonic-xds): A29 TLS section + dead_code cleanup

175c291

fix(tonic-xds): A29 parser accepts deprecated CDS fields (Istio interop)

66d77c5

YutaoMa marked this pull request as ready for review May 12, 2026 23:46

gu0keno0 reviewed May 22, 2026

View reviewed changes

Comment thread tonic-xds/src/xds/bootstrap.rs

Comment thread tonic-xds/src/xds/bootstrap.rs

Comment thread tonic-xds/src/xds/resource/security.rs

gu0keno0 reviewed May 22, 2026

View reviewed changes

gu0keno0 reviewed May 23, 2026

View reviewed changes

gu0keno0 approved these changes May 23, 2026

View reviewed changes

YutaoMa merged commit d16a975 into grpc:master May 26, 2026
21 checks passed

Conversation

YutaoMa commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

Tests

Interop validation

Uh oh!

gu0keno0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gu0keno0 May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YutaoMa commented May 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

YutaoMa commented May 12, 2026 •

edited

Loading

gu0keno0 May 22, 2026 •

edited

Loading