Proxy envoy ADS requests to Consul server#4
Conversation
| // TODO (NET-148): Ensure the server connection here is the one acquired via the server discovery library | ||
| return outCtx, cdp.consulServer.grpcClientConn, nil | ||
| } | ||
| gRPCServer := grpc.NewServer(grpc.UnknownServiceHandler(proxy.TransparentHandler(director))) |
There was a problem hiding this comment.
The library has most of the scaffolding required to proxy grpc requests to a desired target.
The main proxy logic is here.
|
Really like this approach! ⭐ I was a little concerned that we'd end up "manually" proxying the request (i.e. unmarshaling and re-marshaling requests and responses) in a similar way to how we forward RPCs in Consul. This is much better because it means we don't need to consume and upgrade the Envoy protobufs. I guess one concern is that the library is marked as "proof of concept" but as we control our gRPC version here I don't think it's a particularly big issue. |
|
Elaborating a bit on the library background (will update the PR with comments):
Agreed. But, I did get some confidence in it seeing this - https://github.com/siderolabs/grpc-proxy, that uses mwitkow/grpc-proxy as the core foundation to build on more features.
|
|
I realized while reviewing hashicorp/consul-server-connection-manager#3, that when the server terminates an xDS stream to rebalance load, we may need to also close Envoy's connection to get it to reset state (nonces etc.) I'm not quite sure where that logic would go if using this library. Will do some digging! Edit: I think it should "just work" and we'll pass the error along to Envoy as-is, which should cause it to retry. |
jshahriddhi
force-pushed
the
envoy-ads
branch
from
8c5cefc to
dc8c349
Compare
boxofrad
left a comment
boxofrad
left a comment
There was a problem hiding this comment.
Looks good to me! Left a couple of small points, but nothing blocking 👏🏻
boxofrad
left a comment
boxofrad
left a comment
There was a problem hiding this comment.
Lookin' good! I've left a handful of (mostly optional) comments 😅
| func (cdp *ConsulDataplane) checkAndEnableLocalXDSServer() { | ||
| if checkLocalXDSServer(cdp.cfg.XDSServer.BindAddress) { | ||
| cdp.localXDSServer.enabled = true | ||
| } | ||
| } |
There was a problem hiding this comment.
It seems odd that somebody would want to use consul-dataplane without the xDS proxy, other than the example in your comment above - which I think is unlikely because they could run consul connect envoy against the server instead.
What do you think about just erroring out when a non-local bind address is given?
There was a problem hiding this comment.
I was split about this too. Will check with Matt about the use case of allowing non local xds-bind-address as mentioned in the RFC.
| return errors.New("envoy xDS bind address not specified") | ||
| case cfg.XDSServer.BindPort == 0 && !checkLocalXDSServer(cfg.XDSServer.BindAddress): | ||
| return errors.New("envoy xDS bind port not specified") | ||
| case !strings.HasPrefix(cfg.XDSServer.BindAddress, "unix://") && cfg.XDSServer.BindAddress != "127.0.0.1" && cfg.XDSServer.BindAddress != "localhost": |
There was a problem hiding this comment.
Optional: We might want to parse the address and use IsLoopback in case the user provides an IPv6 address.
| // For now we just give the server address directly. | ||
| AgentAddress: cdp.consulServer.address.String(), | ||
| AgentPort: strconv.Itoa(cdp.cfg.Consul.GRPCPort), | ||
| AgentAddress: cdp.cfg.XDSServer.BindAddress, |
There was a problem hiding this comment.
It would be nice to eventually refactor this to not call things Agent*. Its very non-urgent though.
There was a problem hiding this comment.
Yea for now we have copy-pasted a lot of this from the existing consul connect command in the consul repo. We can diverge to a more suitable naming at a later point.
3 participants
consul-dataplane will allow configuring an xDS server for dynamic envoy configuration. By default, this xDS server will be hosted locally by the consul-dataplane process itself.
This PR adds functionality to host a gRPC server to serve xDS requests within consul-dataplane. The gRPC server mainly acts like a proxy to forward the envoy xDS requests to a Consul server the consul-dataplane process is connected to.
Related JIRA: https://hashicorp.atlassian.net/browse/NET-99