Skip to content

Commit 1b67f16

Browse files
ovadiagalovadiagal
authored
Add dns_gateway dynamic module example (#56)
Dynamic module approach to solving this issue: [Retrieve DNS resolution results at runtime #41605](envoyproxy/envoy#41605) This dynamic module can be used to implement egress policies by hostname ([see this blog by Cloudflare](https://blog.cloudflare.com/egress-policies-by-hostname/) which has some good context) <img width="1978" height="971" alt="diagram" src="https://github.com/user-attachments/assets/554aee77-6038-4a22-8fdb-1c276b2b4eeb" /> In this dynamic module, you'll find: ### `dns_gateway/mod.rs` DNS gateway UDP listener filter. Intercepts DNS queries and returns synthetic responses for domains matching configured egress policies (exact or *. wildcard). - For matching A-record queries, allocates a virtual IP from the shared cache and responds with it - For matching non-A queries (e.g. AAAA), responds with NODATA (no IPv6 support yet) - For non-matching queries, passes through to the next filter ### `virtual_ip_cache.rs` Maintains a mapping between FQDN and virtual IPs. - dns_gateway calls `virtual_ip_cache::allocate()` to get a virtual IP for a domain (reuses existing IP if the domain was seen before) - cache_lookup calls `virtual_ip_cache::lookup()` to resolve a virtual IP back to its domain and metadata - IP space is bounded by the configured prefix_len; no eviction currently. ### `cache_lookup.rs` Network filter that runs on new TCP connections. Looks up the destination virtual IP in the shared cache and writes the resolved domain and policy metadata into Envoy filter state, making them available to downstream filters via: - `%FILTER_STATE(envoy.dns_gateway.domain:PLAIN)%` - `%FILTER_STATE(envoy.dns_gateway.metadata.<key>:PLAIN)%` Additional details can be found in the README file in this PR --------- Signed-off-by: Gal Ovadia <ggalovadia@gmail.com>
1 parent 2e468f3 commit 1b67f16

8 files changed

Lines changed: 1895 additions & 8 deletions

File tree

‎rust/Cargo.lock‎

Lines changed: 654 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎rust/Cargo.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,10 @@ serde = { version = "1.0", features = ["derive"] }
1212
serde_json = "1.0"
1313
rand = "0.9.0"
1414
matchers = "0.2.0"
15+
dashmap = "6.1.0"
16+
once_cell = "1.20.2"
17+
hickory-proto = "0.24"
18+
parking_lot = "0.12"
1519

1620
[dev-dependencies]
1721
tempfile = "3.16.0"

‎rust/src/dns_gateway/README.md‎

Lines changed: 329 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,329 @@
1+
# DNS Gateway
2+
3+
Envoy dynamic module filters that intercept DNS queries and route TCP connections to external
4+
domains via virtual IP allocation.
5+
6+
![DNS Gateway diagram](diagram.png)
7+
8+
## Prerequisites
9+
10+
Requires iptables/nftables rules to redirect application traffic to Envoy:
11+
12+
- **DNS**: UDP port 53 redirected to Envoy's DNS listener (e.g. port 15053)
13+
- **TCP**: Outbound connections redirected to Envoy's TCP listener (e.g. port 15001)
14+
15+
See [`connectivity-iptables`](../../../../../connectivity-iptables) for setup scripts.
16+
17+
## How it works
18+
19+
1. **`dns_gateway`** (UDP listener filter) — Intercepts DNS queries. If the queried domain matches
20+
a configured pattern, allocates a virtual IP from a private subnet and responds with an A record.
21+
Caches the mapping from virtual IP to domain and metadata. Non-matching queries pass through.
22+
23+
2. **`cache_lookup`** (network filter) — On new TCP connections, looks up the destination virtual IP
24+
in the shared cache and sets the resolved domain and metadata as Envoy
25+
[filter state](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/data_sharing_between_filters#primitives)
26+
for use in routing.
27+
28+
```
29+
Application
30+
| DNS query: "bucket-1.aws.com"
31+
v
32+
dns_gateway
33+
| matches "*.aws.com", allocates 10.10.0.1, responds with A record
34+
v
35+
Application
36+
| TCP connect to 10.10.0.1:443
37+
v
38+
cache_lookup
39+
| resolves 10.10.0.1 -> domain="bucket-1.aws.com", metadata.cluster="aws"
40+
v
41+
tcp_proxy
42+
| routes to upstream cluster using filter state
43+
v
44+
External service (bucket-1.aws.com)
45+
```
46+
47+
## Filter state
48+
49+
`cache_lookup` sets the following keys, accessible via `%FILTER_STATE(...)%`:
50+
51+
| Key | Example |
52+
| ---------------------------------- | -------------------------------- |
53+
| `envoy.dns_gateway.domain` | `bucket-1.aws.com` |
54+
| `envoy.dns_gateway.metadata.<key>` | value from matched domain config |
55+
56+
Usage in Envoy config:
57+
58+
- `%FILTER_STATE(envoy.dns_gateway.domain:PLAIN)%`
59+
- `%FILTER_STATE(envoy.dns_gateway.metadata.cluster:PLAIN)%`
60+
- `%FILTER_STATE(envoy.dns_gateway.metadata.auth_token:PLAIN)%`
61+
62+
## Domain matching
63+
64+
- **Exact**: `"example.com"` — matches only `example.com`
65+
- **Wildcard**: `"*.aws.com"` — matches any subdomain (e.g. `bucket-1.aws.com`,
66+
`sub.api.aws.com`) but not `aws.com` itself
67+
68+
## Configuration reference
69+
70+
### `dns_gateway`
71+
72+
| Field | Type | Description |
73+
| -------------------- | ------- | ---------------------------------------------------------------- |
74+
| `base_ip` | string | Base IPv4 address for virtual IP allocation (e.g. `"10.10.0.0"`) |
75+
| `prefix_len` | integer | CIDR prefix length (1-32). A `/24` gives 256 IPs. |
76+
| `domains` | array | Domain matchers |
77+
| `domains[].domain` | string | Exact (`"example.com"`) or wildcard (`"*.example.com"`) pattern |
78+
| `domains[].metadata` | object | String key-value pairs exposed via filter state |
79+
80+
### `cache_lookup`
81+
82+
No configuration. Use `filter_config: {}`.
83+
84+
## Manual testing
85+
86+
End-to-end test with docker-compose.
87+
88+
Create the following files:
89+
90+
**docker-compose.yml**:
91+
92+
```yaml
93+
services:
94+
envoy:
95+
image: <your-envoy-image>
96+
network_mode: host
97+
volumes:
98+
- ./envoy.yaml:/etc/envoy/envoy.yaml
99+
command: ["envoy", "-c", "/etc/envoy/envoy.yaml", "-l", "debug"]
100+
101+
upstream-1:
102+
image: python:3.12-slim
103+
network_mode: host
104+
volumes:
105+
- ./upstream_1.py:/app/server.py
106+
command: ["python3", "/app/server.py"]
107+
108+
upstream-2:
109+
image: python:3.12-slim
110+
network_mode: host
111+
volumes:
112+
- ./upstream_2.py:/app/server.py
113+
command: ["python3", "/app/server.py"]
114+
```
115+
116+
**upstream_1.py** (port 18001):
117+
118+
```python
119+
from http.server import HTTPServer, BaseHTTPRequestHandler
120+
121+
class Handler(BaseHTTPRequestHandler):
122+
def do_CONNECT(self):
123+
print(f"\nCONNECT {self.path}")
124+
for key, value in self.headers.items():
125+
print(f" {key}: {value}")
126+
127+
self.send_response(200)
128+
self.end_headers()
129+
130+
request = self.connection.recv(4096)
131+
132+
body = f"cluster_1\nCONNECT: {self.path}\n"
133+
for key, value in self.headers.items():
134+
body += f"{key}: {value}\n"
135+
136+
resp = f"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nX-Upstream: cluster_1\r\nContent-Length: {len(body)}\r\n\r\n{body}"
137+
self.connection.sendall(resp.encode())
138+
139+
HTTPServer(("0.0.0.0", 18001), Handler).serve_forever()
140+
```
141+
142+
**upstream_2.py** (port 18002):
143+
144+
```python
145+
from http.server import HTTPServer, BaseHTTPRequestHandler
146+
147+
class Handler(BaseHTTPRequestHandler):
148+
def do_CONNECT(self):
149+
print(f"\nCONNECT {self.path}")
150+
for key, value in self.headers.items():
151+
print(f" {key}: {value}")
152+
153+
self.send_response(200)
154+
self.end_headers()
155+
156+
request = self.connection.recv(4096)
157+
158+
body = f"cluster_2\nCONNECT: {self.path}\n"
159+
for key, value in self.headers.items():
160+
body += f"{key}: {value}\n"
161+
162+
resp = f"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nX-Upstream: cluster_2\r\nContent-Length: {len(body)}\r\n\r\n{body}"
163+
self.connection.sendall(resp.encode())
164+
165+
HTTPServer(("0.0.0.0", 18002), Handler).serve_forever()
166+
```
167+
168+
**envoy.yaml**:
169+
170+
```yaml
171+
static_resources:
172+
listeners:
173+
- name: dns_listener
174+
address:
175+
socket_address:
176+
address: 0.0.0.0
177+
port_value: 15053
178+
protocol: UDP
179+
listener_filters:
180+
- name: envoy.filters.udp_listener.dynamic_modules
181+
typed_config:
182+
"@type": type.googleapis.com/envoy.extensions.filters.udp.dynamic_modules.v3.DynamicModuleUdpListenerFilter
183+
dynamic_module_config:
184+
name: connectivity_envoy_module
185+
do_not_close: true
186+
filter_name: dns_gateway
187+
filter_config:
188+
"@type": type.googleapis.com/google.protobuf.Struct
189+
value:
190+
base_ip: "10.10.0.0"
191+
prefix_len: 24
192+
domains:
193+
- domain: "*.aws.com"
194+
metadata:
195+
cluster: cluster_1
196+
auth_token: "abc123"
197+
- domain: "example.com"
198+
metadata:
199+
cluster: cluster_2
200+
auth_token: "def456"
201+
- name: envoy.filters.udp_listener.dns_filter
202+
typed_config:
203+
"@type": type.googleapis.com/envoy.extensions.filters.udp.dns_filter.v3.DnsFilterConfig
204+
stat_prefix: dns_fallback
205+
client_config:
206+
max_pending_lookups: 256
207+
dns_resolution_config:
208+
resolvers:
209+
- socket_address:
210+
protocol: TCP
211+
address: 172.20.0.10
212+
port_value: 53
213+
dns_resolver_options:
214+
no_default_search_domain: true
215+
use_tcp_for_dns_lookups: true
216+
server_config:
217+
inline_dns_table: {}
218+
219+
- name: tcp_listener
220+
address:
221+
socket_address:
222+
address: 0.0.0.0
223+
port_value: 15001
224+
listener_filters:
225+
- name: envoy.filters.listener.original_dst
226+
typed_config:
227+
"@type": type.googleapis.com/envoy.extensions.filters.listener.original_dst.v3.OriginalDst
228+
filter_chains:
229+
- filters:
230+
- name: envoy.filters.network.dynamic_modules
231+
typed_config:
232+
"@type": type.googleapis.com/envoy.extensions.filters.network.dynamic_modules.v3.DynamicModuleNetworkFilter
233+
dynamic_module_config:
234+
name: connectivity_envoy_module
235+
do_not_close: true
236+
filter_name: cache_lookup
237+
filter_config: {}
238+
# Setting an upstream cluster directly in the TCP proxy tunneling config with FILTER_STATE(...)
239+
# is not supported. Instead, write the value of FILTER_STATE(...) to 'envoy.tcp_proxy.cluster'
240+
- name: envoy.filters.network.set_filter_state
241+
typed_config:
242+
"@type": type.googleapis.com/envoy.extensions.filters.network.set_filter_state.v3.Config
243+
on_new_connection:
244+
- object_key: envoy.tcp_proxy.cluster
245+
format_string:
246+
text_format_source:
247+
inline_string: "%FILTER_STATE(envoy.dns_gateway.metadata.cluster:PLAIN)%"
248+
- name: envoy.filters.network.tcp_proxy
249+
typed_config:
250+
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
251+
stat_prefix: egress
252+
cluster: default
253+
tunneling_config:
254+
hostname: "%FILTER_STATE(envoy.dns_gateway.domain:PLAIN)%"
255+
headers_to_add:
256+
- header:
257+
key: "X-Auth-Token"
258+
value: "%FILTER_STATE(envoy.dns_gateway.metadata.auth_token:PLAIN)%"
259+
260+
clusters:
261+
- name: cluster_1
262+
type: STATIC
263+
load_assignment:
264+
cluster_name: cluster_1
265+
endpoints:
266+
- lb_endpoints:
267+
- endpoint:
268+
address:
269+
socket_address:
270+
address: 127.0.0.1
271+
port_value: 18001
272+
273+
- name: cluster_2
274+
type: STATIC
275+
load_assignment:
276+
cluster_name: cluster_2
277+
endpoints:
278+
- lb_endpoints:
279+
- endpoint:
280+
address:
281+
socket_address:
282+
address: 127.0.0.1
283+
port_value: 18002
284+
```
285+
286+
### 2. Start
287+
288+
```bash
289+
docker-compose up
290+
```
291+
292+
### 3. Set up iptables redirect
293+
294+
```bash
295+
# Redirect DNS (UDP 53) to Envoy's DNS listener
296+
sudo iptables -t nat -A OUTPUT -p udp --dport 53 -j DNAT --to-destination 127.0.0.1:15053
297+
298+
# Redirect TCP to virtual IPs (10.10.0.0/24) to Envoy's TCP listener
299+
sudo iptables -t nat -A OUTPUT -p tcp -d 10.10.0.0/24 -j DNAT --to-destination 127.0.0.1:15001
300+
```
301+
302+
### 4. Test
303+
304+
```bash
305+
# Will allocate sequentially increasing virtual IPs
306+
dig one.s3.aws.com
307+
dig two.s3.aws.com
308+
dig example.com
309+
310+
# Unmatched domain, will defer to external DNS
311+
dig github.com
312+
313+
# Will reach cluster_1
314+
curl http://s3.aws.com./
315+
316+
# Will reach cluster_2
317+
curl http://example.com./
318+
319+
# See logs for upstream-1 and upstream-2
320+
docker-compose logs upstream-1
321+
docker-compose logs upstream-2
322+
```
323+
324+
### 5. Clean up iptables
325+
326+
```bash
327+
sudo iptables -t nat -D OUTPUT -p udp --dport 53 -j DNAT --to-destination 127.0.0.1:15053
328+
sudo iptables -t nat -D OUTPUT -p tcp -d 10.10.0.0/24 -j DNAT --to-destination 127.0.0.1:15001
329+
```

0 commit comments

Comments
 (0)