Docker cli partially ignores HTTPS_PROXY setting

[bastiaanb]

Description

The Docker CLI partially honors, partially ignores HTTP_PROXY settings. This results in the following

Steps to reproduce the issue:

1. Docker Daemon on public internet, exposes certificate auth protected engine endpoint. Docker client connected to a network with access only to a http proxy server, not the internet.
2. set environment:

export https_proxy=http://proxy-server:8080
export HTTPS_PROXY=http://proxy-server:8080
export DOCKER_HOST=tcp://docker-server.public.inter.net:8443
export DOCKER_TLS_VERIFY=1

3. Run

docker --debug run -ti --rm busybox echo success

Describe the results you received:

The CLI segfaults:

DEBU[0000] Trusting 1 certs                             
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x457858]

goroutine 1 [running]:
github.com/docker/cli/cli/command/container.runContainer(0xc4201f7980, 0xc4203e0000, 0xc4203b5900, 0xc4200ec480, 0x1589f20, 0xc42045c140)
	/go/src/github.com/docker/cli/cli/command/container/run.go:171 +0xa62
github.com/docker/cli/cli/command/container.runRun(0xc4201f7980, 0xc420099560, 0xc4203e0000, 0xc4203b5900, 0x0, 0x0)
	/go/src/github.com/docker/cli/cli/command/container/run.go:94 +0x141
github.com/docker/cli/cli/command/container.NewRunCommand.func1(0xc4203dd8c0, 0xc420438660, 0x4, 0x6, 0x0, 0x0)
	/go/src/github.com/docker/cli/cli/command/container/run.go:48 +0x109
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).execute(0xc4203dd8c0, 0xc420098030, 0x6, 0x6, 0xc4203dd8c0, 0xc420098030)
	/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:646 +0x44e
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4203c0000, 0xc4203c0480, 0xc420361f50, 0xc4200944b8)
	/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:742 +0x349
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4203c0000, 0xc4203c0000, 0x158a760)
	/go/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:695 +0x2b
main.main()
	/go/src/github.com/docker/cli/cmd/docker/docker.go:168 +0xc3

Furthermore, a wireshark capture shows that it will connect to the proxy server, but then also tries to DNS lookup docker-server.public.inter.net.
A run with connectivity to both internet and the proxy server does succeed, with the CLI directly accessing the Docker Engine endpoint (as well as using the proxy)

Describe the results you expected:

expected output:

DEBU[0000] Trusting 1 certs                             
success

Additional information you deem important (e.g. issue happens only occasionally):

Performed the same test with a freshly built docker cli https://github.com/docker/cli.git
@ d861a1c . This version appears to completely ignore proxy settings.

Output of docker version:

Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:52:11 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:51:55 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 22
 Running: 11
 Paused: 0
 Stopped: 11
Images: 23
Server Version: 17.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: 3sykgiia0o7vaa34ud21qiaa4
 Is Manager: true
 ClusterID: j8gxz0xg8d26s4xs1e9531sly
 Managers: 3
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 172.31.20.131
 Manager Addresses:
  172.31.1.235:2377
  172.31.20.131:2377
  172.31.36.30:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.36-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.854GiB
Name: ip-172-31-20-131.eu-west-1.compute.internal
ID: WHL5:MYUF:RMPA:LA7D:7YBC:ZZ36:JI56:H4NL:5MAC:S3EW:27D3:IFEI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 160
 Goroutines: 341
 System Time: 2017-09-04T10:06:51.098448233Z
 EventsListeners: 8
Username: redacted
Registry: https://index.docker.io/v1/
Labels:
 os=linux
 region=eu-west-1
 availability_zone=eu-west-1b
 instance_type=t2.medium
 node_type=manager
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
The used Docker Engine is Docker for AWS 17.06.0 CE

NB. First (wrongly) submitted in moby moby/moby#34718

[pdxkzeerkti]

Blocked by what appears to be this issue.

[bastiaanb]

the segfault has already been fixed, presumably in 45b0e7c#diff-8868a5325011d98de3834041d1d2a913

Apparently the Docker CLI does not use a HTTPS proxy but a socks5 proxy for docker attach.
It expects it to be configured in env var ALL_PROXY. E.g.

export ALL_PROXY=socks5://proxy-server:1080

This needs to be properly documented.
Also I would expect Docker to use the HTTP(S) proxy if configured, as often a socks proxy is not available.

[bastiaanb]

@dnephin: I would expect that since the Docker engine exposes an HTTP API, I would only need a HTTP(S) proxy, not a socks proxy.
You have relabled this as a documentation issue rather than a bug. Do you mean that to have 'docker attach' working through an HTTP(S) proxy, would be a new feature request?

[dnephin]

The Docker API is actually not entirely HTTP. There are a few places where it uses https://golang.org/pkg/net/http/#Hijacker and then starts communicating with some other protocol, which makes me think this is not a bug. I'm not exactly sure where that is implemented.

[thaJeztah]

I think @nathanleclaire did some things for the socks5 proxy stuff perhaps he's around to help out 😅

[nathanleclaire]

What's the specific issue? It's that the DNS lookup does not go through the HTTP proxy? Or that attach is not working through an HTTP proxy?

If it's the former, that does strike me as a bug on Docker CLI's side if it doesn't because https://serverfault.com/questions/169816/how-dns-lookups-work-when-using-an-http-proxy-or-not-in-ie suggests that a browser configured to use a HTTP proxy will do the lookup using the proxy when possible. The Docker client should do the same.

If it's the latter, it's likely to stay WONTFIX because even though an attach request starts life as an HTTP request, as Dan noted Docker rapidly hijacks the connection to do the bidirectional communication required for a pseudo-terminal attach. HTTP/1.1 doesn't really work bidirectionally out of the box (see Comet for a variety of ways folks have worked on hacking around this in the browser) so it's very likely that whatever HTTP proxy you're attempting to go through simply doesn't support this. If it did I'd be impressed though.

Is this weird behavior on Docker's end? Sure, but you can't really have your cake and eat it too. HTTP/1.1 is originally designed as a request => response model without supporting bidirectional communication. Docker could try and move to something like HTTP2, whether via gRPC or otherwise, to fix this, but there's no guarantee your proxy will support that either (however, it's likely a better bet than bespoke protocol over HTTP/1.1). Or, Docker could consider changing the API to support separate methods to create a terminal session and read/write to it vs. one big attach (this would make run -it laggier though). However these are both big engineering efforts.

Workaround: Consider avoiding attach when running through a proxy if possible, e.g., by using docker run -d to run containers in the background.

BTW, HTTP_PROXY and ALL_PROXY are separate things. One deals with HTTP, and the other deals with SOCKS traffic, which can support raw TCP as well as (I think) UDP. That's why attach can work over a SOCKS proxy (IIRC - you can verify this by whipping a quick one up with ssh if desired), it's just relaying the raw TCP packets.

[thaJeztah]

Thanks Nathan!

[beornf]

I investigated this issue further and found a viable path toward supporting proxied hijacked connections.

Normal Docker API requests use the HTTP transport's proxy configuration including HTTP_PROXY and ALL_PROXY. However operations such as attach use the hijack path and establish the connection through Moby client's dialer:

• Hijack connection setup: https://github.com/moby/moby/blob/11ee7e07df4203b65b9cb23302aed9f8d81af0b6/client/hijack.go#L33-L43
• Client dialer: https://github.com/moby/moby/blob/11ee7e07df4203b65b9cb23302aed9f8d81af0b6/client/client.go#L425-L444

A possible solution would be to make the hijack connection path proxy-aware. When a HTTP proxy is configured, the client could establish a HTTP CONNECT tunnel to the Docker daemon and then continue with the existing TCP flow over that tunnel.

The CONNECT handling could follow the same general approach used by go-http-dialer:
https://github.com/mwitkow/go-http-dialer/blob/378f744fb2b81a6b96e3f40cde4f3bcab5a9cff0/dialer.go#L96-L138

This would allow attach, exec, and other hijacked operations to behave consistently with normal Docker API requests.

Would you be open to a PR in Moby client implementing this approach?