Close the #796 auth gaps in docs and tooling

[kvinwang]

Problem

#796 added a shared dstack-api-auth crate that fail-closes and protects the full external surface of VMM ([auth]), Gateway admin ([core.admin]), and KMS admin (admin_token_hash). The code landed correctly, but the docs and tools never caught up, so several official paths still produce or describe an unauthenticated deployment:

• Flagship deploy guide is insecure. docs/deployment.md binds address = "tcp:0.0.0.0:9080" with no [auth] section at all — post-feat(auth): protect VMM with shared API authentication #796 that exposes the entire VMM control API (create/stop CVM, UI, pRPC) to the network with no credential. The doc's "Auth Server Options" only covers the KMS boot webhook, which reads as if auth were handled.
• dstackup ships auth off. The generated vmm.toml hardcoded [auth] enabled = false / tokens = [], so the automated installer's default contradicts the hand-written tutorial path (which generates a token and enables auth).
• No VMM warning. dstack-vmm silently allowed enabled = false on a non-loopback bind — no log, no localhost check (Gateway warns on its equivalent insecure_no_auth).
• Tools can't authenticate. dstack-cli's --token flag was accepted then discarded (let _ = &cli.token;), so it could not talk to an auth-enabled remote VMM. vmm-cli.py supported only Basic and required both user and password — setting only DSTACK_VMM_AUTH_PASSWORD (as several tutorials did) silently sent no auth header, so requests went out unauthenticated and failed confusingly.
• htpasswd_file / gateway & KMS admin auth undocumented. The new htpasswd support and the KMS admin_token_hash appear nowhere in any user-facing deploy/tutorial/security doc; docs/dstack-gateway.md never mentions [core.admin].
• Test configs unlabeled. Hardcoded admin_token = "e2e-admin-token", insecure_no_auth = true, and admin_token_hash = "00" carried no "do not use in production" marker.

Fix

Tools now authenticate, the installer is secure-by-default, the server warns, and every operator-facing doc explains the auth added by #796.

• dstackup install mints a 256-bit token (/dev/urandom, written 0600 to <config>/vmm-auth-token), renders [auth] enabled = true with it, records client_token_path in the install state, and threads the token through every internal VMM connect (preflight, reachability, KMS deploy, destroy). Re-runs reuse the existing token so credentials stay stable.
• dstack-cli wires --token for real (precedence: --token → DSTACK_VMM_TOKEN → the installed token file), sending Authorization: Bearer over both the pRPC client and the /logs route. http-client gained header support + a prpc bearer token.
• vmm-cli.py adds --token / DSTACK_VMM_TOKEN (Bearer), and now errors on a half-set Basic credential instead of silently sending nothing.
• dstack-vmm logs a warn! when the management API binds a non-loopback TCP address while [auth] enabled = false.
• Docs: deployment.md gains a real [auth] block + prose; new "Admin API authentication" sections in docs/dstack-gateway.md, dstack/kms/README.md, and docs/security/security-best-practices.md; the password-only tutorial snippets are fixed (add DSTACK_VMM_AUTH_USER=admin / switch to DSTACK_VMM_TOKEN); vmm-cli-user-guide.md documents the token form and drops the stale "guards only /logs" framing; CHANGELOG.md records feat(auth): protect VMM with shared API authentication #796.
• Test configs get TEST ONLY - do not use in production annotations.

Verification

• cargo build + cargo test for dstackup, dstack-cli-core, dstack-cli, dstack-vmm, http-client — all pass; cargo clippy clean; cargo fmt applied.
• New unit tests assert the vmm.toml renders enabled = true + tokens = ["…"] when a token is set and a valid empty-token list otherwise.
• vmm-cli.py: python -m py_compile passes; verified VmmClient raises on user-only and password-only credentials, and constructs cleanly for token-only and full-Basic.
• Cross-checked the KMS route guards: ensure_admin protects clear_image_cache only (onboarding is attestation-gated), so the deployment.md admin-auth note is scoped to admin RPCs, not onboarding.

End-to-end verification on a TDX host

Ran a real dstack-vmm against dstackup-generated artifacts in an isolated prefix (non-default ports, --no-start so no systemd units and no interference with the host's existing install), then exercised the full auth round-trip and cleaned up.

Generated artifacts:

• vmm-auth-token written -rw------- (0600), 64 hex chars.
• vmm.toml rendered [auth] enabled = true with tokens = ["<same token>"].

Round-trip against a live VMM on 127.0.0.1:19080 (auth fairing active):

Case	Expect	Result
curl, no token	401	✅ 401
curl, wrong token	401	✅ 401
curl, correct Authorization: Bearer	200	✅ 200
curl, X-Admin-Token header	200	✅ 200
dstack --token	ok	✅
dstack, no token	denied	✅ 401
dstack via DSTACK_VMM_TOKEN	ok	✅
vmm-cli.py --token	ok	✅
vmm-cli.py, password only	error	✅ fail-fast error
vmm-cli.py, user only	error	✅ fail-fast error
vmm-cli.py, Basic admin:token	ok	✅
dstack --prefix (auto-reads token from install state, no --token/--host)	ok	✅
control: state points at a missing token file	401	✅ 401 (confirms the token is what authorizes)

The full installer chain (dstackup install → running VMM → KMS-in-CVM) still needs a systemd-level install and is not covered here.

[Copilot]

Pull request overview

This PR aligns dstack’s operator-facing docs and tooling with the post-#796 “auth guards the full management surface” reality, making installs and clients authenticate correctly by default and warning operators when the VMM is exposed without auth.

Changes:

• Secure-by-default dstackup install: generates/reuses a VMM management API bearer token, renders [auth] enabled = true, persists client_token_path, and threads the token through internal VMM calls (incl. destroy).
• Tooling support for auth-enabled VMMs: dstack-cli and vmm-cli.py now send Authorization: Bearer <token> (and vmm-cli.py fails fast on half-configured Basic auth).
• Docs + test config hygiene: deployment/security/gateway/KMS docs document the auth surfaces; test configs are annotated as “TEST ONLY”.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
dstack/vmm/src/vmm-cli.py	Add bearer token support; reject incomplete Basic creds; wire --token/DSTACK_VMM_TOKEN.
dstack/vmm/src/main.rs	Warn when binding the management API publicly with [auth] enabled = false.
dstack/test-scripts/snp-e2e-smoke.sh	Annotate weak test-only KMS admin credential.
dstack/kms/README.md	Document KMS admin authentication configuration (token hash).
dstack/http-client/src/prpc.rs	Add bearer token support for pRPC client requests.
dstack/http-client/src/lib.rs	Add request helper that can attach extra headers.
dstack/gateway/test-run/test_suite.sh	Annotate test-only insecure_no_auth.
dstack/gateway/test-run/e2e/configs/gateway-{1,2,3}.toml	Annotate test-only hardcoded admin token + debug RPC enablement.
dstack/crates/dstackup/src/state.rs	Persist client_token_path in install state.
dstack/crates/dstackup/src/install.rs	Generate/reuse token, write it to disk, enable [auth], and pass token into VMM connectivity checks.
dstack/crates/dstackup/src/destroy.rs	Use recorded token file to authenticate VMM calls during teardown.
dstack/crates/dstack-cli/src/main.rs	Wire --token/env/token-file fallback into VMM connects and /logs.
dstack/crates/dstack-cli-core/src/vmm.rs	Add connect_with_token and send bearer auth for both pRPC and /logs.
dstack/crates/dstack-cli-core/src/config.rs	Render [auth] enabled/tokens into generated vmm.toml + add unit tests.
docs/vmm-cli-user-guide.md	Update auth guidance to reflect full-surface auth + new token support.
docs/deployment.md	Add required [auth] block and explain surface-wide VMM auth and KMS admin auth distinction.
docs/dstack-gateway.md	Document gateway admin API authentication configuration.
docs/security/security-best-practices.md	Add management/admin API auth best practices section.
docs/tutorials/*	Fix tutorial snippets that previously sent no auth due to password-only Basic env.
CHANGELOG.md	Add unreleased entry documenting the #796 auth surface.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.