If deploy fails, rollback state

Author: wadewilliams

SPEC.md

@@ -29,7 +29,19 @@ Every service in `docker-compose.yml` is classified by a label:
 
 ### 2.2 Deploy Lifecycle
 
-For each service with `deploy.role=app`, in the order they appear in the compose file:
+The deploy runs as a single transaction — `flow-deploy deploy --tag <sha>` owns the full lifecycle including git operations. The `--tag` value serves double duty: it is both the Docker image tag and the git SHA to checkout.
+
+**Pre-flight and git checkout (before any service work):**
+
+```
+0a. Dirty check             git status --porcelain
+                             If non-empty → log "working tree is dirty — deploy aborted", exit 1
+0b. Fetch                   git fetch origin
+0c. Record previous SHA     previous_sha = git rev-parse HEAD
+0d. Checkout (detached)     git checkout --detach <sha>
+```
+
+**For each service with `deploy.role=app`, in the order they appear in the compose file:**
 
 ```
 1.  Pull new image           <compose-command> pull <service>
@@ -43,9 +55,14 @@ For each service with `deploy.role=app`, in the order they appear in the compose
 4b. If unhealthy:
       Stop new container     docker stop <new_id> && docker rm <new_id>
       Scale back to 1        <compose-command> up -d --no-deps --scale <service>=1
+      Restore repo           git checkout --detach <previous_sha>
       ✗ Abort deploy, exit 1
 ```
 
+**On success:** the server is in detached HEAD at `<sha>`. Log `HEAD detached at <sha>`.
+
+**On failure:** the repo is restored to `<previous_sha>` before exiting. The invariant is preserved: `git rev-parse HEAD` always matches the image SHA that is actively serving traffic.
+
 Where `<compose-command>` is the project's compose wrapper (see §3.1).
 
 ### 2.3 Graceful Shutdown
@@ -398,9 +415,16 @@ Failure output:
 [12:37:14]   rollback complete, old container still serving
 [12:37:14]   ✗ worker FAILED
 [12:37:14]
+[12:37:14]   restoring repo to a1b2c3d...
 [12:37:14] ── FAILED (deploy aborted) ─────────────
 ```
 
+Dirty-tree output:
+
+```
+[12:34:56] ERROR: working tree is dirty — deploy aborted
+```
+
 ### 6.1 GitHub Actions Integration
 
 Since the tool runs over SSH, output naturally appears in Actions logs. For richer integration, the tool emits GitHub Actions log commands when it detects the `GITHUB_ACTIONS=true` environment variable (passed through SSH):
@@ -421,7 +445,7 @@ For multi-host deploys or when you want host discovery from compose labels, use
 2. Runs `<command> config` to get the fully merged compose YAML
 3. Parses `x-deploy` and `deploy.*` labels to discover hosts
 4. Groups services by host
-5. SSHes to each host: `git pull` → `flow-deploy deploy --tag <tag>`
+5. SSHes to each host: `flow-deploy deploy --tag <tag>`
 6. Streams logs back to GitHub Actions
 
 ```yaml
@@ -504,11 +528,10 @@ For single-host projects, the action is optional. A raw SSH command works:
         run: |
           ssh -o StrictHostKeyChecking=no deploy@${{ secrets.PROD_HOST }} \
             "cd /srv/myapp && \
-             git fetch && git checkout -B main origin/main && \
              GITHUB_ACTIONS=true flow-deploy deploy --tag ${{ needs.build.outputs.tag }}"
 ```
 
-This is the simplest possible deploy: one SSH call, no action, no host discovery. The tool runs locally on the server, calls `script/prod`, and handles the rolling deploy.
+This is the simplest possible deploy: one SSH call, no action, no host discovery. The tool handles git operations (fetch, detached checkout), calls `script/prod`, and runs the rolling deploy. No separate `git pull` or `git checkout` is needed — the tool owns the full transaction.
 
 ---
 

docs/github-actions.md

@@ -8,15 +8,15 @@ The deploy pipeline:
 
 1. CI builds your Docker image and pushes to GHCR
 2. The deploy action discovers hosts from your `docker-compose.yml`
-3. For each host: authenticates with GHCR, pulls the repo, and runs `flow-deploy deploy`
+3. For each host: authenticates with GHCR and runs `flow-deploy deploy` (which handles git fetch and checkout internally)
 
 ## Prerequisites
 
 On your deploy server:
 
 - Docker and Docker Compose
 - Traefik (or your reverse proxy) running
-- Git (the server repo is updated via `git pull --ff-only` before each deploy)
+- Git (`flow-deploy` handles `git fetch` and detached checkout internally during each deploy)
 - `flow-deploy` installed:
 
 ```sh
@@ -164,8 +164,7 @@ For each host group discovered from your compose config:
 1. **SSH agent** — loads your deploy key
 2. **Discover hosts** — parses `docker-compose.yml` for `x-deploy` and `deploy.*` labels, groups services by `(host, user, port, dir)`
 3. **GHCR login** — authenticates Docker on the server (and logs out after)
-4. **Git pull** — fast-forward only, fails safely if the server has diverged
-5. **Deploy** — runs `flow-deploy deploy --tag <tag>` on the server
+4. **Deploy** — runs `flow-deploy deploy --tag <tag>` on the server (git fetch and detached checkout are handled by the tool)
 
 ## Host Discovery
 
@@ -221,8 +220,8 @@ To cut releases with binaries and changelogs, see the release workflow in this r
 
 ## Troubleshooting
 
-**`git pull --ff-only` fails:**
-The server repo has diverged from the remote. SSH into the server and resolve manually — check for local commits or uncommitted changes.
+**`working tree is dirty — deploy aborted`:**
+The server repo has uncommitted changes. SSH into the server and resolve manually — `git status` will show what's dirty.
 
 **`unauthorized` pulling from GHCR:**
 Pass `registry-token: ${{ secrets.GITHUB_TOKEN }}` to the deploy action. The job needs `packages: write` (or at least `packages: read`) permission.

src/flow_deploy/deploy.py

@@ -3,7 +3,7 @@
 import signal
 import time
 
-from flow_deploy import compose, config, containers, lock, log, tags
+from flow_deploy import compose, config, containers, git, lock, log, tags
 
 
 def deploy(
@@ -12,7 +12,7 @@ def deploy(
     dry_run: bool = False,
     cmd: list[str] | None = None,
 ) -> int:
-    """Perform a rolling deploy. Returns exit code (0=success, 1=failure, 2=locked)."""
+    """Perform a rolling deploy. Returns exit code (0=success, 1=failure, 2=locked, 3=skipped)."""
     compose_cmd = cmd or compose.resolve_command()
 
     # Parse compose config
@@ -47,11 +47,18 @@ def deploy(
         _dry_run(tag, app_services)
         return 0
 
+    # Git pre-flight: dirty check, fetch, checkout detached
+    git_code, previous_sha = git.preflight_and_checkout(tag)
+    if git_code != 0:
+        return 1
+
     # Acquire lock
     if not lock.acquire():
         lock_info = lock.read_lock()
         pid = lock_info["pid"] if lock_info else "unknown"
         log.error(f"Deploy lock held by PID {pid}")
+        # Restore repo to previous state before exiting
+        git.restore(previous_sha)
         return 2
 
     # Register signal handlers for cleanup
@@ -80,6 +87,7 @@ def _cleanup_handler(signum, frame):
             result = _deploy_service(svc, tag, compose_cmd, project=project)
             if result != 0:
                 log.info("")
+                git.restore(previous_sha)
                 log.footer("FAILED (deploy aborted)")
                 lock.release()
                 return 1
@@ -88,6 +96,7 @@ def _cleanup_handler(signum, frame):
         tags.write_tag(tag)
 
         log.info("")
+        log.info(f"HEAD detached at {tag}")
         log.footer(f"complete ({elapsed:.1f}s)")
     finally:
         lock.release()

src/flow_deploy/git.py

@@ -0,0 +1,65 @@
+"""Git operations for detached-HEAD deploy strategy."""
+
+from flow_deploy import log, process
+
+
+def is_dirty() -> bool:
+    """Return True if the working tree has uncommitted changes."""
+    result = process.run(["git", "status", "--porcelain"])
+    return bool(result.stdout.strip())
+
+
+def fetch() -> process.Result:
+    """Fetch from origin."""
+    return process.run(["git", "fetch", "origin"])
+
+
+def current_sha() -> str:
+    """Return the current HEAD SHA."""
+    result = process.run(["git", "rev-parse", "HEAD"])
+    return result.stdout.strip()
+
+
+def checkout_detached(sha: str) -> process.Result:
+    """Checkout a specific SHA in detached HEAD mode."""
+    return process.run(["git", "checkout", "--detach", sha])
+
+
+def preflight_and_checkout(tag: str) -> tuple[int, str | None]:
+    """Run git pre-flight checks and checkout the deploy SHA.
+
+    Returns (exit_code, previous_sha).
+    - (0, previous_sha) on success — repo is now at `tag` in detached HEAD.
+    - (1, None) on error — git operation failed or working tree is dirty.
+    """
+    # 1. Dirty check
+    if is_dirty():
+        log.error("working tree is dirty — deploy aborted")
+        return 1, None
+
+    # 2. Fetch
+    result = fetch()
+    if result.returncode != 0:
+        log.error(f"git fetch failed: {result.stderr.strip()}")
+        return 1, None
+
+    # 3. Record previous SHA
+    previous_sha = current_sha()
+
+    # 4. Checkout new SHA (detached HEAD)
+    result = checkout_detached(tag)
+    if result.returncode != 0:
+        log.error(f"git checkout failed: {result.stderr.strip()}")
+        return 1, None
+
+    return 0, previous_sha
+
+
+def restore(previous_sha: str) -> bool:
+    """Restore repo to a previous SHA after a failed deploy."""
+    log.step(f"restoring repo to {previous_sha[:7]}...")
+    result = checkout_detached(previous_sha)
+    if result.returncode != 0:
+        log.error(f"git restore failed: {result.stderr.strip()}")
+        return False
+    return True

tests/test_deploy.py

@@ -6,6 +6,7 @@
 from flow_deploy.deploy import deploy, rollback
 
 COMPOSE_CMD = ["docker", "compose"]
+PREV_SHA = "prev123abc"
 
 COMPOSE_CONFIG_YAML = """\
 services:
@@ -70,13 +71,25 @@ def _err(stderr="error"):
     return process.Result(1, "", stderr)
 
 
+def _git_preflight():
+    """Return the 4 mock responses for a clean git preflight."""
+    return [
+        _ok(""),  # git status --porcelain (clean)
+        _ok(),  # git fetch origin
+        _ok(PREV_SHA + "\n"),  # git rev-parse HEAD
+        _ok(),  # git checkout --detach <sha>
+    ]
+
+
 def _setup_happy_path(mock_process, monkeypatch, tmp_path):
     """Set up mock responses for a successful 2-service deploy."""
     monkeypatch.chdir(tmp_path)
     mock_process.responses.extend(
         [
             # compose config
             _ok(COMPOSE_CONFIG_YAML),
+            # git preflight
+            *_git_preflight(),
             # web: pull
             _ok(),
             # web: scale to 2
@@ -124,6 +137,7 @@ def test_deploy_service_filter(mock_process, monkeypatch, tmp_path):
     mock_process.responses.extend(
         [
             _ok(COMPOSE_CONFIG_YAML),
+            *_git_preflight(),
             # web only: pull, scale, ps, health, stop, rm, scale back
             _ok(),
             _ok(),
@@ -158,6 +172,7 @@ def test_deploy_health_check_failure(mock_process, monkeypatch, tmp_path):
     mock_process.responses.extend(
         [
             _ok(COMPOSE_CONFIG_YAML),
+            *_git_preflight(),
             # web: pull
             _ok(),
             # web: scale to 2
@@ -170,6 +185,8 @@ def test_deploy_health_check_failure(mock_process, monkeypatch, tmp_path):
             _ok(),
             # web: scale back to 1
             _ok(),
+            # git restore to previous SHA
+            _ok(),
         ]
     )
     result = deploy(tag="abc123", cmd=COMPOSE_CMD)
@@ -181,7 +198,10 @@ def test_deploy_pull_failure(mock_process, monkeypatch, tmp_path):
     mock_process.responses.extend(
         [
             _ok(COMPOSE_CONFIG_YAML),
+            *_git_preflight(),
             _err("pull failed"),
+            # git restore to previous SHA
+            _ok(),
         ]
     )
     result = deploy(tag="abc123", cmd=COMPOSE_CMD)
@@ -190,7 +210,14 @@ def test_deploy_pull_failure(mock_process, monkeypatch, tmp_path):
 
 def test_deploy_lock_held(mock_process, monkeypatch, tmp_path):
     monkeypatch.chdir(tmp_path)
-    mock_process.responses.append(_ok(COMPOSE_CONFIG_YAML))
+    mock_process.responses.extend(
+        [
+            _ok(COMPOSE_CONFIG_YAML),
+            *_git_preflight(),
+            # git restore after lock rejection
+            _ok(),
+        ]
+    )
     # Pre-acquire lock with current PID
     from flow_deploy import lock
 
@@ -245,10 +272,12 @@ def test_deploy_container_count_mismatch(mock_process, monkeypatch, tmp_path):
     mock_process.responses.extend(
         [
             _ok(single_svc_config),
+            *_git_preflight(),
             _ok(),  # pull
             _ok(),  # scale to 2
             _ok(WEB_CONTAINER_OLD + "\n"),  # only 1 container returned
             _ok(),  # scale back to 1
+            _ok(),  # git restore
         ]
     )
     result = deploy(tag="abc123", cmd=COMPOSE_CMD)
@@ -286,6 +315,7 @@ def test_rollback(mock_process, monkeypatch, tmp_path):
     mock_process.responses.extend(
         [
             _ok(single_svc_config),
+            *_git_preflight(),
             _ok(),
             _ok(),
             _ok(WEB_CONTAINER_OLD + "\n" + WEB_CONTAINER_NEW + "\n"),
@@ -299,6 +329,21 @@ def test_rollback(mock_process, monkeypatch, tmp_path):
     assert result == 0
 
 
+def test_deploy_dirty_tree_fails(mock_process, monkeypatch, tmp_path, capsys):
+    monkeypatch.chdir(tmp_path)
+    mock_process.responses.extend(
+        [
+            _ok(COMPOSE_CONFIG_YAML),
+            # git status --porcelain returns dirty
+            _ok(" M somefile.py\n"),
+        ]
+    )
+    result = deploy(tag="abc123", cmd=COMPOSE_CMD)
+    assert result == 1
+    err = capsys.readouterr().err
+    assert "dirty" in err
+
+
 def test_rollback_no_previous(monkeypatch, tmp_path):
     monkeypatch.chdir(tmp_path)
     result = rollback(cmd=COMPOSE_CMD)