Skip to content

Minor change to how docker cleans up resources#184

Open
andreroc wants to merge 2 commits intoreleasefrom
nit-4763-adjust-testnode-docker-to-improve-resource-release
Open

Minor change to how docker cleans up resources#184
andreroc wants to merge 2 commits intoreleasefrom
nit-4763-adjust-testnode-docker-to-improve-resource-release

Conversation

@andreroc
Copy link
Copy Markdown

@andreroc andreroc commented Apr 2, 2026

I had claude do some analysis on our docker usage and here is what it found:

Now I have a clear picture. Here's my analysis and fix:

Analysis

The error occurs during --init cleanup at lines 462-467 of test-node.bash. The root cause is a race condition:

  1. docker compose down sends SIGTERM to containers and tries to remove them and the network
  2. blockscout and postgres have restart: always — Docker restarts them immediately after they're stopped, so they're still running when docker compose down tries to remove the network
  3. The subsequent docker rm on line 467 also fails because these containers restarted and are running

The fix has two parts:

  1. Use docker compose down --remove-orphans --timeout 10 to give containers time and clean up orphans.

  2. Change the leftover container cleanup to force-remove (docker rm -f), since any container still alive at this point is one that down couldn't stop (due to restart policy) and needs to
    be killed.

⏺ Update(test-node.bash)
⎿  Added 3 lines, removed 3 lines
464 - docker compose down
464 + docker compose down --remove-orphans --timeout 10 2>/dev/null || true

  467 -        docker rm $leftoverContainers                                                                                                                                     
  467 +        docker rm -f $leftoverContainers                                                                                                                                  

  472 -        docker volume rm $leftoverVolumes                                                                                                                                 
  472 +        docker volume rm -f $leftoverVolumes                                                                                                                              

⏺ Now change restart: always to restart: unless-stopped for blockscout and postgres. This prevents them from restarting during a docker compose down while still auto-restarting on crashes
during normal operation.

⏺ Update(docker-compose.yaml)
⎿  Added 1 line, removed 1 line
6 image: blockscout-testnode
7 - restart: always
7 + restart: unless-stopped

⏺ Update(docker-compose.yaml)
⎿  Added 1 line, removed 1 line
30
31 postgres:
32 image: postgres:13.6
33 - restart: always
33 + restart: unless-stopped

⏺ Summary of changes

Root cause: restart: always on blockscout and postgres causes Docker to restart them immediately after docker compose down sends SIGTERM, so the containers are running again when the
script tries to remove them and the network.

Changes made:

  1. docker-compose.yaml — Changed restart: always to restart: unless-stopped for both blockscout and postgres. This preserves auto-restart during normal operation (crash recovery) but
    allows docker compose down / docker stop to stop them permanently without restart.
  2. test-node.bash (init cleanup block):
    - Added --remove-orphans --timeout 10 to docker compose down — cleans up orphaned containers and gives services 10 seconds for graceful shutdown
    - Added 2>/dev/null || true so a partial failure in down doesn't abort the script (the subsequent force-remove handles stragglers)
    - Changed docker rm to docker rm -f — force-kills any leftover containers that survived down (e.g., from a previous interrupted run)
    - Changed docker volume rm to docker volume rm -f — same defensive approach for volumes still referenced by dying containers

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts the test node Docker teardown behavior to reduce cleanup failures during --init, addressing containers restarting during docker compose down.

Changes:

  • Update docker compose down invocation to remove orphans and use a shorter, explicit shutdown timeout.
  • Force-remove leftover containers and volumes during init cleanup.
  • Change restart policy for blockscout and postgres from always to unless-stopped to prevent restart during teardown.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
test-node.bash Makes init cleanup more aggressive (remove-orphans/timeout + force removal) to avoid teardown races.
docker-compose.yaml Adjusts restart policies for key services to reduce restart-induced teardown failures.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@andreroc andreroc assigned joshuacolvin0 and unassigned andreroc Apr 2, 2026
- Add try/finally around all WebSocket provider lifecycles to prevent
  connection leaks on errors (ethcommands.ts)
- Replace fragile `tail | tr` output parsing with capture_output and
  capture_last_field helpers that fail on empty results (test-node.bash)
- Replace `sleep 45` with deterministic wait_for_chain_progress that
  confirms block production before proceeding (test-node.bash)
- Add timeouts to previously-infinite polling loops in bridgeFunds,
  bridgeNativeToken, and waitForSync (ethcommands.ts)
- Add Docker healthchecks for geth, sequencer, redis, postgres and use
  service_healthy conditions in depends_on for proper startup ordering
  (docker-compose.yaml)
- Track background PIDs and clean up on exit via trap (test-node.bash)
- Add Redis connection cleanup via finally blocks (redis.ts)
- Separate l3node-data volume from validator-data (docker-compose.yaml)
- Replace sed-based JSON rewriting with jq for timeboost config
- Move S3 credential warning to point of use instead of module level
- Add .gitignore, enable set -euo pipefail, fix --dev-contracts shift
  bug, fix yargs l3 boolean type, validate flag dependencies post-parse

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joshuacolvin0
Copy link
Copy Markdown
Member

Added a big update, go ahead and test and see what you think

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants