Skip to content

Extend test coverage: more agent endpoints + Playwright UI suite#248

Merged
misterbh-saashup merged 8 commits into
mainfrom
test_improvements
Jun 2, 2026
Merged

Extend test coverage: more agent endpoints + Playwright UI suite#248
misterbh-saashup merged 8 commits into
mainfrom
test_improvements

Conversation

@misterbh-saashup
Copy link
Copy Markdown
Collaborator

@misterbh-saashup misterbh-saashup commented Jun 2, 2026

Extend test coverage: more agent endpoints + a Playwright UI suite

Builds on the dockerd version-compat harness (#247) with two layers of new coverage:

  1. More hurl tests for agent behaviour that had no coverage, wired into the existing per-version matrix.
  2. A Playwright end-to-end suite for the web dashboard (public/index.html), which had no tests at all.

All green against dockerd 29.5.2 / 28.5.2 / 27.5.1 (hurl) and 29.5.2 (UI).

New hurl tests (run in the version matrix)

File Coverage
volumes.hurl real volume lifecycle: create → verify → delete
networks.hurl real network lifecycle (create is gated on state "creating")
images.hurl image pull → verify → remove
container-ops.hurl kill / restart / rename (lifecycle.hurl already covers start/stop)
metrics.hurl asserts the Prometheus /metrics content, not just that it is a string
errors.hurl bad-credentials → 401; unknown resource type handled gracefully
events.hurl the dockerd event watcher: pause a container directly on the daemon (out-of-band) and assert the agent syncs the new state to netbox

New Playwright UI suite (tests/e2e/, 20 tests)

Drives the real dashboard against a running agent + daemon via the official Playwright image on the compose network (auth via httpCredentials).

  • P1 navigation — loads with no JS errors, tabs switch, Home renders host info + the usage chart.
  • P2 lists — a resource seeded via the API shows in the matching UI list with the right count.
  • P3 container actions — Start/Stop/Kill/Restart buttons drive real ops (click → modal → verify daemon state).
  • P4 logs — the Logs tab renders container logs.
  • P5 exec & terminal — the Exec tab runs a command; the Terminal tab opens the xterm.js websocket console and echoes a typed command.
  • P6 images — per-image Remove / Pull buttons.
  • P7 notifications — the notification modal shows then auto-dismisses.

Things the tests surfaced (worth a look, separate from this PR)

  • Network create is not broken — it just requires state: "creating" (containers/volumes use "none"). An earlier "bug" suspicion was a wrong-parameter false alarm.
  • POST /api/engine/registries has no handler (the POST route table omits registries) — so a registry "contract" test isn't possible; registries are read-only from the agent's side.
  • The exec endpoint expects cmd, not Cmd — sending the raw Docker exec config silently runs nothing; the lifecycle exec test now sends the right shape and asserts real output.
  • Container delete has no force — it can't remove a running container; the UI correctly disables Remove while running, but cleanup has to force-remove on the daemon.
  • A kill issued immediately after restart can race the agent's async restart/wait (container-ops orders kill before restart to avoid it).

How to run

# hurl matrix (per version)
./tests/compat/run.sh            # all versions, or pass one e.g. 29.5.2

# UI suite (needs the compat stack up)
cd tests/e2e && NETWORK=nda-compat_default ./run.sh

The UI suite is intentionally kept out of the per-version matrix — it talks to the agent API, so it's version-agnostic and running it once is enough.

houinbe added 5 commits June 2, 2026 19:48
Extend test coverage: volumes, image removal, metrics, error paths
db16b28 
- volumes.hurl: real volume lifecycle (create -> verify -> delete).
- images.hurl: real image pull -> verify -> remove (busybox).
- metrics.hurl: assert the Prometheus /metrics content (daemon-connectivity
  gauge with HELP/TYPE and value, netbox error counter) instead of just
  checking it is a string.
- errors.hurl: negative paths — a bad-credentials request is rejected (401)
  and an unknown resource type is handled gracefully (202, no crash).

Wired into run.sh, the compose tester command, and the compat README.
All green on dockerd 29.5.2.
Add dockerd event-watcher test
5cb44db 
events.hurl verifies the agent's /events watcher: it pauses a container
directly on the daemon over its TCP endpoint (out-of-band, not via the agent
API) and asserts the agent detects the event, looks the container up in
netbox, and PATCHes the new "paused" state back — proving the agent reacts to
actions taken directly on dockerd.

Runs in the isolated (restarted-agent) phase alongside netbox.hurl; run.sh and
the compose tester now pass a `dind` variable (the daemon's TCP API). Uses
pause/unpause because the agent always processes those events. Green on 29.5.2.
Add container-ops and network lifecycle tests
4ac62fa 
- container-ops.hurl (I): the container operations beyond start/stop —
  kill, restart, rename — asserting each state transition. kill is ordered
  before restart so the agent's async restart/wait can't race the kill.
- networks.hurl: a real network lifecycle (create -> verify -> delete).
  The network create path is gated on state "creating" (containers/volumes
  use "none"); the earlier "network create is broken" report was a wrong-
  parameter false alarm, not an agent bug.

Registry contract (L) was dropped: POST /api/engine/registries has no handler
(the POST route table omits registries), so there is no create behaviour or
netbox callback to assert.

All green on dockerd 29.5.2.
Add Playwright UI end-to-end tests (P1-P3)
1001194 
Browser tests for the agent dashboard (public/index.html), driving the real UI
against a running agent + daemon:

- 01-navigation: page loads with no JS errors, the title is correct, every tab
  switches sections, and Home renders host info + the system-usage chart.
- 02-lists: a container/image/network seeded via the agent API shows up in the
  matching UI list with the right count.
- 03-container-actions: the Start/Stop/Kill/Restart buttons in a container card
  drive real operations — the test clicks the button, confirms the notification
  modal, and verifies the resulting daemon state via the API.

Runs via the official Playwright image (browsers bundled) on the compat compose
network; tests/e2e/run.sh points BASE_URL at the agent. Auth is supplied with
Playwright httpCredentials. 14/14 green against dockerd 29.5.2.

Card action buttons are located by their onclick handler (the visible label
carries a font-awesome icon and "disabled" is a CSS class, not the attribute),
which makes role+name matching unreliable.
Add Playwright UI tests P4-P7 (logs, exec/terminal, images, notificat…
810828d 
…ions)

- 04-logs: the Logs tab fetches and renders a container's log output.
- 05-exec-terminal: the Exec tab runs a one-shot command and shows stdout; the
  Terminal tab opens an interactive xterm.js console over the websocket exec
  channel and echoes a typed command (waits for the shell prompt before typing
  so early keystrokes are not dropped).
- 06-images: the per-image Remove and Pull buttons drive real delete / re-pull.
- 07-notifications: the notification modal shows with the right title/message
  for an operation, then auto-dismisses.

Cleanup force-removes containers straight on the daemon (the agent's delete has
no force, so it cannot remove a running container); raised the per-test timeout
for these container/image + browser flows. 20/20 green on dockerd 29.5.2.
houinbe added 3 commits June 2, 2026 20:17
CI: run the Playwright UI tests
8d354a3 
Add a "Web UI (Playwright)" job to pr_ci.yml: it stands up the agent stack
(dind + netbox + agent) for the first pinned dockerd version, runs the e2e
suite via tests/e2e/run.sh against the agent, uploads Playwright artifacts on
failure, and tears the stack down. The UI is version-agnostic (it talks to the
agent API), so it runs once rather than across the matrix.
e2e: tolerate ApexCharts' intermittent load-time error
65da510 
The navigation smoke test asserted zero uncaught JS errors, but ApexCharts
intermittently throws "Cannot read properties of undefined (reading 'type')"
from inside its own create() while drawing the system-usage chart (the chart
still renders). This surfaced as a flaky CI failure. Filter that one known
error so the check still catches any new uncaught errors. TODO noted to revisit
the chart setup in public/index.html.
e2e: link the tolerated ApexCharts error to issue #249
f816568
@misterbh-saashup misterbh-saashup mentioned this pull request Jun 2, 2026
Merged
@misterbh-saashup misterbh-saashup merged commit 0ab4a4e into main Jun 2, 2026
8 checks passed
@misterbh-saashup misterbh-saashup deleted the test_improvements branch June 2, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lvenier lvenier lvenier approved these changes

@fanshan fanshan Awaiting requested review from fanshan

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@misterbh-saashup @lvenier