Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions docs/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,12 @@ For debugging against a server you started yourself, set
`E2E_PUBLIC_BASE_URL` as needed. Do not use reuse mode for CI or for
regression runs that need a clean database.

Local trace and video capture are opt-in because a complete run keeps many SSE
connections in one worker and can otherwise accumulate gigabytes of temporary
artifacts before Playwright discards passing tests. Set `E2E_TRACE=1` and/or
`E2E_VIDEO=1` for a focused debugging run. CI still records trace/video on the
first retry automatically.

### Suite structure

- **`tests/e2e/helpers/`** — small, user-behaviour-shaped helpers (setup/login,
Expand All @@ -71,19 +77,28 @@ regression runs that need a clean database.
disposable DB is set up once per run, so first-run setup happens here (proving
SETUP-001) and the owner's authenticated `storageState` is saved. Every spec
depends on it.
- **`dashboard.e2e.ts` preflight** — runs immediately after setup, before any
persona/plugin/content mutations, so clean-install onboarding facts stay
deterministic. The normal E2E project excludes this file to avoid a second run.
- **`account-persona.setup.ts`** — the next setup project creates the
dedicated Admin used by `account.e2e.ts`. Password changes, MFA changes, and
sign-out-everywhere flows revoke other sessions for their target account; the
persona keeps those intentional side effects away from the saved owner session.
- **Session rule.** Specs default to the shared owner `storageState` (fast).
Specs that run a **step-up-gated action** (publish, profile basics, etc.) or
**sign out** rotate the session token server-side, so they opt into
`ANONYMOUS_STATE` and `login()` fresh — otherwise they would invalidate the
shared state for later specs.
`ANONYMOUS_STATE` and log in fresh — otherwise they would invalidate the shared
state for later specs. Account-global self-management specs log in as the
account persona, never the owner.
- **Selectors.** Durable user-facing selectors first (roles, labels, accessible
names). `data-testid` only for stable editor/canvas controls where an
accessible name is not practical (canvas notch, toolbar publish actions, the
step-up dialog).
- **Isolation.** With `workers: 1` all specs share one database; each spec works
on its own uniquely-named page/post (only the core lifecycle spec edits the
homepage), and publish→assert happens within a single test so cross-spec order
never matters.
homepage), clean-install dashboard coverage runs before shared mutations,
account-global security changes stay on the account persona, fixture-owned AI
defaults are removed, and publish→assert happens within a single test.

### Automated coverage map

Expand Down Expand Up @@ -111,7 +126,7 @@ Playwright specs unless a focused Bun test path is included:
| CAP-005 (plugin read/install/configure/lifecycle/schedule/pack + AI chat rail + provider/audit tab gates + AI write-tool filtering) | `capabilities.e2e.ts`, `ai.e2e.ts` |
| ADMIN-002 (profile basics), ACCOUNT-001 (display name + profile API edges + mobile cancel/no-op), ACCOUNT-002 (avatar upload/removal/invalid upload feedback/API edges/storage error/mobile layout), ADMIN-003 (MFA setup cancel), ACCOUNT-003 (password change + mobile dialog), AUTH-002 (MFA TOTP/mobile challenge/pending-session API edges/unknown-cookie rejection/empty+wrong-code feedback/recovery-code login/reuse rejection), AUTH-004 (active-device sign-out + mobile table + session API edges), ACCOUNT-004 (MFA invalid-code feedback, QR fallback, enable/login/recovery regenerate/disable + mobile setup/login containment), ACCOUNT-005 (step-up window + mobile controls + disabled/invalid API edges), AUTH-006 (failed/successful login activity feed, disposable-account lockout/rate-limit browser flow, and suspicious activity banner) | `account.e2e.ts`, `src/__tests__/admin/accountPage.test.tsx`, `src/__tests__/server/accountSecurity.test.ts`, `src/__tests__/server/authSessions.test.ts`, `src/__tests__/server/authSessionEdgeCases.test.ts` |
| PLUGIN-001 (ZIP package install via packaged fixtures), PLUGIN-002 (enable/disable/remove lifecycle), PLUGIN-003 (settings/secrets), PLUGIN-004 (packaged admin pages/resources/runtime route), PLUGIN-005 (schedules), PLUGIN-006 (pack install/re-sync), PLUGIN-008 (invalid manifest upload recovery) | `plugins.e2e.ts` |
| AI-001 (Ollama credential create/delete + offline default guard), AI-002 (Data-scope default save/reload/clear), AI-003 (Site chat history load/new/delete), AI-004 (fixture-backed Site assistant streaming), AI-005 (browser `read_document` tool-result bridge), AI-006 (Audit tab rollups from streamed usage) | `ai.e2e.ts` |
| AI-001 (Ollama credential create/delete + offline default guard), AI-002 (Data-scope default save/reload/clear), AI-003 (Site chat history load/new/delete), AI-004 (fixture-backed Site assistant streaming), AI-005 (browser `site_read_document` tool-result bridge), AI-006 (Audit tab rollups from streamed usage) | `ai.e2e.ts` |
| A11Y-001, A11Y-002, RESP-001, RESP-002 | `accessibility.e2e.ts` |
| PERF-001, PERF-002 (performance/reliability smokes) | `performance.e2e.ts` |
| REL-001 | `reliability.e2e.ts` |
Expand Down Expand Up @@ -377,7 +392,7 @@ durable assertion brittle:
are automated in `ai.e2e.ts`; AI-002 Data-scope default save/reload/clear
coverage is automated there too. AI-003 Site chat history load/new/delete,
AI-004 fixture-backed Site assistant streaming, AI-005 browser
`read_document` tool-result bridge, CAP-005 request-level AI write-tool filtering, and AI-006 Audit tab rollups from that
`site_read_document` tool-result bridge, CAP-005 request-level AI write-tool filtering, and AI-006 Audit tab rollups from that
streamed usage are also automated in `ai.e2e.ts`. AI-007 direct
Anthropic/OpenAI/Ollama/OpenRouter driver mapping and pricing coverage is
automated in focused Bun tests; live-provider network behaviour remains
Expand Down
14 changes: 7 additions & 7 deletions docs/e2e/feature-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ USERS-001 note: non-owner user creation, step-up protection, edit, suspend, acti

AUTH-003 note: account-menu logout/login is covered in the core owner lifecycle, and `auth.e2e.ts` automates stale-tab session invalidation by cloning a live session into a second browser context, signing out in the first, then verifying the stale context is forced back to Admin Login on the next authenticated admin route without the stale Site route logging the module-inserter Unauthorized preference-load error. The same spec covers 390px mobile account-menu containment and sign-out reachability. `authSessionEdgeCases.test.ts` covers a repeated stale `POST /logout` returning `{ ok: true }`, clearing the cookie again, and leaving `/me` unauthorized.

ACCOUNT-003 note: password form validation, exact-minimum local validation, mobile password card/dialog layout, step-up-gated password rotation, old-password rejection, new-password login, and shared owner credential restoration are automated in `account.e2e.ts`; multi-session revocation and autofill variants remain lower-level or future browser coverage.
ACCOUNT-003 note: password form validation, exact-minimum local validation, mobile password card/dialog layout, step-up-gated password rotation, old-password rejection, new-password login, and dedicated account-persona credential restoration are automated in `account.e2e.ts`; multi-session revocation and autofill variants remain lower-level or future browser coverage. The account persona isolates password, MFA, and sign-out-everywhere side effects from the shared owner session used by later specs.

AUTH-002 note: TOTP MFA login, mobile MFA challenge layout, empty-code required-field validation, wrong-code feedback, recovery-code login burn, and recovery-code reuse rejection are automated in `account.e2e.ts`; pending-cookie API denial, expired pending-session rejection, and unknown pending-cookie rejection are covered in `accountSecurity.test.ts`.

Expand Down Expand Up @@ -97,7 +97,7 @@ DASH-001 note: `dashboard.e2e.ts` verifies the default owner dashboard route, Ov

DASH-002 note: `dashboard.e2e.ts` verifies customize mode, the Block library, adding the built-in AI usage widget, server-backed `dashboard-layout` preference persistence, reload restoration, grid drag move, right-edge resize, drag-to-library removal, final reload absence, and 390px customize/library containment. DEF-20260623-DASH002-01 fixed invalid nested buttons in Block library live previews by rendering preview widget chrome in edit mode.

DASH-003 note: `dashboard.e2e.ts` verifies onboarding progress from clean E2E state (2/5: identity and first page complete, framework active, plugin/team not started), all five step labels/actions, the Settings modal action, workspace routes for New page/Browse plugins/Add members, 390px mobile containment, and server-backed dismiss persistence through `dashboard-layout`.
DASH-003 note: `dashboard.e2e.ts` runs in a dedicated post-setup, pre-persona project and verifies onboarding progress from genuinely clean E2E state (1/5: identity complete; framework in progress; first page, plugin, and team not started), all five step labels/actions, the Settings modal action, workspace routes for New page/Browse plugins/Add members, 390px mobile containment, and server-backed dismiss persistence through `dashboard-layout`.

## Capabilities And Access Control

Expand Down Expand Up @@ -153,15 +153,15 @@ Visual builder note: `visual-builder.e2e.ts` inserts notch and picker modules, s

SITE-005 note: full module picker coverage is automated in `visual-builder.e2e.ts`. The desktop regression opens Add to canvas, verifies Grid view, searches `button`, inserts Button with Enter, verifies the Button layer, reopens Recent to find the inserted item, switches to List view, closes/reopens, and verifies the persisted view preference. The drag regression filters for Text, drags the picker item into the center of an existing canvas Container, verifies the inside drop preview, edits the inserted Text, and verifies it renders inside the Container. The 390px mobile regression verifies the Add to canvas dialog, Search modules field, Modules/Recent category buttons, and filtered Button item stay viewport-contained without page-level horizontal overflow, then inserts Button with Enter and verifies the Layer row. Lower-level module-inserter suites cover corrupted localStorage fallback, recent dedupe, server-backed favorites, disabled/hidden module availability, and responsive category accessible names.

BUILDER-005 note: undo/redo history is automated in `visual-builder.e2e.ts`. The regression creates and reloads a disposable page to prove clean loaded history state, inserts Text, uses Ctrl/Cmd+Z and Ctrl/Cmd+Shift+Z to undo/redo the visible layer, uses the notch Undo button, inserts Container to prove Redo is cleared by a new edit, then saves/reloads and verifies the saved Container persists while transient history controls reset.
BUILDER-005 note: undo/redo history is automated in `visual-builder.e2e.ts`. The regression creates and reloads a disposable page to prove clean loaded history state, inserts Text, uses Ctrl/Cmd+Z and Ctrl/Cmd+Shift+Z to undo/redo the visible layer, uses the notch Undo button, inserts Container to prove Redo is cleared by a new edit, then saves/reloads and verifies the saved Container persists while transient history controls reset. Focused store regressions also prove undo/redo prunes selection ids for nodes removed by restored history, so the next insertion cannot silently target a stale node.

BUILDER-007 note: breakpoint-scoped selector styles are automated in `visual-builder.e2e.ts`; the regression verifies desktop/mobile canvas frame separation and published CSS at default and 360px visitor widths.

BUILDER-008 note: rich-body bold/italic persistence and public entry-template rendering are automated in `content.e2e.ts`; custom binding confusion, media/unsafe HTML sanitization, and visual typography review remain agent-run or lower-level.

SITE-017 note: `visual-builder.e2e.ts` componentizes a Text node, adds a Slot Outlet in VC mode, returns to the page, inserts Text into the locked generated slot, saves, publishes, and verifies an anonymous visitor sees both component body and slot fill while editor-only slot labels/component names stay absent. DEF-20260623-SITE017-001 fixed the page/component incremental-save ordering bug by writing component rows before page rows.

SITE-018 note: `visual-builder.e2e.ts` creates a high-priority Posts template from the Site panel, sets Template settings to Post types/Posts, inserts `{currentEntry.title}` through the binding picker, verifies synthetic canvas preview for title/body, saves and publishes the template snapshot, publishes a post, and verifies an anonymous `/posts/<slug>` visitor route renders the custom template with the published row data and no unresolved tokens. `content.e2e.ts` extends the binding coverage by adding a custom Posts field, inserting `{currentEntry.<customField>}` from the binding picker, verifying canvas preview resolution, publishing the template, and verifying the public post route resolves the custom value.
SITE-018 note: `visual-builder.e2e.ts` first publishes a disposable post, creates a higher-priority Posts template from the Site panel, explicitly selects that real row through Preview source, inserts `{currentEntry.title}` through the binding picker plus the content outlet, verifies the selected row title/body in canvas, saves and publishes the template snapshot, and verifies an anonymous `/posts/<slug>` visitor route renders the custom template with no unresolved tokens. `content.e2e.ts` extends the binding coverage by adding a custom Posts field, inserting `{currentEntry.<customField>}` from the binding picker, verifying canvas preview resolution, publishing the template, and verifying the public post route resolves the custom value.

SITE-019 note: `visual-builder.e2e.ts` saves a styled Container subtree as a layout, verifies blank and duplicate-name validation, confirms the Layouts category remains reachable at 390px, inserts the saved layout into another page with its captured class styling, renames and deletes the saved layout from the inserter manage menu, saves/reloads, publishes, and verifies anonymous public output. DEF-20260623-SITE019-001 fixed stale node selection on `addPage`; DEF-20260623-SITE019-002 fixed count-only mobile category buttons; DEF-20260623-SITE019-003 fixed the saved-layout manage menu z-index under the spotlight inserter.

Expand Down Expand Up @@ -253,7 +253,7 @@ MEDIA-007 note: unsafe SVG upload sanitization and public `/uploads` serving are

CONTENT-003 note: slash-menu Heading 2 and Data token placeholder insertion with save/reload persistence are automated in `content.e2e.ts`; media picker insertion and sanitization edge cases remain lower-level or future browser coverage.

CONTENT-005 note: draft title/body rendering through the seeded entry template in Live mode is automated in `content.e2e.ts`; missing-template error handling, stale public-state comparison, and mobile live-canvas checks remain future browser coverage.
CONTENT-005 note: `content.e2e.ts` explicitly authors and publishes a low-priority Posts title/outlet template, then verifies unsaved draft title/body rendering through that owned fixture in Live mode. Missing-template error handling, stale public-state comparison, and mobile live-canvas checks remain future browser coverage.

CONTENT-006 note: content built-in field toggles are automated in `content.e2e.ts`; custom field-schema edge cases and destructive collection deletion remain covered by lower-level tests or future browser expansion.

Expand All @@ -269,7 +269,7 @@ CONTENT-008 note: `content.e2e.ts` adds a custom text field to the system Posts
| AI-002 | P1 | partial | Defaults | Set default credential/model per AI scope | Credential with models | AI Defaults tab | A scope saves the intended credential/model pair, reloads with the saved selection resolved, and can clear the default so the credential can be deleted | no credentials, deleted credential, stale model list, all-scope coverage |
| AI-003 | P2 | partial | Conversations | Continue saved AI conversations per scope | AI chat permission and configured default | AI assistant history | Saved Site chats can be listed, reloaded, and deleted from the history popover | cross-user access, deleted credential, title update, empty history |
| AI-004 | P2 | partial | Chat | Stream scoped AI replies and tool-loop events | Configured provider/default | Site/content AI assistant | Prompt sends to a fixture-backed local provider, streamed text renders, and text-only usage persists for audit | provider SSE failure, aborts, empty prompt, large context, write-tool UI |
| AI-005 | P2 | partial | Tool Bridge | Return browser-executed tool results to the AI runtime | Active tool request | Browser bridge + `/ai/tool-result` | Browser `read_document` tool result correlates to the pending request and resumes the model loop | lost tab, timeout, duplicate result, malformed ids, write tools |
| AI-005 | P2 | partial | Tool Bridge | Return browser-executed tool results to the AI runtime | Active tool request | Browser bridge + `/ai/tool-result` | Browser `site_read_document` tool result correlates to the pending request and resumes the model loop | lost tab, timeout, duplicate result, malformed ids, write tools |
| AI-006 | P2 | partial | Audit | Review AI usage rollups by user/scope/model/day | Usage generated by chat | AI Audit tab/dashboard widget | Audit tab renders model, scope, token, and daily rollups from a real streamed chat turn | empty usage, deleted labels, bad timezone, dashboard widget, mobile table |
| AI-007 | P2 | partial | Drivers | Use provider REST drivers without SDK lock-in | Mocked or local provider | AI runtime/provider drivers | Direct drivers map messages/tools, stream events, usage, model catalogues, context windows, and prices without provider SDKs | malformed SSE, rate limits, unknown pricing |

Expand All @@ -279,7 +279,7 @@ AI-003 note: Site assistant conversation creation, new-chat reset, history reloa

AI-004 note: site assistant streaming against a fixture-backed local Ollama-compatible server is automated in `ai.e2e.ts`; write-tool/tool-loop UI, abort/error recovery, and content/data/plugin provider-backed permutations remain future coverage.

AI-005 note: fixture-backed local Ollama tool-loop coverage is automated in `ai.e2e.ts` for the browser-executed `read_document` path: tool request, browser result POST, provider second turn, completed tool badge, and final assistant text. CAP-005 also verifies request-level write-tool filtering for `ai.chat` without `ai.tools.write`. Mutating write-tool bridge success coverage, timeout/abort UX, duplicate/malformed result handling, permission-denied execution defense, and unknown-tool provider requests remain lower-level or future browser coverage.
AI-005 note: fixture-backed local Ollama tool-loop coverage is automated in `ai.e2e.ts` for the browser-executed `site_read_document` path: tool request, browser result POST, provider second turn, completed tool badge, and final assistant text. CAP-005 also verifies request-level write-tool filtering for `ai.chat` without `ai.tools.write`. Mutating write-tool bridge success coverage, timeout/abort UX, duplicate/malformed result handling, permission-denied execution defense, and unknown-tool provider requests remain lower-level or future browser coverage.

AI-006 note: usage generated by the AI-004 site chat is verified in the Audit tab by model, surface, prompt/completion tokens, and daily spend in `ai.e2e.ts`; dashboard widget, range switching, timezone/user permission variants, deleted-label display, and mobile table layout remain future coverage.

Expand Down
Loading
Loading