Skip to content

ci: benchmark built suites with unsharded parallel Endform - #2

Closed
ostenbom wants to merge 9 commits into
codex/playwright-baselinefrom
codex/endform-product-suite
Closed

ostenbom wants to merge 9 commits into
codex/playwright-baselinefrom
codex/endform-product-suite

Conversation

@ostenbom

@ostenbom ostenbom commented Oct 8, 2026 •

Copy link
Copy Markdown

What does this PR do?

The existing product harness serves the Astro/Vite development graph: its login loads about 67.4 MB across 418 resources, and regular Endform navigation exceeds the original heading assertion. Build the private fixtures and serve the compiled Node standalone application and Cloudflare Worker instead. A local built-login diagnostic loaded 12 resources and about 9.4 MB, without Vite/source-module requests.

Keep the native Playwright baseline on its existing eight CI shards. Run Endform in ONE CI job on ONE GitHub runner, with one complete unsharded invocation per target, no test-list partitioning and no concurrent-test cap. Node and Cloudflare use the built applications on that runner; Endform distributes the tests remotely at maximum parallelism. Endform alone receives a 240-second outer test budget; passing tests finish immediately. Existing assertions, explicit test timeouts and native browser installation remain unchanged.

The previous eight-shard, one-at-a-time Endform configuration prevented its intended execution model. Its timings do not establish how Endform performs at full parallelism and are superseded by this correction.

The application compiles in production mode. Narrow private-fixture adaptations retain setup/reset/auth helpers, captured emails, and the localhost mock registry. Manual locale routing, the canonical fixture URL, runtime registry initialization, and a direct Sharp dependency make the existing assertions work against the builds. No published package source changed.

Fork-only work in endformdev/emdash, stacked on native baseline PR #1 at 40152078f550ba50e012812450f175799d88db43. The original baseline used development servers and is not an equivalent timing comparator. This PR's native matrix supplies the matching built baseline. No upstream merge is requested.

Type of change

  • Bug fix
  • Feature (requires maintainer-approved Discussion)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes — local package checks and the CI Typecheck job passed
  • pnpm lint passes — local quick/type-aware checks returned zero diagnostics; CI lint passed
  • pnpm test passes (or targeted tests for my change) — full unsharded Endform execution is under verification; prior serial local built suites passed
  • pnpm format has been run
  • I have added/updated tests for my changes (if applicable) — existing assertions exercise the builds; HTTPS image mocks satisfy production CSP
  • User-visible strings in the admin UI are wrapped for translation — n/a, no admin UI changes or catalog extraction
  • I have added and reviewed the user-facing changeset — n/a, no published package source changes
  • New features link to an approved Discussion — n/a, test infrastructure
  • I have included screenshots below if this PR changes the UI — n/a, no application UI changes

AI-generated code disclosure

  • This PR includes AI-generated code — model/tool: GPT-6 (Codex)

Screenshots / test output

Screenshots: not applicable to this infrastructure change. Failure screenshots and Playwright traces are retained in the CI artifacts.

Root build, formatting, quick/type-aware lint (zero diagnostics), package type checks and 362-test discovery passed for the architecture correction.

Tested revision: 36723f59e9f52b8cf4426e7e8453f783598bf042.
Completed CI run and standalone Endform job.

Runner Time from common CI dispatch until its benchmark jobs finish Node pass/fail/skip Cloudflare pass/fail/skip
Native, eight existing shards 13:40 326 / 2 / 34 314 / 1 / 47
Endform, one machine, full parallelism 18:51 197 / 131 / 34 85 / 230 / 47

Counts include recovered retries as passed and cover the same 362 selected tests per target. Both configurations failed; these results do not establish the speed of a clean passing suite. The native lane includes one recovered retry. Endform has many recovered retries. One cohort was measured, so no stable median or cost claim is made.

Native job window: 13:17. Endform's single job: 18:27, comprising 0:34 setup/install, 1:12 package builds, 9:09 Node command, 0:28 Node failure-artifact upload, 6:07 Cloudflare command, 0:55 Cloudflare failure-artifact upload and about 0:02 cleanup. Both targets run sequentially on the one Endform host. Service execution windows were 8:16 Node and 5:43 Cloudflare. The combined workflow finished in 18:59 and includes both benchmark lanes and unrelated checks.

The corrected parallel architecture changes the failure analysis: failures and retries now dominate the critical path. Three Node setup cases each exhausted a 240-second timeout twice, accounting for almost the full 8:16 service window. In trace 0551937c8c1be375adfafc1dba51a687, the setup page navigates to login and the test then waits 222 seconds to fill the missing Site Title field. In Cloudflare trace f92c8948133e483d174322915ba7ff15, auth/me and media APIs return 401, followed by a 200-second wait to click Add new folder. Increasing timeouts cannot restore deleted authentication state.

Two distinct correctness issues need resolution before a clean benchmark:

  • The existing setup-wizard group declares serial mode, but Endform starts its cases concurrently under fullyParallel=true. Their overlapping HTTP reset calls are visible in the traces. Endform needs to preserve explicit serial groups while distributing independent tests.
  • Setup/reset tests delete shared users while other tests authenticate against the same application/database. The test harness needs isolated backend state for destructive setup groups, plus investigation of other shared-data assumptions. Preserving serial execution inside that group alone will not isolate it from other files. Reintroducing a suite-wide concurrency cap would defeat this benchmark's purpose.

Additional native failures remain the byline alignment and marketplace-toast obstruction described in the earlier run. Neither test assertions nor suite coverage were weakened to make the comparison green.

Local full parallel runs also exposed SOCKS failures, relay-capacity drops and a delayed finalization_timeout on CLI 0.81.5. Those transport/reporting symptoms did NOT reproduce in this CI Endform job: zero relay-capacity-drop messages, no finalization_timeout, and both commands returned promptly after service execution. They should be investigated as a separate local-network reproduction.

Endform evidence:

Prior run 38824536265 used the incorrect serialized/sharded Endform architecture. Its performance conclusions are superseded. The current configuration is the intended unsharded architecture, but reliability and shared-state isolation must be fixed before presenting a successful-suite performance demonstration.

@ostenbom ostenbom closed this Oct 15, 2026
@ostenbom ostenbom reopened this Oct 15, 2026
@ostenbom ostenbom changed the title ci: verify Endform against the product Playwright baseline ci: compare native and Endform suites against built fixtures Oct 15, 2026
@github-actions github-actions Bot added size/XL and removed size/L labels Oct 15, 2026
@github-actions

Copy link
Copy Markdown

PR template validation failed

Please fix the following issues by editing your PR description:

  • Check the "I have read CONTRIBUTING.md" checkbox.

See CONTRIBUTING.md for the full contribution policy.

@ostenbom ostenbom changed the title ci: compare native and Endform suites against built fixtures ci: benchmark built suites with unsharded parallel Endform Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant