Skip to content

Commit 80c6ea9

Browse files
committed
Add performance testing docs and integration
Introduce performance testing features and documentation: add a new performance testing guide, document Core Web Vitals and supplementary metrics (LCP, FCP, TTFB, CLS, INP, JS heap, DOM nodes), and explain budget enforcement via [PerformanceBudget] attribute or motus.config.json. Update plugin interfaces to include IPerformanceReporter and register reporter behavior; add CLI flag (--perf-budget) and environment/config options for performance collection. Document browser lifecycle health detection (heartbeat and crash handling), pool proactive replenishment, and test fixture auto-restart on crash. Update various README files and config docs to surface the new functionality and reporter integration.
1 parent 5fb13f9 commit 80c6ea9

9 files changed

Lines changed: 528 additions & 18 deletions

File tree

README.md

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,48 @@ Or in `motus.config.json`:
127127
}
128128
```
129129

130+
### Performance Testing
131+
132+
Motus collects Core Web Vitals (LCP, FCP, TTFB, CLS, INP) and supplementary metrics (JS heap size, DOM node count) directly from the browser during test execution. Set budget thresholds via attributes or config, and performance regressions fail your tests automatically.
133+
134+
```csharp
135+
// Assert all metrics are within the class-level budget
136+
[PerformanceBudget(Lcp = 2500, Fcp = 1800, Cls = 0.1)]
137+
[TestClass]
138+
public class DashboardTests : MotusTestBase
139+
{
140+
[TestMethod]
141+
public async Task DashboardLoadsWithinBudget()
142+
{
143+
await Page.GotoAsync("https://app.example.com/dashboard");
144+
await Expect.That(Page).ToMeetPerformanceBudgetAsync();
145+
}
146+
}
147+
148+
// Or assert individual metrics
149+
await Expect.That(page).ToHaveLcpBelowAsync(2500);
150+
await Expect.That(page).ToHaveClsBelowAsync(0.1);
151+
```
152+
153+
Enable budget enforcement from the CLI:
154+
155+
```bash
156+
motus run ./bin/Debug/net8.0/MyTests.dll --perf-budget
157+
```
158+
159+
Or in `motus.config.json`:
160+
161+
```json
162+
{
163+
"performance": {
164+
"enable": true,
165+
"lcp": 2500,
166+
"cls": 0.1,
167+
"inp": 200
168+
}
169+
}
170+
```
171+
130172
## How It Works
131173

132174
Motus communicates directly with the browser over WebSocket. For Chromium-based browsers, it speaks the Chrome DevTools Protocol (CDP). For Firefox, it uses WebDriver BiDi. There is no Node.js sidecar, no driver binary, and no process boundary between your test code and the protocol layer.
@@ -158,6 +200,7 @@ Five interfaces define every point of extensibility, all registered through `IPl
158200
| `IReporter` | Receive test run events for custom reporting (multiple reporters run simultaneously) |
159201
| `IAccessibilityRule` | Define custom WCAG accessibility rules evaluated against the browser's accessibility tree |
160202
| `IAccessibilityReporter` | Opt-in interface for reporters to receive per-violation accessibility events |
203+
| `IPerformanceReporter` | Opt-in interface for reporters to receive performance metrics and budget results |
161204
| `IMotusLogger` | Structured logging for plugin diagnostics |
162205

163206
### Plugin Discovery
@@ -228,6 +271,7 @@ Launch the Blazor-based visual runner with `motus run --visual`:
228271
| **Recording** | Capture browser interactions and emit idiomatic C# test code (MSTest, xUnit, NUnit) |
229272
| **Codegen** | Generate Page Object Model classes from live pages with selector inference |
230273
| **Accessibility** | Built-in WCAG 2.1 Level A/AA audits via CDP accessibility tree, lifecycle hook, page and locator assertions, custom rules via `IAccessibilityRule` |
274+
| **Performance** | Core Web Vitals collection (LCP, FCP, TTFB, CLS, INP), configurable budgets via `[PerformanceBudget]` attribute or config, auto-retry assertions, reporter integration via `IPerformanceReporter` |
231275
| **Reporters** | Console, HTML, JUnit XML, TRX, plus custom reporters via `IReporter` (with opt-in `IAccessibilityReporter` for violation events) |
232276
| **Tracing** | Screenshots, DOM snapshots, network logs, HAR export, and WebM video recording |
233277
| **Parallel** | Context-level, browser-level, and worker-level parallel execution |
@@ -237,7 +281,7 @@ Launch the Blazor-based visual runner with `motus run --visual`:
237281
## CLI Reference
238282

239283
```
240-
motus run <assemblies> Run tests with optional --visual, --filter, --workers, --reporter, --a11y
284+
motus run <assemblies> Run tests with optional --visual, --filter, --workers, --reporter, --a11y, --perf-budget
241285
motus record Record a browser session and emit test code
242286
motus codegen Generate POM classes from live pages (--headed, --connect, --detect-listeners)
243287
motus screenshot Capture a screenshot (--full-page, --delay, --hide-banners, --width, --height)

docs/architecture/browser-lifecycle.md

Lines changed: 28 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -188,10 +188,13 @@ The sequence:
188188

189189
`CloseAsync` is the graceful shutdown path:
190190

191-
1. All contexts are closed in sequence by calling `context.CloseAsync()`.
192-
2. `Browser.close` is sent over the browser-level CDP session. A `CdpDisconnectedException` is expected and caught because the browser closes the WebSocket as part of its shutdown.
193-
3. The owned OS process (if any) is awaited for up to **5 seconds**. If it has not exited within that window, `process.Kill(entireProcessTree: true)` is called.
194-
4. `IsConnected` is set to `false`.
191+
1. `IsConnected` is set to `false`.
192+
2. The `BrowserHeartbeat` (if running) is disposed, stopping background health pings.
193+
3. Signal handlers and the `Process.Exited` handler are unregistered.
194+
4. All contexts are closed in sequence by calling `context.CloseAsync()`.
195+
5. `Browser.close` is sent over the browser-level CDP session. A `CdpDisconnectedException` is expected and caught because the browser closes the WebSocket as part of its shutdown.
196+
6. The transport is disposed.
197+
7. The owned OS process (if any) is awaited for up to **5 seconds**. If it has not exited within that window, `process.Kill(entireProcessTree: true)` is called.
195198

196199
### `IBrowserContext.CloseAsync`
197200

@@ -204,12 +207,25 @@ The sequence:
204207

205208
`DisposeAsync` is the non-graceful teardown path, used when an exception occurs during launch or when the pool discards a disconnected browser:
206209

207-
1. Signal handlers are unregistered.
208-
2. `IsConnected` is set to `false`.
209-
3. The transport is disposed asynchronously.
210-
4. If the process has not exited, `process.Kill(entireProcessTree: true)` is called immediately, without the 5-second grace period.
211-
5. The process handle is disposed.
212-
6. If a temp user data directory is owned, `Directory.Delete(path, recursive: true)` is attempted. Failures are silently ignored (best-effort cleanup).
210+
1. The `BrowserHeartbeat` (if running) is disposed.
211+
2. Signal handlers and the `Process.Exited` handler are unregistered.
212+
3. `IsConnected` is set to `false`.
213+
4. The transport is disposed asynchronously.
214+
5. If the process has not exited, `process.Kill(entireProcessTree: true)` is called immediately, without the 5-second grace period.
215+
6. The process handle is disposed.
216+
7. If a temp user data directory is owned, `Directory.Delete(path, recursive: true)` is attempted. Failures are silently ignored (best-effort cleanup).
217+
218+
### Health detection
219+
220+
When a browser is launched (not connected via `ConnectAsync`), Motus monitors the browser process for two failure modes: crashes and freezes.
221+
222+
**Crash detection via `Process.Exited`.** The `Browser` constructor subscribes to `_process.Exited`. When Chrome crashes or is killed externally, the handler fires immediately, sets `IsConnected = false`, disposes the transport (which faults all pending CDP commands with `CdpDisconnectedException`), and raises the `Disconnected` event. This is faster than waiting for the WebSocket receive loop to detect the broken connection.
223+
224+
**Freeze detection via `BrowserHeartbeat`.** After `InitializeAsync` completes, a background `BrowserHeartbeat` is started for launched browsers. The heartbeat sends `Browser.getVersion` every 5 seconds with a 10-second timeout. If the ping times out or fails (and the browser has not been intentionally shut down), the `OnHeartbeatFailed` callback triggers the same disconnect path as `Process.Exited`. This catches cases where Chrome becomes unresponsive without closing the WebSocket, such as GPU process hangs or runaway JavaScript.
225+
226+
Both handlers are guarded by an `Interlocked.CompareExchange` on `_disconnectedFlag` so that only one path fires the `Disconnected` event, regardless of which detection mechanism triggers first.
227+
228+
**`IBrowser.IsHealthy`.** The `IsHealthy` property (defined as a default interface member on `IBrowser`) returns `true` when the browser is connected. The `Browser` implementation overrides this with a process-aware check: `_isConnected && (_process is null || !_process.HasExited)`. For browsers obtained via `ConnectAsync` (no process ownership), `IsHealthy` is equivalent to `IsConnected`.
213229

214230
### Signal handlers
215231

@@ -268,6 +284,8 @@ IPage page = await lease.Browser.NewPageAsync();
268284

269285
The returned `IBrowserLease` holds the browser and a return callback. Disposing the lease returns the browser to the idle channel if it is still connected and the pool has not been disposed. A disconnected or post-dispose browser is discarded and the semaphore slot is released.
270286

287+
**Proactive replenishment.** The pool subscribes to each browser's `Disconnected` event. When a browser crashes or becomes unresponsive, `OnBrowserDisconnected` disposes the dead browser, releases its capacity semaphore slot, and (if the idle count has dropped below `MinInstances`) launches a replacement in the background. A `ConcurrentDictionary<IBrowser, byte>` deduplicates the disconnect and lease-return paths to prevent double-dispose when both fire for the same browser.
288+
271289
`IBrowserPool.ActiveCount` and `IdleCount` expose live counters for observability.
272290

273291
`DisposeAsync` on the pool drains and disposes all idle browsers. Browsers that are currently leased out are not forcibly recalled; callers must dispose their leases before the pool can be fully cleaned up.

docs/extensions/plugin-interfaces.md

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Plugin Interfaces
22

3-
Motus exposes seven extensibility interfaces that plugins can implement to participate in browser automation, test reporting, and accessibility auditing. All interfaces live in the `Motus.Abstractions` NuGet package. Plugins register their implementations through `IPluginContext` during initialization.
3+
Motus exposes eight extensibility interfaces that plugins can implement to participate in browser automation, test reporting, accessibility auditing, and performance monitoring. All interfaces live in the `Motus.Abstractions` NuGet package. Plugins register their implementations through `IPluginContext` during initialization.
44

55
---
66

@@ -271,6 +271,55 @@ public sealed class A11yReporter : IReporter, IAccessibilityReporter
271271

272272
---
273273

274+
## IPerformanceReporter
275+
276+
`IPerformanceReporter` is an opt-in companion to `IReporter` for reporters that want to receive performance metrics collected during test execution. Motus checks `reporter is IPerformanceReporter` at runtime on each registered reporter; only those that implement both interfaces receive performance callbacks. This follows the same NativeAOT-safe pattern as `IAccessibilityReporter`.
277+
278+
### Members
279+
280+
| Member | Return Type | Description |
281+
|---|---|---|
282+
| `OnPerformanceMetricsCollectedAsync(PerformanceMetrics metrics, PerformanceBudgetResult? budgetResult, TestInfo test)` | `Task` | Called when performance metrics have been collected for a test. |
283+
284+
### Supporting Types
285+
286+
`PerformanceMetrics` carries the collected values: `Lcp`, `Fcp`, `Ttfb`, `Cls`, `Inp` (all `double?`), `JsHeapSize` (`long?`), `DomNodeCount` (`int?`), `LayoutShifts` (`IReadOnlyList<LayoutShiftEntry>`), `CollectedAtUtc` (`DateTime`), and `DiagnosticMessage` (`string?`).
287+
288+
`PerformanceBudgetResult` carries the evaluation: `Entries` (`IReadOnlyList<PerformanceBudgetEntry>`) and `Passed` (`bool`). Each `PerformanceBudgetEntry` has `MetricName`, `Threshold`, `ActualValue`, `Passed`, and `Delta`.
289+
290+
`budgetResult` is `null` when no budget is configured.
291+
292+
### Lifecycle Notes
293+
294+
`OnPerformanceMetricsCollectedAsync` is called once per test when metrics have been collected (typically after a navigation). It is called before `OnTestEndAsync` for the same test. Exceptions are caught and logged; they do not suppress remaining reporter events.
295+
296+
### Example
297+
298+
```csharp
299+
using Motus.Abstractions;
300+
301+
public sealed class PerfReporter : IReporter, IPerformanceReporter
302+
{
303+
public Task OnTestRunStartAsync(TestSuiteInfo suite) => Task.CompletedTask;
304+
public Task OnTestStartAsync(TestInfo test) => Task.CompletedTask;
305+
public Task OnTestEndAsync(TestInfo test, TestResult result) => Task.CompletedTask;
306+
public Task OnTestRunEndAsync(TestRunSummary summary) => Task.CompletedTask;
307+
308+
public Task OnPerformanceMetricsCollectedAsync(
309+
PerformanceMetrics metrics,
310+
PerformanceBudgetResult? budgetResult,
311+
TestInfo test)
312+
{
313+
Console.WriteLine(
314+
$"[{test.TestName}] LCP={metrics.Lcp:F0}ms " +
315+
$"Budget: {(budgetResult?.Passed == true ? "PASS" : budgetResult?.Passed == false ? "FAIL" : "none")}");
316+
return Task.CompletedTask;
317+
}
318+
}
319+
```
320+
321+
---
322+
274323
## IAccessibilityRule
275324

276325
`IAccessibilityRule` evaluates a single WCAG rule against one node in the accessibility tree. The engine walks every non-ignored node in the tree and calls each registered rule's `Evaluate` method. Rules are synchronous to keep the audit loop efficient; use the `AccessibilityAuditContext` for any cross-node data rather than performing additional async page queries.
@@ -384,7 +433,7 @@ public sealed class MyPlugin : IMotusPlugin
384433
| `RegisterSelectorStrategy(ISelectorStrategy strategy)` | `void` | Registers a custom selector strategy. |
385434
| `RegisterWaitCondition(IWaitCondition condition)` | `void` | Registers a custom wait condition. |
386435
| `RegisterLifecycleHook(ILifecycleHook hook)` | `void` | Registers a lifecycle hook. |
387-
| `RegisterReporter(IReporter reporter)` | `void` | Registers a test reporter. Reporters that also implement `IAccessibilityReporter` automatically receive violation events. |
436+
| `RegisterReporter(IReporter reporter)` | `void` | Registers a test reporter. Reporters that also implement `IAccessibilityReporter` or `IPerformanceReporter` automatically receive the corresponding events. |
388437
| `RegisterAccessibilityRule(IAccessibilityRule rule)` | `void` | Registers a custom accessibility rule that is invoked during accessibility audits. |
389438
| `CreateLogger(string categoryName)` | `IMotusLogger` | Creates a logger scoped to the given category name. |
390439

docs/guides/configuration.md

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,17 @@ When Motus initializes, it walks up from `Environment.CurrentDirectory`, checkin
6666
"auditAfterActions": false,
6767
"includeWarnings": true,
6868
"skipRules": []
69+
},
70+
"performance": {
71+
"enable": false,
72+
"collectAfterNavigation": true,
73+
"lcp": null,
74+
"fcp": null,
75+
"ttfb": null,
76+
"cls": null,
77+
"inp": null,
78+
"jsHeapSize": null,
79+
"domNodeCount": null
6980
}
7081
}
7182
```
@@ -154,6 +165,22 @@ Controls the built-in accessibility audit hook. When enabled, Motus runs axe-sty
154165
| `includeWarnings` | `bool` | `true` | Treat warning-severity findings as failures alongside errors (when mode is `Enforce`). |
155166
| `skipRules` | `string[]` | `null` | Rule IDs to exclude from every audit, e.g. `["a11y-color-contrast"]`. |
156167

168+
### `performance` Section
169+
170+
Controls the built-in performance metrics collector and budget thresholds. When enabled, Motus collects Core Web Vitals and supplementary metrics during test execution. Setting metric thresholds activates budget enforcement.
171+
172+
| Property | Type | Default | Description |
173+
|---|---|---|---|
174+
| `enable` | `bool` | `false` | Enable the performance metrics collector. |
175+
| `collectAfterNavigation` | `bool` | `true` | Collect metrics after each page navigation. |
176+
| `lcp` | `number` | `null` | Maximum Largest Contentful Paint in milliseconds. |
177+
| `fcp` | `number` | `null` | Maximum First Contentful Paint in milliseconds. |
178+
| `ttfb` | `number` | `null` | Maximum Time to First Byte in milliseconds. |
179+
| `cls` | `number` | `null` | Maximum Cumulative Layout Shift score (unitless). |
180+
| `inp` | `number` | `null` | Maximum Interaction to Next Paint in milliseconds. |
181+
| `jsHeapSize` | `number` | `null` | Maximum JavaScript heap size in bytes. |
182+
| `domNodeCount` | `number` | `null` | Maximum DOM node count. |
183+
157184
---
158185

159186
## Environment Variables
@@ -179,8 +206,16 @@ Boolean variables accept `true`, `false`, `1`, or `0` (case-insensitive). Intege
179206
| `MOTUS_FAILURES_TRACE_PATH` | `failure.tracePath` | `string` | Directory for trace archives. |
180207
| `MOTUS_ACCESSIBILITY_ENABLE` | `accessibility.enable` | `bool` | Enable the accessibility audit hook. |
181208
| `MOTUS_ACCESSIBILITY_MODE` | `accessibility.mode` | `string` | Audit violation handling mode (`Off`, `Warn`, `Enforce`). |
209+
| `MOTUS_PERFORMANCE_ENABLE` | `performance.enable` | `bool` | Enable the performance metrics collector. |
210+
| `MOTUS_PERFORMANCE_LCP` | `performance.lcp` | `double` | Maximum LCP in milliseconds. |
211+
| `MOTUS_PERFORMANCE_FCP` | `performance.fcp` | `double` | Maximum FCP in milliseconds. |
212+
| `MOTUS_PERFORMANCE_TTFB` | `performance.ttfb` | `double` | Maximum TTFB in milliseconds. |
213+
| `MOTUS_PERFORMANCE_CLS` | `performance.cls` | `double` | Maximum CLS score. |
214+
| `MOTUS_PERFORMANCE_INP` | `performance.inp` | `double` | Maximum INP in milliseconds. |
215+
| `MOTUS_PERFORMANCE_JS_HEAP_SIZE` | `performance.jsHeapSize` | `long` | Maximum JS heap size in bytes. |
216+
| `MOTUS_PERFORMANCE_DOM_NODE_COUNT` | `performance.domNodeCount` | `int` | Maximum DOM node count. |
182217

183-
> Not all config file sections have environment variable coverage. `reporter`, `recorder`, and the locator `selectorPriority` arrays are file-only settings. Use the config file for those.
218+
> Not all config file sections have environment variable coverage. `reporter`, `recorder`, the locator `selectorPriority` arrays, and `performance.collectAfterNavigation` are file-only settings. Use the config file for those.
184219
185220
---
186221

@@ -215,6 +250,7 @@ The merge checks are per-property:
215250
- `IgnoreHTTPSErrors` -- a config value of `true` is applied when `options.IgnoreHTTPSErrors` is `false`. A config value of `false` never overrides a caller-set `true`.
216251
- `Viewport` -- applied only when `options.Viewport is null`. Both `width` and `height` must be non-null in the config for a viewport to be applied.
217252
- `Accessibility` -- the entire `AccessibilityOptions` object is applied only when `options.Accessibility is null`.
253+
- `Performance` -- the entire `PerformanceOptions` object is applied only when `options.Performance is null`.
218254

219255
---
220256

@@ -237,6 +273,7 @@ The merge checks are per-property:
237273
| `DownloadsPath` | `string?` | `null` | Directory for browser-initiated file downloads. |
238274
| `Plugins` | `IReadOnlyList<IPlugin>?` | `null` | Plugin instances loaded into every browser context created from this launch. |
239275
| `Accessibility` | `AccessibilityOptions?` | `null` | Accessibility audit hook configuration. Disabled when `null`. |
276+
| `Performance` | `PerformanceOptions?` | `null` | Performance metrics collector configuration. Disabled when `null`. |
240277

241278
### AccessibilityOptions
242279

@@ -251,6 +288,17 @@ Nested under `LaunchOptions.Accessibility`. A `null` value disables the hook ent
251288
| `IncludeWarnings` | `bool` | `true` | Count warning-severity findings as failures when mode is `Enforce`. |
252289
| `SkipRules` | `IReadOnlyList<string>?` | `null` | Rule IDs excluded from every audit. |
253290

291+
### PerformanceOptions
292+
293+
Nested under `LaunchOptions.Performance`. A `null` value disables the collector entirely.
294+
295+
| Property | Type | Default | Description |
296+
|---|---|---|---|
297+
| `Enable` | `bool` | `false` | Enable the performance metrics collector. |
298+
| `CollectAfterNavigation` | `bool` | `true` | Collect metrics after each page navigation. |
299+
300+
Budget thresholds are not part of `PerformanceOptions`. They are configured via the `[PerformanceBudget]` attribute or the `performance` section in `motus.config.json`.
301+
254302
---
255303

256304
## ContextOptions
@@ -292,6 +340,9 @@ export MOTUS_FAILURES_TRACE=true
292340
export MOTUS_FAILURES_TRACE_PATH=test-results/traces
293341
export MOTUS_ACCESSIBILITY_ENABLE=true
294342
export MOTUS_ACCESSIBILITY_MODE=Enforce
343+
export MOTUS_PERFORMANCE_ENABLE=true
344+
export MOTUS_PERFORMANCE_LCP=2500
345+
export MOTUS_PERFORMANCE_CLS=0.1
295346
```
296347

297348
No `motus.config.json` is required on the CI agent. These variables provide a complete runtime configuration without touching source control.
@@ -335,4 +386,5 @@ Commit this file or add it to `.gitignore` depending on whether the whole team u
335386
- [Getting Started](../getting-started.md)
336387
- [Plugins and Extensibility](../extensions/getting-started.md)
337388
- [Accessibility Testing](./accessibility-testing.md)
389+
- [Performance Testing](./performance-testing.md)
338390
- [Testing Frameworks](./testing-frameworks.md)

0 commit comments

Comments
 (0)