You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Introduce performance testing features and documentation: add a new performance testing guide, document Core Web Vitals and supplementary metrics (LCP, FCP, TTFB, CLS, INP, JS heap, DOM nodes), and explain budget enforcement via [PerformanceBudget] attribute or motus.config.json. Update plugin interfaces to include IPerformanceReporter and register reporter behavior; add CLI flag (--perf-budget) and environment/config options for performance collection. Document browser lifecycle health detection (heartbeat and crash handling), pool proactive replenishment, and test fixture auto-restart on crash. Update various README files and config docs to surface the new functionality and reporter integration.
Copy file name to clipboardExpand all lines: README.md
+45-1Lines changed: 45 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -127,6 +127,48 @@ Or in `motus.config.json`:
127
127
}
128
128
```
129
129
130
+
### Performance Testing
131
+
132
+
Motus collects Core Web Vitals (LCP, FCP, TTFB, CLS, INP) and supplementary metrics (JS heap size, DOM node count) directly from the browser during test execution. Set budget thresholds via attributes or config, and performance regressions fail your tests automatically.
133
+
134
+
```csharp
135
+
// Assert all metrics are within the class-level budget
motus run ./bin/Debug/net8.0/MyTests.dll --perf-budget
157
+
```
158
+
159
+
Or in `motus.config.json`:
160
+
161
+
```json
162
+
{
163
+
"performance": {
164
+
"enable": true,
165
+
"lcp": 2500,
166
+
"cls": 0.1,
167
+
"inp": 200
168
+
}
169
+
}
170
+
```
171
+
130
172
## How It Works
131
173
132
174
Motus communicates directly with the browser over WebSocket. For Chromium-based browsers, it speaks the Chrome DevTools Protocol (CDP). For Firefox, it uses WebDriver BiDi. There is no Node.js sidecar, no driver binary, and no process boundary between your test code and the protocol layer.
@@ -158,6 +200,7 @@ Five interfaces define every point of extensibility, all registered through `IPl
158
200
|`IReporter`| Receive test run events for custom reporting (multiple reporters run simultaneously) |
159
201
|`IAccessibilityRule`| Define custom WCAG accessibility rules evaluated against the browser's accessibility tree |
160
202
|`IAccessibilityReporter`| Opt-in interface for reporters to receive per-violation accessibility events |
203
+
|`IPerformanceReporter`| Opt-in interface for reporters to receive performance metrics and budget results |
161
204
|`IMotusLogger`| Structured logging for plugin diagnostics |
162
205
163
206
### Plugin Discovery
@@ -228,6 +271,7 @@ Launch the Blazor-based visual runner with `motus run --visual`:
228
271
|**Recording**| Capture browser interactions and emit idiomatic C# test code (MSTest, xUnit, NUnit) |
229
272
|**Codegen**| Generate Page Object Model classes from live pages with selector inference |
230
273
|**Accessibility**| Built-in WCAG 2.1 Level A/AA audits via CDP accessibility tree, lifecycle hook, page and locator assertions, custom rules via `IAccessibilityRule`|
274
+
|**Performance**| Core Web Vitals collection (LCP, FCP, TTFB, CLS, INP), configurable budgets via `[PerformanceBudget]` attribute or config, auto-retry assertions, reporter integration via `IPerformanceReporter`|
231
275
|**Reporters**| Console, HTML, JUnit XML, TRX, plus custom reporters via `IReporter` (with opt-in `IAccessibilityReporter` for violation events) |
232
276
|**Tracing**| Screenshots, DOM snapshots, network logs, HAR export, and WebM video recording |
233
277
|**Parallel**| Context-level, browser-level, and worker-level parallel execution |
@@ -237,7 +281,7 @@ Launch the Blazor-based visual runner with `motus run --visual`:
237
281
## CLI Reference
238
282
239
283
```
240
-
motus run <assemblies> Run tests with optional --visual, --filter, --workers, --reporter, --a11y
284
+
motus run <assemblies> Run tests with optional --visual, --filter, --workers, --reporter, --a11y, --perf-budget
241
285
motus record Record a browser session and emit test code
242
286
motus codegen Generate POM classes from live pages (--headed, --connect, --detect-listeners)
243
287
motus screenshot Capture a screenshot (--full-page, --delay, --hide-banners, --width, --height)
Copy file name to clipboardExpand all lines: docs/architecture/browser-lifecycle.md
+28-10Lines changed: 28 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -188,10 +188,13 @@ The sequence:
188
188
189
189
`CloseAsync` is the graceful shutdown path:
190
190
191
-
1. All contexts are closed in sequence by calling `context.CloseAsync()`.
192
-
2.`Browser.close` is sent over the browser-level CDP session. A `CdpDisconnectedException` is expected and caught because the browser closes the WebSocket as part of its shutdown.
193
-
3. The owned OS process (if any) is awaited for up to **5 seconds**. If it has not exited within that window, `process.Kill(entireProcessTree: true)` is called.
194
-
4.`IsConnected` is set to `false`.
191
+
1.`IsConnected` is set to `false`.
192
+
2. The `BrowserHeartbeat` (if running) is disposed, stopping background health pings.
193
+
3. Signal handlers and the `Process.Exited` handler are unregistered.
194
+
4. All contexts are closed in sequence by calling `context.CloseAsync()`.
195
+
5.`Browser.close` is sent over the browser-level CDP session. A `CdpDisconnectedException` is expected and caught because the browser closes the WebSocket as part of its shutdown.
196
+
6. The transport is disposed.
197
+
7. The owned OS process (if any) is awaited for up to **5 seconds**. If it has not exited within that window, `process.Kill(entireProcessTree: true)` is called.
195
198
196
199
### `IBrowserContext.CloseAsync`
197
200
@@ -204,12 +207,25 @@ The sequence:
204
207
205
208
`DisposeAsync` is the non-graceful teardown path, used when an exception occurs during launch or when the pool discards a disconnected browser:
206
209
207
-
1. Signal handlers are unregistered.
208
-
2.`IsConnected` is set to `false`.
209
-
3. The transport is disposed asynchronously.
210
-
4. If the process has not exited, `process.Kill(entireProcessTree: true)` is called immediately, without the 5-second grace period.
211
-
5. The process handle is disposed.
212
-
6. If a temp user data directory is owned, `Directory.Delete(path, recursive: true)` is attempted. Failures are silently ignored (best-effort cleanup).
210
+
1. The `BrowserHeartbeat` (if running) is disposed.
211
+
2. Signal handlers and the `Process.Exited` handler are unregistered.
212
+
3.`IsConnected` is set to `false`.
213
+
4. The transport is disposed asynchronously.
214
+
5. If the process has not exited, `process.Kill(entireProcessTree: true)` is called immediately, without the 5-second grace period.
215
+
6. The process handle is disposed.
216
+
7. If a temp user data directory is owned, `Directory.Delete(path, recursive: true)` is attempted. Failures are silently ignored (best-effort cleanup).
217
+
218
+
### Health detection
219
+
220
+
When a browser is launched (not connected via `ConnectAsync`), Motus monitors the browser process for two failure modes: crashes and freezes.
221
+
222
+
**Crash detection via `Process.Exited`.** The `Browser` constructor subscribes to `_process.Exited`. When Chrome crashes or is killed externally, the handler fires immediately, sets `IsConnected = false`, disposes the transport (which faults all pending CDP commands with `CdpDisconnectedException`), and raises the `Disconnected` event. This is faster than waiting for the WebSocket receive loop to detect the broken connection.
223
+
224
+
**Freeze detection via `BrowserHeartbeat`.** After `InitializeAsync` completes, a background `BrowserHeartbeat` is started for launched browsers. The heartbeat sends `Browser.getVersion` every 5 seconds with a 10-second timeout. If the ping times out or fails (and the browser has not been intentionally shut down), the `OnHeartbeatFailed` callback triggers the same disconnect path as `Process.Exited`. This catches cases where Chrome becomes unresponsive without closing the WebSocket, such as GPU process hangs or runaway JavaScript.
225
+
226
+
Both handlers are guarded by an `Interlocked.CompareExchange` on `_disconnectedFlag` so that only one path fires the `Disconnected` event, regardless of which detection mechanism triggers first.
227
+
228
+
**`IBrowser.IsHealthy`.** The `IsHealthy` property (defined as a default interface member on `IBrowser`) returns `true` when the browser is connected. The `Browser` implementation overrides this with a process-aware check: `_isConnected && (_process is null || !_process.HasExited)`. For browsers obtained via `ConnectAsync` (no process ownership), `IsHealthy` is equivalent to `IsConnected`.
The returned `IBrowserLease` holds the browser and a return callback. Disposing the lease returns the browser to the idle channel if it is still connected and the pool has not been disposed. A disconnected or post-dispose browser is discarded and the semaphore slot is released.
270
286
287
+
**Proactive replenishment.** The pool subscribes to each browser's `Disconnected` event. When a browser crashes or becomes unresponsive, `OnBrowserDisconnected` disposes the dead browser, releases its capacity semaphore slot, and (if the idle count has dropped below `MinInstances`) launches a replacement in the background. A `ConcurrentDictionary<IBrowser, byte>` deduplicates the disconnect and lease-return paths to prevent double-dispose when both fire for the same browser.
288
+
271
289
`IBrowserPool.ActiveCount` and `IdleCount` expose live counters for observability.
272
290
273
291
`DisposeAsync` on the pool drains and disposes all idle browsers. Browsers that are currently leased out are not forcibly recalled; callers must dispose their leases before the pool can be fully cleaned up.
Copy file name to clipboardExpand all lines: docs/extensions/plugin-interfaces.md
+51-2Lines changed: 51 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Plugin Interfaces
2
2
3
-
Motus exposes seven extensibility interfaces that plugins can implement to participate in browser automation, test reporting, and accessibility auditing. All interfaces live in the `Motus.Abstractions` NuGet package. Plugins register their implementations through `IPluginContext` during initialization.
3
+
Motus exposes eight extensibility interfaces that plugins can implement to participate in browser automation, test reporting, accessibility auditing, and performance monitoring. All interfaces live in the `Motus.Abstractions` NuGet package. Plugins register their implementations through `IPluginContext` during initialization.
4
4
5
5
---
6
6
@@ -271,6 +271,55 @@ public sealed class A11yReporter : IReporter, IAccessibilityReporter
271
271
272
272
---
273
273
274
+
## IPerformanceReporter
275
+
276
+
`IPerformanceReporter` is an opt-in companion to `IReporter` for reporters that want to receive performance metrics collected during test execution. Motus checks `reporter is IPerformanceReporter` at runtime on each registered reporter; only those that implement both interfaces receive performance callbacks. This follows the same NativeAOT-safe pattern as `IAccessibilityReporter`.
277
+
278
+
### Members
279
+
280
+
| Member | Return Type | Description |
281
+
|---|---|---|
282
+
|`OnPerformanceMetricsCollectedAsync(PerformanceMetrics metrics, PerformanceBudgetResult? budgetResult, TestInfo test)`|`Task`| Called when performance metrics have been collected for a test. |
`PerformanceBudgetResult` carries the evaluation: `Entries` (`IReadOnlyList<PerformanceBudgetEntry>`) and `Passed` (`bool`). Each `PerformanceBudgetEntry` has `MetricName`, `Threshold`, `ActualValue`, `Passed`, and `Delta`.
289
+
290
+
`budgetResult` is `null` when no budget is configured.
291
+
292
+
### Lifecycle Notes
293
+
294
+
`OnPerformanceMetricsCollectedAsync` is called once per test when metrics have been collected (typically after a navigation). It is called before `OnTestEndAsync` for the same test. Exceptions are caught and logged; they do not suppress remaining reporter events.
`IAccessibilityRule` evaluates a single WCAG rule against one node in the accessibility tree. The engine walks every non-ignored node in the tree and calls each registered rule's `Evaluate` method. Rules are synchronous to keep the audit loop efficient; use the `AccessibilityAuditContext` for any cross-node data rather than performing additional async page queries.
@@ -384,7 +433,7 @@ public sealed class MyPlugin : IMotusPlugin
384
433
|`RegisterSelectorStrategy(ISelectorStrategy strategy)`|`void`| Registers a custom selector strategy. |
385
434
|`RegisterWaitCondition(IWaitCondition condition)`|`void`| Registers a custom wait condition. |
386
435
|`RegisterLifecycleHook(ILifecycleHook hook)`|`void`| Registers a lifecycle hook. |
387
-
|`RegisterReporter(IReporter reporter)`|`void`| Registers a test reporter. Reporters that also implement `IAccessibilityReporter` automatically receive violation events. |
436
+
|`RegisterReporter(IReporter reporter)`|`void`| Registers a test reporter. Reporters that also implement `IAccessibilityReporter`or `IPerformanceReporter`automatically receive the corresponding events. |
388
437
|`RegisterAccessibilityRule(IAccessibilityRule rule)`|`void`| Registers a custom accessibility rule that is invoked during accessibility audits. |
389
438
|`CreateLogger(string categoryName)`|`IMotusLogger`| Creates a logger scoped to the given category name. |
Copy file name to clipboardExpand all lines: docs/guides/configuration.md
+53-1Lines changed: 53 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -66,6 +66,17 @@ When Motus initializes, it walks up from `Environment.CurrentDirectory`, checkin
66
66
"auditAfterActions": false,
67
67
"includeWarnings": true,
68
68
"skipRules": []
69
+
},
70
+
"performance": {
71
+
"enable": false,
72
+
"collectAfterNavigation": true,
73
+
"lcp": null,
74
+
"fcp": null,
75
+
"ttfb": null,
76
+
"cls": null,
77
+
"inp": null,
78
+
"jsHeapSize": null,
79
+
"domNodeCount": null
69
80
}
70
81
}
71
82
```
@@ -154,6 +165,22 @@ Controls the built-in accessibility audit hook. When enabled, Motus runs axe-sty
154
165
|`includeWarnings`|`bool`|`true`| Treat warning-severity findings as failures alongside errors (when mode is `Enforce`). |
155
166
|`skipRules`|`string[]`|`null`| Rule IDs to exclude from every audit, e.g. `["a11y-color-contrast"]`. |
156
167
168
+
### `performance` Section
169
+
170
+
Controls the built-in performance metrics collector and budget thresholds. When enabled, Motus collects Core Web Vitals and supplementary metrics during test execution. Setting metric thresholds activates budget enforcement.
171
+
172
+
| Property | Type | Default | Description |
173
+
|---|---|---|---|
174
+
|`enable`|`bool`|`false`| Enable the performance metrics collector. |
175
+
|`collectAfterNavigation`|`bool`|`true`| Collect metrics after each page navigation. |
176
+
|`lcp`|`number`|`null`| Maximum Largest Contentful Paint in milliseconds. |
177
+
|`fcp`|`number`|`null`| Maximum First Contentful Paint in milliseconds. |
178
+
|`ttfb`|`number`|`null`| Maximum Time to First Byte in milliseconds. |
179
+
|`cls`|`number`|`null`| Maximum Cumulative Layout Shift score (unitless). |
180
+
|`inp`|`number`|`null`| Maximum Interaction to Next Paint in milliseconds. |
181
+
|`jsHeapSize`|`number`|`null`| Maximum JavaScript heap size in bytes. |
182
+
|`domNodeCount`|`number`|`null`| Maximum DOM node count. |
|`MOTUS_PERFORMANCE_ENABLE`|`performance.enable`|`bool`| Enable the performance metrics collector. |
210
+
|`MOTUS_PERFORMANCE_LCP`|`performance.lcp`|`double`| Maximum LCP in milliseconds. |
211
+
|`MOTUS_PERFORMANCE_FCP`|`performance.fcp`|`double`| Maximum FCP in milliseconds. |
212
+
|`MOTUS_PERFORMANCE_TTFB`|`performance.ttfb`|`double`| Maximum TTFB in milliseconds. |
213
+
|`MOTUS_PERFORMANCE_CLS`|`performance.cls`|`double`| Maximum CLS score. |
214
+
|`MOTUS_PERFORMANCE_INP`|`performance.inp`|`double`| Maximum INP in milliseconds. |
215
+
|`MOTUS_PERFORMANCE_JS_HEAP_SIZE`|`performance.jsHeapSize`|`long`| Maximum JS heap size in bytes. |
216
+
|`MOTUS_PERFORMANCE_DOM_NODE_COUNT`|`performance.domNodeCount`|`int`| Maximum DOM node count. |
182
217
183
-
> Not all config file sections have environment variable coverage. `reporter`, `recorder`, and the locator `selectorPriority` arrays are file-only settings. Use the config file for those.
218
+
> Not all config file sections have environment variable coverage. `reporter`, `recorder`, the locator `selectorPriority` arrays, and `performance.collectAfterNavigation` are file-only settings. Use the config file for those.
184
219
185
220
---
186
221
@@ -215,6 +250,7 @@ The merge checks are per-property:
215
250
-`IgnoreHTTPSErrors` -- a config value of `true` is applied when `options.IgnoreHTTPSErrors` is `false`. A config value of `false` never overrides a caller-set `true`.
216
251
-`Viewport` -- applied only when `options.Viewport is null`. Both `width` and `height` must be non-null in the config for a viewport to be applied.
217
252
-`Accessibility` -- the entire `AccessibilityOptions` object is applied only when `options.Accessibility is null`.
253
+
-`Performance` -- the entire `PerformanceOptions` object is applied only when `options.Performance is null`.
218
254
219
255
---
220
256
@@ -237,6 +273,7 @@ The merge checks are per-property:
237
273
|`DownloadsPath`|`string?`|`null`| Directory for browser-initiated file downloads. |
238
274
|`Plugins`|`IReadOnlyList<IPlugin>?`|`null`| Plugin instances loaded into every browser context created from this launch. |
239
275
|`Accessibility`|`AccessibilityOptions?`|`null`| Accessibility audit hook configuration. Disabled when `null`. |
276
+
|`Performance`|`PerformanceOptions?`|`null`| Performance metrics collector configuration. Disabled when `null`. |
240
277
241
278
### AccessibilityOptions
242
279
@@ -251,6 +288,17 @@ Nested under `LaunchOptions.Accessibility`. A `null` value disables the hook ent
251
288
|`IncludeWarnings`|`bool`|`true`| Count warning-severity findings as failures when mode is `Enforce`. |
252
289
|`SkipRules`|`IReadOnlyList<string>?`|`null`| Rule IDs excluded from every audit. |
253
290
291
+
### PerformanceOptions
292
+
293
+
Nested under `LaunchOptions.Performance`. A `null` value disables the collector entirely.
294
+
295
+
| Property | Type | Default | Description |
296
+
|---|---|---|---|
297
+
|`Enable`|`bool`|`false`| Enable the performance metrics collector. |
298
+
|`CollectAfterNavigation`|`bool`|`true`| Collect metrics after each page navigation. |
299
+
300
+
Budget thresholds are not part of `PerformanceOptions`. They are configured via the `[PerformanceBudget]` attribute or the `performance` section in `motus.config.json`.
0 commit comments