[perf-improver] perf: add dotnet test server-mode scenario to performance runner

[Evangelink]

Goal and Rationale

The nightly perf-timing workflow collects PlainProcess timing for MSTest's direct-run mode (MTP native process). This exercises the plain-process path where the test host runs standalone as an executable.

The MTP server-mode path (triggered by dotnet test) follows a completely different code path: the test host is invoked with --server --protocol dotnet-test-protocol, communicates results over a named pipe, and the JSON-RPC serialisation/framing/pipe I/O overhead is included in the user-visible wall-clock time. Multiple recent efficiency-improver PRs targeted exactly this path (#9274, #9300, #9353, #9380, #9408, #9436), but there has been no perf scenario to verify or measure their impact, and no regression detection for it.

This PR adds a Scenario1_DotnetTest_PlainProcess pipeline that runs the same 100×100 test asset via dotnet test --no-build, bridging that measurement gap. Addresses the gap identified in #9480.

Approach

New step: Steps/DotnetTestProcess.cs

Mirrors PlainProcess in structure but invokes dotnet test "<projectDir>" --no-build using the repo-local SDK, rather than running the compiled executable directly. Key design points:

• Uses the same {root}/.dotnet/dotnet muxer as DotnetMuxer (consistent toolchain)
• Sets the same SDK environment variables (DOTNET_ROOT, DOTNET_INSTALL_DIR, etc.)
• Drains stdout/stderr asynchronously to prevent pipe-buffer deadlocks
• Records ElapsedTime (wall-clock) and TotalProcessorTime in the same JSON format as PlainProcess

Measurement note (documented in XML doc): TotalProcessorTime covers only the dotnet test parent process; the spawned test-host child's CPU time is not included. Wall-clock ElapsedTime is the primary metric and represents what users observe.

New pipeline entry in Program.cs:

pipelineRunner.AddPipeline("Default", "Scenario1_DotnetTest_PlainProcess",
    [OSPlatform.Windows, OSPlatform.Linux, OSPlatform.OSX], ...);

The name contains "PlainProcess" so the existing nightly workflow filter --pipelineNameFilter "*PlainProcess*" picks it up automatically — no workflow file changes required.

Performance Evidence

This is a measurement infrastructure change, not a code optimization. It adds observability for a previously unmeasured path. Comparable baseline (PlainProcess, 100 classes × 100 methods, Debug, Linux): ~3.5 s wall-clock per run. Server-mode overhead is expected to add protocol setup time (typically 0.5–1.5 s for pipe init + JSON-RPC negotiation) on top of test execution.

Trade-offs

• Adds ~3 × dotnet test invocations to the nightly perf run (one per _numberOfRun). Each run takes ~5–10 s. Total additional nightly time: ~30–60 s per platform.
• No production source code is modified; risk is limited to the performance runner.

Test Status

Local SDK not available in CI agent (pinned SDK 11.0.100-preview). Build and scenario execution delegated to CI.

The implementation follows the established PlainProcess pattern exactly, substituting only the launched process (dotnet test vs. the direct test-host executable). The DotnetTestProcess step correctly handles output draining and produces the same Result.json format.

Reproducibility

# After ./build.sh -pack
.dotnet/dotnet run --project test/Performance/MSTest.Performance.Runner -c Release \
  -- execute --pipelineNameFilter "*DotnetTest*"

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.8K AIC · ⌖ 26.3 AIC · ⊞ 57.6K · [◷]( · ◷)

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

[Copilot]

Pull request overview

Adds a new performance-runner scenario to measure the dotnet test (MTP server-mode / JSON-RPC) execution path, complementing the existing direct executable (plain-process) timing scenario so nightly perf runs can detect regressions/improvements in the server-mode pipeline.

Changes:

• Introduces a new DotnetTestProcess step that runs dotnet test --no-build and records timing metrics into Result.json.
• Registers a new pipeline Scenario1_DotnetTest_PlainProcess in the performance runner so it’s picked up by the existing nightly *PlainProcess* filter.

Show a summary per file

File	Description
test/Performance/MSTest.Performance.Runner/Steps/DotnetTestProcess.cs	New step that launches dotnet test and captures wall-clock/CPU metrics + zips the asset for upload.
test/Performance/MSTest.Performance.Runner/Program.cs	Adds a new pipeline wiring Scenario1 + build + DotnetTestProcess into the default runner set.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Low

[Evangelink]

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

#	Dimension	Verdict
1	Algorithmic Correctness	🟠 1 MAJOR — no exit-code check, silent bad-data on dotnet test failure
1	Algorithmic Correctness	🟡 1 MODERATE — missing --configuration flag, fragile implicit Debug coupling

✅ 21/22 dimensions clean.

• Exit-code check — dotnet test has many more failure modes than a pre-located binary (project not found, SDK mismatch, JIT error). Without checking process.ExitCode, a failed run records a short "elapsed time" that silently contaminates the perf baseline. See inline comment on line 77.
• --configuration propagation — DotnetTestProcess implicitly relies on dotnet test --no-build defaulting to Debug. If a pipeline ever wires DotnetMuxer(BuildConfiguration.Release) before this step, the step will silently measure stale Debug artefacts (or fail). Adding a BuildConfiguration constructor parameter keeps this class self-contained. See inline comment on line 54.

---

Non-blocking design note: The scenario name Scenario1_DotnetTest_PlainProcess embeds "PlainProcess" only to satisfy the nightly workflow's *PlainProcess* filter — as the Program.cs comment explains. This works but inverts the normal naming contract: the name describes the measurement class (PlainProcess) rather than the invocation path (DotnetTest). A cleaner long-term solution would be to extend the nightly workflow filter to *PlainProcess*|*DotnetTest* and rename the scenario to Scenario1_DotnetTest so it says what it actually does. The current approach isn't wrong, but the comment dependency (why does a DotnetTest scenario have "PlainProcess" in its name?) will surprise the next contributor who looks at the file in isolation.

[Evangelink]

🧪 Test quality grade — PR #9486

No new or modified test methods were identified in the changed regions of this PR. Nothing to grade.

_{Re-run with /grade-tests.}

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Grade Tests on PR (on open / sync) workflow. · 108.6 AIC · ⌖ 13.6 AIC · ⊞ 45.7K · [◷]( · ◷)