diff --git a/docs/test/configure-unit-tests-by-using-a-dot-runsettings-file.md b/docs/test/configure-unit-tests-by-using-a-dot-runsettings-file.md index e7e74b989c8..40fcd7c3764 100644 --- a/docs/test/configure-unit-tests-by-using-a-dot-runsettings-file.md +++ b/docs/test/configure-unit-tests-by-using-a-dot-runsettings-file.md @@ -1,7 +1,7 @@ --- title: Configure unit tests with a .runsettings file description: Learn how to use the .runsettings file in Visual Studio to configure unit tests that are run from the command line, from the IDE, or in a build workflow. -ms.date: 12/17/2025 +ms.date: 07/15/2026 ms.topic: how-to ms.author: mikejo @@ -150,6 +150,42 @@ To run tests from the command line, use *vstest.console.exe*, and specify the se For more information, see [VSTest.Console.exe command-line options](vstest-console-options.md). +## Override run settings from the command line + +You can also override some run settings directly from the command line. Specify run settings overrides after `--`. Include the space after `--`; all arguments after `--` are treated as run settings overrides, so these arguments must be last. Command-line run settings overrides take precedence over values from a file passed with `/Settings`. + +```cmd +vstest.console.exe test.dll -- MSTest.MapInconclusiveToFailed=True MSTest.DeploymentEnabled=False +``` + +The same syntax works with `dotnet test` (use `dotnet test -- MSTest.MapInconclusiveToFailed=True`). + +You can set **TestRunParameters** from the command line: + +```cmd +vstest.console.exe test.dll -- TestRunParameters.Parameter(name="myParam", value="value") +``` + +Shell quoting rules differ. In Command Prompt, escape quotation marks as needed. In PowerShell 7.3 and later, native command argument handling supports this syntax more directly; in earlier PowerShell versions, or when `$PSNativeCommandArgumentPassing` is set to `legacy`, use the `--%` stop-parsing token. In Bash, escape characters such as parentheses, quotation marks, spaces, and semicolons. + +In zsh (the default shell on macOS), use the same escaping as Bash: escape quotation marks, spaces, and semicolons. Because zsh also treats unquoted parentheses as globbing and array syntax, always escape or quote the parentheses in `TestRunParameters.Parameter(...)`. For example: + +```zsh +vstest.console.exe test.dll -- TestRunParameters.Parameter\(name=\"myParam\",\ value=\"value\"\) +``` + +Parameter values can contain semicolons and other special characters, such as connection strings, but the shell must pass them through literally. For complex values, consider using a *.runsettings* file instead: + +```xml + + + + + +``` + +For more command-line examples and shell-specific quoting details, see [Passing runsettings arguments through the command line](https://github.com/microsoft/vstest/blob/main/docs/RunSettingsArguments.md). + ## The *.runsettings file The *.runsettings file is an XML file that contains different configuration elements within the **RunSettings** element. The sections that follow detail the different elements. For a complete sample, see [Example *.runsettings file](#example-runsettings-file). @@ -511,6 +547,6 @@ The **RunConfiguration** node should contain an **EnvironmentVariables** node. A ## Related content -- [Configure a test run](https://github.com/microsoft/vstest-docs/blob/main/docs/configure.md) +- [Configure a test run](https://github.com/microsoft/vstest/blob/main/docs/configure.md) - [Customize code coverage analysis](../test/customizing-code-coverage-analysis.md) - [Visual Studio test task (Azure Test Plans)](/azure/devops/pipelines/tasks/test/vstest?view=vsts&preserve-view=true) diff --git a/docs/test/migrate-testsettings-to-runsettings.md b/docs/test/migrate-testsettings-to-runsettings.md index 280056f1585..6bb3cdcfd43 100644 --- a/docs/test/migrate-testsettings-to-runsettings.md +++ b/docs/test/migrate-testsettings-to-runsettings.md @@ -1,7 +1,7 @@ --- title: Migrate testsettings to runsettings description: Learn how to migrate testsettings to runsettings -ms.date: 03/18/2021 +ms.date: 07/17/2026 ms.topic: upgrade-and-migration-article f1_keywords: - vs.UnitTest.Migrate @@ -34,7 +34,7 @@ SettingsMigrator.exe E:\MyTest\MyTestSettings.testsettings SettingsMigrator.exe E:\MyTest\MyTestSettings.testsettings E:\MyTest\MyNewRunSettings.runsettings ``` -If you're interested in reading more on how the *.testsettings* options convert to *.runsettings* you can find more implementation details in the [open source test platform repository](https://github.com/microsoft/vstest-docs/blob/main/RFCs/0023-TestSettings-Deprecation.md#migration) on GitHub. +If you're interested in reading more on how the *.testsettings* options convert to *.runsettings*, you can find more implementation details in the [open source test platform repository](https://github.com/microsoft/vstest/blob/main/docs/RFCs/0023-TestSettings-Deprecation.md#migration) on GitHub. ## Related content diff --git a/docs/test/vstest-console-options.md b/docs/test/vstest-console-options.md index f4223c3ec17..78c8f495c5c 100644 --- a/docs/test/vstest-console-options.md +++ b/docs/test/vstest-console-options.md @@ -1,7 +1,7 @@ --- title: VSTest.Console.exe command-line options description: Learn about the VSTest.Console.exe command-line tool that runs tests. This article includes the General command-line options. -ms.date: 02/28/2025 +ms.date: 07/17/2026 ms.topic: reference helpviewer_keywords: - vstest.console.exe @@ -24,7 +24,7 @@ Open [Developer Command Prompt](../ide/reference/command-prompt-powershell.md) t ## General command-line options -The following table lists all the options for *VSTest.Console.exe* and short descriptions of them. You can see a similar summary by typing `VSTest.Console/?` at a command line. +The following table lists the commonly used options for *VSTest.Console.exe* and short descriptions of them. You can see a similar summary by typing `VSTest.Console/?` at a command line. For the complete reference, including internal and legacy switches that aren't listed here, see [vstest.console.exe command line options](https://github.com/microsoft/vstest/blob/main/docs/commandline.md) and specifically [Omitted switches](https://github.com/microsoft/vstest/blob/main/docs/commandline.md#omitted-switches) in the vstest repository. | Option | Description | |---|---| @@ -32,26 +32,22 @@ The following table lists all the options for *VSTest.Console.exe* and short des |**/Settings:[*file name*]**|Run tests with additional settings such as data collectors. For more information, see [Configure unit tests using a .runsettings file](../test/configure-unit-tests-by-using-a-dot-runsettings-file.md)
Example: `/Settings:local.runsettings`| |**/Tests:[*test name*]**|Run tests with names that contain the provided values. This command matches against the full test name, including the namespace. To provide multiple values, separate them by commas.
Example: `/Tests:TestMethod1,testMethod2`
The **/Tests** command-line option can't be used with the **/TestCaseFilter** command-line option.| |**/Parallel**|Specifies that the tests be executed in parallel. By default, up to all available cores on the machine can be used. You can configure the number of cores to use in a settings file.| -|**/Enablecodecoverage**|Enables data diagnostic adapter CodeCoverage in the test run.
Default settings are used if not specified using settings file.| |**/InIsolation**|Runs the tests in an isolated process.
This isolation makes the *vstest.console.exe* process less likely to be stopped on an error in the tests, but tests might run slower.| -|**/UseVsixExtensions**|This option makes the *vstest.console.exe* process use or skip the VSIX extensions installed (if any) in the test run.
This option is deprecated. Starting from the next major release of Visual Studio this option may be removed. Move to consuming extensions made available as a NuGet package.
Example: `/UseVsixExtensions:true`| |**/TestAdapterPath:[*path*]**|Forces the *vstest.console.exe* process to use custom test adapters from a specified path (if any) in the test run.
Example: `/TestAdapterPath:[pathToCustomAdapters]`| -|**/Platform:[*platform type*]**|Forces the given platform to be used, instead of the platform determined from the current runtime. This option is able to force only x86, and x64 platforms on Windows. The ARM option is broken and will result in x64 on most systems.
Do NOT specify this option to run on runtimes that aren't in the list of valid values such as ARM64.
Valid values are x86, x64, and ARM.
-|**/Framework: [*framework version*]**|Target .NET version to be used for test execution.
Example values are `Framework35`, `Framework40`, `Framework45`, `FrameworkUap10`, `.NETCoreApp,Version=v1.1`.
TargetFrameworkAttribute is used to automatically detect this option from your assembly, and defaults to `Framework40` when the attribute isn't present. You must specify this option explicitly if you remove the [TargetFrameworkAttribute](/dotnet/api/system.runtime.versioning.targetframeworkattribute) from your .NET Core assemblies.
If the target framework is specified as **Framework35**, the tests run in CLR 4.0 "compatibility mode".
Example: `/Framework:framework40`| -|**/TestCaseFilter:[*expression*]**|Run tests that match the given expression.
is of the format =[\|].
Example: `/TestCaseFilter:"Priority=1"`
Example: `/TestCaseFilter:"TestCategory=Nightly|FullyQualifiedName=Namespace.ClassName.MethodName"`
The **/TestCaseFilter** command-line option can't be used with the **/Tests** command-line option.
For information about creating and using expressions, see [TestCase filter](https://github.com/Microsoft/vstest-docs/blob/main/docs/filter.md).| +|**/Platform:[*platform type*]**|Forces the given platform architecture to be used, instead of the platform determined from the current runtime. Values are case-insensitive; the accepted values are `x86`, `x64`, `ARM`, `ARM64`, `S390x`, `Ppc64le`, `RiscV64`, and `LoongArch64`.
On Windows, only x86 and x64 can be reliably forced; specifying `ARM` results in x64 on most systems. Don't specify this option to run on a runtime that isn't in the list of valid values.| +|**/Framework: [*framework version*]**|Target .NET version to be used for test execution.
Modern framework short forms are accepted and parsed by the NuGet framework parser, for example `net48`, `net6.0`, or `net10.0` (as well as the long forms such as `.NETFramework,Version=v4.8` and `.NETCoreApp,Version=v10.0`).
The legacy aliases `Framework35`, `Framework40`, `Framework45`, `FrameworkCore10`, and `FrameworkUap10` are also accepted.
TargetFrameworkAttribute is used to automatically detect this option from your assembly, and defaults to `Framework40` when the attribute isn't present. You must specify this option explicitly if you remove the [TargetFrameworkAttribute](/dotnet/api/system.runtime.versioning.targetframeworkattribute) from your .NET Core assemblies.
If the target framework is specified as **Framework35**, the tests run in CLR 4.0 "compatibility mode".
Example: `/Framework:net8.0`| +|**/TestCaseFilter:[*expression*]**|Run tests that match the given expression.
is of the format =[\|].
Example: `/TestCaseFilter:"Priority=1"`
Example: `/TestCaseFilter:"TestCategory=Nightly|FullyQualifiedName=Namespace.ClassName.MethodName"`
The **/TestCaseFilter** command-line option can't be used with the **/Tests** command-line option.
For information about creating and using expressions, see [TestCase filter](https://github.com/microsoft/vstest/blob/main/docs/filter.md). When you type a filter directly in a shell, see [Escape filter expressions in the shell](#escape-filter-expressions-in-the-shell).| +|**/Environment:[*NAME*]=[*VALUE*]**|Sets the value of an environment variable for the test host process. Creates the variable if it doesn't exist, and overrides it if it does. This option implies **/InIsolation** and forces the tests to run in an isolated process. Specify the option multiple times to set multiple variables. Short form: **/e**.
Example: `/e:VARIABLE1=VALUE1`| |**/?**|Displays usage information.| -|**/Logger:[*uri/friendlyname*]**|Specify a logger for test results. Specify the parameter multiple times to enable multiple loggers.
Example: To log results into a Visual Studio Test Results File (TRX), use
**/Logger:trx**
**[;LogFileName=\]**
For more information, see [Logging example](#logging-example).| -|**/ListTests:[*file name*]**|Lists discovered tests from the given test container.
Note: The `/TestCaseFilters` option has no effect when listing tests; it only controls which tests get run.| -|**/ListDiscoverers**|Lists installed test discoverers.| -|**/ListExecutors**|Lists installed test executors.| -|**/ListLoggers**|Lists installed test loggers.| -|**/ListSettingsProviders**|Lists installed test settings providers.| -|**/Blame**|Runs the tests in blame mode. This option is helpful in isolating problematic tests that cause the test host to crash. When a crash is detected, it creates a sequence file in `TestResults//_Sequence.xml` that captures the order of tests that were run before the crash. For more information, see [Blame data collector](https://github.com/Microsoft/vstest-docs/blob/main/docs/extensions/blame-datacollector.md).| -|**/Diag:[*file name*]**|Writes diagnostic trace logs to the specified file.| +|**/Logger:[*uri/friendlyname*]**|Specify a logger for test results. Specify the parameter multiple times to enable multiple loggers.
Example: To log results into a Visual Studio Test Results File (TRX), use
**/Logger:trx**
**[;LogFileName=\]**
Use `LogFilePrefix=` instead of `LogFileName` to keep a separate, timestamped file per run. `LogFileName` sets an explicit name and overwrites the previous file, whereas `LogFilePrefix` doesn't.
For more information, see [Logging example](#logging-example).| +|**/ListTests:[*file name*]**|Lists discovered tests from the given test container. Short form: **/lt**.
Note: The `/TestCaseFilter` option has no effect when listing tests; it only controls which tests get run.| +|**/Blame**|Runs the tests in blame mode. This option is helpful in isolating problematic tests that cause the test host to crash. When a crash is detected, it creates a sequence file in `TestResults//_Sequence.xml` that captures the order of tests that were run before the crash.
You can also collect a crash or hang dump, for example `/Blame:CollectDump;DumpType=full` or `/Blame:CollectHangDump;TestTimeout=90m;HangDumpType=mini`. The equivalent `dotnet test` switches are `--blame-crash` and `--blame-hang`.
For the full option matrix and dump-collection requirements, see [Blame data collector](https://github.com/microsoft/vstest/blob/main/docs/extensions/blame-datacollector.md).| +|**/Diag:[*file name*]**|Writes diagnostic trace logs to the specified file.
Set the trace level with `/Diag:;tracelevel=` (default is `verbose`).| |**/ResultsDirectory:[*path*]**|Test results directory will be created in specified path if not exists.
Example: `/ResultsDirectory:`| |**/ParentProcessId:[*parentProcessId*]**|Process ID of the Parent Process responsible for launching current process.| |**/Port:[*port*]**|The Port for socket connection and receiving the event messages.| -|**/Collect:[*dataCollector friendlyName*]**|Enables data collector for the test run. [More information](https://github.com/Microsoft/vstest-docs/blob/main/docs/analyze.md).| +|**/Collect:[*dataCollector friendlyName*]**|Enables data collector for the test run. [More information](https://github.com/microsoft/vstest/blob/main/docs/analyze.md).| +|**@[*file*]**|Reads additional options from the specified response file. Arguments in the file are separated by whitespace (spaces or newlines) and quoting is supported, so options can span multiple lines.
Example: `vstest.console.exe @options.rsp`| > [!TIP] > The options and values aren't case-sensitive. @@ -101,6 +97,34 @@ Example of a generated *sequence.xml* file: ``` +In this case, the `` listed last is the test that was running at the time of the crash. + +## Exit codes + +*vstest.console.exe* returns one of two exit codes: + +| Code | Meaning | +|---|---| +| `0` | Success. The requested operation completed and, for a test run, all executed tests passed. | +| `1` | Failure. For example, one or more tests failed, a run error was reported, the command line was invalid or missing, a test source couldn't be loaded, or the run was aborted or canceled. | + +The process never returns any other value. When you run tests through `dotnet test`, the .NET SDK surfaces a non-zero exit code when the run fails in the same way. + +When discovery finds no matching tests, the runner prints a *warning* rather than an error, and by default still returns `0`. To make a run that discovers or selects zero tests return `1` instead, set `true` in the **RunConfiguration** element of your *.runsettings* file. For more information, see [Configure unit tests using a .runsettings file](configure-unit-tests-by-using-a-dot-runsettings-file.md). + +## Escape filter expressions in the shell + +A **/TestCaseFilter** expression is parsed by both your shell and the test platform, so some characters need shell-specific escaping before *vstest.console.exe* receives them. Quoting the whole expression, as in the examples earlier in this article, avoids most problems. The following cases need extra care: + +- **PowerShell**: The comma (`,`) is the array operator and the semicolon (`;`) is a statement separator. Quote the entire filter expression so it's passed through literally, for example `/TestCaseFilter:"FullyQualifiedName=MyNamespace.MyClass.MyMethod"`. +- **Bash and zsh (Linux and macOS)**: Escape `!` with a backslash when you use the `!~` (*not contains*) operator, for example `--filter FullyQualifiedName\!~IntegrationTests` with `dotnet test`. Also quote values that contain characters with special meaning to the shell, such as `<`, `>`, or `,` in a generic type argument list: + + ```bash + dotnet test --filter "FullyQualifiedName=MyNamespace.MyClass.MyMethod" + ``` + +For the full filtering reference and supported properties per test framework, see [TestCase filter](https://github.com/microsoft/vstest/blob/main/docs/filter.md). + ## Logging example Each logger defines its own parameters. Unlike trx, the console logger allows you to set verbosity level. For additional information, type `VSTest.Console/?` at the command line. @@ -119,6 +143,8 @@ In PowerShell, you need to use quotes: vstest.console.exe myTestFile.dll /logger:"console;verbosity=detailed" ``` +For the full list of available loggers, as well as instructions for authoring your own logger, see [Reporting test results](https://github.com/microsoft/vstest/blob/main/docs/report.md) in the vstest repository. + ## UWP example For UWP, the appxrecipe file must be referenced instead of a DLL. @@ -126,3 +152,26 @@ For UWP, the appxrecipe file must be referenced instead of a DLL. ```cmd vstest.console.exe /Logger:trx /Platform:x64 /framework:frameworkuap10 UnitTestsUWP\bin\x64\Release\UnitTestsUWP.build.appxrecipe ``` + +## Environment variables + +The test platform recognizes several environment variables. The following are the ones most useful when you run tests from the command line. For the complete list, see [Environment variables understood by the test platform](https://github.com/microsoft/vstest/blob/main/docs/environment-variables.md) in the vstest repository. + +| Variable | Description | +|---|---| +|`VSTEST_CONNECTION_TIMEOUT`|Timeout, in seconds, for establishing connections between test platform components (*vstest.console.exe*, testhost, and data collector). The default is 90. Increase it on slow machines or when network latency causes connection timeouts.| +|`VSTEST_DIAG`|Enables diagnostic logging and specifies the path to the log file. Equivalent to the **/Diag** option.| +|`VSTEST_DIAG_VERBOSITY`|Sets the verbosity of diagnostic logging when `VSTEST_DIAG` is enabled. Valid values are `Verbose`, `Info`, `Warning`, and `Error` (default is `Verbose`).| +|`VSTEST_HOST_DEBUG`|Set to any non-empty value to enable debugging of the testhost process.| +|`VSTEST_RUNNER_DEBUG`|Set to any non-empty value to enable debugging of the runner (*vstest.console.exe*).| +|`VSTEST_DUMP_PATH`|Overrides the default directory where blame crash dumps are stored.| +|`VSTEST_DUMP_FORCEPROCDUMP`|Set to any non-empty value to force ProcDump to be used for crash dump collection.| +|`VSTEST_DISABLE_UTF8_CONSOLE_ENCODING`|Set to `1` to disable setting UTF-8 encoding on console output.| +|`VSTEST_CONSOLE_PATH`|Path to the *vstest.console.exe* executable used by the .NET SDK's `dotnet test` forwarding app. Equivalent to `-p:VSTestConsolePath` when you run `dotnet test` on a project.| + +## Related content + +- [Quickstart: run tests from the command line](https://github.com/microsoft/vstest/blob/main/docs/quickstart.md) in the vstest repository +- [Configure unit tests using a .runsettings file](configure-unit-tests-by-using-a-dot-runsettings-file.md) +- [Create a data collector](https://github.com/microsoft/vstest/blob/main/docs/extensions/datacollector.md) in the vstest repository +- [dotnet test command reference](/dotnet/core/tools/dotnet-test)