I was adding NUnit tests lately that were not supposed to run every time, but only when explicitly specified. They start and end an external process in order to test some IPC. I did not want those tests executed by default, especially not on the build server. Of course I could have changed the --filter on the build server. But I do run tests locally from the command line sometimes, and do not want to have to remember (and even less type) to filter them every time.
Luckily, there is a way involving just two steps:
- Create a runsettings file
- Make the CLI use the runsettings file by default
Create a runsettings file
Now, this sounds like it would be pretty straightforward with all the Microsoft documentation. Unfortunately, that will not tell you how to define filters. Extrapolating from the part in the documentation about defining test-runner-specific configuration items in the runsettings file, I looked at the NUnit documentation for its own “where” filter. This tells you right away how to define NUnit filters in your runsettings. There is an entire blog post about it. Hooray….
… except it does not work when using dotnet test. I got strange errors about tests 0-1035 not being in some cache and it would not run any test at all.
So I duck-duck-goed (or is it duck-duck-went?) a bit with different search phrases, The solution to the problem is described in a github PR comment for vstest (yes, vstest) of all places. Just wow, Microsoft.
The magic spell uses a TestCaseFilter element.
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<TestCaseFilter>TestCategory!=badcategory</TestCaseFilter>
</RunConfiguration>
</RunSettings>
This is all fine as long as you remember to always write dotnet test MySolution.sln -s .runsettings. Not good enough.
Use runsettings by default
One way to get the CLI to respect the runsettings by default is to include the RunSettingsFilePath element in all test projects and have it point to the one runsettings file. Fortunately with the SDK project format, this can be just a line in a Directory.Build.props file in the root folder of all test projects, e. g. <reporoot>\test. For me it turned out like this
<PropertyGroup> <RunSettingsFilePath>$(MSBuildProjectDirectory)\..\..\.runsettings</RunSettingsFilePath> </PropertyGroup>
Now I can just type dotnet test MySolution.sln or dotnet test test\SomeTestProject and the runsettings will be honoured.
Visual Studio remains a problem though. Although the documentation states that having the RunSettingsFilePath set in the project should cause VS to honor it, this does not work, at least not for filtering. Turning on auto-detection or even explicitly specifying a solution-wide runsettings file has no effect when running “All Tests” or running a single test project from the UI.
But hey, at least the build server and local CLI is taken care of :-). You can always just do the two clicks excluding the categories from the Test Explorer.
