Running Code Analyzers on Build Server
Diagnostic Analyzers in Visual Studio 2015
The introduction of diagnostic analyzers in Visual Studio 2015 significantly lowered the bar for development of custom static code analyzers. In less than a year this resulted in an impressive selection of third party analyzers that can be used in Visual Studio to warn you about bad practices and potential issues in your code. Finally, this is no longer limited to high-end Visual Studio extensions, such as ReSharper.
On top of that, the infrastructure around diagnostic analyzers makes it much easier to have the same static analysis run after every commit as part of continuous integration on your build server. While this was theoretically also possible with ReSharper Command Line Tools, it required additional effort to correctly setup build configurations on the server, and even more so if you wanted the same set of inspections to be run in Visual Studio and on the server.
With diagnostic analyzers, you can achieve all that without any reconfiguration of the build server. You just need to configure your Visual Studio projects correctly and the same analyzers will automatically run in Visual Studio and on the build server.
Configuring Your Projects
Most of diagnostic analyzers are available in two different forms:
- as a Visual Studio extension, and
- as a NuGet package.
If you install the extension, its analyzers will automatically be run for any project you load in Visual Studio. Although this might seem a better choice, it actually isn't. The extension will only work for you on the machine where you installed it. The analyzers won't run for other developers, nor on the build server. On the other hand, if you install the NuGet package in your project, the information about it will automatically be written to the project configuration files. This way the configuration will be preserved and available wherever the project will be built.
Each analyzer comes with a default severity for each of its rules. error, warning, info or hidden. This defines how a broken rule will affect your build. If you want to put in the extra effort, you can customize these settings for every rule by right clicking on it in Solution Explorer and selecting the desired severity from the context menu. These changes will also be saved with the project so that the behavior will be the same for everyone.
Configuring the Build Server
Your build server is probably already using MSBuild to build the Visual Studio projects. In this case it should automatically also run the analyzers, as configured above. If it doesn't, the most common reasons are:
- The MSBuild version used by the build server is not recent enough. It must be at least version 14, i.e. the one released in Visual Studio 2015 time frame. Depending on the analyzers used, a higher version might be required, therefore you should make sure, the build server has the same Visual Studio update installed as the developers.
- NuGet packages are not being restored before the build. With the recent versions of NuGet, this needs to be done by a separate call to NuGet before the build. In Visual Studio, The NuGet extension does this automatically for you.
By default, MSBuild will fail the build for any broken rule of error severity and write the information about broken rules with lower severity to the build log. This behavior can be changed with the Treat warnings as errors project setting in Visual Studio.
How the broken rules that haven't failed the build affect the final outcome, will depend on the build server and its configuration. In TeamCity warnings are only visible in build log by default. Although warnings are treated as important messages, hence rendered in a different color and available for filtering by that criteria, their presence isn't noted anywhere else in the build report. With some additional configuration, you can make them much more meaningful:
- Show their count in the build overview.
- List them in a separate build report page.
- Create a custom metric and e.g. fail a build when their number increases.
With some effort, similar results should be achievable with most build servers.
