Continuous Testing in Visual Studio
Continuous Testing?
The smaller the overhead of running unit tests, the more often you are going to run them and the sooner you will notice that the changes you have made to the code have broken existing tests. Quick feedback loop is very important to make development more effective; even more so in test driven development.
Ever since I have tried out NCrunch, I never looked back. Continuous running of tests as implemented in NCrunch significantly changed how I look at tests. At that time the only competition was Mighty Moose, a.k.a. Continuous Tests, but it didn't manage to convince me. Since then continuous testing has become much more mainstream and today both ReSharper and stock Visual Studio offer support for it. It was about time I took another look at the other tools and see how they compare to NCrunch today. Is NCrunch still worth the price it is sold for?
NCrunch
Being a regular user of NCrunch, I see it as a baseline for comparing the other tools. The basic premise hasn't really changed since its initial release: once enabled for a solution, it starts monitoring the source code files and as soon as it detects a change (even without saving the file), it rebuilds the affected parts of the solution and runs the tests. If a test fails, this is immediately visualized in 2 locations:
- The test is marked as failed in the tests window. This window is very configurable and I prefer to have only the failing tests visible. This makes the window empty most of the time, hence failing tests are all the more visible.
- The coverage markers in code editor turn from green to red in all lines that were covered by the failing tests. Usually this happens right along the lines where you are currently editing the code, warning you about the issue that you have just caused.
Of course, continuous rebuilding and running of tests requires a significant amount of resources. NCrunch does a good job at prioritizing the tests to provide the most feedback even before all the tests are run. If that's not enough, it provides the means to limit automatic running to only a subset of tests. There's even a remote runner available that can be installed on other machines allowing you to offload rebuilding and running of tests to other machines, reducing the load on your development computer.
As you can tell, I like the tool a lot, but there are still things that bother me:
- Not all target frameworks are supported, most notably support for Windows Phone, Windows Store and Universal applications is lacking. It's still possible to test business logic implemented in portable class libraries, but for testing the platform specific code another runner is required.
- The tool isn't cheap and even requires an upgrade license to keep up with new versions released more than 12 months after buying the original license.
ReSharper
ReSharper might be the most popular commercial Visual Studio available. It offers a very extensive set of features, including a test runner and code coverage tool. Since its latest version (ReSharper 10), the test runner has been enhanced with support for continuous testing by taking advantage of another new high profile feature: ReSharper Build.
It offers a lot of what is available in NCrunch, although it is not yet completely on a par with it:
- Changes are only detected when a file is saved.
- In spite of using optimized building and running only affected tests, the feedback in general seems slower than in NCrunch.
- There's no support for distributed processing.
- Code coverage indicators don't change because of failed tests.
To be fair, there are some advantages to ReSharper as well. Most noticeably, continuous testing is supported in all target frameworks, including the modern platforms. Being priced similarly to NCrunch it also offers much more for this price. As it is a part of dotCover, you need ReSharper Ultimate, so you might need to upgrade your license even if you are already an existing ReSharper user.
Visual Studio
As one would expect, Visual Studio offering is the weakest. I have a hard time calling it continuous testing at all. It's just a flag to run the tests automatically after every build, i.e. it doesn't monitor the files for changes or run the tests when you save a file. You actually need to trigger a rebuild to have the tests run, which is not all that different from just running the tests manually, which causes a rebuild any way. I guess, this way you won't forget to run the tests, but if you're at least a little bit serious about testing, you wouldn't do that any way.
On top of that the feature is not available in the free Community edition of Visual Studio 2015, giving it a higher price tag than both NCrunch and ReSharper.
Mighty Moose
I remember Mighty Moose being a serious alternative to NCrunch a couple of years ago. I can't say it still is, though. In the meantime it transformed from a commercial tool to an open source one with not all that many activity or contributors. There's not even a binary distribution available; you need to clone the repository and build it yourself. Well, at least the build script worked flawlessly the first time.
With all the warnings about the Visual Studio extension being unreliable I decided against installing it and only tried out the standalone executable. I know that because of this I opted out of features, such as debugging and coverage visualization, but I didn't feel like crashing my Visual Studio instance just to give this extension a try. I suppose most potential users will be in a similar state of mind.
Even as an external tool, Mighty Moose can track saved changes to source code files and rebuilds the projects and runs the tests automatically when it detects a change. However, it doesn't support other target frameworks than the classic .NET framework, and can't even handle C# 6.
Verdict
I'm sticking with NCrunch for the time being. Its features and user experience are still superior to the alternatives. As a ReSharper Ultimate user, I will probably enable its continuous testing feature when working with target frameworks, not supported by NCrunch, such as Windows Phone and universal applications. Both Visual Studio and Mighty Moose are not worth seriously considering in their current state. As it seems continuous testing still has a non-negligible price tag in the .NET ecosystem.
