Damir's Corner.

Quick Guide to Unit Testing Diagnostic Analyzers

Unit Testing Project

Effective development of diagnostic analyzers strongly depends on unit testing. Even if you're not a proponent of TDD or testing in general, you'll start to share my opinion as soon as you'll attempt to debug an analyzer for the first time. Debugging diagnostic analyzers requires a second instance of Visual Studio to be started which will host the debugged analyzer as an extension. This takes far too long for being useful during development, therefore using tests instead is a must.

Fortunately a test project is automatically created for you by the template which gets you going without having to understand all the details of how it actually works. You can base your tests on the two examples that are included in the generated test project:

//No diagnostics expected to show up
[TestMethod]
public void TestMethod1()
{
    var test = @"";

    VerifyCSharpDiagnostic(test);
}

//Diagnostic and CodeFix both triggered and checked for
[TestMethod]
public void TestMethod2()
{
    var test = @"
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    using System.Diagnostics;

    namespace ConsoleApplication1
    {
        class TypeName
        {   
        }
    }";
    var expected = new DiagnosticResult
    {
        Id = Analyzer9Analyzer.DiagnosticId,
        Message = String.Format("Type name '{0}' contains lowercase letters", 
            "TypeName"),
        Severity = DiagnosticSeverity.Warning,
        Locations =
            new[] {
                    new DiagnosticResultLocation("Test0.cs", 11, 15)
                }
    };

    VerifyCSharpDiagnostic(test, expected);

    var fixtest = @"
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    using System.Diagnostics;

    namespace ConsoleApplication1
    {
        class TYPENAME
        {   
        }
    }";
    VerifyCSharpFix(test, fixtest);
}

Creating a Unit Test

To create your own test class, you need to inherit it from DiagnosticVerifier or CodeFixVerifier, based on whether you will only test a diagnostic analyzer or a code fix as well. Make sure, you instantiate the right analyzer, include some source code to try it out on, and you're ready to test:

[TestClass]
public class UnitTest : CodeFixVerifier
{

    //No diagnostics expected to show up
    [TestMethod]
    public void TestMethod1()
    {
        var test = @"
    using System.Text.RegularExpressions;

    namespace RegExSample
    {
        public class Class1
        {
            public void Foo()
            {
                Regex.Match("""", """");
            }
        }
    }";

        VerifyCSharpDiagnostic(test);
    }

    //Diagnostic triggered and checked for
    [TestMethod]
    public void TestMethod2()
    {
        var test = @"
    using System.Text.RegularExpressions;

    namespace RegExSample
    {
        public class Class1
        {
            public void Foo()
            {
                Regex.Match("""", ""["");
            }
        }
    }";
        var expected = new DiagnosticResult
        {
            Id = RegexAnalyzerAnalyzer.DiagnosticId,
            Message = String.Format("Regular expression is invalid: {0}", 
                @"parsing ""["" - Unterminated [] set."),
            Severity = DiagnosticSeverity.Error,
            Locations =
                new[] {
                        new DiagnosticResultLocation("Test0.cs", 10, 33)
                    }
        };

        VerifyCSharpDiagnostic(test, expected);
    }

    protected override DiagnosticAnalyzer GetCSharpDiagnosticAnalyzer()
    {
        return new RegexAnalyzerAnalyzer();
    }
}

The only tricky part is, getting the correct values for the expected diagnostic. In my experience, it's best to set your best guess there at first, let the test fail because of mismatched values and set the expected properties one by one, until the test passes. At least for getting the correct location, this should be easier than trying to calculate it yourself.

Adding Assembly References

The above described approach should work fine, as long as your code snippets for testing the analyzers require no assembly references. If they do, your analyzer will stop reporting the diagnostic (or misbehave differently depending on how you have implemented it) because it won't find the expected symbols. With the current templates (RC) there is no simple way to add those references per test method or test class - you'll need to modify the plumbing code that comes with the templates.

If you take a closer look at that plumbing, you will soon drill down to CreateProject method inside DiagnosticVerifier.Helper.cs:

/// <summary>
/// Create a project using the inputted strings as sources.
/// </summary>
/// <param name="sources">Classes in the form of strings</param>
/// <param name="language">The language the source code is in</param>
/// <returns>
/// A Project created out of the Documents created from the source strings
/// </returns>
private static Project CreateProject(string[] sources,
    string language = LanguageNames.CSharp)
{
    string fileNamePrefix = DefaultFilePathPrefix;
    string fileExt = language == LanguageNames.CSharp ? 
        CSharpDefaultFileExt : VisualBasicDefaultExt;

    var projectId = ProjectId.CreateNewId(debugName: TestProjectName);

    var solution = new AdhocWorkspace()
        .CurrentSolution
        .AddProject(projectId, TestProjectName, TestProjectName, language)
        .AddMetadataReference(projectId, CorlibReference)
        .AddMetadataReference(projectId, SystemCoreReference)
        .AddMetadataReference(projectId, CSharpSymbolsReference)
        .AddMetadataReference(projectId, CodeAnalysisReference);

    int count = 0;
    foreach (var source in sources)
    {
        var newFileName = fileNamePrefix + count + "." + fileExt;
        var documentId = DocumentId.CreateNewId(projectId, debugName: newFileName);
        solution = solution.AddDocument(documentId, newFileName, 
            SourceText.From(source));
        count++;
    }
    return solution.GetProject(projectId);
}

Here a project is created from your sample source code and references are added, before the compilation is done. By default only 4 assemblies are referenced: mscorlib.dll, System.Core.dll, Microsoft.CodeAnalysis.CSharp.dll, and Microsoft.CodeAnalysis.dll. Since the method is static, you'll need to add additional references for all your tests directly to this method, unless you want to do some major refactoring of that code.

You can find existing MetadataReference instances initialized at the top of the class. I suggest you add your own right there beside them:

// create System.dll reference
private static readonly MetadataReference SystemReference = 
    MetadataReference.CreateFromAssembly(
        typeof(System.Text.RegularExpressions.Regex).Assembly);

Now you can include it in the project using the fluent API:

var solution = new AdhocWorkspace()
    .CurrentSolution
    .AddProject(projectId, TestProjectName, TestProjectName, language)
    .AddMetadataReference(projectId, CorlibReference)
    .AddMetadataReference(projectId, SystemCoreReference)
    .AddMetadataReference(projectId, CSharpSymbolsReference)
    .AddMetadataReference(projectId, CodeAnalysisReference)
    // include System.dll reference in the project
    .AddMetadataReference(projectId, SystemReference);

This should be enough to get you over the initial humps of unit testing your analyzers. I suggest you still try them out in Visual Studio in the end, but for most of the development time, you should be able to manage without it. Doing it like this should save you a lot of time and make the development a much more pleasant experience. Now, start writing that analyzer, you always wanted to have.