Damir's Corner.

Ignored failing assertions in mock callbacks

When you write unit tests, make sure not only that they succeed if the tested code works as expected, but also that they fail if the code does not work as expected. Otherwise, these tests will give you a false sense of confidence in your code.

The following test is one such example. It might succeed even if the assertion fails:

[Test]
public void PassesWithFailedAssertionInCallback()
{
    var mocker = new AutoMocker();
    var service = mocker.CreateInstance<Service>();

    var dependency = mocker.GetMock<IDependency>();
    dependency
        .Setup(d => d.DependencyCall(It.IsAny<int>()))
        .Callback((int input) =>
        {
            input.Should().BeLessThan(5);
        });

    service.MethodCall(6);
    mocker.VerifyAll();
}

Why would that happen? Because a failed assertion throws an exception. And if this exception is caught and doesn't bubble up all the way to the test, the unit test framework will not recognize the failed assertion.

By asserting a value and hence throwing the exception in the callback of the mocked service dependency, the service under test can catch that exception. Let us try this out and print the details of the exception:

public bool MethodCall(int input)
{
    try
    {
        this.dependency.DependencyCall(input);
        return true;
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
        return false;
    }
}

This way we can check the output of the test for the exception, even if the test was successful. This is what we would find:

NUnit.Framework.AssertionException: Expected input to be less than 5, but found 6.

You can argue that the method should not catch all exceptions, and in most cases that is true. But the test should not depend on that to do its job correctly. So how can we fix the test to avoid this?

One way would be to avoid the callback altogether and check the input value with mock verification instead of assertion:

[Test]
public void FailsWithFailedMockVerification()
{
    var mocker = new AutoMocker();
    var service = mocker.CreateInstance<Service>();

    var dependency = mocker.GetMock<IDependency>();
    dependency
        .Setup(d => d.DependencyCall(It.Is<int>(input => input < 5)));

    service.MethodCall(6);
    mocker.VerifyAll();
}

This is a simpler approach, but it may have some drawbacks:

• Depending on how complex the value check is, it could be much easier to implement and understand with an assertion.

• The error messages of a good assertion library like FluentAssertions are much better than a Moq output for a failed verification:

  This mock failed verification due to the following:
  
     IDependency d => d.DependencyCall(It.Is<int>(input => input < 5)):
     This setup was not matched.
  

To assert the input value of a mocked method, the callback should only store the value in a local variable so that the assertion can be performed directly in the test:

[Test]
public void FailsWithFailedAssertionOutsideCallback()
{
    var mocker = new AutoMocker();
    var service = mocker.CreateInstance<Service>();

    var dependency = mocker.GetMock<IDependency>();
    int? actualInput = null;
    dependency
        .Setup(d => d.DependencyCall(It.IsAny<int>()))
        .Callback((int input) =>
        {
            actualInput = input;
        });

    service.MethodCall(6);

    actualInput.Should().BeLessThan(5);
    mocker.VerifyAll();
}

This way, it does not matter how the method under test handles exceptions. The assertion throws the exception outside the method, so there is no way to catch it.

You can see the full sample code for these tests in my GitHub repository.

It's just as important for tests to fail if the code being tested does not work correctly as it is for them to succeed if the code works correctly. In this post, I showed an example where the code under test has the ability to catch the exception of a failed assertion, and suggested solutions to prevent this from happening. To make sure you do not miss such problems, you should always check if a newly written test fails when the expectations in assertions and verifications are not met.