Background

A while ago I wrote a post about a tool called ApprovalTests. I've included it in my workshop on unit testing Umbraco, and people are amazed at its usefulness. Having Visual Studio pop a diff in your face when stuff breaks is a real timesaver. However, when I ran the workshop at CodeGarden 18, I realized people were concerned that their tests would be impossible to run in CI environments and the like. Not to worry - ApprovalTests have you covered. (Pun intended)

Environmentalism

When declaring reporters with ApprovalTests, you can specify multiple reporters. That's not all. The reporters have an extensive API on them, which caters for us tailoring everything. There is one particularly interesting interface on all "diff-reporters" named IEnvironmentAwareReporter.
The VisualStudioReporter one has a nice little one-liner implementation:

public override bool IsWorkingInThisEnvironment(string forFile)
{
    return OsUtils.IsWindowsOs() && base.IsWorkingInThisEnvironment(forFile) && LaunchedFromVisualStudio();
}

The last function call there checks if the current process is a descendent of devenv.exe. If it isn't, execution will just be passed to the next reporter in the chain. So there! It won't break your CI build.

But what do we want in our CI build, then? Probably a regular WhateverUnit assertion. I use NUnit, so that'll be our example.

Falling back

Take this little test. I approved the result for seed 1, and we'll examine the output for an invalid result by swapping to 2:

[TestFixture]
[UseReporter(typeof(VisualStudioReporter))]
public class When_Running_In_Different_Environments
{
    [Test]
    public void Delegates_To_Most_Appropriate_Reporter()
    {
        var rnd = new Random(1);
        var items = Enumerable
            .Range(0, 10)
            .Select(x => rnd.Next());
        Approvals.VerifyAll(items, "");
    }
}

Let's change the seed to 2 to fail the test:

var rnd = new Random(2);

Now if we run the test with NUnit3-Console.exe, we'll get an exception saying that ApprovalTests can't find Visual Studio:

nunit3-console.exe .\bin\debug\environmentaltests.dll

...
1) Error : EnvironmentalTests.When_Running_In_Different_Environments.Delegates_To_Most_Appropriate_Reporter
System.Exception : Couldn't find Visual Studio at
at ApprovalTests.Reporters.GenericDiffReporter.Report(String approved, String received) in C:\code\ApprovalTests.Net\ApprovalTests\Reporters\GenericDiffReporter.cs:line 142
...

It sounds worse than it is. ApprovalTests insists that it has something to do. We can add a reporter to the fixture to fix it:

[TestFixture]
[UseReporter(typeof(VisualStudioReporter), typeof(NUnitReporter))]
public class When_Running_In_Different_Environments

Now when we run the test, we get a pure assertion failure:

1) Failed : EnvironmentalTests.When_Running_In_Different_Environments.Delegates_To_Most_Appropriate_Reporter
Expected string length 165 but was 162. Strings differ at index 6.
Expected: "[0] = 534011718\n[1] = 237820880\n[2] = 1002897798\n[3] = 165700..."
But was:  "[0] = 1655911537\n[1] = 867932563\n[2] = 356479430\n[3] = 211537..."
-----------------^

Notice it didn't even mention Visual Studio. Going back into our favorite IDE will start popping diffs again, and the NUnit one will govern the output.

Cleaning up

So should we go around declaring at least two reporters on all our fixtures, then? Luckily not. There are two more tricks that are nice to know.

First, the reporter attribute is found by iterating up the inheritance hierarchy. It can be defined as high up as all your [assembly:XAttribute] metadata. You can create a file in your test project root called ApprovalsConfig.cs for instance. Within it, you declare your reporters on the assembly level:

using ApprovalTests.Reporters;

[assembly:UseReporter(typeof(VisualStudioReporter), typeof(NUnitReporter))]

The second is that you might not want the NUnit assertion when you run in VS (for some reason), or maybe some other tool might be in your way. You might even want composite reporters. In any case, it also makes for a bit nicer setup if you implement the FirstWorkingReporter class:

public class EnvironmentReporter : FirstWorkingReporter
{
    public static readonly EnvironmentReporter INSTANCE = new EnvironmentReporter();

    public EnvironmentReporter()
        : base(
            VisualStudioReporter.INSTANCE,
            NUnitReporter.INSTANCE
        )
    {
    }
}

With it, we can change our UseReporter to be:

[assembly:UseReporter(typeof(EnvironmentReporter))]

Now the first reporter to confirm its environment is executed, and the rest are ignored. You might want to keep the NUnit one around, though. In that case, you can implement MultiReporter in the same way - which coincidentally is the same that UseReporterAttribute does.

I encourage you to go have a look at all the built-in reporters and get some inspiration for even more helpful reporting.

I'd rejoice for any cool usages in the discussion thread. :)