Lars-Erik's blog

Environmental ApprovalTests

Sun, 03 Jun 2018 21:24:46 Z

Background

A while ago I wrote a post about a tool called ApprovalTests. I've included it in my workshop on unit testing Umbraco, and people are amazed at its usefulness. Having Visual Studio pop a diff in your face when stuff breaks is a real timesaver. However, when I ran the workshop at CodeGarden 18, I realized people were concerned that their tests would be impossible to run in CI environments and the like. Not to worry - ApprovalTests have you covered. (Pun intended)

Environmentalism

When declaring reporters with ApprovalTests, you can specify multiple reporters. That's not all. The reporters have an extensive API on them, which caters for us tailoring everything. There is one particularly interesting interface on all "diff-reporters" named IEnvironmentAwareReporter.
The VisualStudioReporter one has a nice little one-liner implementation:

public override bool IsWorkingInThisEnvironment(string forFile)
{
    return OsUtils.IsWindowsOs() && base.IsWorkingInThisEnvironment(forFile) && LaunchedFromVisualStudio();
}

The last function call there checks if the current process is a descendent of devenv.exe. If it isn't, execution will just be passed to the next reporter in the chain. So there! It won't break your CI build.

But what do we want in our CI build, then? Probably a regular WhateverUnit assertion. I use NUnit, so that'll be our example.

Falling back

Take this little test. I approved the result for seed 1, and we'll examine the output for an invalid result by swapping to 2:

[TestFixture]
[UseReporter(typeof(VisualStudioReporter))]
public class When_Running_In_Different_Environments
{
    [Test]
    public void Delegates_To_Most_Appropriate_Reporter()
    {
        var rnd = new Random(1);
        var items = Enumerable
            .Range(0, 10)
            .Select(x => rnd.Next());
        Approvals.VerifyAll(items, "");
    }
}

Let's change the seed to 2 to fail the test:

var rnd = new Random(2);

Now if we run the test with NUnit3-Console.exe, we'll get an exception saying that ApprovalTests can't find Visual Studio:

nunit3-console.exe .\bin\debug\environmentaltests.dll

...
1) Error : EnvironmentalTests.When_Running_In_Different_Environments.Delegates_To_Most_Appropriate_Reporter
System.Exception : Couldn't find Visual Studio at
at ApprovalTests.Reporters.GenericDiffReporter.Report(String approved, String received) in C:\code\ApprovalTests.Net\ApprovalTests\Reporters\GenericDiffReporter.cs:line 142
...

It sounds worse than it is. ApprovalTests insists that it has something to do. We can add a reporter to the fixture to fix it:

[TestFixture]
[UseReporter(typeof(VisualStudioReporter), typeof(NUnitReporter))]
public class When_Running_In_Different_Environments

Now when we run the test, we get a pure assertion failure:

1) Failed : EnvironmentalTests.When_Running_In_Different_Environments.Delegates_To_Most_Appropriate_Reporter
Expected string length 165 but was 162. Strings differ at index 6.
Expected: "[0] = 534011718\n[1] = 237820880\n[2] = 1002897798\n[3] = 165700..."
But was:  "[0] = 1655911537\n[1] = 867932563\n[2] = 356479430\n[3] = 211537..."
-----------------^

Notice it didn't even mention Visual Studio. Going back into our favorite IDE will start popping diffs again, and the NUnit one will govern the output.

Cleaning up

So should we go around declaring at least two reporters on all our fixtures, then? Luckily not. There are two more tricks that are nice to know.

First, the reporter attribute is found by iterating up the inheritance hierarchy. It can be defined as high up as all your [assembly:XAttribute] metadata. You can create a file in your test project root called ApprovalsConfig.cs for instance. Within it, you declare your reporters on the assembly level:

using ApprovalTests.Reporters;

[assembly:UseReporter(typeof(VisualStudioReporter), typeof(NUnitReporter))]

The second is that you might not want the NUnit assertion when you run in VS (for some reason), or maybe some other tool might be in your way. You might even want composite reporters. In any case, it also makes for a bit nicer setup if you implement the FirstWorkingReporter class:

public class EnvironmentReporter : FirstWorkingReporter
{
    public static readonly EnvironmentReporter INSTANCE = new EnvironmentReporter();

    public EnvironmentReporter()
        : base(
            VisualStudioReporter.INSTANCE,
            NUnitReporter.INSTANCE
        )
    {
    }
}

With it, we can change our UseReporter to be:

[assembly:UseReporter(typeof(EnvironmentReporter))]

Now the first reporter to confirm its environment is executed, and the rest are ignored. You might want to keep the NUnit one around, though. In that case, you can implement MultiReporter in the same way - which coincidentally is the same that UseReporterAttribute does.

I encourage you to go have a look at all the built-in reporters and get some inspiration for even more helpful reporting.

I'd rejoice for any cool usages in the discussion thread. :)

Exploiting ApprovalTests for clearer tests

Mon, 23 Oct 2017 20:14:08 Z

What's this for?

Ever written several asserts in one test because you have a big graph you want to verify? How 'bout files or maybe razor views? Several asserts often clutter up the tests. Big strings also make the actuall calls hard to see for all that content. Putting big strings in files is a good idea to avoid that, but few people do. It adds another "menial" task when you're in the zone. But what if it was dead easy?

I once maintained a Resharper extension, and their example tests had so called "gold" files that they compared to output. The squigglies was represented by special characters around terms. So they just compared the output of a text renderer to the text in a file. Great idea. One limitation though: with a regular Assert.Equals you just see the segment around the first mismatch. Guess what, there's a tool that's been around for at least 10 years that solves all those issues, and more.

Approval tests, eh?

Sounds a bit like acceptance tests, right? Don't be fooled, it's purpose is to serve all the way down to the "unit test layer" of your tests. I've found it to be a huge timesaver, as well as making my tests so much more clear.

I know you're thinking "Shut up and give me an example, then!", so let's have a look. I've got this unit test from my post about testing razor views that I never really asserted anything in. I just output the result to console and assert inconclusive. Here it is for reference:

[TestFixture]
public class When_Displaying_An_Event
{
    [Test]
    public void It_Is_Rendered_With_A_Name_Date_And_A_Link()
    {
        var view = new _Views_Partials_Event_cshtml();
        var actionResult = GetConcertEvent();

        Assert.AreEqual("Event", actionResult.ViewName);

        var renderedResult = view.Render((Event) actionResult.Model);

        Console.WriteLine(renderedResult);
        Assert.Inconclusive("Way too big to assert here.");
    }
}

It outputs the following HTML:

<div>
    <a href="https://eventsite/123">
        <label>
            Concert of your life
        </label>
        <span>
            fredag 31. desember 2049 23.59
        </span>
    </a>
</div>

Let's see it

Now let's assert this with ApprovalTests. To use it, you just install-package ApprovalTests with your trusty package manager console. Make sure to install it in your test project. ;)

Now instead of Assert, we ask ApprovalTests to Verify our data instead. It even has a special overload for this concrete case: Approvals.VerifyHtml. So we rewrite the test as such:

[Test]
public void It_Is_Rendered_With_A_Name_Date_And_A_Link()
{
    var view = new _Views_Partials_Event_cshtml();
    var actionResult = GetConcertEvent();

    Assert.AreEqual("Event", actionResult.ViewName);

    var renderedResult = view.Render((Event) actionResult.Model);

    Approvals.VerifyHtml(renderedResult);
}

Now when we run our test, we get this nice little welcoming message from ApprovalTests:

It tells us we're missing a vital part: A reporter. It's possible to use DiffReporter to launch your favorite configured difftool. But if you're in Visual Studio, there's a special treat: VisualStudioReporter. Let's add that to our fixture and see what happens:

[UseReporter(typeof(VisualStudioReporter))]
[TestFixture]
public class When_Displaying_An_Event
{
    [Test]
    public void It_Is_Rendered_With_A_Name_Date_And_A_Link()
    {
        // ...
    }
}

And we run it:

What hits you is probably the big failure statement there, but look again - up at the top there. We've got a diff opened, the result on the left hand and a big green field on the right side.
What just happened is that ApprovalTests took our result, stored it in a received file, and at the same time wrote an empty "approved" file. It then proceeded to compare, and finally pop a diff of those two files.
Isn't that just beautiful? Everything that makes your test fail in one clear diagram.

The "procedure to follow" here, is to "approve" results when you're happy. To do that, you just copy and paste. Let's do that now:

If you've got ReSharper, it's probably gonna try to format everything nicely when you paste. To have it in the original, ugly indentation state, press Ctrl+Z (undo) once after pasting.

Regarding the indentation that's off. It seems to be a bug with HTML in the current version of ApprovalTests, so stupid me for choosing this example. I'll update the post if it gets fixed.

We can now run our test again, and it's gonna pass. When it passes, it doesn't bother to open the diff.

This means we're just gonna get diffs for whatever is currently failing. Even if we run our entire suite of tests. Now there's a couple of housekeeping things to keep in mind:

Commit approvals only

If you were paying attention, you noticed we got two files adjacent to our test source file. One is named [TestFixtureName].[TestMethodName].received.html and one is named [TestFixtureName].[TestMethodName].approved.html. If you ran a test that passed, you'll actually just have your approved file. You want those approved files in source control!

The received files though, might end up not being cleaned up for one or the other reason. Hopefully just that you didn't bother to commit a fully passing build. I'm sure you didn't do that to the master branch, though. In any case, make sure to ignore your received files. This pattern in .gitignore generally does the trick:

*.received.*

That's just the tip of the iceberg

We've seen the VerifyHtml bit. One of my personal favorites is it's sibling VerifyJson. I keep an extension on object in my tests, called .ToJson(). With it, I can just go:

Approvals.VerifyJson(myBigGraph.ToJson());

The diff is done with prettified JSON, so it's super simple to find the property / area that has changed or doesn't work. Knowing the area of the graph should also make it easier to find the usage that is wrong.

There's a vanilla Verify method too, and it saves plain text files. It's useful in cases where you have nice ToString() implementations. Let's try with a "person":

public class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }

    public override string ToString()
    {
        return $"{FirstName} {LastName} ({Age})";
    }
}

[Test]
public void Comparing_Objects()
{
    var person = new Person
    {
        FirstName = "Lars-Erik",
        LastName = "Aabech",
        Age = 19
    };

    Approvals.Verify(person);
}

It produces the following .received file:

Lars-Erik Aabech (19)

It can be approved like that.
We can also do lists with "big brother" VerifyAll:

[Test]
public void Comparing_Lists()
{
    var list = new List<Person>
    {
        new Person {FirstName = "Lars-Erik", LastName = "Aabech"},
        new Person {FirstName = "Dear", LastName = "Reader"}
    };

    Approvals.VerifyAll(list, "");
}

Which, unsurprisingly outputs:

[0] = Lars-Erik Aabech
[1] = Dear Reader

Now how bout that?
I don't know how quickly you got hooked, but I certainly find it sneaking into more and more of my tests.

It can even do images, but I'll let James blog about that one.

So what are you lingering around here for? Run over to nuget and get your copy, or lurk around some more at the ApprovalTests project site. There's great examples, even though they might not be in your favorite language.

Happy approving! :)