Lars-Erik's blog

More efficient integration tests with Umbraco

Tue, 25 Nov 2025 07:11:08 Z

TLDR;

I made a package that removes the "database and Umbraco instance" per fixture constraint from Umbraco's Integration Test library. It also enables reuse of seeded database snapshots to avoid executing the same setup / seeding over an over per fixture or test.

An itch

I've spent way too long waiting for similar setup code for my integration tests the last couple of years. The modern .NET version of Umbraco's integration test library is super nice, and real effective with SQLite. But it's got a few constraints:

All things Umbraco is instantiated and spun up per test, granted a configurable amount.
Databases have to be "installed" and seeded for each TestFixture at the least.
All test fixtures need to inherit from Umbraco's base classes

So we spend a long time waiting for suff, even if it's only the little green box we care about:

We're also stuck either juggling a singleton of some sorts and/or stuffing a lot of "units" into the same fixture.
Any notion of intermediate base classes quickly become a sore spot.

An idea

I had recently discovered, likely for the n-th time in my life, that NUnit supports namespace scoped setup fixtures. Those live as long as all that other tests in that namespace and deeper. They bring an opportunity to share initial state across several fixtures. I often find it nice to have several fixtures for the same systems under tests because it makes sense to group by use-case rather than tech.

So I set out to see if I could hack together something that made Umbraco's base tests possible to use as setup fixtures rather than "base" fixtures. As things would have it it was early october, and the CanCon gang hosted a virtual hackathon on a friday. I took the day "off" regular work and gave it a go.

Dark magic

In the end I've used all the dirty tricks you can imagine in today's .NET landscape. You'll find Lib.Harmony for messing with Umbraco's tests' IL (Intermediate Language), and there's Castle DynamicProxy doing a lot of fooling around with the NUnit test hierarchy. The Harmony bit can likely be removed by making a few slight changes to Umbraco's test core, but I wanted to get this working satisfactory before suggesting such changes.

The bottom line is that those two tools let us run code before, after or even instead of inherited code. And that in turn enables all the following features.

A new way of composing tests

With the package I've cooked together we can leave the dark world of peach and purple above in favor of a way greener scenery like such:

So without further ado, let's dig into how you can get there too.

Umbraco's attributes

We rely on Umbraco's Tests:Database config and the [UmbracoTest] attribute to provide databases (or not). The Database option works almost as expected, but the four non-none options end up doing the same: they prepare one db for the lifetime of the setup fixture.

[ExtendableSetUpFixture]

The first attribute you have to know is [ExtendableSetUpFixture]. It's sole purpose is to enable the rest of them. Since it's inherited from [SetUpFixture], it tells NUnit we won't add any [SetUp], [TearDown] or [Test] methods in the marked class. But this is the attribute we'll add to any UmbracoIntegrationTest or UmbracoTestServerTestBase, and those have a bunch of each. That's why we have...

[MakeOneTimeLifecycle]

The first attribute in the package that does something. [MakeOneTimeLifecycle] lets you mark methods in otherwise not accessible base classes like UmbracoTestBase to become [OneTimeSetUp] methods rather than [SetUp] methods, and [TearDown] methods becomes [OneTimeTearDown].
Here's an example of what we'd feed it for an UmbracoIntegrationTest inheritor:

[UmbracoTest(Database=SchemaPerFixture, Boot=true)]
[ExtendableSetUpFixture]
[MakeOneTimeLifecycle(
    setUpNames: [nameof(UmbracoIntegrationTest.Setup), nameof(UmbracoIntegrationTest.SetUp_Logging)],
    tearDownNames: [
        nameof(UmbracoIntegrationTest.TearDown), nameof(UmbracoIntegrationTest.TearDownAsync), 
        nameof(UmbracoIntegrationTest.FixtureTearDown), nameof(UmbracoIntegrationTest.TearDown_Logging)
    ]
)]
public class MyLongLivedUmbracoSetUp : UmbracoIntegrationTest
{
}

I found it a bother to write that long attribute all the time, so there's a derived one called:

[OneTimeUmbracoSetUp]

It makes it slightly less messy by letting us write as follows:

[UmbracoTest(Database=SchemaPerFixture, Boot=true)]
[ExtendableSetUpFixture]
[OneTimeUmbracoSetUp]
public class MyLongLivedUmbracoSetUp : UmbracoIntegrationTest
{
}

Note however that the current version doesn't come with a 100% compatible one for UmbracoTestServerTestBase or ManagementApiTest<>, but it's easy to make one yourself, and they're likely suddenly in the package.

Now if you're into hacking around the environment and hidden, protected members lurking around in memory, we're quite alike. But when doing professional work that's often not what we're paid for. So even though the above code will start up a perfectly fine Umbraco instance with a SQLite database it doesn't do much good. We need to get at the IServiceProvider somehow...

[InjectionProvider(string)] and [ServiceProvider]

There is a limit to how many attributes we'll stick on our setup fixtures, I promise. I complained about intermediate base classes earlier, but a few for attribute consolidation is cool enough. OK, these two are a basic re-usable one and a "defaulted" one. The [InjectionProvider(string)] attribute stores a reference to a property on the setup fixture that exposes an IServiceProvider instance. Umbraco's base classes exposea property called Services, and [ServiceProvider] is just a derived [InjectionProvider(nameof(Services))]. To be specific, the service provider reference goes into the test context property bag as a factory method.
By adding one to our growing tower we're ready to meet our first actual test fixture enjoying the power of:

[Inject(string)]

To enjoy the benefits of our stored service provider reference, we must expose an instance method on our "scoped" test fixture that accepts services. We could've used constructor injection, but that's already "in use" by NUnit [TestFixture(args)] functionality. Once you start composing tests like this package allows, you can get much more value from things like that. So a void method will have to do. A fresh "standalone" and "scoped" test fixture will look like this:

using Same.Namespace.As.SetupFixture;

[Inject(nameof(Inject))]
public class LookMaImFreeAsABird
{
    IDataTypeService dataTypeService = null!;

    public void Inject(IDataTypeService dataTypeService)
    {
        this.dataTypeService = dataTypeService;
    }

    [Test]
    public void FiddleWithDataTypeA()
    {
        // ...
    }

    [Test]
    public void CreateDataTypeB()
    {
        // ...
    }
}

Notice the beauty of no forced ctors, no base class and no overrides in that class! Not to mention, you can have several of those!

Interlude: Transactions

With the above setup the individual fixtures are free to create ICoreScope instances from Umbraco and commit or dispose them as you wish. Just writing three sentences about it here doesn't really give the method justice. Suffice to say, just give it a go!
However it won't work when we use UmbracoTestServerTestBase. There's no way to spread a core scope across threads, and there's not even a way to spread gool old .NET transaction scopes across. I guess you could go full old school and bring DCOM into the picture, but after a lot of failed attempts I finally stumbled over an obvious solution.

Snapshots

The final itch has taken me the longest, and even longer to stuff into an attribute rather than yet another base class. I must admit it took a desperate (but accurate) final plea with ChatGPT 5.1 to see the obvious solution just lying there to implement. All database engines, or at least the two I've implemented, support some sort of fast backup. VACUUM INTO in SQLite and CREATE/RESTORE DATABASE AS SNAPSHOT in SQL Server.

I had two intermediate base classes for a few weeks, but this last weekend I ran over the goal line:

`[ReusableDatabase(type, string)]`

If you configure Tests:Database:DatabaseType in your appSettings.Tests.Local.json and set any of the non-none values in the [UmbracoTest] database parameter, you can also add [ReusableDatabase(typeof(SeedClass), nameof(SeedClass.Configure))] to get a new type of ITestDatabase.

As of writing it requires a method to be implemented in the setup fixture, or on some supporting type. Its mission is to configure whether the database needs a fresh seed and how to seed the initial data for all the tests. Only if we say so, the database is installed and re-seeded. For all other scopes it's just restored from that initial snapshot.

The seeding can be done in any way you please. My favorite is importing things using uSync, and that might just become another blog post. For simplicity let's say we're testing variants of property editors based on a datatype we want all tests to start with:

[UmbracoTest(
    Database = UmbracoTestOptions.Database.NewSchemaPerTest,
    Boot = true,
    Logger = UmbracoTestOptions.Logger.Console
)]
[ExtendableSetUpFixture]
[OneTimeUmbracoSetUp]
[ServiceProvider]
[ReusableDatabase(nameof(ConfigureSeeding))]
public class ReusedDbAttributeSetUp : UmbracoIntegrationTest
{
    public static void ConfigureSeeding(ReusableTestDatabaseOptions options)
    {
        options.NeedsNewSeed = _ => Task.FromResult(true);
        options.SeedData = async (services) =>
        {
            await TestContext.Progress.WriteLineAsync("Creating datatype");
            await services.GetRequiredService<IDataTypeService>().CreateAsync(
                new DataType
                (
                    new TextboxPropertyEditor
                    (
                        services.GetRequiredService<IDataValueEditorFactory>(),
                        services.GetRequiredService<IIOHelper>()
                    ),
                    services.GetRequiredService<IConfigurationEditorJsonSerializer>()
                )
                {
                    Name = "A seeded textbox"
                },
                Umbraco.Cms.Core.Constants.Security.SuperUserKey
            );
        };
    }
}

The method has to be static void. If it's on the setup fixture you can omit the type from the attribute parameters.
Now all tests within this "scope" will have access to the seeded textbox. Such seeding code might quickly grow out of hand though, and that's why I prefer just importing the content schema from the web project in the same repo as my tests.

Which ones to choose

As you've hopefully realized by now you can introduce really slim intermediate base classes that set up the initial environment for a bunch of use-cases and scenarios. Leaving your actual test fixtures free to be composed just as you wish, and with all the features of NUnit at your disposal.

I'm sure you can see that the last setup fixture example above makes for a nice base class to avoid repeating that tower of attributes. The repository has a few examples using a few or all of the attributes.

The final trick I haven't disclosed above is how to do a "mid-fixture" rollback. The reusable database implementations have a "RestoreSnapshot" method, so in a [TearDown] you can go:

await serviceProvider.GetRequiredService<IReusableDatabase>().RestoreSnapshot();

That makes it so that whatever test comes next it's unaffected by the changes you've made. This is of course what happens during [OneTimeSetUp] for any setup fixture using [ReusableDatabase].

If you want to try this out, I recommend going with [ReusableDatabase] (and uSync) for everything that use more than 10ish artifacts like data and content types, not to mention content. Whenever you want to run changes through the Management API, or definitely if you want to test how all your sites' elements look in the Delivery API. It's a no-brainer when you go full integration with UmbracoTestServerTestBase and more than one HTTP call.

If all your tests have all they need in a fresh Umbraco database, but they all mutate it so you need cleanup, it'll be blazing fast if you can use ICoreScope with autocommit off. As long as you don't need to test via HTTP endpoints, this is a better option than restoring the snapshot. Using a snapshot restore between fixtures however is still a nice option to keep around, so I still opt for [ReusableDatabase].

Want more?

I'm definitely going to be in "only fix what breaks for me" mode for a little while, as this was just gonna be a hacktoberfest weekend project. Turned out to take too much of my spare time for one and a half month instead.

But I honestly believe this'll save some trees if applied well to CI pipelines running all day long.
And I've gone from 90 sec to 26 sec to execute a full test suite, a lot of which would run for any filter.

The package is currently listed as a beta (pre-release) on nuget and if it doesn't crash much I might just promote it to full visibility. It's a bit furry on the edges and has a bit of legacy to it (already), but if you stick the namespaces in the project file and leave the stray base classes alone I think you'll be good.

All this is to say I'd love for conversation about what it solves and if it could do it better. I'd love even more to throw out a bit of the code because Umbraco suddenly fixed the core two issues. Let me hear from you if you try it out. Feel free to clone the code and mess about, although I reserve the right to refactor the mess myself that beautiful day I get to keep going with this.

Looking forward to hearing how (and if) you like it. Hope you have a go!

Links

Unit Testing the IIS URL Rewrite Module

Mon, 19 Sep 2022 13:09:15 Z

Overview

I can't count on two hands the amount of times I've messed up some IIS redirect rule on our sites. "We need all these 50 old top traffic drivers from the old site, and of course we want five domains per country in case someone mistypes it and Google don't show us as first hit.
Oh, and by the way, everybody should be able to use these three slugs for our new campaigs."

Bloody marketeers!

Wouldn't it be nice if we could write some automated tests that verifies that we've configured the UrlRewriteModule correctly? I've been half-heartedly looking for a way to do this with the proprietary .NET Framework IIS Module, but recently realized while researching it that it's a public middleware in .NET Core.
This means we can start to write unit tests against it, and verify that stuff we want to use with our old .NET Framework sites will work even if we test for - and redirect to domains.

Test dependencies

Since I'm testing the config of a .NET Framework app, I just add a separate .NET Core 6 NUnit test project. Its sole purpose is verifying these redirects.

All the dependencies are nuget packages, but I just went and used some fixed paths to the SDK for now. Here are the test project dependencies:

<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.1.0" />
    <PackageReference Include="Moq" Version="4.18.2" />
    <PackageReference Include="NUnit" Version="3.13.3" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
    <PackageReference Include="NUnit.Analyzers" Version="3.3.0" />
    <PackageReference Include="Verify.NUnit" Version="17.10.2" />
</ItemGroup>

<ItemGroup>
    <Reference Include="Microsoft.AspNetCore.Http">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.AspNetCore.Http.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.AspNetCore.Http.Abstractions">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.AspNetCore.Http.Abstractions.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.AspNetCore.Rewrite">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.AspNetCore.Rewrite.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.AspNetCore.Http.Features">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.AspNetCore.Http.Features.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.AspNetCore.Hosting.Abstractions">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.AspNetCore.Hosting.Abstractions.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.Extensions.Features">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.Extensions.Features.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.Extensions.Hosting.Abstractions">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.Extensions.Hosting.Abstractions.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.Extensions.Logging.Abstractions">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.Extensions.Logging.Abstractions.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.Extensions.FileProviders.Abstractions">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.Extensions.FileProviders.Abstractions.dll</HintPath>
    </Reference>
    <Reference Include="Microsoft.Extensions.Options">
        <HintPath>C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.9\Microsoft.Extensions.Options.dll</HintPath>
    </Reference>
</ItemGroup>

Now that we have them, we can start setting up a totally mocked HttpContext and look at what the IIS UrlRewritingModule middleware does.

The middleware

In order to use the IIS UrlRewriteModule in ASP.NET Core we do app.UseRewriter(RewriteOptions) and on the RewriteOptions we do .AddIISUrlRewrite(FileProvider, "web.config"). All these two things do is camouflaging a middleware and parsing of the config.
What we need in our test is this:

// omitted quite a few web hosting mocks
var reader = // some way to read our config as a web config file
var options = new RewriteOptions().AddIISUrlRewrite(reader);
var middleware = new RewriteMiddleware(_ => Task.CompletedTask, Mock.Of<IWebHostEnvironment>(), loggerFactory, new OptionsWrapper<RewriteOptions>(options))
middleware.Invoke(httpContext);
Verify(state);

All necessary code is at the bottom of this post.

Reading the configuration

We usually put our rewrite rules in a separate file in the web project. It's linked from the web.config file using the configSource attribute as such:

<system.webServer>
    <rewrite>
        <rules configSource="config\RewriteRules.config" />
    </rewrite>
</system.webServer>

To be frank it's actually done on publish using a transform file, so we don't have it configured when working locally.
The rules are then managed in a file looking like this:

<?xml version="1.0" encoding="utf-8" ?>
<rules>
    <rule enabled="true" name="test">
        <match url="(.*)" />
        <conditions logicalGrouping="MatchAll">
            <add input="{HTTP_URL}" pattern=".*abc.*" matchType="Pattern"  />
        </conditions>
        <action type="Redirect" redirectType="Found" url="/tada" />
    </rule>
</rules>

However the AddIISUrlRewrite extension for the RewriteOptions require a stream of a full web.config file. I've just hacked together a small factory method that wraps our config in a slim web.config envelope:

var path = Path.Join(AppDomain.CurrentDomain.BaseDirectory, @"..\..\..\..\Customer.Web\config\rewriterules.config");
var contents = File.ReadAllText(path)
    .Replace("<?xml version=\"1.0\" encoding=\"utf-8\" ?>", "");
var fullContents = "<?xml version=\"1.0\" encoding=\"utf-8\" ?><configuration><system.webServer><rewrite>" + 
                    contents +
                   "</rewrite></system.webServer></configuration>";
var reader = new StringReader(fullContents);

It can now be passed to AddIISUrlRewrite and we're good to go.

A mock HttpContext

The UrlRewriteModule reads from the HttpRequest instance and possibly writes to the HttpResponse instance. We need to mock both these. However the stuff we want to vary is all he URL data, so we pass that to a (fairly large) factory method:

private static HttpContext CreateHttpContext(Uri uri)
{
    var ctx = Mock.Of<HttpContext>();

    var resp = Mock.Of<HttpResponse>();
    var headers = new HeaderDictionary();
    Mock.Get(resp).Setup(x => x.Headers).Returns(headers);
    Mock.Get(ctx).Setup(x => x.Response).Returns(resp);

    var req = Mock.Of<HttpRequest>();
    Mock.Get(ctx).Setup(x => x.Request).Returns(req);
    Mock.Get(req).Setup(x => x.Path).Returns(new PathString(uri.AbsolutePath));
    Mock.Get(req).Setup(x => x.Scheme).Returns(uri.Scheme);
    Mock.Get(req).Setup(x => x.Host).Returns(new HostString(uri.Host, uri.Port));
    Mock.Get(req).Setup(x => x.PathBase).Returns(new PathString("/"));
    Mock.Get(req).Setup(x => x.QueryString).Returns(new QueryString(uri.Query));

    var variableFeature = Mock.Of<IServerVariablesFeature>();
    var features = new FeatureCollection();
    features.Set(variableFeature);
    Mock.Get(ctx).Setup(x => x.Features).Returns(features);

    Mock.Get(variableFeature).Setup(x => x["HTTP_HOST"]).Returns(uri.Host);
    Mock.Get(variableFeature).Setup(x => x["HTTP_URL"]).Returns(uri.AbsolutePath);
    Mock.Get(variableFeature).Setup(x => x["HTTPS"]).Returns(uri.Scheme == "https" ? "on" : "off");
    return ctx;
}

What to verify?

I was initially trying to verify the HttpResponse. We can do that and test the StatusCode and headers etc. However, I was Console.WriteLineing all the logging and it turns out the UrlRewrite middleware logs perfectly nice descriptions of what happens. It also (usually) stops at the rule that handles the URL, so if we take the last two outputs, we're quite good to go:

Request matched current UrlRewriteRule 'test'.
Location header '//tada' with status code '302'.

Request did not match current rule 'test'.
Current url is http://localhost:80/

If I could have my will, I'd like something like this, and I can have it. :)

http://localhost/abc => Request matched current UrlRewriteRule 'test'. Location header '//tada' with status code '302'.
http://localhost/ => Request did not match current rule 'test'. Current url is http://localhost:80/

So what loggerFactory sets up is an instance that keeps a local message log, and of course write to the console:

public class RedirectLogger : ILogger
{
    public List<string> Messages { get; } = new List<string>();

    public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
    {
        var output = formatter(state, exception);
        output = output.Replace("Request is done processing. ", "");
        output = output.Replace("Request is continuing in applying rules. ", "");
        Console.WriteLine(output);
        Messages.Add(output);
    }

    public bool IsEnabled(LogLevel logLevel)
    {
        return true;
    }

    public IDisposable BeginScope<TState>(TState state)
    {
        throw new NotImplementedException();
    }

    public override string ToString()
    {
        return String.Join(" ", Messages);
    }

    public string Last(int count = 2)
    {
        return String.Join(" ", Messages.TakeLast(count));
    }
}

I keep that instance around for the test to verify what was written to it.

Trying it together

The last piece here is Verify which we can use for larger "assertEquals" blocks.

I'd like one big report of different URLs I expect hit the server, and then the result for each url. So I set up a bunch of testcases like so:

public static Uri[] TestCases() => new[]
{
    new Uri("http://localhost/abc"),
    new Uri("https://localhost/abc"),
    new Uri("http://localhost/tadabcada"),
    new Uri("http://localhost/"),
    new Uri("http://localhost/tada")
};

Now we can loop over those and verify a report in our test:

[Test]
public async Task Handles_All_Rules()
{
    List<string> results = new List<string>();
    foreach(var uri in TestCases())
    {
        var ctx = CreateHttpContext(uri);

        redirectLogger.Messages.Clear();
        await middleware.Invoke(ctx);

        results.Add(uri + " => " + redirectLogger);
    }

    await Verify(String.Join(Environment.NewLine, results));
}

Our output becomes:

http://localhost/abc => Request matched current UrlRewriteRule 'test'. Location header '//tada' with status code '302'.
https://localhost/abc => Request matched current UrlRewriteRule 'test'. Location header '//tada' with status code '302'.
http://localhost/tadabcada => Request matched current UrlRewriteRule 'test'. Location header '//tada' with status code '302'.
http://localhost/ => Request did not match current rule 'test'. Current url is http://localhost:80/
http://localhost/tada => Request did not match current rule 'test'. Current url is http://localhost:80/tada

Fairly easy to read, and now we'll blow up if ever we break our rules.

I think the readability can be tweaked for even better readability and intuition, but for now I'm super happy I can test it. There are tons of other cases I haven't covered of course, but this is a very good starting place.

One quirk I've noticed is that the location header always gets a leading slash, but i think that's due to running in a test.

Hope you like it.
I'd love comments about usage or extensions. :)

All the code

Here's all the code reading the config from a resource file instead:

using Microsoft.AspNetCore.Rewrite;
using Moq;
using DiffEngine;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Features;
using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Newtonsoft.Json.Linq;
using NUnit.Framework;

namespace UrlRewrite.Tests
{
    public class UrlRewriting
    {
        public static Uri[] TestCases() => new[]
        {
            new Uri("http://localhost/abc"),
            new Uri("https://localhost/abc"),
            new Uri("http://localhost/tadabcada"),
            new Uri("http://localhost/"),
            new Uri("http://localhost/tada")
        };

        [Test]
        public async Task Handles_All_Rules()
        {
            List<string> results = new List<string>();
            foreach(var uri in TestCases())
            {
                var ctx = CreateHttpContext(uri);

                redirectLogger.Messages.Clear();
                await middleware.Invoke(ctx);

                results.Add(uri + " => " + redirectLogger);
            }

            await Verify(String.Join(Environment.NewLine, results));
        }

        private RewriteMiddleware middleware = null!;
        private RedirectLogger redirectLogger = null!;

        [SetUp]
        public void Setup()
        {
            DiffTools.UseOrder(DiffTool.VisualStudio, DiffTool.Rider, DiffTool.VisualStudioCode);

            var options = CreateOptions(GetType().Assembly.GetManifestResourceStream("UrlRewrite.Tests.testroutes.xml")!);
            redirectLogger = new RedirectLogger();
            var loggerFactory = CreateLoggerFactory(redirectLogger);
            middleware = CreateMiddleware(loggerFactory, options);
        }

        private static HttpContext CreateHttpContext(Uri uri)
        {
            var ctx = Mock.Of<HttpContext>();

            var resp = Mock.Of<HttpResponse>();
            var headers = new HeaderDictionary();
            Mock.Get(resp).Setup(x => x.Headers).Returns(headers);
            Mock.Get(ctx).Setup(x => x.Response).Returns(resp);

            var req = Mock.Of<HttpRequest>();
            Mock.Get(ctx).Setup(x => x.Request).Returns(req);
            Mock.Get(req).Setup(x => x.Path).Returns(new PathString(uri.AbsolutePath));
            Mock.Get(req).Setup(x => x.Scheme).Returns(uri.Scheme);
            Mock.Get(req).Setup(x => x.Host).Returns(new HostString(uri.Host, uri.Port));
            Mock.Get(req).Setup(x => x.PathBase).Returns(new PathString("/"));
            Mock.Get(req).Setup(x => x.QueryString).Returns(new QueryString(uri.Query));

            var variableFeature = Mock.Of<IServerVariablesFeature>();
            var features = new FeatureCollection();
            features.Set(variableFeature);
            Mock.Get(ctx).Setup(x => x.Features).Returns(features);

            Mock.Get(variableFeature).Setup(x => x["HTTP_HOST"]).Returns(uri.Host);
            Mock.Get(variableFeature).Setup(x => x["HTTP_URL"]).Returns(uri.AbsolutePath);
            Mock.Get(variableFeature).Setup(x => x["HTTPS"]).Returns(uri.Scheme == "https" ? "on" : "off");
            return ctx;
        }

        private RewriteMiddleware CreateMiddleware(ILoggerFactory loggerFactory, RewriteOptions options)
        {
            return new RewriteMiddleware(_ => Task.CompletedTask, Mock.Of<IWebHostEnvironment>(), loggerFactory, new OptionsWrapper<RewriteOptions>(options));
        }

        private static ILoggerFactory CreateLoggerFactory(ILogger logger)
        {
            var loggerFactory = Mock.Of<ILoggerFactory>();
            Mock.Get(loggerFactory).Setup(x => x.CreateLogger(It.IsAny<string>())).Returns(logger);
            return loggerFactory;
        }

        private static RewriteOptions CreateOptions(Stream stream)
        {
            using var reader = new StreamReader(stream);
            var options = new RewriteOptions().AddIISUrlRewrite(reader);
            options.StaticFileProvider = Mock.Of<IFileProvider>();
            return options;
        }
    }

    public class RedirectLogger : ILogger
    {
        public List<string> Messages { get; } = new List<string>();

        public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
        {
            var output = formatter(state, exception);
            output = output.Replace("Request is done processing. ", "");
            output = output.Replace("Request is continuing in applying rules. ", "");
            Console.WriteLine(output);
            Messages.Add(output);
        }

        public bool IsEnabled(LogLevel logLevel)
        {
            return true;
        }

        public IDisposable BeginScope<TState>(TState state)
        {
            throw new NotImplementedException();
        }

        public override string ToString()
        {
            return String.Join(" ", Messages);
        }
    }
}

Stubbing the internet with Postman, HttpClient and HttpMessageHandler

Fri, 05 Jun 2020 10:52:32 Z

Using Postman or Insomnia to test code and integrations

Recently Tim Heuer at Microsoft posted a poll on Twitter asking people using tools like Postman and Insomnia why they "just don't rely on test frameworks".

Folks who use tools like postman/insomnia...why don't you just rely on test frameworks?
— Tim Heuer (@timheuer) May 29, 2020

I wanted to pitch in with the method I use, which is a combination of both.

Whenever I integrate with third parties, I almost always start off with a spike to learn how their APIs work.
Right now for instance, I'm building some integrations with the norwegian business registry.
They have a couple of nice and simple endpoints. Here's a Postman request searching for our company, MarkedsPartner.

I also had to run this code from a public webserver in order to find the right request messages from a second third party, so I threw in some HttpClient usage for the above, while logging incoming requests.

I could then replicate the requests in Postman to continue spiking with my mediating API controller.

As soon as I knew all about the incoming and outgoing messages, I was ready to start stabilizing with tests so I could proceed with refactoring and extending.

Stubbing the entire internet

The .NET HttpClient is fully stubbable, but the seam is fairly well hidden. I'm not convinced enough people know about this.
The HttpClient has a parameterized constructor taking the abstract HttpMessageHandler as its only parameter. By default this is a concrete HttpClientHandler when the parameterless ctor is used. Within HttpMessageHandler there is a protected SendAsync method which is the final method called by all the Get, Put, Post methods etc. It receives a HttpRequestMessage and returns a HttpResponseMessage.

You can use mocking frameworks like Moq to stub protected methods. But I really like to build my own fake HttpMessageHandler implementation for HTTP integration tests. In my experience it's actually (way) less code, and provides both stubbing and mocking mechanisms.

The simplest way to build it is to just return the same message for any call, but with a few extensions it's almost like a small stub internet with static resources. Here's how you could build one that varies its responses by request URI:

public class StubHttpClientHandler : HttpClientHandler
{
    public readonly Dictionary<string, HttpResponseMessage> UriResponses = new Dictionary<string, HttpResponseMessage>();

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var url = request.RequestUri.ToString();
        if (UriResponses.ContainsKey(url))
        {
            return await Task.FromResult(UriResponses[url]);
        }

        return await Task.FromResult(new HttpResponseMessage(HttpStatusCode.NotFound));
    }
}

To make sure my production call hits this integration border instead of going off to the internet, I have to abstract away the creation of a HttpClient in my code. Often I set up a Func<HttpClient> factory with a DI container, but in my current case it's enough to add a static property to the SUT (system under test):

public static Func<HttpClient> HttpClientFactory = () => new HttpClient();

In my tests, I then swap out that factory with a function that returns a reference to a HttpClient using my StubHttpClientHandler:

var handler = new StubHttpClientHandler();
HttpClient httpClient = new HttpClient(handler);
BrregLookupController.HttpClientFactory = () => httpClient;

The final thing to do is to register a URI with a corresponding response message:

handler.UriResponses.Add(
    "https://data.brreg.no/enhetsregisteret/api/enheter?navn=markedspartner",
    new HttpResponseMessage(HttpStatusCode.OK)
    {
        Content = new StringContent("{\"status\":\"OK\"}", Encoding.UTF8)
    }
);

Whenever the controller now fires a call to that URL, it'll get a response with the JSON provided in the response message. You could add a content-type header and such for good measure, but it's not really necessary depending on how you handle the result.

Bringing Postman responses into the tests

Of course a slim JSON object with a status message isn't what I'm after here. I'm after the exact response the Brreg APIs would give me. Instead of pasting the entire JSON graph seen in the screenshot above, I like to keep them in files instead.

Those kinds of files are excellent candidates for Embedded Resources in the test project. So I paste the entire graph into a JSON file in a "TestResources" folder and make sure the file's build action is "Embedded Resource":

In order to easily provide that content to my stub I usually employ a simple helper class for resources:

public static class TestData
{
    public static string ReadAsString(string resourceName, string suffix = "json")
    {
        using (var resourceStream = typeof(TestData).Assembly.GetManifestResourceStream(
            $"Full.TestProject.Namespace.TestResources.{resourceName}.{suffix}"))
        {
            if (resourceStream == null)
            {
                throw new InvalidOperationException($"Couldn't find resource '{resourceName}'");
            }
            return new StreamReader(resourceStream, Encoding.UTF8).ReadToEnd();
        }
    }
}

Now we can change the stub response as follows:

handler.UriResponses.Add(
    "https://data.brreg.no/enhetsregisteret/api/enheter?navn=markedspartner",
    new HttpResponseMessage(HttpStatusCode.OK)
    {
        Content = new StringContent(TestData.ReadAsString("NameSearchResult.MarkedsPartner"), Encoding.UTF8)
    }
);

Verifying the integration

As long as the SUT returns something serializable, in this case it's an API controller returning a strongly typed transformed version of the data from Brreg, we can verify the entire graph using ApprovalTests. I've blogged about approvaltests before if you don't know what it is. Here's the entire act / assert part of the test:

var response = await controller.Lookup(new LookupRequest
{
    Name = "MarkedsPartner"
});

Approvals.VerifyJson(JsonConvert.SerializeObject(response, JsonConfigurationAttribute.SerializerSettings));

I want to make sure I serialize exactly as the controller formatter will, so I pass the production serialization settings to ApprovalTests. On a sidenote, Approvals.VerifyJson works best with non-indented JSON, so make sure to use settings that have formatting set to None.

Expanding on the stubbed world wide web

There are a couple of other mechanisms I enjoy to use with the StubHttpClientHandler. First of all, we can use it as a mock to verify that our code sends out one-way messages as well. In order to do that we simply record every request in the SendAsync method:

public class StubHttpClientHandler : HttpClientHandler
{
    public readonly List<HttpRequestMessage> Requests = new List<HttpRequestMessage>();
    public readonly Dictionary<string, HttpResponseMessage> UriResponses = new Dictionary<string, HttpResponseMessage>();

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        Requests.Add(request);

        // ...
    }
}

We can then assert that URLs and even bodies were sent off by our code:

Assert.That(handler.Requests, Has.One
    .Property("RequestUri")
    .EqualTo(new Uri("https://data.brreg.no/enhetsregisteret/api/enheter?navn=MarkedsPartner"))
);

The second mechanism I enjoy is to provide predicates and response factories in order to simplify setup.
For my current code I haven't refined it very much, so it might look a bit complex, but I'm sure you can follow the code. 😊

public class StubHttpClientHandler : HttpClientHandler
{
    public readonly List<HttpRequestMessage> Requests = new List<HttpRequestMessage>();
    public readonly Dictionary<string, HttpResponseMessage> UriResponses = new Dictionary<string, HttpResponseMessage>();
    public readonly List<(Func<HttpRequestMessage, bool> predicate, Func<HttpRequestMessage, HttpResponseMessage> responseFactory)> PredicateResponses = 
        new List<(Func<HttpRequestMessage, bool> predicate, Func<HttpRequestMessage, HttpResponseMessage> responseFactory)>();

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        Requests.Add(request);

        // Return any match for full URIs
        var url = request.RequestUri.ToString();
        if (UriResponses.ContainsKey(url))
        {
            return await Task.FromResult(UriResponses[url]);
        }

        // Run all predicates and see if one matches
        var predicateEntry = PredicateResponses.FirstOrDefault(x => x.predicate(request));
        if (predicateEntry.responseFactory != null)
        {
            // Create the response using the provided factory
            return await Task.FromResult(predicateEntry.responseFactory(request));
        }

        return await Task.FromResult(new HttpResponseMessage(HttpStatusCode.NotFound));
    }
}

Now we can set up a list of different responses as a dictionary, and then provide different responses based on the parameter. Here we could also add checks that authorization headers or other request requirements are verified by our production code:

Dictionary<string, Func<string>> searchResults = new Dictionary<string, Func<string>>
{
    { "MarkedsPartner", () => TestData.ReadAsString("NameSearchResult.MarkedsPartner")}
};

handler.PredicateResponses.Add((
    predicate: request => 
        request.RequestUri.ToString().StartsWith("https://data.brreg.no/enhetsregisteret/api/enheter?navn="),
    responseFactory: request =>
        searchResults.ContainsKey(request.RequestUri.ParseQueryString()["navn"])
            ? new HttpResponseMessage(HttpStatusCode.OK)
            {
                Content = new StringContent(searchResults[request.RequestUri.ParseQueryString()["navn"]](), Encoding.UTF8)
            }
            : new HttpResponseMessage(HttpStatusCode.NotFound)
));

We can now hide away the slightly unrefactored and ugly stubbing code, while focusing on the parameter / response pairs in the searchResults dictionary.

A search for "MarkedsPartner" will return the expected response, while a search for "Umbraco" for instance would return a 404 response.

Conclusion

Using Postman or Insomnia is really useful for spiking, and it's invaluable when we need stub data for our tests. By leveraging files and embedded resources, we can easily build strong sets of integration tests, stubbed at the very frontier before the code runs off to the internet.

Making sure your own API controllers behave correctly is a matter of verifying with your production binding and serialization settings. Then you won't have to call your own code via tools such as Postman or Insomnia.

This is all necessary for .NET Framework users. For light .NET Core solutions, it should be fairly simple to spin up an inproc webserver and do the entire HTTP process in a speedy manner. However, the stubbed version of the internet is gold here as well.

Hope you found the code interesting and thanks for reading!

Here's a copy of the complete StubHttpClientHandler described in this post:

public class StubHttpClientHandler : HttpClientHandler
{
    public readonly List<HttpRequestMessage> Requests = new List<HttpRequestMessage>();
    public readonly Dictionary<string, HttpResponseMessage> UriResponses = new Dictionary<string, HttpResponseMessage>();
    public readonly List<(Func<HttpRequestMessage, bool> predicate, Func<HttpRequestMessage, HttpResponseMessage> responseFactory)> PredicateResponses = 
        new List<(Func<HttpRequestMessage, bool> predicate, Func<HttpRequestMessage, HttpResponseMessage> responseFactory)>();

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        Requests.Add(request);

        var url = request.RequestUri.ToString();
        if (UriResponses.ContainsKey(url))
        {
            return await Task.FromResult(UriResponses[url]);
        }

        var predicateEntry = PredicateResponses.FirstOrDefault(x => x.predicate(request));
        if (predicateEntry.responseFactory != null)
        {
            return await Task.FromResult(predicateEntry.responseFactory(request));
        }

        return await Task.FromResult(new HttpResponseMessage(HttpStatusCode.NotFound));
    }
}

Environmental ApprovalTests

Sun, 03 Jun 2018 21:24:46 Z

Background

A while ago I wrote a post about a tool called ApprovalTests. I've included it in my workshop on unit testing Umbraco, and people are amazed at its usefulness. Having Visual Studio pop a diff in your face when stuff breaks is a real timesaver. However, when I ran the workshop at CodeGarden 18, I realized people were concerned that their tests would be impossible to run in CI environments and the like. Not to worry - ApprovalTests have you covered. (Pun intended)

Environmentalism

When declaring reporters with ApprovalTests, you can specify multiple reporters. That's not all. The reporters have an extensive API on them, which caters for us tailoring everything. There is one particularly interesting interface on all "diff-reporters" named IEnvironmentAwareReporter.
The VisualStudioReporter one has a nice little one-liner implementation:

public override bool IsWorkingInThisEnvironment(string forFile)
{
    return OsUtils.IsWindowsOs() && base.IsWorkingInThisEnvironment(forFile) && LaunchedFromVisualStudio();
}

The last function call there checks if the current process is a descendent of devenv.exe. If it isn't, execution will just be passed to the next reporter in the chain. So there! It won't break your CI build.

But what do we want in our CI build, then? Probably a regular WhateverUnit assertion. I use NUnit, so that'll be our example.

Falling back

Take this little test. I approved the result for seed 1, and we'll examine the output for an invalid result by swapping to 2:

[TestFixture]
[UseReporter(typeof(VisualStudioReporter))]
public class When_Running_In_Different_Environments
{
    [Test]
    public void Delegates_To_Most_Appropriate_Reporter()
    {
        var rnd = new Random(1);
        var items = Enumerable
            .Range(0, 10)
            .Select(x => rnd.Next());
        Approvals.VerifyAll(items, "");
    }
}

Let's change the seed to 2 to fail the test:

var rnd = new Random(2);

Now if we run the test with NUnit3-Console.exe, we'll get an exception saying that ApprovalTests can't find Visual Studio:

nunit3-console.exe .\bin\debug\environmentaltests.dll

...
1) Error : EnvironmentalTests.When_Running_In_Different_Environments.Delegates_To_Most_Appropriate_Reporter
System.Exception : Couldn't find Visual Studio at
at ApprovalTests.Reporters.GenericDiffReporter.Report(String approved, String received) in C:\code\ApprovalTests.Net\ApprovalTests\Reporters\GenericDiffReporter.cs:line 142
...

It sounds worse than it is. ApprovalTests insists that it has something to do. We can add a reporter to the fixture to fix it:

[TestFixture]
[UseReporter(typeof(VisualStudioReporter), typeof(NUnitReporter))]
public class When_Running_In_Different_Environments

Now when we run the test, we get a pure assertion failure:

1) Failed : EnvironmentalTests.When_Running_In_Different_Environments.Delegates_To_Most_Appropriate_Reporter
Expected string length 165 but was 162. Strings differ at index 6.
Expected: "[0] = 534011718\n[1] = 237820880\n[2] = 1002897798\n[3] = 165700..."
But was:  "[0] = 1655911537\n[1] = 867932563\n[2] = 356479430\n[3] = 211537..."
-----------------^

Notice it didn't even mention Visual Studio. Going back into our favorite IDE will start popping diffs again, and the NUnit one will govern the output.

Cleaning up

So should we go around declaring at least two reporters on all our fixtures, then? Luckily not. There are two more tricks that are nice to know.

First, the reporter attribute is found by iterating up the inheritance hierarchy. It can be defined as high up as all your [assembly:XAttribute] metadata. You can create a file in your test project root called ApprovalsConfig.cs for instance. Within it, you declare your reporters on the assembly level:

using ApprovalTests.Reporters;

[assembly:UseReporter(typeof(VisualStudioReporter), typeof(NUnitReporter))]

The second is that you might not want the NUnit assertion when you run in VS (for some reason), or maybe some other tool might be in your way. You might even want composite reporters. In any case, it also makes for a bit nicer setup if you implement the FirstWorkingReporter class:

public class EnvironmentReporter : FirstWorkingReporter
{
    public static readonly EnvironmentReporter INSTANCE = new EnvironmentReporter();

    public EnvironmentReporter()
        : base(
            VisualStudioReporter.INSTANCE,
            NUnitReporter.INSTANCE
        )
    {
    }
}

With it, we can change our UseReporter to be:

[assembly:UseReporter(typeof(EnvironmentReporter))]

Now the first reporter to confirm its environment is executed, and the rest are ignored. You might want to keep the NUnit one around, though. In that case, you can implement MultiReporter in the same way - which coincidentally is the same that UseReporterAttribute does.

I encourage you to go have a look at all the built-in reporters and get some inspiration for even more helpful reporting.

I'd rejoice for any cool usages in the discussion thread. :)

Umbraco Unit Testing Workshop Material

Thu, 09 Nov 2017 10:09:16 Z

For this years Umbraco UK Festival I was honored to be invited to run a unit testing workshop. It's a culmination of my talks and workshops until now, and covers everything you need to get going with testing the Umbraco backoffice, controller logic, external calls and finally front-end UI.

I know there's demand for this kind of training, and it might be difficult to get to the conferences where it's happening. For those of you, and those who feel bold and confident, I've released the material under the MIT license.

I can't promise I'll be able to help out if you go at it, but keep an eye out for a conference near you and encourage the organizers to host a workshop. ;)

You can clone the "workshop-start" branch from Bitbucket. The slides and workbook are included at the root of the repo.

https://bitbucket.org/bleedo/umbraco-testing-workshop-h2-2017/src/?at=workshop-start

You'll need

node
npm task runner (visual studio extension)
razorgenerator (visual studio extension)

Happy testing!

Exploiting ApprovalTests for clearer tests

Mon, 23 Oct 2017 20:14:08 Z

What's this for?

Ever written several asserts in one test because you have a big graph you want to verify? How 'bout files or maybe razor views? Several asserts often clutter up the tests. Big strings also make the actuall calls hard to see for all that content. Putting big strings in files is a good idea to avoid that, but few people do. It adds another "menial" task when you're in the zone. But what if it was dead easy?

I once maintained a Resharper extension, and their example tests had so called "gold" files that they compared to output. The squigglies was represented by special characters around terms. So they just compared the output of a text renderer to the text in a file. Great idea. One limitation though: with a regular Assert.Equals you just see the segment around the first mismatch. Guess what, there's a tool that's been around for at least 10 years that solves all those issues, and more.

Approval tests, eh?

Sounds a bit like acceptance tests, right? Don't be fooled, it's purpose is to serve all the way down to the "unit test layer" of your tests. I've found it to be a huge timesaver, as well as making my tests so much more clear.

I know you're thinking "Shut up and give me an example, then!", so let's have a look. I've got this unit test from my post about testing razor views that I never really asserted anything in. I just output the result to console and assert inconclusive. Here it is for reference:

[TestFixture]
public class When_Displaying_An_Event
{
    [Test]
    public void It_Is_Rendered_With_A_Name_Date_And_A_Link()
    {
        var view = new _Views_Partials_Event_cshtml();
        var actionResult = GetConcertEvent();

        Assert.AreEqual("Event", actionResult.ViewName);

        var renderedResult = view.Render((Event) actionResult.Model);

        Console.WriteLine(renderedResult);
        Assert.Inconclusive("Way too big to assert here.");
    }
}

It outputs the following HTML:

<div>
    <a href="https://eventsite/123">
        <label>
            Concert of your life
        </label>
        <span>
            fredag 31. desember 2049 23.59
        </span>
    </a>
</div>

Let's see it

Now let's assert this with ApprovalTests. To use it, you just install-package ApprovalTests with your trusty package manager console. Make sure to install it in your test project. ;)

Now instead of Assert, we ask ApprovalTests to Verify our data instead. It even has a special overload for this concrete case: Approvals.VerifyHtml. So we rewrite the test as such:

[Test]
public void It_Is_Rendered_With_A_Name_Date_And_A_Link()
{
    var view = new _Views_Partials_Event_cshtml();
    var actionResult = GetConcertEvent();

    Assert.AreEqual("Event", actionResult.ViewName);

    var renderedResult = view.Render((Event) actionResult.Model);

    Approvals.VerifyHtml(renderedResult);
}

Now when we run our test, we get this nice little welcoming message from ApprovalTests:

It tells us we're missing a vital part: A reporter. It's possible to use DiffReporter to launch your favorite configured difftool. But if you're in Visual Studio, there's a special treat: VisualStudioReporter. Let's add that to our fixture and see what happens:

[UseReporter(typeof(VisualStudioReporter))]
[TestFixture]
public class When_Displaying_An_Event
{
    [Test]
    public void It_Is_Rendered_With_A_Name_Date_And_A_Link()
    {
        // ...
    }
}

And we run it:

What hits you is probably the big failure statement there, but look again - up at the top there. We've got a diff opened, the result on the left hand and a big green field on the right side.
What just happened is that ApprovalTests took our result, stored it in a received file, and at the same time wrote an empty "approved" file. It then proceeded to compare, and finally pop a diff of those two files.
Isn't that just beautiful? Everything that makes your test fail in one clear diagram.

The "procedure to follow" here, is to "approve" results when you're happy. To do that, you just copy and paste. Let's do that now:

If you've got ReSharper, it's probably gonna try to format everything nicely when you paste. To have it in the original, ugly indentation state, press Ctrl+Z (undo) once after pasting.

Regarding the indentation that's off. It seems to be a bug with HTML in the current version of ApprovalTests, so stupid me for choosing this example. I'll update the post if it gets fixed.

We can now run our test again, and it's gonna pass. When it passes, it doesn't bother to open the diff.

This means we're just gonna get diffs for whatever is currently failing. Even if we run our entire suite of tests. Now there's a couple of housekeeping things to keep in mind:

Commit approvals only

If you were paying attention, you noticed we got two files adjacent to our test source file. One is named [TestFixtureName].[TestMethodName].received.html and one is named [TestFixtureName].[TestMethodName].approved.html. If you ran a test that passed, you'll actually just have your approved file. You want those approved files in source control!

The received files though, might end up not being cleaned up for one or the other reason. Hopefully just that you didn't bother to commit a fully passing build. I'm sure you didn't do that to the master branch, though. In any case, make sure to ignore your received files. This pattern in .gitignore generally does the trick:

*.received.*

That's just the tip of the iceberg

We've seen the VerifyHtml bit. One of my personal favorites is it's sibling VerifyJson. I keep an extension on object in my tests, called .ToJson(). With it, I can just go:

Approvals.VerifyJson(myBigGraph.ToJson());

The diff is done with prettified JSON, so it's super simple to find the property / area that has changed or doesn't work. Knowing the area of the graph should also make it easier to find the usage that is wrong.

There's a vanilla Verify method too, and it saves plain text files. It's useful in cases where you have nice ToString() implementations. Let's try with a "person":

public class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }

    public override string ToString()
    {
        return $"{FirstName} {LastName} ({Age})";
    }
}

[Test]
public void Comparing_Objects()
{
    var person = new Person
    {
        FirstName = "Lars-Erik",
        LastName = "Aabech",
        Age = 19
    };

    Approvals.Verify(person);
}

It produces the following .received file:

Lars-Erik Aabech (19)

It can be approved like that.
We can also do lists with "big brother" VerifyAll:

[Test]
public void Comparing_Lists()
{
    var list = new List<Person>
    {
        new Person {FirstName = "Lars-Erik", LastName = "Aabech"},
        new Person {FirstName = "Dear", LastName = "Reader"}
    };

    Approvals.VerifyAll(list, "");
}

Which, unsurprisingly outputs:

[0] = Lars-Erik Aabech
[1] = Dear Reader

Now how bout that?
I don't know how quickly you got hooked, but I certainly find it sneaking into more and more of my tests.

It can even do images, but I'll let James blog about that one.

So what are you lingering around here for? Run over to nuget and get your copy, or lurk around some more at the ApprovalTests project site. There's great examples, even though they might not be in your favorite language.

Happy approving! :)

UmbracoSupport got typed content

Tue, 17 Oct 2017 21:18:06 Z

What's UmbracoSupport?

UmbracoSupport is a class I've been introducing to my unit tests over the last year or so. It allows me to have my own hierarchy for tests, as well as re-using all of Umbraco's own stubbing code. I've written about it in a post called Unit testing Umbraco just got simpler, and its gut's code is described in details in The basics of unit testing Umbraco.

A quick primer on what's already available

The BaseDatabaseFactoryTest in Umbraco.Tests has a method called GetXmlContent. It replaces the umbraco.config file that acts as the cache at runtime. It makes UmbracoContext.Current.ContentCache tick in unit tests. The base tests out of the box has a small flaw though. They can't "popuplate" properties. All you get is the hierarchy.

Usually I've injected an IPublishedContentCache to my controllers. When testing them, I've created a mock instance of the IPublishedContentCache. However, all my code has to use the non-context aware overloads. For instance cache.GetById(umbracoContext, false, id). There's also a whole lot of ugly mocking code going on to set up queries and stubbed content. How to stub properties on stubbed content is described in Slides from CG 2016 and testing IPublishedContent properties.

So what's new?

As mentioned, I've been throwing around all kinds of ugly stubbing code for content and I've also been tampering with Umbraco.Tests's GetXmlContent() to use the "built-in" content stubs. It's all been done before in misc. tests in Umbraco. I finally got my s**t together and refactored all my setup spaghetti into a few small helpers on the UmbracoSupport class.

Let's go over them in increasing "integrationness".

Pure hierarchy

Your basic hierarchy structure can be set up by simply returning a string from an overload of BaseDatabaseFactoryTest.GetXmlContent. UmbracoSupport overloads this method and returns whatever you've set on the UmbracoSupport.ContentCacheXml property. I recommend using the technique described in Automating creating of source data for tests with this. You can even extend that code to have fixture specific content caches.

In any case, to make this work, you just need to set the XML in the setup method.

Note: I've got some probs with the markdown parsing here, imagine the CDATA parts of the XML is correctly written.

[SetUp]
public void Setup()
{
    umbracoSupport = new UmbracoSupport();
    umbracoSupport.SetupUmbraco();

    // This XML is what the ContentCache will represent
    umbracoSupport.ContentCacheXml = @"
        <?xml version=""1.0"" encoding=""utf-8""?>
        <!DOCTYPE root [
          <!ELEMENT contentBase ANY>
          <!ELEMENT home ANY>
          <!ATTLIST home id ID #REQUIRED>
          <!ELEMENT page ANY>
          <!ATTLIST page id ID #REQUIRED>
        ]>
        <root id=""-1="""">
          <home id=""1103="""" key=""156f1933-e327-4dce-b665-110d62720d03="""" parentID=""-1="""" level=""1="""" creatorID=""0="""" sortOrder=""0="""" createDate=""2017-10-17T20:25:12="""" updateDate=""2017-10-17T20:25:17="""" nodeName=""Home="""" urlName=""home="""" path=""-1,1103="""" isDoc="""" nodeType=""1093="""" creatorName=""Admin="""" writerName=""Admin="""" writerID=""0="""" template=""1064="""" nodeTypeAlias=""home="""">
            <title>Welcome!</title>
            <excerptCount>4</excerptCount>
            <page id=""1122="""" key=""1cb33e0a-400a-4938-9547-b05a35739b8b="""" parentID=""1103="""" level=""2="""" creatorID=""0="""" sortOrder=""0="""" createDate=""2017-10-17T20:25:12="""" updateDate=""2017-10-17T20:25:17="""" nodeName=""Page="" 1="""" urlName=""page1="""" path=""-1,1103,1122="""" isDoc="""" nodeType=""1095="""" creatorName=""Admin="""" writerName=""Admin="""" writerID=""0="""" template=""1060="""" nodeTypeAlias=""page="""">
              <title>Welcome!</title>
              <excerpt>[CDATA[Lorem ipsum dolor...]]</excerpt>
              <body>
                [CDATA[<p>Lorem ipsum dolor...</p>]]
              </body>
              <image>123</image>
            </page>
            <page id=""1123="""" key=""242928f6-a1cf-4cd3-ac34-f3ddf3526b2e="""" parentID=""1103="""" level=""2="""" creatorID=""0="""" sortOrder=""1="""" createDate=""2017-10-17T20:25:12="""" updateDate=""2017-10-17T20:25:17="""" nodeName=""Page="" 2="""" urlName=""page2="""" path=""-1,1103,1123="""" isDoc="""" nodeType=""1095="""" creatorName=""Admin="""" writerName=""Admin="""" writerID=""0="""" template=""1060="""" nodeTypeAlias=""page="""">
              <title>More welcome!</title>
              <excerpt>[CDATA[More lorem ipsum dolor...]]</excerpt>
              <body>[CDATA[Even more lorem ipsum dolor...]]</body>
              <image>234</image>
            </page>
          </home>
        </root>
    ".Trim();
}

In our tests, we can now query by anything. The returned content has the hierarchy and everything, so we can traverse it with Children(), Parent() and whatnot. The only missing piece is the properties. Here's a test showing that we have everything but the title property of Page 1:

const int Page1Id = 1122;

[Test]
public void Returns_Empty_Documents()
{
    var contentCache = umbracoSupport.UmbracoContext.ContentCache;
    var page1 = contentCache.GetById(Page1Id);

    Assert.That(page1, Is
        .Not.Null
        .And
        .InstanceOf<PublishedContentWithKeyBase>()
        .And
        .Property("Name").EqualTo("Page 1")
        .And
        .Matches<IPublishedContent>(c => c["title"] == null)
        .And
        .Property("Parent")
            .Property("Children")
                .With.Count.EqualTo(2)
    );
}

Don't be discouraged though. This method is excellent for testing URL providers, ContentFinders, Menus, Sitemaps. You name it. I know I've written my fair share of hierarchy traversing code or fancy XPath queries. Unless of course, you need property values.

Instead of pulling your leg about it, here's how we fix that.

Put some meat on the content

The reason the properties are not there isn't because they weren't read. It's because the XmlPublishedContent that we get out ultimately relies on the PublishedContentType for it's respective document type. Luckily, all Umbraco's services are already stubbed up for us, so we can give it what it needs.

The gory guts of it is that it needs an IContentType from the ContentTypeService. We can easily stub one up with Moq: var contentType = Mock.Of<IContentType>(). Further, it uses the IContentType.CompositionPropertyTypes collection to iterate the properties. These PropertyType instances are actually completely dependency-less, so we can just create some:

Mock.Get(contentType)
    .Setup(t => t.CompositionPropertyTypes)
    .Returns(new[] {
        new PropertyType("Umbraco.TinyMCEv3", DataTypeDatabaseType.Nvarchar, "body")
    });

Finally, we set it up on the ContentTypeService stub:

Mock.Get(umbracoSupport.ServiceContext.ContentTypeService)
    .Setup(s => s.GetContentType(alias))
    .Returns(contentType);

If only it were so easy. We depend on the BaseWebTest class from Umbraco.Tests. It sets up a content type factory that's being used somewhere in the hierarchy. It feeds AutoPublishedContent instances instead of what we've stubbed up. We need to turn that off. There's a trick here. UmbracoSupport should now live in an assembly called Umbraco.UnitTests.Adapter. If you're pre 7.6.4 you need to go with Umbraco.VisualStudio. This is because the factory we need to reset is internal to Umbraco. By having UmbracoSupport in an assembly with one of these two names, we're able to do it. (Otherwise, you use reflection.) By no means do this with production code. Just... forget it!
This paragraph should also get it's own blog post. :)

But I digress. Here's the line you need to have the content use the ContentTypeService to fetch its type:

PublishedContentType.GetPublishedContentTypeCallback = null;

It's tempting to leave setup code like this lying around in all our SetUp methods or even in our "Arrange" sections. I've sinned too much, so those few lines are now part of UmbracoSupport and can be used to set up multiple types for your fixture or test.

Here's a test that fetches the same document as before, but can now read properties:

[Test]
public void With_DocumentTypes_Setup_Returns_Full_Blown_Documents()
{
    umbracoSupport.SetupContentType("page", new[]
    {
        new PropertyType("textstring", DataTypeDatabaseType.Nvarchar, "title"),
        new PropertyType("textarea", DataTypeDatabaseType.Nvarchar, "excerpt"),
        new PropertyType("Umbraco.TinyMCEv3", DataTypeDatabaseType.Nvarchar, "body"),
        new PropertyType("media", DataTypeDatabaseType.Integer, "image")
    });

    var page1 = contentCache.GetById(Page1Id);

    Assert.Multiple(() =>
    {
        Assert.That(page1["title"], Is.EqualTo("Welcome!"));
        Assert.That(page1["excerpt"], Is.EqualTo("Lorem ipsum dolor..."));
        Assert.That(page1["body"].ToString(), Is.EqualTo("<p>Lorem ipsum dolor...</p>"));
        Assert.That(page1["image"], Is.EqualTo(123));
    });
}

Notice the .ToString() on the body. It's actually not a string, but some weird dynamic Umbraco thingy. I never saw that type before, but I didn't pursue it in time for this post. I don't want anything to do with it though, so let's storm on to the grand finale.

Let's make them strong already!

We're finally there. The last piece of the puzzle. Strongly typed content!

It's managed by two resolvers: PublishedContentModelFactoryResolver and PropertyValueConvertersResolver. I won't go into details about those now, but suffice to say all resolvers have to be initialized before BaseWebTest.Initialize and its ancestors. I've added an InitializeResolvers method to the UmbracoSupport class where these two are initialized. The PublishedContentModelFactoryResolver is set to a FakeModelFactoryResolver that lets you register constructors for document type aliases. The code for this is available in my "Umbraco unit testing samples" repo on github.

To set up property value converters, we also need to do that before registering the resolver. The resolver takes all the converters as constructor arguments. I've added a list of those types as a property on UmbracoSupport, so we can add IPropertyValueConverter implementing types before calling UmbracoSupport.SetupUmbraco:

[SetUp]
public void Setup()
{
    umbracoSupport = new UmbracoSupport();

    // Converter types need to be added before setup
    umbracoSupport.ConverterTypes.Add(typeof(TinyMceValueConverter));

    umbracoSupport.SetupUmbraco();

    //...
}

To register the typed model, there's just one line you can do in your setup, or even in your tests. Here I've refactored out the setup for the content type from earlier, and I register a model type for the document type alias:

private void SetupContentType()
{
    umbracoSupport.SetupContentType("page", new[]
    {
        new PropertyType("textstring", DataTypeDatabaseType.Nvarchar, "title"),
        new PropertyType("textarea", DataTypeDatabaseType.Nvarchar, "excerpt"),
        new PropertyType("Umbraco.TinyMCEv3", DataTypeDatabaseType.Nvarchar, "body"),
        new PropertyType("media", DataTypeDatabaseType.Integer, "image")
    });
}

[Test]
public void With_DocumentTypes_And_Models_Setup_Returns_Fully_Functional_Typed_Content()
{
    SetupContentType();

    // Register strongly typed models with the ModelFactory
    umbracoSupport.ModelFactory.Register("page", c => new Page(c));

    var page1 = contentCache.GetById(Page1Id);

    Assert.That(page1, Is
        .InstanceOf<Page>()
        .And.Property("Body")
            .Matches<IHtmlString>(s => 
                s.ToString() == "<p>Lorem ipsum dolor...</p>"
            )
    );
}

public class Page : PublishedContentModel
{
    public Page(IPublishedContent content) : base((IPublishedContentWithKey)content)
    {
    }

    public string Title => Content.GetPropertyValue<string>("title");
    public string Excerpt => Content.GetPropertyValue<string>("excerpt");
    public IHtmlString Body => Content.GetPropertyValue<IHtmlString>("body");
    public int Image => Content.GetPropertyValue<int>("image");
}

There you go! There's nothing more to it. Well, there is...

The Page class here is bundled with the test. If we use a common interface both for our runtime model and our test model, we're safe. But we should really use the runtime models. This means you shouldn't use runtime generated models. Go through the instructions for ModelsBuilder to have your models compiled and accessible from the tests.

Conclusion

And although the XML is pretty ugly, you can flush it out into files bundled with your tests. You can also exploit the umbraco.config file and just copy segments from there into your test source files. That way, you spend no time writing the stubs, and the content is cleanly separated from your tests.

That's really all there is to it! It is. Now go test a bit, or a byte, or a string, or even a view.

The new version of UmbracoSupport including the fake model factory is available here.

Testing views with RazorGenerator

Mon, 09 Oct 2017 21:08:20 Z

Razor what?

RazorGenerator is a hidden gem that lets you generate and pre-compile what would otherwise be generated and compiled at runtime. Your Razor. Not only does it give you a startup time boost, but it lets you unit test your views. The latter is the focus of this post.

We'll continue to build on the project I described in my post on generating documentation with NUnit. It's a simple use case where an imaginary CMS has a feature to display events from a third party site. Here are the tests I've got so far:

In my previous post, I left the rendering test inconclusive. I like keeping inconclusive tests around as reminders of stuff I've got left to do. Let's have a quick look at the passing code before we dive into the rendering bits.

Basic HTTP integration and conversion tests

The first large piece of what's in there for now is a way to remove the physical dependency on the third party site. I like to stub away IO as far out as I can so I can test as much of my code quickly, yet as integrated as possible. In other words, as many participating classes as possible. Since we're making an example here, I'm just using the HttpClient directly from the controller. The HttpClient is hard to mock or fake, but it has an inner dependency that we can pass as an argument to a constructor: HttpMessageHandler. It has the innermost function that the HttpClient uses for any operation. It also has the rare trait of being protected virtual, so we can stub it out. For this example, I'm just using a fake one that records requests and returns responses for registered URLs. Here it is:

public class FakeMessageHandler : HttpMessageHandler
{
    private readonly Dictionary<string, string> responses = new Dictionary<string,string>();
    public List<string> Calls { get; } = new List<string>();

    public void Register(string url, string response)
    {
        responses.Add(url, response);
    }

    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var url = request.RequestUri.ToString();

        Calls.Add(url);

        if (!responses.ContainsKey(url))
        { 
            return Task.FromResult(new HttpResponseMessage
            {
                StatusCode = HttpStatusCode.NotFound,
                Content = new StringContent("{}")
            });
        }

        return Task.FromResult(new HttpResponseMessage
        {
            Content = new StringContent(responses[url])
        });
    }
}

We'll not dwell further on this one in this post, but you'll need it to run the following examples if you want to tag along.

The tests that verifies that we call the right URL on the third party site is pretty simple. It checks the Calls collection on the FakeMessageHandler. Here's the test and the setup code needed:

[TestFixture]
public class When_Displaying_An_Event
{
    [Test]
    [DocumentationOrder(0)]
    [Description(@"
    Events are provided at eventsite with a REST api at the URL:
    https://eventsite/api/{id}
    ")]
    public void It_Is_Fetched_By_Id_From_The_Event_Server()
    {
        eventController.Event(234).Wait();

        Assert.AreEqual(
            "https://eventsite/api/234", 
            httpMessageHandler.Calls.Single()
        );
    }

    // ... omitted other tests

    [SetUp]
    public void Setup()
    {
        httpMessageHandler = new FakeMessageHandler();
        httpClient = new HttpClient(httpMessageHandler);
        eventController = new EventController(httpClient);
    }

    private FakeMessageHandler httpMessageHandler;
    private HttpClient httpClient;
    private EventController eventController;
}

And the controller:

public class EventController : Controller
{
    private readonly HttpClient httpClient;

    public EventController(HttpClient httpClient)
    {
        this.httpClient = httpClient;
    }

    public async Task<PartialViewResult> Event(int id)
    {
        var requestUri = "https://eventsite/api/" + id;
        var result = await httpClient.GetAsync(requestUri);
        var response = await result.Content.ReadAsStringAsync();
        // ... omitted further processing for now
    }
}

We pass the FakeMessageHandler instance to a new HttpClient effectively making it a stub. We can control the response for individual URLs, and verify that the right ones were called.

Next step is to convert it to a valid model we can use in our view. We've got a second test asserting that:

[Test]
[DocumentationOrder(1)]
[Description("The JSON response from the event server is deserialized as the Event type.")]
public void It_Is_Converted_To_An_Event_Model()
{
    var result = GetConcertEvent();
    var model = (Event)result.Model;

    Assert.AreEqual("Concert of your life", model.Name);
    Assert.AreEqual(new DateTime(2049,12,31,23,59,59), model.Time);
    Assert.AreEqual("https://eventsite/123", model.Url);
}

private PartialViewResult GetConcertEvent()
{
    httpMessageHandler.Register(
        "https://eventsite/api/123",
        "{\"name\":\"Concert of your life\", \"time\":2524607999}"
    );

    var result = eventController.Event(123).Result;
    return result;
}

And here's the rest of the controller code creating that model:

public async Task<PartialViewResult> Event(int id)
{
    var requestUri = "https://eventsite/api/" + id;
    var result = await httpClient.GetAsync(requestUri);
    var response = await result.Content.ReadAsStringAsync();
    var eventJObj = JsonConvert.DeserializeObject<JObject>(response);
    var evt = new Event
    {
        Name = eventJObj.Value<string>("name"),
        Time = DateTimeOffset.FromUnixTimeSeconds(eventJObj.Value<long>("time")).DateTime,
        Url = "https://eventsite/" + id
    };
    return PartialView("Event", evt);
}

The test is a tedious verification of every property on the event object. There are several ways to get around that. Amongst others, equality members. I've got a way better trick, but that's for an upcoming post.

Now that we're through those tests, let's dive into how we can test the actual HTML output of this whole shenanigan.

Generating some Razor

As mentioned initially, RazorGenerator is a hidden gem in the ASP.NET OSS wilderness. There's a Visual Studio plugin that you need to exploit it fully. It's aptly called RazorGenerator and can be installed from here. There's also a couple of Nuget packages that we want:

RazorGenerator.Mvc
Referenced in your website to make use of a special ViewEngine for pre-compiled views.
RazorGenerator.Testing
Referenced in our test projects to be able to automagically render views in tests.

Armed with these tools, there's nothing left to do but add a custom tool to our view:

As soon as you've blured the "Custom Tool" field, a .cs file will be generated beneath the view. Upon inspection it yields a class in the ASP namespace making a whole lot of Write* statements. This is what clutters up your ASP.NET Temporary files all the time. And the nice part: it can be instantiated.

Here's the initial markup from Event.cshtml:

@model Example.WebSite.Models.Event

<div>
    <a href="@Model.Url">
        <label>
            @Model.Name
        </label>
        <span>
            @Model.Time.ToString("f")
        </span>
    </a>
</div>

Over in our unit test, we can now start adding some actual rendering of HTML:

[Test]
[DocumentationOrder(2)]
public void It_Is_Rendered_With_A_Name_Date_And_A_Link()
{
    var view = new _Views_Partials_Event_cshtml();
    var result = GetConcertEvent();

    Assert.AreEqual("Event", result.ViewName);

    var renderedResult = view.Render((Event) result.Model);

    Console.WriteLine(renderedResult);
    Assert.Inconclusive("Rendering has not been implemented yet.");
}

private PartialViewResult GetConcertEvent()
{
    httpMessageHandler.Register(
        "https://eventsite/api/123",
        "{\"name\":\"Concert of your life\", \"time\":2524607999}"
    );

    var result = eventController.Event(123).Result;
    return result;
}

I know, the view type name ain't too pretty. But it's necessary for the view engine to find the right one based on the path given. The cool part is the Render statement. It's an extension method from the RazorGenerator.Tests package. It returns the final HTML as a string.

The Console.WriteLine statement yields the following in our test output now:

<div>
    <a href="https://eventsite/123">
        <label>
            Concert of your life
        </label>
        <span>
            fredag 31. desember 2049 23.59
        </span>
    </a>
</div>

(Yes, I force my locale upon thee!)

What we've just done is to test our system almost end to end, cut off at the uttermost borders to external IO. Specifically, the third party site and the end-user's browser.

Granted, we could do Selenium tests here to really test it via the browser, but my rule of thumb is that RazorGenerator is the best choice unless you've got a lot of JavaScript interactivity you need to test in integration. These are subjects for another post.

There is a remaining issue though. We should assert the HTML we got rendered. We could store that HTML we got in a const string expected in our test, but it's gonna foul up the code quite a bit. We could go with so-called "gold files" and implement something to compare a file to the actual output. There's a magical tool for that called ApprovalTests, which I'll cover in my next post.

There's also the option of using HtmlAgilityPack to query the rendered view. The RazorGenerator.Tests package have built-in alternative for Render called RenderAsHtml that returns HtmlDocument instances for you to query. It's quite useful when your only Assert.That is for some specific element in a big page.

Resources and a small limitation

You've seen how you can use RazorGenerator to test your views. There are several posts by David Ebbo (one of the authors of RazorGenerator) on how to use RazorGenerator. Please check them out for more details than I was able to give here.

For now it doesn't do nested Html.Action or Html.Partial calls. I've got a PR in the works, but I need to polish it for it to get in there. Some day soon. ;) If you really want to, you'll find my fork and build your own, but you'll be on your own.

There's also a tiny performance hit. You'll have to wait a second longer for your tests to execute, since a lot of the ASP.NET MVC framework is spun up to render views. It's still less than the magical 2 second focus cap though, so you should be able to work effectively with it.

I hope this piqued your interest in writing broader tests even up to the UI layers. There's even cooler tricks in store for you if you're already on .net Core, but the rest of us will have to make due until then.

Generating documentation from NUnit tests

Sun, 08 Oct 2017 21:06:26 Z

What's the catch?

There's a lot of benefits with unit testing. One of the less exploited ones would be documentation. I've heard it said, and I've said it myself countless times: Unit Tests can be used as documentation. They are inherently documentation by being a good place to look for sample code. But what if they could also serve as actual go-to documentation, a backlog, a technical specification and all those things that otherwise go rot in a Word document on some server?

Lately, I've been experimenting with generating good documentation from NUnit tests for our ASP.NET MVC sites. (Based on Umbraco CMS, but that's not important to this article.)
I've been baffled by the lack of good information when I've googled for NUnit result transformations. So I gave it a go, and I'll share some techniques with you in this post.

Many unit tests won't do for documentation though. I believe that testing should be done as integrated as possible, stopping just before they become slow. This allows for more user-centric, or holistic descriptions of the system. I'm doing my best to get better at naming them. They should read like stories describing the system. That way it'll be easy for other devs to find the right code to extend or debug. Namespacing and organization play a strong role, as well as good, descriptive names. It's hard, but I believe striving for this might actually also aid in the ability to use them as documentation.

An example

So for this example, let's have a look at a pretty simple case. We're integrating with a third-party event site to display relevant events on pages in our site. We have some kind of module that lets editors pick events and place them in pages. The part I'll use in this post is the part about fetching and displaying it.

The tests I'm writing are feature specific, but technical enough in nature to belong in the realm of technical documentation. For business targeted documentation, you're better off looking into BDD and all the practices that follows. If you don't have the team though, you might be able to get something working for you with the following techniques.

Here's the test output I've got so far:

I think it reads fairly well. There shouldn't be too much doubt what this feature does. Granted, it's a really simple one, and complex systems will have thousands. Even more reason to group by feature, and add areas and other useful grouping levels as namespaces.

You'll also notice I've got a test that is inconclusive saying not implemented. I've started doing this to keep a kind of backlog of things to do. It's easy to forget about technical debt or rarer cases we've ignored. Having inconclusive tests glaring at you makes it easy to remember. It may also serve as a discussion point to bring up with the customer. "Should we bother to handle this weird case? It'll increase the budget with $...".

There are still a couple of readability issues in the tests. When we add more features, it'd be nice to find all event-related tests in their own group. That's a quick fix. We'll add a namespace. (Put common tests in the same folder)

There's also a matter of order. Obviously fetching from server happens before converting to a model and then rendering it. There are concepts like ordered tests, but I believe atomic tests are faster and more flexible. I've got a solution for that, but let's live with it for now.

Having this output from the testrunner or on the build server, even in your inbox when someone commits, is useful. We have a broad overview of what the system does. The tests names are decoupled from any implementation details so we're free to change the architecture while keeping our descriptive tests.

But we could do more.

NUnit console runner output

When working on code, we use built-in test runners in our IDEs. We're able to run one or a few test in isolation and focus on just the right things. Build servers use command line runners and ususally produce some form of textual or HTML output. We can exploit the CLI tools as well. The NUnit console runner outputs an XML format by default. You can even do it from the package manager console in Visual Studio like so:

PM> .\packages\NUnit.ConsoleRunner.3.7.0\tools\nunit3-console.exe .\NUnitDocs.Tests\bin\debug\NUnitDocs.Tests.dll

If you're using Nuget to organize your dependencies, you can install the NUnit.ConsoleRunner package to get the executable in the packages folder. Otherwise, you can download it from the NUnit downloads page.

In addition to logging the results to the output, NUnit will write some metadata, and the following (more interesting) to TestOutput.xml:

<test-run id="2" testcasecount="3" result="Passed" total="3" passed="2" failed="0" inconclusive="1" >

  <!-- Attributes and metadata elements stripped for clarity -->

  <test-suite type="TestSuite" id="0-1005" name="NUnitDocs" fullname="NUnitDocs" runstate="Runnable" testcasecount="3" result="Passed" total="3" passed="2" failed="0" warnings="0" inconclusive="1">
    <test-suite type="TestSuite" id="0-1006" name="Tests" fullname="NUnitDocs.Tests">
      <test-suite type="TestSuite" id="0-1007" name="Events" fullname="NUnitDocs.Tests.Events">
        <test-suite type="TestFixture" id="0-1000" name="When_Displaying_An_Event" fullname="NUnitDocs.Tests.Events.When_Displaying_An_Event">
          <test-case id="0-1002" name="It_Is_Converted_To_An_Event_Model" fullname="NUnitDocs.Tests.Events.When_Displaying_An_Event.It_Is_Converted_To_An_Event_Model" methodname="It_Is_Converted_To_An_Event_Model" classname="NUnitDocs.Tests.Events.When_Displaying_An_Event" runstate="Runnable" seed="1189788506" result="Passed" start-time="2017-10-08 19:44:11Z" end-time="2017-10-08 19:44:12Z" duration="0.413085" asserts="3" />
          <test-case id="0-1001" name="It_Is_Fetched_By_Id_From_The_Event_Server" fullname="NUnitDocs.Tests.Events.When_Displaying_An_Event.It_Is_Fetched_By_Id_From_The_Event_Server" methodname="It_Is_Fetched_By_Id_From_The_Event_Server" classname="NUnitDocs.Tests.Events.When_Displaying_An_Event" runstate="Runnable" seed="486335788" result="Passed" start-time="2017-10-08 19:44:12Z" end-time="2017-10-08 19:44:12Z" duration="0.001595" asserts="1" />
          <test-case id="0-1003" name="It_Is_Rendered_With_A_Name_Date_And_A_Link" fullname="NUnitDocs.Tests.Events.When_Displaying_An_Event.It_Is_Rendered_With_A_Name_Date_And_A_Link" methodname="It_Is_Rendered_With_A_Name_Date_And_A_Link" classname="NUnitDocs.Tests.Events.When_Displaying_An_Event" runstate="Runnable" seed="804806648" result="Inconclusive" start-time="2017-10-08 19:44:12Z" end-time="2017-10-08 19:44:12Z" duration="0.015905" asserts="0">
            <reason>
              <message>
                <![CDATA[Rendering has not been implemented yet.]]/>
              </message>
            </reason>
          </test-case>
        </test-suite>
      </test-suite>
    </test-suite>
  </test-suite>
</test-run>

I know, I know. It's not cool anymore. I don't like it any more than you, but there is one durable ancient technology we could use to do something with this output. Did you guess it yet?

XSLT!

There. I said it.

Now let's look at a simple output before we dive into the more beefy things:

Now ain't that starting to look readable? There's not that much to it. The NUnit console runner has a built-in parameter with the syntax --result=SPEC, where SPEC can point to an XSLT sheet. Here's the command to generate the former:

.\packages\NUnit.ConsoleRunner.3.7.0\tools\nunit3-console.exe .\NUnitDocs.Tests\bin\debug\NUnitDocs.Tests.dll  --result:"TestSummary.htm;transform=TestSummary.xslt"

The XSLT isn't much either. There's Bootstrap and (our trusty) JQuery from CDN. Some simple HTML formatting. And I wrote a one-liner JavaScript function to strip out all the underscores as well. Here's the first pass:

<?xml version="1.0" encoding="iso-8859-1"?>
<xsl:stylesheet
    version="1.0"
    exclude-result-prefixes="msxsl"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:msxsl="urn:schemas-microsoft-com:xslt"
    >
  <xsl:output method="html" indent="yes"/>

  <xsl:template match="@* | node()">
    <html>
      <head>
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous"/>
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous"/>
        <style type="text/css">
          .Passed { color: green; }
          .Inconclusive { color: #BBAA00; }
          .Failed { color: red; }

          ul {
          margin-left: 0px;
          list-style-type: none;
          padding-left: 0;
          }

          ul ul {
          margin-left: 15px;
          }

          .counts {
          font-size: .7em;
          color: gray;
          }
        </style>

      </head>
      <body>
        <div class="container">
          <xsl:apply-templates select="/test-run/test-suite"/>
        </div>
        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous">// Force closing tag</script>
        <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js">// Force closing tag</script>
        <script type="text/javascript">
          $("td").each(function(i, e) {
          $(e).text($(e).text().replace(/_/g, " "));
          });
        </script>
      </body>
    </html>
  </xsl:template>

  <xsl:template match="/test-run/test-suite">
    <h1>
      <xsl:value-of select="./test-run/@name"/>
    </h1>
    <table class="table table-striped">
      <xsl:apply-templates select="./test-suite"/>
    </table>
  </xsl:template>

  <xsl:template match="test-suite">
    <tr>
      <td>
        <xsl:attribute name="class">
          <xsl:choose>
            <xsl:when test="./@failed > 0">Failed</xsl:when>
            <xsl:when test="./@inconclusive > 0">Inconclusive</xsl:when>
            <xsl:otherwise>Passed</xsl:otherwise>
          </xsl:choose>
        </xsl:attribute>
        <xsl:attribute name="style">
          padding-left: <xsl:value-of select="count(ancestor::test-suite)*15"/>px;
        </xsl:attribute>
        <xsl:value-of select="./@name"/>
      </td>
      <td class="counts">
        <xsl:value-of select="./@passed"/> passed,
        <xsl:value-of select="./@inconclusive"/> inconclusive,
        <xsl:value-of select="./@failed"/> failed
      </td>
    </tr>
    <xsl:for-each select="./test-suite">
      <xsl:apply-templates select="."/>
    </xsl:for-each>
    <xsl:for-each select="./test-case">
      <xsl:apply-templates select="."/>
    </xsl:for-each>
  </xsl:template>

  <xsl:template match="test-case">
    <tr>
      <td colspan="2">
        <xsl:attribute name="style">
          padding-left: <xsl:value-of select="count(ancestor::test-suite)*15"/>px;
        </xsl:attribute>
        <xsl:attribute name="class">
          <xsl:value-of select="./@result"/>
        </xsl:attribute>
        <xsl:value-of select="./@name"/>
      </td>
    </tr>

  </xsl:template>
</xsl:stylesheet>

This is fairly good as a starting point, but we're stuck with the brief test names. Having some free text, at least at feature level or use-case level would be nice. And we still have this ordering problem to deal with. NUnit has the perfect tool for this:

NUnit PropertyAttribute

NUnit comes with a PropertyAttribute that we can use to decorate our tests with metadata. It's main purpose is to add properties to fixtures and tests in the test output. It's an abstract class we can use to create our own metadata, and there's a few built-in like DescriptionAttribute. There are uses of properties at runtime too, but that's way too advanced for this post.

The DescriptionAttribute is just what we want for the free text part of our documentation. It's added to the method just like the Test attribute:

[Test]
[Description(@"
Events are provided at eventsite with a REST api at the URL:
https://eventsite/api/{id}
")]
public void It_Is_Fetched_By_Id_From_The_Event_Server()
{
    // ...
}

[Test]
[Description("The JSON response from the event server is deserialized as the Event type.")]
public void It_Is_Converted_To_An_Event_Model()
{
    // ...
}

The Description is written to the XML output like so:

<test-case id="0-1002" name="It_Is_Converted_To_An_Event_Model" ...>
    <properties>
        <property name="Description" value="The JSON response from the event server is deserialized as the Event type." />
    </properties>
</test-case>

By adding a few more voodoo lines to the XSLT and another one-liner JavaScript, we get the following output:
(Don't worry, complete XSLT is available at the bottom fo the post.)

Now we're really getting somewhere. We can provide that little extra that tells the reader what's really happening and stuff that might be obscured by too many, or hopefully just the right amount of abstractions.

Let's tackle the ordering problem while we're at it.

Implementing our own properties

The PropertyAttribute isn't much more than a metadata container. An ordering property is as simple as such:

public class DocumentationOrderAttribute : PropertyAttribute
{
    public DocumentationOrderAttribute(int order)
        : base("DocumentationOrder", order)
    {
    }
}

It's applied to the tests just like the descriptions:

[Test]
[DocumentationOrder(0)]
[Description(@"
Events are provided at eventsite with a REST api at the URL:
https://eventsite/api/{id}
")]
public void It_Is_Fetched_By_Id_From_The_Event_Server()
{
    // ...
}

And it's output just like descriptions:

<test-case id="0-1002" name="It_Is_Converted_To_An_Event_Model" ...>
    <properties>
        <property name="DocumentationOrder" value="1" />
        <property name="Description" value="The JSON response from the event server is deserialized as the Event type." />
    </properties>
</test-case>

Now we can tackle the ordering problem with a simple XSLT sort element:

<xsl:for-each select="./test-case">
  <xsl:sort select="./properties/property[@name='DocumentationOrder']/@value"/>
  <xsl:apply-templates select="."/>
</xsl:for-each>

And we get a nice ordered output:

Summary

By structuring and naming our tests by feature we get nice headings for potential documentation. We also untie them completely from implementation so we're free to change our code. Applying a bit of metadata to our tests adds that little extra to make quite meaningful documentation.

The fact that the results can be tranformed means we can create rich documentation UIs with search, tags, navigation structure and a bit less than just enough prose.

Next steps

We've still got that rendering test to pass. There's also a few too many asserts in one of the tests. In an upcoming post I'll share a couple of other tools with you and how to extend those to enrich the technical documentation even more.

Complete XSLT

Here's the complete XSLT for the examples above:

<?xml version="1.0" encoding="iso-8859-1"?>
<xsl:stylesheet
    version="1.0"
    exclude-result-prefixes="msxsl"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:msxsl="urn:schemas-microsoft-com:xslt"
>
  <xsl:output method="html" indent="yes"/>

  <xsl:template match="@* | node()">
    <html>
      <head>
        <!-- Latest compiled and minified CSS -->
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous"/>
        <!-- Optional theme -->
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous"/>
        <style type="text/css">
          .Passed { color: green; }
          .Inconclusive { color: #BBAA00; }
          .Failed { color: red; }

          ul {
          margin-left: 0px;
          list-style-type: none;
          padding-left: 0;
          }

          ul ul {
          margin-left: 15px;
          }

          label {
          font-weight: normal;
          }

          .counts {
          font-size: .7em;
          color: gray;
          }
        </style>

      </head>
      <body>
        <div class="container">
          <xsl:apply-templates select="/test-run/test-suite"/>
        </div>
        <!-- Latest compiled and minified JavaScript -->
        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous">// Force closing tag</script>
        <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js">// Force closing tag</script>
        <script type="text/javascript">
          $("td label").each(function(i, e) {
          $(e).text($(e).text().replace(/_/g, " "));
          });
          $(".description").each(function(i, e) {
          $(e).html($(e).html().trim().replace(/\n/g, '<br/>'));
          });
        </script>
      </body>
    </html>
  </xsl:template>

  <xsl:template match="/test-run/test-suite">
    <h1>
      <xsl:value-of select="./test-run/@name"/>
    </h1>
    <table class="table table-striped">
      <xsl:apply-templates select="./test-suite"/>
    </table>
  </xsl:template>

  <xsl:template match="test-suite">
    <tr>
      <td>
        <xsl:attribute name="class">
          <xsl:choose>
            <xsl:when test="./@failed > 0">Failed</xsl:when>
            <xsl:when test="./@inconclusive > 0">Inconclusive</xsl:when>
            <xsl:otherwise>Passed</xsl:otherwise>
          </xsl:choose>
        </xsl:attribute>
        <xsl:attribute name="style">
          padding-left: <xsl:value-of select="count(ancestor::test-suite)*15"/>px;
        </xsl:attribute>
        <xsl:value-of select="./@name"/>
      </td>
      <td class="counts">
        <xsl:value-of select="./@passed"/> passed,
        <xsl:value-of select="./@inconclusive"/> inconclusive,
        <xsl:value-of select="./@failed"/> failed
      </td>
    </tr>
    <xsl:for-each select="./test-suite">
      <xsl:apply-templates select="."/>
    </xsl:for-each>
    <xsl:for-each select="./test-case">
      <xsl:sort select="./properties/property[@name='DocumentationOrder']/@value"/>
      <xsl:apply-templates select="."/>
    </xsl:for-each>
  </xsl:template>

  <xsl:template match="test-case">
    <tr>
      <td colspan="2">
        <xsl:attribute name="style">
          padding-left: <xsl:value-of select="count(ancestor::test-suite)*15"/>px;
        </xsl:attribute>
        <label>
          <xsl:attribute name="class">
            <xsl:value-of select="./@result"/>
          </xsl:attribute>
          <xsl:value-of select="./@name"/>
        </label>
        <xsl:if test="./properties/property[@name='Description']">
          <div class="description">
            <xsl:value-of select="./properties/property[@name='Description']/@value"/>
            <xsl:text> </xsl:text>
          </div>
        </xsl:if>

      </td>
    </tr>

  </xsl:template>
</xsl:stylesheet>