Lars-Erik's blog

More efficient integration tests with Umbraco

Tue, 25 Nov 2025 07:11:08 Z

TLDR;

I made a package that removes the "database and Umbraco instance" per fixture constraint from Umbraco's Integration Test library. It also enables reuse of seeded database snapshots to avoid executing the same setup / seeding over an over per fixture or test.

An itch

I've spent way too long waiting for similar setup code for my integration tests the last couple of years. The modern .NET version of Umbraco's integration test library is super nice, and real effective with SQLite. But it's got a few constraints:

All things Umbraco is instantiated and spun up per test, granted a configurable amount.
Databases have to be "installed" and seeded for each TestFixture at the least.
All test fixtures need to inherit from Umbraco's base classes

So we spend a long time waiting for suff, even if it's only the little green box we care about:

We're also stuck either juggling a singleton of some sorts and/or stuffing a lot of "units" into the same fixture.
Any notion of intermediate base classes quickly become a sore spot.

An idea

I had recently discovered, likely for the n-th time in my life, that NUnit supports namespace scoped setup fixtures. Those live as long as all that other tests in that namespace and deeper. They bring an opportunity to share initial state across several fixtures. I often find it nice to have several fixtures for the same systems under tests because it makes sense to group by use-case rather than tech.

So I set out to see if I could hack together something that made Umbraco's base tests possible to use as setup fixtures rather than "base" fixtures. As things would have it it was early october, and the CanCon gang hosted a virtual hackathon on a friday. I took the day "off" regular work and gave it a go.

Dark magic

In the end I've used all the dirty tricks you can imagine in today's .NET landscape. You'll find Lib.Harmony for messing with Umbraco's tests' IL (Intermediate Language), and there's Castle DynamicProxy doing a lot of fooling around with the NUnit test hierarchy. The Harmony bit can likely be removed by making a few slight changes to Umbraco's test core, but I wanted to get this working satisfactory before suggesting such changes.

The bottom line is that those two tools let us run code before, after or even instead of inherited code. And that in turn enables all the following features.

A new way of composing tests

With the package I've cooked together we can leave the dark world of peach and purple above in favor of a way greener scenery like such:

So without further ado, let's dig into how you can get there too.

Umbraco's attributes

We rely on Umbraco's Tests:Database config and the [UmbracoTest] attribute to provide databases (or not). The Database option works almost as expected, but the four non-none options end up doing the same: they prepare one db for the lifetime of the setup fixture.

[ExtendableSetUpFixture]

The first attribute you have to know is [ExtendableSetUpFixture]. It's sole purpose is to enable the rest of them. Since it's inherited from [SetUpFixture], it tells NUnit we won't add any [SetUp], [TearDown] or [Test] methods in the marked class. But this is the attribute we'll add to any UmbracoIntegrationTest or UmbracoTestServerTestBase, and those have a bunch of each. That's why we have...

[MakeOneTimeLifecycle]

The first attribute in the package that does something. [MakeOneTimeLifecycle] lets you mark methods in otherwise not accessible base classes like UmbracoTestBase to become [OneTimeSetUp] methods rather than [SetUp] methods, and [TearDown] methods becomes [OneTimeTearDown].
Here's an example of what we'd feed it for an UmbracoIntegrationTest inheritor:

[UmbracoTest(Database=SchemaPerFixture, Boot=true)]
[ExtendableSetUpFixture]
[MakeOneTimeLifecycle(
    setUpNames: [nameof(UmbracoIntegrationTest.Setup), nameof(UmbracoIntegrationTest.SetUp_Logging)],
    tearDownNames: [
        nameof(UmbracoIntegrationTest.TearDown), nameof(UmbracoIntegrationTest.TearDownAsync), 
        nameof(UmbracoIntegrationTest.FixtureTearDown), nameof(UmbracoIntegrationTest.TearDown_Logging)
    ]
)]
public class MyLongLivedUmbracoSetUp : UmbracoIntegrationTest
{
}

I found it a bother to write that long attribute all the time, so there's a derived one called:

[OneTimeUmbracoSetUp]

It makes it slightly less messy by letting us write as follows:

[UmbracoTest(Database=SchemaPerFixture, Boot=true)]
[ExtendableSetUpFixture]
[OneTimeUmbracoSetUp]
public class MyLongLivedUmbracoSetUp : UmbracoIntegrationTest
{
}

Note however that the current version doesn't come with a 100% compatible one for UmbracoTestServerTestBase or ManagementApiTest<>, but it's easy to make one yourself, and they're likely suddenly in the package.

Now if you're into hacking around the environment and hidden, protected members lurking around in memory, we're quite alike. But when doing professional work that's often not what we're paid for. So even though the above code will start up a perfectly fine Umbraco instance with a SQLite database it doesn't do much good. We need to get at the IServiceProvider somehow...

[InjectionProvider(string)] and [ServiceProvider]

There is a limit to how many attributes we'll stick on our setup fixtures, I promise. I complained about intermediate base classes earlier, but a few for attribute consolidation is cool enough. OK, these two are a basic re-usable one and a "defaulted" one. The [InjectionProvider(string)] attribute stores a reference to a property on the setup fixture that exposes an IServiceProvider instance. Umbraco's base classes exposea property called Services, and [ServiceProvider] is just a derived [InjectionProvider(nameof(Services))]. To be specific, the service provider reference goes into the test context property bag as a factory method.
By adding one to our growing tower we're ready to meet our first actual test fixture enjoying the power of:

[Inject(string)]

To enjoy the benefits of our stored service provider reference, we must expose an instance method on our "scoped" test fixture that accepts services. We could've used constructor injection, but that's already "in use" by NUnit [TestFixture(args)] functionality. Once you start composing tests like this package allows, you can get much more value from things like that. So a void method will have to do. A fresh "standalone" and "scoped" test fixture will look like this:

using Same.Namespace.As.SetupFixture;

[Inject(nameof(Inject))]
public class LookMaImFreeAsABird
{
    IDataTypeService dataTypeService = null!;

    public void Inject(IDataTypeService dataTypeService)
    {
        this.dataTypeService = dataTypeService;
    }

    [Test]
    public void FiddleWithDataTypeA()
    {
        // ...
    }

    [Test]
    public void CreateDataTypeB()
    {
        // ...
    }
}

Notice the beauty of no forced ctors, no base class and no overrides in that class! Not to mention, you can have several of those!

Interlude: Transactions

With the above setup the individual fixtures are free to create ICoreScope instances from Umbraco and commit or dispose them as you wish. Just writing three sentences about it here doesn't really give the method justice. Suffice to say, just give it a go!
However it won't work when we use UmbracoTestServerTestBase. There's no way to spread a core scope across threads, and there's not even a way to spread gool old .NET transaction scopes across. I guess you could go full old school and bring DCOM into the picture, but after a lot of failed attempts I finally stumbled over an obvious solution.

Snapshots

The final itch has taken me the longest, and even longer to stuff into an attribute rather than yet another base class. I must admit it took a desperate (but accurate) final plea with ChatGPT 5.1 to see the obvious solution just lying there to implement. All database engines, or at least the two I've implemented, support some sort of fast backup. VACUUM INTO in SQLite and CREATE/RESTORE DATABASE AS SNAPSHOT in SQL Server.

I had two intermediate base classes for a few weeks, but this last weekend I ran over the goal line:

`[ReusableDatabase(type, string)]`

If you configure Tests:Database:DatabaseType in your appSettings.Tests.Local.json and set any of the non-none values in the [UmbracoTest] database parameter, you can also add [ReusableDatabase(typeof(SeedClass), nameof(SeedClass.Configure))] to get a new type of ITestDatabase.

As of writing it requires a method to be implemented in the setup fixture, or on some supporting type. Its mission is to configure whether the database needs a fresh seed and how to seed the initial data for all the tests. Only if we say so, the database is installed and re-seeded. For all other scopes it's just restored from that initial snapshot.

The seeding can be done in any way you please. My favorite is importing things using uSync, and that might just become another blog post. For simplicity let's say we're testing variants of property editors based on a datatype we want all tests to start with:

[UmbracoTest(
    Database = UmbracoTestOptions.Database.NewSchemaPerTest,
    Boot = true,
    Logger = UmbracoTestOptions.Logger.Console
)]
[ExtendableSetUpFixture]
[OneTimeUmbracoSetUp]
[ServiceProvider]
[ReusableDatabase(nameof(ConfigureSeeding))]
public class ReusedDbAttributeSetUp : UmbracoIntegrationTest
{
    public static void ConfigureSeeding(ReusableTestDatabaseOptions options)
    {
        options.NeedsNewSeed = _ => Task.FromResult(true);
        options.SeedData = async (services) =>
        {
            await TestContext.Progress.WriteLineAsync("Creating datatype");
            await services.GetRequiredService<IDataTypeService>().CreateAsync(
                new DataType
                (
                    new TextboxPropertyEditor
                    (
                        services.GetRequiredService<IDataValueEditorFactory>(),
                        services.GetRequiredService<IIOHelper>()
                    ),
                    services.GetRequiredService<IConfigurationEditorJsonSerializer>()
                )
                {
                    Name = "A seeded textbox"
                },
                Umbraco.Cms.Core.Constants.Security.SuperUserKey
            );
        };
    }
}

The method has to be static void. If it's on the setup fixture you can omit the type from the attribute parameters.
Now all tests within this "scope" will have access to the seeded textbox. Such seeding code might quickly grow out of hand though, and that's why I prefer just importing the content schema from the web project in the same repo as my tests.

Which ones to choose

As you've hopefully realized by now you can introduce really slim intermediate base classes that set up the initial environment for a bunch of use-cases and scenarios. Leaving your actual test fixtures free to be composed just as you wish, and with all the features of NUnit at your disposal.

I'm sure you can see that the last setup fixture example above makes for a nice base class to avoid repeating that tower of attributes. The repository has a few examples using a few or all of the attributes.

The final trick I haven't disclosed above is how to do a "mid-fixture" rollback. The reusable database implementations have a "RestoreSnapshot" method, so in a [TearDown] you can go:

await serviceProvider.GetRequiredService<IReusableDatabase>().RestoreSnapshot();

That makes it so that whatever test comes next it's unaffected by the changes you've made. This is of course what happens during [OneTimeSetUp] for any setup fixture using [ReusableDatabase].

If you want to try this out, I recommend going with [ReusableDatabase] (and uSync) for everything that use more than 10ish artifacts like data and content types, not to mention content. Whenever you want to run changes through the Management API, or definitely if you want to test how all your sites' elements look in the Delivery API. It's a no-brainer when you go full integration with UmbracoTestServerTestBase and more than one HTTP call.

If all your tests have all they need in a fresh Umbraco database, but they all mutate it so you need cleanup, it'll be blazing fast if you can use ICoreScope with autocommit off. As long as you don't need to test via HTTP endpoints, this is a better option than restoring the snapshot. Using a snapshot restore between fixtures however is still a nice option to keep around, so I still opt for [ReusableDatabase].

Want more?

I'm definitely going to be in "only fix what breaks for me" mode for a little while, as this was just gonna be a hacktoberfest weekend project. Turned out to take too much of my spare time for one and a half month instead.

But I honestly believe this'll save some trees if applied well to CI pipelines running all day long.
And I've gone from 90 sec to 26 sec to execute a full test suite, a lot of which would run for any filter.

The package is currently listed as a beta (pre-release) on nuget and if it doesn't crash much I might just promote it to full visibility. It's a bit furry on the edges and has a bit of legacy to it (already), but if you stick the namespaces in the project file and leave the stray base classes alone I think you'll be good.

All this is to say I'd love for conversation about what it solves and if it could do it better. I'd love even more to throw out a bit of the code because Umbraco suddenly fixed the core two issues. Let me hear from you if you try it out. Feel free to clone the code and mess about, although I reserve the right to refactor the mess myself that beautiful day I get to keep going with this.

Looking forward to hearing how (and if) you like it. Hope you have a go!

Links

Environmental ApprovalTests

Sun, 03 Jun 2018 21:24:46 Z

Background

A while ago I wrote a post about a tool called ApprovalTests. I've included it in my workshop on unit testing Umbraco, and people are amazed at its usefulness. Having Visual Studio pop a diff in your face when stuff breaks is a real timesaver. However, when I ran the workshop at CodeGarden 18, I realized people were concerned that their tests would be impossible to run in CI environments and the like. Not to worry - ApprovalTests have you covered. (Pun intended)

Environmentalism

When declaring reporters with ApprovalTests, you can specify multiple reporters. That's not all. The reporters have an extensive API on them, which caters for us tailoring everything. There is one particularly interesting interface on all "diff-reporters" named IEnvironmentAwareReporter.
The VisualStudioReporter one has a nice little one-liner implementation:

public override bool IsWorkingInThisEnvironment(string forFile)
{
    return OsUtils.IsWindowsOs() && base.IsWorkingInThisEnvironment(forFile) && LaunchedFromVisualStudio();
}

The last function call there checks if the current process is a descendent of devenv.exe. If it isn't, execution will just be passed to the next reporter in the chain. So there! It won't break your CI build.

But what do we want in our CI build, then? Probably a regular WhateverUnit assertion. I use NUnit, so that'll be our example.

Falling back

Take this little test. I approved the result for seed 1, and we'll examine the output for an invalid result by swapping to 2:

[TestFixture]
[UseReporter(typeof(VisualStudioReporter))]
public class When_Running_In_Different_Environments
{
    [Test]
    public void Delegates_To_Most_Appropriate_Reporter()
    {
        var rnd = new Random(1);
        var items = Enumerable
            .Range(0, 10)
            .Select(x => rnd.Next());
        Approvals.VerifyAll(items, "");
    }
}

Let's change the seed to 2 to fail the test:

var rnd = new Random(2);

Now if we run the test with NUnit3-Console.exe, we'll get an exception saying that ApprovalTests can't find Visual Studio:

nunit3-console.exe .\bin\debug\environmentaltests.dll

...
1) Error : EnvironmentalTests.When_Running_In_Different_Environments.Delegates_To_Most_Appropriate_Reporter
System.Exception : Couldn't find Visual Studio at
at ApprovalTests.Reporters.GenericDiffReporter.Report(String approved, String received) in C:\code\ApprovalTests.Net\ApprovalTests\Reporters\GenericDiffReporter.cs:line 142
...

It sounds worse than it is. ApprovalTests insists that it has something to do. We can add a reporter to the fixture to fix it:

[TestFixture]
[UseReporter(typeof(VisualStudioReporter), typeof(NUnitReporter))]
public class When_Running_In_Different_Environments

Now when we run the test, we get a pure assertion failure:

1) Failed : EnvironmentalTests.When_Running_In_Different_Environments.Delegates_To_Most_Appropriate_Reporter
Expected string length 165 but was 162. Strings differ at index 6.
Expected: "[0] = 534011718\n[1] = 237820880\n[2] = 1002897798\n[3] = 165700..."
But was:  "[0] = 1655911537\n[1] = 867932563\n[2] = 356479430\n[3] = 211537..."
-----------------^

Notice it didn't even mention Visual Studio. Going back into our favorite IDE will start popping diffs again, and the NUnit one will govern the output.

Cleaning up

So should we go around declaring at least two reporters on all our fixtures, then? Luckily not. There are two more tricks that are nice to know.

First, the reporter attribute is found by iterating up the inheritance hierarchy. It can be defined as high up as all your [assembly:XAttribute] metadata. You can create a file in your test project root called ApprovalsConfig.cs for instance. Within it, you declare your reporters on the assembly level:

using ApprovalTests.Reporters;

[assembly:UseReporter(typeof(VisualStudioReporter), typeof(NUnitReporter))]

The second is that you might not want the NUnit assertion when you run in VS (for some reason), or maybe some other tool might be in your way. You might even want composite reporters. In any case, it also makes for a bit nicer setup if you implement the FirstWorkingReporter class:

public class EnvironmentReporter : FirstWorkingReporter
{
    public static readonly EnvironmentReporter INSTANCE = new EnvironmentReporter();

    public EnvironmentReporter()
        : base(
            VisualStudioReporter.INSTANCE,
            NUnitReporter.INSTANCE
        )
    {
    }
}

With it, we can change our UseReporter to be:

[assembly:UseReporter(typeof(EnvironmentReporter))]

Now the first reporter to confirm its environment is executed, and the rest are ignored. You might want to keep the NUnit one around, though. In that case, you can implement MultiReporter in the same way - which coincidentally is the same that UseReporterAttribute does.

I encourage you to go have a look at all the built-in reporters and get some inspiration for even more helpful reporting.

I'd rejoice for any cool usages in the discussion thread. :)