Lars-Erik's blog

More efficient integration tests with Umbraco

Tue, 25 Nov 2025 07:11:08 Z

TLDR;

I made a package that removes the "database and Umbraco instance" per fixture constraint from Umbraco's Integration Test library. It also enables reuse of seeded database snapshots to avoid executing the same setup / seeding over an over per fixture or test.

An itch

I've spent way too long waiting for similar setup code for my integration tests the last couple of years. The modern .NET version of Umbraco's integration test library is super nice, and real effective with SQLite. But it's got a few constraints:

All things Umbraco is instantiated and spun up per test, granted a configurable amount.
Databases have to be "installed" and seeded for each TestFixture at the least.
All test fixtures need to inherit from Umbraco's base classes

So we spend a long time waiting for suff, even if it's only the little green box we care about:

We're also stuck either juggling a singleton of some sorts and/or stuffing a lot of "units" into the same fixture.
Any notion of intermediate base classes quickly become a sore spot.

An idea

I had recently discovered, likely for the n-th time in my life, that NUnit supports namespace scoped setup fixtures. Those live as long as all that other tests in that namespace and deeper. They bring an opportunity to share initial state across several fixtures. I often find it nice to have several fixtures for the same systems under tests because it makes sense to group by use-case rather than tech.

So I set out to see if I could hack together something that made Umbraco's base tests possible to use as setup fixtures rather than "base" fixtures. As things would have it it was early october, and the CanCon gang hosted a virtual hackathon on a friday. I took the day "off" regular work and gave it a go.

Dark magic

In the end I've used all the dirty tricks you can imagine in today's .NET landscape. You'll find Lib.Harmony for messing with Umbraco's tests' IL (Intermediate Language), and there's Castle DynamicProxy doing a lot of fooling around with the NUnit test hierarchy. The Harmony bit can likely be removed by making a few slight changes to Umbraco's test core, but I wanted to get this working satisfactory before suggesting such changes.

The bottom line is that those two tools let us run code before, after or even instead of inherited code. And that in turn enables all the following features.

A new way of composing tests

With the package I've cooked together we can leave the dark world of peach and purple above in favor of a way greener scenery like such:

So without further ado, let's dig into how you can get there too.

Umbraco's attributes

We rely on Umbraco's Tests:Database config and the [UmbracoTest] attribute to provide databases (or not). The Database option works almost as expected, but the four non-none options end up doing the same: they prepare one db for the lifetime of the setup fixture.

[ExtendableSetUpFixture]

The first attribute you have to know is [ExtendableSetUpFixture]. It's sole purpose is to enable the rest of them. Since it's inherited from [SetUpFixture], it tells NUnit we won't add any [SetUp], [TearDown] or [Test] methods in the marked class. But this is the attribute we'll add to any UmbracoIntegrationTest or UmbracoTestServerTestBase, and those have a bunch of each. That's why we have...

[MakeOneTimeLifecycle]

The first attribute in the package that does something. [MakeOneTimeLifecycle] lets you mark methods in otherwise not accessible base classes like UmbracoTestBase to become [OneTimeSetUp] methods rather than [SetUp] methods, and [TearDown] methods becomes [OneTimeTearDown].
Here's an example of what we'd feed it for an UmbracoIntegrationTest inheritor:

[UmbracoTest(Database=SchemaPerFixture, Boot=true)]
[ExtendableSetUpFixture]
[MakeOneTimeLifecycle(
    setUpNames: [nameof(UmbracoIntegrationTest.Setup), nameof(UmbracoIntegrationTest.SetUp_Logging)],
    tearDownNames: [
        nameof(UmbracoIntegrationTest.TearDown), nameof(UmbracoIntegrationTest.TearDownAsync), 
        nameof(UmbracoIntegrationTest.FixtureTearDown), nameof(UmbracoIntegrationTest.TearDown_Logging)
    ]
)]
public class MyLongLivedUmbracoSetUp : UmbracoIntegrationTest
{
}

I found it a bother to write that long attribute all the time, so there's a derived one called:

[OneTimeUmbracoSetUp]

It makes it slightly less messy by letting us write as follows:

[UmbracoTest(Database=SchemaPerFixture, Boot=true)]
[ExtendableSetUpFixture]
[OneTimeUmbracoSetUp]
public class MyLongLivedUmbracoSetUp : UmbracoIntegrationTest
{
}

Note however that the current version doesn't come with a 100% compatible one for UmbracoTestServerTestBase or ManagementApiTest<>, but it's easy to make one yourself, and they're likely suddenly in the package.

Now if you're into hacking around the environment and hidden, protected members lurking around in memory, we're quite alike. But when doing professional work that's often not what we're paid for. So even though the above code will start up a perfectly fine Umbraco instance with a SQLite database it doesn't do much good. We need to get at the IServiceProvider somehow...

[InjectionProvider(string)] and [ServiceProvider]

There is a limit to how many attributes we'll stick on our setup fixtures, I promise. I complained about intermediate base classes earlier, but a few for attribute consolidation is cool enough. OK, these two are a basic re-usable one and a "defaulted" one. The [InjectionProvider(string)] attribute stores a reference to a property on the setup fixture that exposes an IServiceProvider instance. Umbraco's base classes exposea property called Services, and [ServiceProvider] is just a derived [InjectionProvider(nameof(Services))]. To be specific, the service provider reference goes into the test context property bag as a factory method.
By adding one to our growing tower we're ready to meet our first actual test fixture enjoying the power of:

[Inject(string)]

To enjoy the benefits of our stored service provider reference, we must expose an instance method on our "scoped" test fixture that accepts services. We could've used constructor injection, but that's already "in use" by NUnit [TestFixture(args)] functionality. Once you start composing tests like this package allows, you can get much more value from things like that. So a void method will have to do. A fresh "standalone" and "scoped" test fixture will look like this:

using Same.Namespace.As.SetupFixture;

[Inject(nameof(Inject))]
public class LookMaImFreeAsABird
{
    IDataTypeService dataTypeService = null!;

    public void Inject(IDataTypeService dataTypeService)
    {
        this.dataTypeService = dataTypeService;
    }

    [Test]
    public void FiddleWithDataTypeA()
    {
        // ...
    }

    [Test]
    public void CreateDataTypeB()
    {
        // ...
    }
}

Notice the beauty of no forced ctors, no base class and no overrides in that class! Not to mention, you can have several of those!

Interlude: Transactions

With the above setup the individual fixtures are free to create ICoreScope instances from Umbraco and commit or dispose them as you wish. Just writing three sentences about it here doesn't really give the method justice. Suffice to say, just give it a go!
However it won't work when we use UmbracoTestServerTestBase. There's no way to spread a core scope across threads, and there's not even a way to spread gool old .NET transaction scopes across. I guess you could go full old school and bring DCOM into the picture, but after a lot of failed attempts I finally stumbled over an obvious solution.

Snapshots

The final itch has taken me the longest, and even longer to stuff into an attribute rather than yet another base class. I must admit it took a desperate (but accurate) final plea with ChatGPT 5.1 to see the obvious solution just lying there to implement. All database engines, or at least the two I've implemented, support some sort of fast backup. VACUUM INTO in SQLite and CREATE/RESTORE DATABASE AS SNAPSHOT in SQL Server.

I had two intermediate base classes for a few weeks, but this last weekend I ran over the goal line:

`[ReusableDatabase(type, string)]`

If you configure Tests:Database:DatabaseType in your appSettings.Tests.Local.json and set any of the non-none values in the [UmbracoTest] database parameter, you can also add [ReusableDatabase(typeof(SeedClass), nameof(SeedClass.Configure))] to get a new type of ITestDatabase.

As of writing it requires a method to be implemented in the setup fixture, or on some supporting type. Its mission is to configure whether the database needs a fresh seed and how to seed the initial data for all the tests. Only if we say so, the database is installed and re-seeded. For all other scopes it's just restored from that initial snapshot.

The seeding can be done in any way you please. My favorite is importing things using uSync, and that might just become another blog post. For simplicity let's say we're testing variants of property editors based on a datatype we want all tests to start with:

[UmbracoTest(
    Database = UmbracoTestOptions.Database.NewSchemaPerTest,
    Boot = true,
    Logger = UmbracoTestOptions.Logger.Console
)]
[ExtendableSetUpFixture]
[OneTimeUmbracoSetUp]
[ServiceProvider]
[ReusableDatabase(nameof(ConfigureSeeding))]
public class ReusedDbAttributeSetUp : UmbracoIntegrationTest
{
    public static void ConfigureSeeding(ReusableTestDatabaseOptions options)
    {
        options.NeedsNewSeed = _ => Task.FromResult(true);
        options.SeedData = async (services) =>
        {
            await TestContext.Progress.WriteLineAsync("Creating datatype");
            await services.GetRequiredService<IDataTypeService>().CreateAsync(
                new DataType
                (
                    new TextboxPropertyEditor
                    (
                        services.GetRequiredService<IDataValueEditorFactory>(),
                        services.GetRequiredService<IIOHelper>()
                    ),
                    services.GetRequiredService<IConfigurationEditorJsonSerializer>()
                )
                {
                    Name = "A seeded textbox"
                },
                Umbraco.Cms.Core.Constants.Security.SuperUserKey
            );
        };
    }
}

The method has to be static void. If it's on the setup fixture you can omit the type from the attribute parameters.
Now all tests within this "scope" will have access to the seeded textbox. Such seeding code might quickly grow out of hand though, and that's why I prefer just importing the content schema from the web project in the same repo as my tests.

Which ones to choose

As you've hopefully realized by now you can introduce really slim intermediate base classes that set up the initial environment for a bunch of use-cases and scenarios. Leaving your actual test fixtures free to be composed just as you wish, and with all the features of NUnit at your disposal.

I'm sure you can see that the last setup fixture example above makes for a nice base class to avoid repeating that tower of attributes. The repository has a few examples using a few or all of the attributes.

The final trick I haven't disclosed above is how to do a "mid-fixture" rollback. The reusable database implementations have a "RestoreSnapshot" method, so in a [TearDown] you can go:

await serviceProvider.GetRequiredService<IReusableDatabase>().RestoreSnapshot();

That makes it so that whatever test comes next it's unaffected by the changes you've made. This is of course what happens during [OneTimeSetUp] for any setup fixture using [ReusableDatabase].

If you want to try this out, I recommend going with [ReusableDatabase] (and uSync) for everything that use more than 10ish artifacts like data and content types, not to mention content. Whenever you want to run changes through the Management API, or definitely if you want to test how all your sites' elements look in the Delivery API. It's a no-brainer when you go full integration with UmbracoTestServerTestBase and more than one HTTP call.

If all your tests have all they need in a fresh Umbraco database, but they all mutate it so you need cleanup, it'll be blazing fast if you can use ICoreScope with autocommit off. As long as you don't need to test via HTTP endpoints, this is a better option than restoring the snapshot. Using a snapshot restore between fixtures however is still a nice option to keep around, so I still opt for [ReusableDatabase].

Want more?

I'm definitely going to be in "only fix what breaks for me" mode for a little while, as this was just gonna be a hacktoberfest weekend project. Turned out to take too much of my spare time for one and a half month instead.

But I honestly believe this'll save some trees if applied well to CI pipelines running all day long.
And I've gone from 90 sec to 26 sec to execute a full test suite, a lot of which would run for any filter.

The package is currently listed as a beta (pre-release) on nuget and if it doesn't crash much I might just promote it to full visibility. It's a bit furry on the edges and has a bit of legacy to it (already), but if you stick the namespaces in the project file and leave the stray base classes alone I think you'll be good.

All this is to say I'd love for conversation about what it solves and if it could do it better. I'd love even more to throw out a bit of the code because Umbraco suddenly fixed the core two issues. Let me hear from you if you try it out. Feel free to clone the code and mess about, although I reserve the right to refactor the mess myself that beautiful day I get to keep going with this.

Looking forward to hearing how (and if) you like it. Hope you have a go!

Links

Migrating Umbraco Forms from 7 to modern

Wed, 28 Feb 2024 11:58:06 Z

No migration, no cry

There is no uSync Migrations extension for moving forms from 7 to modern. There isn't even a uSync Forms for V7. However the serialization format for forms on disk didn't change (much?) from 7 to 8, so we can do a trick. (Thanks to Kevin for the tip)
We can do 7 to 8, then 8 to modern.
This applies to files. I have not tried with the forms in database method, but that should be possible to do with database migrations from 7 to 8.

7 to 8

Create an empty V8 site and install the following nuget packages:

UmbracoForms
uSync.Forms

To "migrate" your forms from 7 to 8 you just copy the files over. 😁
Your source files should be in /App_Data/UmbracoForms/Data.
The target should be the same in a V8 site.

When your files are copied you're ready to go into the backoffice.
Go to Settings \ uSync and click Export.

You should now have uSync files for the forms under /uSync/v8/Forms.

8 to modern

Install uSync.Forms in your modern site.
Copy your uSync forms files from the V8 site into /uSync/v9/Forms.

The uSync files will have prevalues in a parsedPreValues field instead of preValues.
uSync Forms import in modern does not like this.
So I made a small powershell script that does the necessary "migration":

param(
    $path
)

$files = Get-ChildItem -Path $path -File

$files | % {
    $fileInfo = $_
    $doc = [xml](Get-Content $fileInfo.FullName)
    $pageNode = $doc.DocumentElement.SelectSingleNode("Pages")
    $origCData = $pageNode.FirstChild
    $pages = [array]($origCData.InnerText | ConvertFrom-Json)

    $results = $pages | % {
        $page = $_
        $page.fieldSets | % {
            $fieldSet = $_
            $fieldSet.containers | % {
                $container = $_
                $container.fields | % {
                    $field = $_
                    if ($field.parsedPrevalues.Length -gt 0) {
                        $field.preValues = $field.parsedPrevalues
                    }
                    $field.psobject.Properties.Remove("parsedPreValues")
                }
            }
        }
    }

    $cdata = $doc.CreateCDataSection((ConvertTo-Json -depth 99 -inputobject $pages))
    $replaced = $pageNode.ReplaceChild($cdata, $origCData)

    $doc.Save($fileInfo.FullName)
}

I called the script fix-usync-forms.ps1 and we can call it as such:

./fix-usync-forms.ps1 [path-to-site]/uSync/v9/Forms

That's it!

We can now go into the backoffice in our modern site.
Go to Settings \ uSync and click "Import" under Forms.

Success

Now all that's left is porting your potential code and views.

Happy migrating! 🙃

Search and RSS for our Umbraco ContentList

Wed, 06 May 2020 13:30:33 Z

The ContentList package have gotten a few new improvements over the last couple of weeks. Here's a rundown on what's new since the initial release post.

Theme control

Often a theme is customized to work well with just one or a few of the data sources. This is especially relevant with the changes described later in this post. Also, a theme might not use bootstrap, and as such not have the same rigid column count structure. Themes have therefore got a new config file you can choose to add to a theme folder: list.json.

There are two options you can add to the list config file:

Compatible sources

To specify which data sources a theme goes well with, add a compatibleSources array:

{
    "compatibleSources": [
        "ListablesByLuceneDataSource",
        "ListableByNodeDataSource",
        "ListableChildrenDataSource"
    ] 
}

Disable columns setting

The FlexTheme example view for instance does its own flexbox styling with varying amounts of columns. For this theme, the columns setting with "mobile", "tablet" and "desktop" column counts make no sense.

You can now disable the columns setting as such:

{
    "disableColumnsSetting": true
}

Search

The ListableByLuceneDataSource have gotten a small brushup. The query setting have gotten its textbox expanded for better UX, and a new parameter called "Fulltext Querystring Parameter" have been added.

The latter will combine the "query" with a search in all fields for all terms in a phrase extracted from the querystring parameter specified in the setting.

There is a new sample template for search where a search box is included in the theme list view. It posts and controls the configured querystring parameter. You can of course also just make a form in your layout posting to a given page with a content list. Or even a grid editor.

Rss

I didn't want to bundle RSS with the core component, so there's a new package on the block called Our.Umbraco.ContentList.Rss. By installing it, you get another data source aptly called "Rss". It's also currently in beta, so remember to add the -pre option when installing.

It has one parameter for the URL to the RSS-feed, so I gather it doesn't need that much documentation.

By specifying some theme settings as described earlier, it should be fairly simple to control both RSS and search themes combined with whatever other creative lists you come up with.

Hope y'all enjoy and find this stuff useful! Don't be shy, come over to the github repo and post issues if you want to discuss, ask questions or (please do) contribute.

Our.Umbraco.WysiwygGrid 1.0.0 beta released

Thu, 23 Apr 2020 12:09:36 Z

A while ago I blogged about the state of WYSIWYG in the Umbraco grid editor. We're now about to add this script to a second and likely third Umbraco 8 site, so I had to generalize and package up the code.

There's nothing much to hide or earn from here, so I decided to put it up on github and nuget instead of our internal scm and feeds.

Hope to see some more community members fiddling with it, if nothing else that it inspires the block editor team to make the backoffice experience more similar to the frontend look and feel.

A definite change from the previous blogpost is that we favor grid settings over inline styling. By slightly modifying the bootstrap view (we use v4 anyway), we can add multiple classes instead of attributes as is done the original views. As such we can have settings like these:

Those will generate extra classes named padding-lots and theme-inverse.
Those classes are added to rows or cells in both the backoffice and the front-end.
By doing some fancy sass include ninjaing, we can hopefully limit the amount of duplication, while still mimicing the front-end as much as possible in the backoffice.

There are some samples in the github repo if you're unsure how to get started. The SQLCE database is included with a fairly common test username/password. 😇

Without further ado, here's the nuget and github links:

https://github.com/lars-erik/Our.Umbraco.WysiwygGrid
https://www.nuget.org/packages/Our.Umbraco.WysiwygGrid/1.0.0-beta01

Hope you enjoy it, and as usual, hope for a good discussion or feedback from the community!

The state of WYSIWYG grid editing in Umbraco - a prototype

Fri, 03 Apr 2020 13:26:14 Z

The state of the grid

Since the grid was launched at the CG talk "The sky is the limit" back in 2015 (i think), it has been a joy to use for editors, designers and developers alike. Granted, there's been the other feature that could've gotten more love (configuration and settings), but generally it does the job swell.
However, it's still just a white canvas with "settings applied".
Antoine made LePainter back in 2015, but likely didn't get much traction.
I was just told about it today by Callum. Shame on me for not watching the package space enough, how embarrasing. 😳

Anyway...

Why should we care

Recently, I've been in lots of discussions; near battles IMO; about which CMS to sell to our customers.
The feedback is that editors wonder why what they see while editing does not match what they see in the front-end. We do work with another CMS where the backoffice experience has a near 1:1 layout and design with the front-end. As developers, we tend to think structured data is the better solution. Think nested- and stacked content. I tend to agree. But in a sales meeting, those arguments are total losing arguments. A competing agency will bury you when they oversell the fancy editing experience. Even though the not-so-apparent technical features of the competing CMS are totally useless.
And I do agree - the editing experience is what Umbraco likes to sell as well!

What can we do?

The current HQ efforts are working on the "block based" editor. It's been going on for the better part of a year, but the publicly available information is mostly just an RFC with screenshots looking like stacked / nested content, and that's it. The technical aspects seem super, but there's nothing much (but text) about the general editing experience. Also, and I know this is a hot take, editors DO want to see three columns in the editor when it ends up three columns in the front-end.

Hopefully, as mentioned in the RFC, it can act as a stepping stone towards the "Grid 2.0" (which we started work on at the retreat two years ago).

But as far as I'm concerned, this could just as well end up never happening. The effort is not very visible, Project UniCore 🦄 has (rightly) high priority, and in the mean time we're stuck with Grid 1.0, or maybe, hopefully, a rumored commercial package called Bento by the eminent Peter Gregory and his colleagues at KØBEN Digital.

A prototype

So back using Grid 1.0 for projects to be delivered imminently, I've started doing some prototypes for something similar to what LePainter did for early v7 projects.

I asked the community if it was madness or worth pursuing, and the general feedback is that it's both. Hence this blog post to explore it some more, and hopefully gather some more feedback or at least provide feedback to the block editor team. (Hope they receive it. 😁)

Here's a quick little preview of what I've been doing so far:

The issues

Actually getting the styling into the grid is "just" a matter of including a few pagefuls of JavaScript and a limited and scoped CSS file in a package.manifest file for the site. Both classes and styles are applied as expected, and with some due dilligence it doesn't look half bad.

The biggest issue I have is that the cells and controls are styled white, and while styling one have to mind not messing up the GUI text like "Settings applied".

I attempted to lessen the opacity of the cells and controls, and it works to a degree. The minute one adds a background image however, the entire thing starts to crumble. How to keep the WYSIWYG experience while still seeing the GUI? One could fantasize about doing an average color sample of the image and toggling between white and black text, maybe? Could be done in a short hacking sprint...
Could just keep it fairly white like I've done in the prototype. Could keep just the active rows white. This is where I start struggling. 😳

To be frank, the biggest problem lies in the fact that the "Add content" button is in the grid. Other CMSes have drag drop functionality from a menu over to the side. It could possibly still just be white or slightly transparent. The last problem then is all those "settings applied" and empty cells with the other type of "Add content" button.

I can't help but think this is all within grasp, albeit just out of my limited back-ender reach. 😆

The settings dialog issues

The settings dialog is a chapter of itself. It's a mystery to me why we still have a concept of individual "prevalue editors" in Umbraco. Property editors have the same model, and can be sneaked in there by just providing a full path.

However, as experienced when working on our ContentList package, nesting property editors using built-in directives wreaks total havok on the styling. There's some grey background and drop shadows - that I haven't seen being actively used anywhere. It's generally just a mystery, and a huge candidate for backoffice re-work.

Also, for instance the slider property editor does look a bit weird when used as a setting editor. I do hope that those things will get an overhaul and are made more uniform sooner or later.

I'll just leave that hanging, since I know (and appreciate) there are several HQ and community efforts going on to clean things up.

The code

Since I managed to sneak in a few new events in the grid, making it work was a matter of hooking into the "grid.initialized" event and start watching the model. There might be some performance issues with large grids since it's a deep watch, but with todays computers that might be irrational over engineering to think as well.

It adds the configured styles as expected.
For the configuration, it just adds the value of the configuration setting called "class". If there's more it concats the setting key with the setting value, delimited by a dash.

You can scrutinize the JavaScript here, or in this gist.

angular.module("umbraco").run(["eventsService", function(eventsService) {
    function findModelScope(scope) {
        if (!scope) {
            return null;
        }
        if (scope.model) {
            return scope;
        }
        return findModelScope(scope.$parent);
    }

    function isProtectedClass(className) {
        return className === "umb-row-inner" ||
            className === "umb-cell-inner" ||
            className.substr(0, 2) === "ng-" ||
            className.substr(0, 2) === "ui-";
    }

    function addClasses(element, gridItem) {
        function classNameFromConfig(e) {
            if (e === "class") {
                return gridItem.config[e];
            } else {
                return e + "-" + gridItem.config[e];
            }
        }

        var classes = (element.className || "").split(/\s+/);
        var newClasses = classes.filter(isProtectedClass);
        var nameClass = (gridItem.name || "").toLowerCase().replace(" ", "-");
        var configClasses = Object.keys(gridItem.config || {}).map(classNameFromConfig);
        newClasses.push(nameClass);
        newClasses = newClasses.concat(configClasses);
        element.className = newClasses.join(" ");
    }

    function addStyles(element, gridItem) {
        function styleFromKeyPair(e) {
            return e + ":" + gridItem.styles[e];
        }

        var stylesValues = Object.keys(gridItem.styles || {}).map(styleFromKeyPair);
        element.style = stylesValues.join(";");
    }

    eventsService.on("grid.initialized",
        function(evt, data) {
            var modelScope = findModelScope(data.scope);
            var model = modelScope.model;
            var jqEl = data.element;
            var el = data.element.get(0);
            jqEl.addClass("stylized-grid");

            modelScope.$watch(
                "model",
                function () {
                    var areaElements = el.getElementsByClassName("umb-column");
                    if (areaElements.length === 0) {
                        return;
                    }
                    model.value.sections.forEach(function (area, ai) {
                        var rowElements = areaElements[ai].getElementsByClassName("umb-row-inner");
                        area.rows.forEach(function (row, ri) {
                            var rowElement = rowElements[ri];
                            addClasses(rowElement, row);
                            addStyles(rowElement, row);

                            var cellElements = rowElement.getElementsByClassName("umb-cell-inner");
                            row.areas.forEach(function(cell, ci) {
                                addClasses(cellElements[ci], cell);
                                addStyles(cellElements[ci], cell);
                            });
                        });
                    });
                },
                true
            );
        });

}]);

The CSS

To not interfere with any other backoffice styling, the script adds the class "stylized-grid" to the grid container. Here's the (S)CSS I used to make things more transparent:

.umb-grid.stylized-grid { 

    .umb-cell-content {
        background-color: rgba(255, 255, 255, .8);
    }

    .umb-cell-content:hover {
        background-color: rgba(255, 255, 255, 1);
    }

    .umb-cell-content.-has-editors {
        background-color: rgba(255, 255, 255, .8);
    }

    .umb-cell-content.-has-editors:hover {
        background-color: rgba(255, 255, 255, 1);
    }

    .umb-grid-add-more-content {
        background-color: rgba(255, 255, 255, 1);
    }

    .umb-control-bar, .umb-grid-has-config {
        font-family: Lato, Helvetica Neue, Helvetica, Arial, sans-serif !important;
    }

    iframe {
        background-color: transparent;
    }

    @import "../../../Assets/grid";
}

The import at the bottom is the sass that styles our individual components in the front-end, and that should be reflected in the backoffice.

Conclusion

We'll be diving into deep water and testing this out on a real customer. I'm a bit weary, but we need to get the experience.

I do hope I've been able to provide some inspiration, and that WYSIWYG editing in Umbraco might get the renaissance (I believe) it deserves.

Feel free to drop your comments below or ping me on Twitter or Slack!
Also happy to continue the discussion on our or github, but not sure the best way forward, if any.

Our Umbraco Community ContentList 1.0 (beta) released

Wed, 01 Apr 2020 21:57:27 Z

ContentList is a grid editor we've used internally at MarkedsPartner since 2016. It helps editors and designers insert lists of content in the grid without having to worry about queries, paging or building new templates.
I've finally been able to upgrade it to Umbraco 8 and polished it a bit. It's still slightly in beta, but there's nothing special but a bit of CSS polish lacking, so it should be fairly safe to use in production. 👼

There's a whole documentation piece in the readme over on github, so I won't bother with a whole lot of prose in this blogpost, but I recorded a half an hour demo (below) showing all the aspects of implementing its use in an Umbraco site.

There's really not much to do once you get the hang of it, and our editors and staff have been enjoying using it for five years, so I hope you will too.

Here's the demo, and I really hope to get your feedback here, on github or on youtube. The first 3-4 minutes show the gist of it, so no need to sit through the full half hour if you're just curious. 👍

Morphing UCommerce Products

Tue, 30 Jan 2018 23:48:27 Z

The problem

Today I had a new but fun challenge with UCommerce. Turns out, as usual, it's a great fit for my whims with architecture. I was stuck between a rock and a hard place when I was looking at adding a custom pricing algoritm. I can't go into details, but there's custom client pricing involved of course. To add to the fun, we're mapping UCommerce products to DTOs for wire transfer. We could aslo have been mapping to view models or something else. To map we're using AutoMapper with quite a few configurations and jumps-through-hoops.

I had this code (ish):

var products = productRepository.Select().Where(SomePredicate);
var mapped = products.Select(Mapper.Map<ProductDto>);
return mapped;

I immediately thought of a few options:

Iterate over the products and change prices here
Create a Product adapter with additional logic and map from that
Execute the pricing logic from AutoMapper configuration

They all seemed weird and out of place though. None seemed like they would be easy to find for the next developer. Not even with unit tests. It just didn't seem right. Changing data on the entities would mean I'd have to go out of my way to ensure nobody went and saved those products later in the request. Creating an adapter would mean loads of new instances, bloated wrapper classes and weird names. And finally executing business logic from AutoMapper configuration means I'd been mixing responsibilities en mass.

UCommerce & NHibernate to the rescue

Luckily I've been using EntityFramework a lot and tried to force it into my Domain Driven Design patterns since it's infancy. I've been through the lot (and I enjoy it). So I kind of know what to expect from an ORM. When using UCommerce I'm stuck with NHibernate, but I haven't really been doing it justice by just leaving it in the background. (And fiddling with Entity Framework - which is just as good!) Together the two systems are extremely powerful. UCommerce have even documented the possibilities, though the documentation fails to point out the really juicy benefits.

We have ProductDefintion, right? It allows us to set up product types with different properties and variant options. It even supports inheritance. But we're still stuck with the Product class and its GetProperty() overrides. In my case, I'd like to have ProductWithFancyPricing so I could override that GetPrice() method. If I could have ProductWithFancyPricing and ProductWithEvenFancierPricing that would be totally awesome.

Turns out you can have your cake and eat it too. When properly using an ORM you can exploit OOP like it's supposed to and use polymorphism for varying behavior. It's possible to set up an inheritance tree so the mapper automatically handles creation of different types for you. You've basically got three options:

Table per concrete class (type)
- All classes have a table of their own
- Useful when base classes don't have (much) data
Table per hierarchy
- One table per base class
- Useful when all data is on the base class
Table per subclass
- One common table for base data
- Individual tables per derived class with only additional data
- Useful when there are some data in both classes. (Think umbracoNode)

In my case, I don't need any new data on the derived classes. It's all there in GetProperty() anyway. (I will add some getters though. ModelsBuilder, anyone? )
So for me it's going to be Table per hierarchy. The rest of the options are all viable for this technique if you have other requirements. You can read a bit about it in the UCommerce docs.

Mapping some product types

(I inadvertently wrote document types there. ModelsBuilder, anyone?)

In order to have NHibernate treat products as subclasses with the Table per hierarchy strategy it needs a way to pick the right class for each record. That way is known as discriminator columns. I first thought I could just discriminate by the ProductDefinitionId, but it turns out NHibernate doesn't support discriminating on a column already in use for associations (foreign keys) or other means.
We have to add a column. I just call it "Discriminator" and make it a varchar.

alter table uCommerce_Product add Discriminator nvarchar(max)

Then we need some entities. I added a couple of docu... product types:

public class ProductWithFancyPricing : UCommerce.EntitiesV2.Product
{
    public override Money GetPrice(PriceGroup priceGroup)
    {
        var price = base.GetPrice(priceGroup);
        if (IsChristmas())
        {
            price = new Money(price.Value * 2, price.Culture, price.Currency);
        }
        return base.GetPrice(priceGroup);
    }
}

public class ProductWithEvenFancierPricing : UCommerce.EntitiesV2.Product
{
    public override Money GetPrice(PriceGroup priceGroup)
    {
        var blackMarket = ObjectFactory.Instance.Resolve<IBlackMarketService>();
        var priceValue = blackMarket.GetPrice(Sku);
        return new Money(priceValue, priceGroup.Currency);
    }
}

The next thing you need is to tell NHibernate that these are our new product classes:

public class ProductWithFancyPricingMapping : FluentNHibernate.Mapping.SubclassMap<ProductWithFancyPricing>
{
    public ProductWithFancyPricingMapping()
    {
        DiscriminatorValue("Product with fancy pricing");
    }
}

public class ProductWithEvenFancierPricingMapping : FluentNHibernate.Mapping.SubclassMap<ProductWithEvenFancierPricing>
{
    public ProductWithEvenFancierPricingMapping()
    {
        DiscriminatorValue("Product with naughty pricing");
    }
}

We also need to subclass UCommerce's mapping for Product in order to tell UCommerce which column to use as the discriminator:

public class ProductMap : global::UCommerce.EntitiesV2.Maps.ProductMap
{
    public ProductMap()
    {
        DiscriminateSubClassesOnColumn("Discriminator");
    }
}

Finally we need a class in the same assembly with a tag on it. More on that in the UCommerce docs.

public class MappingMarker : IContainsNHibernateMappingsTag
{
}

To have NHibernate pick the right classes now, we just need to fix the existing products if we have any. I have called my discriminator values the same as my document types, so I can easily construct a query as such:

update
    ucommerce_product
set 
    discriminator = case productdefinitionid
        when 10 then 'Product with fancy pricing'
        when 11 then 'Product with naughty pricing'
        else null
    end

Now if we go...

var products = productRepository.Select();

...we'll get a bounch of ProductWithFancyPricings and ProductWithEvenFancierPricing. If you have more types, you might get into trouble though. You need to have a discriminator on them all.

The final hurdle

So that's cool. That's really cool. But there's one hurdle we have to jump over before we can cross the goal line. From very nasty experiences I knew I had to test everything manually and integrated. So I went and tried to see what happened if I added a product through the UCommerce Admin UI.

boom :)

'Course it didn't work. It actually did, and didn't. Several weird things happened ranging from strange NHibernate mapping exceptions to products getting the discriminator "UCommerce.EntitiesV2.Product". (Which makes a lot of sense if you think about it)

The @#¤%& CreateCategoryOrProduct.as[p|c]x WebForms control is in our way. It instantiates a Product and saves it. It's completely sealed and unconfigurable. We could overwrite it with a custom one, but that would open another can of worms with regards to upgrading, source control and what-not. Luckily it's going away very very soon in UCommerce V8. (2018?)

After hacking at it a bit my final resolve was to add a step right after save in the product saving pipeline. Again, UCommerce is so versatile that even when it sucks, it's got a golden workaround right up its arm. If you're not familiar with UCommerce Pipelines, go read about it here.

Here's the extra configuration. (In a .config file included from UCommerce's custom.config)

<!-- PRODUCT CLASS FIX -->
<component id="SaveProduct"
           service="UCommerce.Pipelines.IPipeline`1[[UCommerce.EntitiesV2.Product, UCommerce]], UCommerce"
           type="UCommerce.Pipelines.Catalog.ProductPipeline, UCommerce.Pipelines">
  <parameters>
    <tasks>
      <array>
        <value>${Product.UpdateRevision}</value>
        <value>${Product.Save}</value>
        <value>${FixProductClass}</value>
        <value>${Product.IndexAsync}</value>
      </array>
    </tasks>
  </parameters>
</component>

<component id="FixProductClass"
           service="UCommerce.Pipelines.IPipelineTask`1[[UCommerce.EntitiesV2.Product, UCommerce]], UCommerce"
           type="My.Awesome.Site.Persistence.FixProductClassTask, My.Awesome.Site.UCommerce"/>

A pipeline task gets a reference to the entity being handled, so we can't just go and replace the entire product with an instance of the right type. But we can fake it and force the database value to be correct after saving. UCommerce uses NHibernate level 2 cache, so we need to flush that as well, but we'll get to that.

Forcing the database is fairly easy. We have to resort to good old ADO code, which was actually a joyful little deja-vu experience (although I'm glad it was brief):

public class FixProductClassTask : IPipelineTask<Product>
{
    private readonly IStatelessSessionProvider sessionProvider;

    public FixProductClassTask(IStatelessSessionProvider sessionProvider)
    {
        this.sessionProvider = sessionProvider;
    }

    public PipelineExecutionResult Execute(Product subject)
    {
        var command = sessionProvider.GetStatelessSession().Connection.CreateCommand();
        command.CommandText = "UPDATE uCommerce_Product SET Discriminator = @discriminator WHERE ProductId = @productId";
        command.CommandType = CommandType.Text;
        var discriminatorParam = command.CreateParameter();
        discriminatorParam.ParameterName = "discriminator";
        discriminatorParam.Value = subject.ProductDefinition.Name;
        var idParam = command.CreateParameter();
        idParam.ParameterName = "productId";
        idParam.Value = subject.Id;
        command.Parameters.Add(discriminatorParam);
        command.Parameters.Add(idParam);
        command.ExecuteNonQuery();

        // TODO: Clear cache

        return PipelineExecutionResult.Success;
    }
}

I'm sure a lot of sazzy devs out there could prettify this a bit, but it does the job. Insert a switch/case (please don't) or whatever you fancy if the product definition name isn't what you discriminate by. I'll leave it up to you to choose between strings, ints or even enums for performance vs. readability.

If you've turned off the level 2 cache, you might be fine with this. Otherwise we'd better "evict" the entity from the cache. We need to do that in order for the cached instance to change type from Product to ProductWithFancyPricing. Sadly the NHibernate SessionFactory in charge of doing this is hidden in an internal static field in UCommerce, so we need to resort to some nasty reflection to do it:

// ...
command.ExecuteNonQuery();

var fieldInfo = typeof(SessionProvider).GetField("_factory", BindingFlags.Static | BindingFlags.NonPublic);
if (fieldInfo == null) throw new Exception("SessionFactory instance has moved in this UCommerce version. %(");
var sessionFactory = (ISessionFactory)fieldInfo.GetValue(null);
sessionFactory.Evict(typeof(Product), subject.Id);

return PipelineExecutionResult.Success;
// ...

Et voilá! We can now save new products, and they immediately morph into the correct derived type. (Except for when being saved to RavenDB for the first time, ref. the config).

I'm now free to go back into the instances and implement however naughty pricing I fancy. \o/

Added bonuses

I already have a custom entity in the database and NHibernate model. It has two associations to Product. Had I realized what I had under my fingertips it would already have been collections on my new shiny subclasses.

I recon you noticed I referenced ModelsBuilder a couple of times. How 'bout having all your properties statically typed on your product instances. How about some interfaces?

I'm sure you're getting the drift.

I for one am quite embarrased I didn't think of this before. I've had the knowledge and tools for years. But there you go. We learn something every day. And I love doing it with Umbraco, UCommerce, EntityFramework and apparently now also... NHibernate. :)

Umbraco Unit Testing Workshop Material

Thu, 09 Nov 2017 10:09:16 Z

For this years Umbraco UK Festival I was honored to be invited to run a unit testing workshop. It's a culmination of my talks and workshops until now, and covers everything you need to get going with testing the Umbraco backoffice, controller logic, external calls and finally front-end UI.

I know there's demand for this kind of training, and it might be difficult to get to the conferences where it's happening. For those of you, and those who feel bold and confident, I've released the material under the MIT license.

I can't promise I'll be able to help out if you go at it, but keep an eye out for a conference near you and encourage the organizers to host a workshop. ;)

You can clone the "workshop-start" branch from Bitbucket. The slides and workbook are included at the root of the repo.

https://bitbucket.org/bleedo/umbraco-testing-workshop-h2-2017/src/?at=workshop-start

You'll need

node
npm task runner (visual studio extension)
razorgenerator (visual studio extension)

Happy testing!

Automating Umbraco with PowerShell

Tue, 07 Nov 2017 19:46:14 Z

Background

This particular example is for warming up a UCommerce site in staging, but the technique can be used for anything you can do in the Umbraco backoffice. Especially with the new REST APIs coming out.

In this particular case, I've sinned and not created a good test environment for the last few integration bits of a project. It was hard to tune the production behavior of some code without actually running it in production. However, it's running Umbraco 7.5.13 and UCommerce 7.7. It's probably missing the other performance fix too, but the result is that it takes quite a while to warm up everything. So we warm it up in staging and then swap slots to get it fresh, awake and blazing fast into production.

Resolve

After having done this a few times, I figured I wanted something to do while waiting. What better activity than automating the whole routine so I could do something else instead? (Like automating automation...) Here's a powershell script I ended up with to warm up everything in the backoffice. I'll go through the pieces below.

Param(
    [string]$password
)

$ws = New-Object Microsoft.PowerShell.Commands.WebRequestSession

$body = @{
    "username"="admin@admin.com"
    "password"=$password
}

$json = $body | ConvertTo-Json

Invoke-RestMethod `
    -Method Post `
    -ContentType "application/json" `
    -Uri "https://customer-staging.azurewebsites.net/umbraco/backoffice/UmbracoApi/Authentication/PostLogin" `
    -WebSession $ws `
    -Body $json

Write-Host "20%"

Invoke-RestMethod -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/catalog/editcategory.aspx?id=718" -WebSession $ws

Write-Host "40%"

Invoke-RestMethod -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/catalog/editproduct.aspx?id=465&parentcategoryId=718" -WebSession $ws

Write-Host "60%"
Invoke-RestMethod -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/settings/orders/editpaymentmethod.aspx?id=8" -WebSession $ws

Write-Host "80%"

Invoke-RestMethod -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/settings/orders/editshippingmethod.aspx?id=10" -WebSession $ws
Write-Host "100%"

The effect is that we log into Umbraco using a provided password, and instead of navigating and clicking everything, we fire a request triggering all the caching and JIT compilation for us. Even though I'm using the Invoke-RestMethod cmdlet, I can do regular web calls. The cmdlet has a sibling called Invoke-WebRequest, but the rest version is better for posting commands. It's mostly a matter of mental context, but they have a few differences.

Log into Umbraco

To set up an authorized session with Umbraco, we can call the PostLogin action. It's the same endpoint that is used from the login screen. An authorized session means that we need to get a cookie and pass it with all our requests. In order for each Invoke-RestMethod to pass this cookie, we can create a WebRequestSession we pass to each call:

$ws = New-Object Microsoft.PowerShell.Commands.WebRequestSession

Invoke-RestMethod -WebSession $ws -Uri "..."

If the response brings a cookie, it's kept in the WebRequestSession, and subsequently passed back with each new request. Just like a browser.

Then we need some JSON to pass our username and password. You can declare dictionaries of sorts in PowerShell like so:

$body = @{
    "username"="admin@admin.com"
    "password"=$password
}

And then convert it to JSON by piping it to the ConvertTo-Json cmdlet:

$json = $body | ConvertTo-Json

Finally we're ready to fire the request off to Umbraco, adding config for HTTP method, ContentType etc.:

Invoke-RestMethod `
    -Method Post `
    -ContentType "application/json" `
    -Uri "https://customer-staging.azurewebsites.net/umbraco/backoffice/UmbracoApi/Authentication/PostLogin" `
    -WebSession $ws `
    -Body $json

Automate Umbraco

For this I just needed to kick off a request to some pages, but posting messages around like rebuilding a grumpy index, running an ad hoc task, even publishing should be just as simple.

It takes a while, so I added a little status message. I'm sure PowerShell wizards would pack this stuff into better reusable parts.

Write-Host "20%"

Now reuse that WebRequestSession object to fire off new authenticated requests:

Invoke-RestMethod -WebSession $ws -Method Get -Uri "https://customer-staging.azurewebsites.net/umbraco/ucommerce/catalog/editcategory.aspx?id=718"

Do it everywhere

With all the love I can give to UCommerce, I ended up naming the script kick-ucommerce.ps1. It's like kicking your old belowed car to get it started, after you've polished it. Really!
Adding my source folder to the path environment variable makes the script available from any shell. Even the Package Manager console in Visual Studio.

Make a note that the password is a parameter. You do change it more often than you'd like to update the code right? How 'bout automating the process? ;)

Smoke test and swap the Azure slot

I'll leave the swap-slot script I run after warming up the site here too. The cool thing about warming up with the script is that it'll fail almost immediately on the Umbraco login if anything isn't like it should. So it doubles as a smoke test.

When everything looks good, I can just go:

swap-slot -from staging -to production -site customer-x

And here's the "simpleness" of that one. There are fairly good docs on all the Azure cmdlets over on Microsoft's sites. (Ask Google. ;) )

Param(
    [string]$site,
    [string]$from,
    [string]$to
)

$subscriptionId = "333112F4-4483-449C-A2DA-727E8D2E428D"
$resourcegroupname = "Common-Group"     # Might need to be param

Login-AzureRmAccount -SubscriptionId $subscriptionId

Swap-AzureRmWebAppSlot `
    -SourceSlotName $from `
    -DestinationSlotName $to `
    -Name $site `
    -ResourceGroupName $resourcegroupname

That's it! Now go automate something so you get more time to do fun stuff! :)

UmbracoSupport got typed content

Tue, 17 Oct 2017 21:18:06 Z

What's UmbracoSupport?

UmbracoSupport is a class I've been introducing to my unit tests over the last year or so. It allows me to have my own hierarchy for tests, as well as re-using all of Umbraco's own stubbing code. I've written about it in a post called Unit testing Umbraco just got simpler, and its gut's code is described in details in The basics of unit testing Umbraco.

A quick primer on what's already available

The BaseDatabaseFactoryTest in Umbraco.Tests has a method called GetXmlContent. It replaces the umbraco.config file that acts as the cache at runtime. It makes UmbracoContext.Current.ContentCache tick in unit tests. The base tests out of the box has a small flaw though. They can't "popuplate" properties. All you get is the hierarchy.

Usually I've injected an IPublishedContentCache to my controllers. When testing them, I've created a mock instance of the IPublishedContentCache. However, all my code has to use the non-context aware overloads. For instance cache.GetById(umbracoContext, false, id). There's also a whole lot of ugly mocking code going on to set up queries and stubbed content. How to stub properties on stubbed content is described in Slides from CG 2016 and testing IPublishedContent properties.

So what's new?

As mentioned, I've been throwing around all kinds of ugly stubbing code for content and I've also been tampering with Umbraco.Tests's GetXmlContent() to use the "built-in" content stubs. It's all been done before in misc. tests in Umbraco. I finally got my s**t together and refactored all my setup spaghetti into a few small helpers on the UmbracoSupport class.

Let's go over them in increasing "integrationness".

Pure hierarchy

Your basic hierarchy structure can be set up by simply returning a string from an overload of BaseDatabaseFactoryTest.GetXmlContent. UmbracoSupport overloads this method and returns whatever you've set on the UmbracoSupport.ContentCacheXml property. I recommend using the technique described in Automating creating of source data for tests with this. You can even extend that code to have fixture specific content caches.

In any case, to make this work, you just need to set the XML in the setup method.

Note: I've got some probs with the markdown parsing here, imagine the CDATA parts of the XML is correctly written.

[SetUp]
public void Setup()
{
    umbracoSupport = new UmbracoSupport();
    umbracoSupport.SetupUmbraco();

    // This XML is what the ContentCache will represent
    umbracoSupport.ContentCacheXml = @"
        <?xml version=""1.0"" encoding=""utf-8""?>
        <!DOCTYPE root [
          <!ELEMENT contentBase ANY>
          <!ELEMENT home ANY>
          <!ATTLIST home id ID #REQUIRED>
          <!ELEMENT page ANY>
          <!ATTLIST page id ID #REQUIRED>
        ]>
        <root id=""-1="""">
          <home id=""1103="""" key=""156f1933-e327-4dce-b665-110d62720d03="""" parentID=""-1="""" level=""1="""" creatorID=""0="""" sortOrder=""0="""" createDate=""2017-10-17T20:25:12="""" updateDate=""2017-10-17T20:25:17="""" nodeName=""Home="""" urlName=""home="""" path=""-1,1103="""" isDoc="""" nodeType=""1093="""" creatorName=""Admin="""" writerName=""Admin="""" writerID=""0="""" template=""1064="""" nodeTypeAlias=""home="""">
            <title>Welcome!</title>
            <excerptCount>4</excerptCount>
            <page id=""1122="""" key=""1cb33e0a-400a-4938-9547-b05a35739b8b="""" parentID=""1103="""" level=""2="""" creatorID=""0="""" sortOrder=""0="""" createDate=""2017-10-17T20:25:12="""" updateDate=""2017-10-17T20:25:17="""" nodeName=""Page="" 1="""" urlName=""page1="""" path=""-1,1103,1122="""" isDoc="""" nodeType=""1095="""" creatorName=""Admin="""" writerName=""Admin="""" writerID=""0="""" template=""1060="""" nodeTypeAlias=""page="""">
              <title>Welcome!</title>
              <excerpt>[CDATA[Lorem ipsum dolor...]]</excerpt>
              <body>
                [CDATA[<p>Lorem ipsum dolor...</p>]]
              </body>
              <image>123</image>
            </page>
            <page id=""1123="""" key=""242928f6-a1cf-4cd3-ac34-f3ddf3526b2e="""" parentID=""1103="""" level=""2="""" creatorID=""0="""" sortOrder=""1="""" createDate=""2017-10-17T20:25:12="""" updateDate=""2017-10-17T20:25:17="""" nodeName=""Page="" 2="""" urlName=""page2="""" path=""-1,1103,1123="""" isDoc="""" nodeType=""1095="""" creatorName=""Admin="""" writerName=""Admin="""" writerID=""0="""" template=""1060="""" nodeTypeAlias=""page="""">
              <title>More welcome!</title>
              <excerpt>[CDATA[More lorem ipsum dolor...]]</excerpt>
              <body>[CDATA[Even more lorem ipsum dolor...]]</body>
              <image>234</image>
            </page>
          </home>
        </root>
    ".Trim();
}

In our tests, we can now query by anything. The returned content has the hierarchy and everything, so we can traverse it with Children(), Parent() and whatnot. The only missing piece is the properties. Here's a test showing that we have everything but the title property of Page 1:

const int Page1Id = 1122;

[Test]
public void Returns_Empty_Documents()
{
    var contentCache = umbracoSupport.UmbracoContext.ContentCache;
    var page1 = contentCache.GetById(Page1Id);

    Assert.That(page1, Is
        .Not.Null
        .And
        .InstanceOf<PublishedContentWithKeyBase>()
        .And
        .Property("Name").EqualTo("Page 1")
        .And
        .Matches<IPublishedContent>(c => c["title"] == null)
        .And
        .Property("Parent")
            .Property("Children")
                .With.Count.EqualTo(2)
    );
}

Don't be discouraged though. This method is excellent for testing URL providers, ContentFinders, Menus, Sitemaps. You name it. I know I've written my fair share of hierarchy traversing code or fancy XPath queries. Unless of course, you need property values.

Instead of pulling your leg about it, here's how we fix that.

Put some meat on the content

The reason the properties are not there isn't because they weren't read. It's because the XmlPublishedContent that we get out ultimately relies on the PublishedContentType for it's respective document type. Luckily, all Umbraco's services are already stubbed up for us, so we can give it what it needs.

The gory guts of it is that it needs an IContentType from the ContentTypeService. We can easily stub one up with Moq: var contentType = Mock.Of<IContentType>(). Further, it uses the IContentType.CompositionPropertyTypes collection to iterate the properties. These PropertyType instances are actually completely dependency-less, so we can just create some:

Mock.Get(contentType)
    .Setup(t => t.CompositionPropertyTypes)
    .Returns(new[] {
        new PropertyType("Umbraco.TinyMCEv3", DataTypeDatabaseType.Nvarchar, "body")
    });

Finally, we set it up on the ContentTypeService stub:

Mock.Get(umbracoSupport.ServiceContext.ContentTypeService)
    .Setup(s => s.GetContentType(alias))
    .Returns(contentType);

If only it were so easy. We depend on the BaseWebTest class from Umbraco.Tests. It sets up a content type factory that's being used somewhere in the hierarchy. It feeds AutoPublishedContent instances instead of what we've stubbed up. We need to turn that off. There's a trick here. UmbracoSupport should now live in an assembly called Umbraco.UnitTests.Adapter. If you're pre 7.6.4 you need to go with Umbraco.VisualStudio. This is because the factory we need to reset is internal to Umbraco. By having UmbracoSupport in an assembly with one of these two names, we're able to do it. (Otherwise, you use reflection.) By no means do this with production code. Just... forget it!
This paragraph should also get it's own blog post. :)

But I digress. Here's the line you need to have the content use the ContentTypeService to fetch its type:

PublishedContentType.GetPublishedContentTypeCallback = null;

It's tempting to leave setup code like this lying around in all our SetUp methods or even in our "Arrange" sections. I've sinned too much, so those few lines are now part of UmbracoSupport and can be used to set up multiple types for your fixture or test.

Here's a test that fetches the same document as before, but can now read properties:

[Test]
public void With_DocumentTypes_Setup_Returns_Full_Blown_Documents()
{
    umbracoSupport.SetupContentType("page", new[]
    {
        new PropertyType("textstring", DataTypeDatabaseType.Nvarchar, "title"),
        new PropertyType("textarea", DataTypeDatabaseType.Nvarchar, "excerpt"),
        new PropertyType("Umbraco.TinyMCEv3", DataTypeDatabaseType.Nvarchar, "body"),
        new PropertyType("media", DataTypeDatabaseType.Integer, "image")
    });

    var page1 = contentCache.GetById(Page1Id);

    Assert.Multiple(() =>
    {
        Assert.That(page1["title"], Is.EqualTo("Welcome!"));
        Assert.That(page1["excerpt"], Is.EqualTo("Lorem ipsum dolor..."));
        Assert.That(page1["body"].ToString(), Is.EqualTo("<p>Lorem ipsum dolor...</p>"));
        Assert.That(page1["image"], Is.EqualTo(123));
    });
}

Notice the .ToString() on the body. It's actually not a string, but some weird dynamic Umbraco thingy. I never saw that type before, but I didn't pursue it in time for this post. I don't want anything to do with it though, so let's storm on to the grand finale.

Let's make them strong already!

We're finally there. The last piece of the puzzle. Strongly typed content!

It's managed by two resolvers: PublishedContentModelFactoryResolver and PropertyValueConvertersResolver. I won't go into details about those now, but suffice to say all resolvers have to be initialized before BaseWebTest.Initialize and its ancestors. I've added an InitializeResolvers method to the UmbracoSupport class where these two are initialized. The PublishedContentModelFactoryResolver is set to a FakeModelFactoryResolver that lets you register constructors for document type aliases. The code for this is available in my "Umbraco unit testing samples" repo on github.

To set up property value converters, we also need to do that before registering the resolver. The resolver takes all the converters as constructor arguments. I've added a list of those types as a property on UmbracoSupport, so we can add IPropertyValueConverter implementing types before calling UmbracoSupport.SetupUmbraco:

[SetUp]
public void Setup()
{
    umbracoSupport = new UmbracoSupport();

    // Converter types need to be added before setup
    umbracoSupport.ConverterTypes.Add(typeof(TinyMceValueConverter));

    umbracoSupport.SetupUmbraco();

    //...
}

To register the typed model, there's just one line you can do in your setup, or even in your tests. Here I've refactored out the setup for the content type from earlier, and I register a model type for the document type alias:

private void SetupContentType()
{
    umbracoSupport.SetupContentType("page", new[]
    {
        new PropertyType("textstring", DataTypeDatabaseType.Nvarchar, "title"),
        new PropertyType("textarea", DataTypeDatabaseType.Nvarchar, "excerpt"),
        new PropertyType("Umbraco.TinyMCEv3", DataTypeDatabaseType.Nvarchar, "body"),
        new PropertyType("media", DataTypeDatabaseType.Integer, "image")
    });
}

[Test]
public void With_DocumentTypes_And_Models_Setup_Returns_Fully_Functional_Typed_Content()
{
    SetupContentType();

    // Register strongly typed models with the ModelFactory
    umbracoSupport.ModelFactory.Register("page", c => new Page(c));

    var page1 = contentCache.GetById(Page1Id);

    Assert.That(page1, Is
        .InstanceOf<Page>()
        .And.Property("Body")
            .Matches<IHtmlString>(s => 
                s.ToString() == "<p>Lorem ipsum dolor...</p>"
            )
    );
}

public class Page : PublishedContentModel
{
    public Page(IPublishedContent content) : base((IPublishedContentWithKey)content)
    {
    }

    public string Title => Content.GetPropertyValue<string>("title");
    public string Excerpt => Content.GetPropertyValue<string>("excerpt");
    public IHtmlString Body => Content.GetPropertyValue<IHtmlString>("body");
    public int Image => Content.GetPropertyValue<int>("image");
}

There you go! There's nothing more to it. Well, there is...

The Page class here is bundled with the test. If we use a common interface both for our runtime model and our test model, we're safe. But we should really use the runtime models. This means you shouldn't use runtime generated models. Go through the instructions for ModelsBuilder to have your models compiled and accessible from the tests.

Conclusion

And although the XML is pretty ugly, you can flush it out into files bundled with your tests. You can also exploit the umbraco.config file and just copy segments from there into your test source files. That way, you spend no time writing the stubs, and the content is cleanly separated from your tests.

That's really all there is to it! It is. Now go test a bit, or a byte, or a string, or even a view.

The new version of UmbracoSupport including the fake model factory is available here.

Building a spell checker for search in Umbraco

Fri, 13 May 2016 13:19:11 Z

I spent the day building a spell checker for a search UI I've been improving. Turns out it was pretty easy when leaning on Lucene.Net Contrib. I'll show you the gist of it in this article.

Make sure you have some nice proofed data to work with tough. While writing the tests for my checker, I kept getting silly suggestions and wrong results. Turned out the data itself had quite a few typos. On the bright side, it allowed us to fix the typos.

Building a dictionary

So the first thing you need to provide spell checking is a dictionary. For my solution I went with using all the words that exists in the site so I wouldn't suggest words that don't have results. We already have something that provides this functionality, namely Examine. Examine lets you add indexes with custom indexers, and that's a perfect fit for our needs. This dictionary however only needs one field: word.

Here's the basic code for an indexer that indexes all the words in a site:

public class SpellCheckIndexer : BaseUmbracoIndexer
{
    // May be extended to find words from more types
    protected override IEnumerable<string> SupportedTypes
    {
        get
        {
            yield return IndexTypes.Content;
        }
    }

    protected override void AddDocument(Dictionary<string, string> fields, IndexWriter writer, int nodeId, string type)
    {
        var doc = new Document();
        List<string> cleanValues = new List<string>();

        // This example just cleans HTML, but you could easily clean up json too
        CollectCleanValues(fields, cleanValues);
        var allWords = String.Join(" ", cleanValues);

        // Make sure you don't stem the words. You want the full terms, but no whitespace or punctuation.
        doc.Add(new Field("word", allWords, Field.Store.NO, Field.Index.ANALYZED));

        writer.UpdateDocument(new Term("__id", nodeId.ToString(CultureInfo.InvariantCulture)), doc);
    }

    protected override IIndexCriteria GetIndexerData(IndexSet indexSet)
    {
        var indexCriteria = indexSet.ToIndexCriteria(DataService);
        return indexCriteria;
    }

    private void CollectCleanValues(Dictionary<string, string> fields, List<string> cleanValues)
    {
        var values = fields.Values;
        foreach (var value in values)
            cleanValues.Add(CleanValue(value));
    }

    private static string CleanValue(string value)
    {
        return HttpUtility.HtmlDecode(value.StripHtml());
    }
}

To make it actually create an index you have to add it and an IndexSet to Examine's configuration.

/config/ExamineSettings.config:

<Examine>
    <ExamineIndexProviders>
        <providers>

                  <!-- Existing providers... -->

                  <add name="SpellCheckIndexer" type="YourAssembly.SpellCheckIndexer, YourAssembly"
                      analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
                  />
        </providers>
    </ExamineIndexProviders>
    <!-- Search Providers - configured later -->
</Examine>

/config/ExamineIndex.config:

<ExamineLuceneIndexSets>
    <IndexSet SetName="SpellCheckIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/{machinename}/SpellCheck/">
        <IndexAttributeFields>
            <add Name="nodeName"/>
        </IndexAttributeFields>
        <IndexUserFields>
            <!-- Add the properties you want to extract words from here -->
            <add Name="body"/>
            <add Name="summary"/>
            <add Name="description"/>
        </IndexUserFields>
    </IndexSet>
</ExamineLuceneIndexSets>

Now we're actually ready to create a word index. Open the backoffice and the developer section. Navigate to the Examine Management tab. You should see the "SpellCheckIndexer". Click it and select "Index info & tools". You might already have documents in the index. If so, we're good to go, otherwise click "Rebuild index". If you want, you can have a look at the data using Luke.

Casting the spell

Now we're ready to let the Lucene Contrib SpellChecker do its magic. It needs to get the dictionary from our index, so let's give it a searcher from Exmine. We'll configure that as a regular searcher in ExamineSettings.config:

<ExamineSearchProviders defaultProvider="ExternalSearcher">
    <providers>
        <add name="SpellCheckSearcher"
            type="UmbracoExamine.UmbracoExamineSearcher, UmbracoExamine"
            analyzer="Lucene.Net.Analysis.Standard.StandardAnalyzer, Lucene.Net"
        />
    </providers>
</ExamineSearchProviders>

Now we've got everything we need to build the checker. I've written some unit-tests for it, so the constructor takes a BaseLuceneSearcher as an argument. For testing, I just create one from an index I know exists. For Umbraco I instantiate it as a singleton using the searcher from Examine.

public class UmbracoSpellChecker
{
    private static readonly object lockObj = new object();
    private static UmbracoSpellChecker instance;
    public static UmbracoSpellChecker Instance
    {
        get
        {
            lock(lockObj)
            { 
                if (instance == null)
                {
                    instance = new UmbracoSpellChecker((BaseLuceneSearcher)ExamineManager.Instance.SearchProviderCollection["SpellCheckSearcher"]);
                    instance.EnsureIndexed();
                }
            }
            return instance;
        }
    }

    private readonly BaseLuceneSearcher searchProvider;
    private readonly SpellChecker.Net.Search.Spell.SpellChecker checker;
    private readonly IndexReader indexReader;
    private bool isIndexed;

    public UmbracoSpellChecker(BaseLuceneSearcher searchProvider)
    {
        this.searchProvider = searchProvider;
        var searcher = (IndexSearcher)searchProvider.GetSearcher();
        indexReader = searcher.GetIndexReader();
        checker = new SpellChecker.Net.Search.Spell.SpellChecker(new RAMDirectory(), new JaroWinklerDistance());
    }

    private void EnsureIndexed()
    {
        if (!isIndexed)
        { 
            checker.IndexDictionary(new LuceneDictionary(indexReader, "word"));
            isIndexed = true;
        }
    }

    public string Check(string value)
    {
        EnsureIndexed();

        var existing = indexReader.DocFreq(new Term("word", value));
        if (existing > 0)
            return value;

        var suggestions = checker.SuggestSimilar(value, 10, null, "word", true);

        var jaro = new JaroWinklerDistance();
        var leven = new LevenshteinDistance();
        var ngram = new NGramDistance();

        var metrics = suggestions.Select(s => new
        {
            word = s,
            freq = indexReader.DocFreq(new Term("word", s)),
            jaro = jaro.GetDistance(value, s),
            leven = leven.GetDistance(value, s),
            ngram = ngram.GetDistance(value, s)
        })
        .OrderByDescending(metric =>
            (
                (metric.freq/100f) +
                metric.jaro +
                metric.leven +
                metric.ngram
            )
            / 4f
        )
        .ToList();

        return metrics.Select(m => m.word).FirstOrDefault();
    }
}

The constructor keeps a reference to our word index and creates a SpellChecker that will do it's work in memory. When we first search for suggestions, EnsureIndex passes our word index to the SpellChecker and lets it create it's magic "back-end". It could also be on disk, but for this site the performance is good enough to do it live.

The only suggestion code you actually need in Check is basically just:

        var suggestion = checker.SuggestSimilar(value, 1);

It will give you a feasible word if the value has a typo. However, in many cases it's not the one you're after. You can increase the number of suggestions you want to get a few more, but for search suggestions, we really just want one. Changing the StringDistance algorithm passed to the SpellChecker also varies the output, so you should experiment with that.

In my case there was a typo in the source data. I was testing against the Norwegian word for bathtub: "badekar". My test phrase was "badkear" and I kept getting suggestions for "badkar". That word snuck in there due to one spelling mistake in one document. So I figured word frequency should trump. But sorting the suggestions by word frequency turned out to be a bad idea. "Baderom", the Norwegian word for "bathroom", is more frequent and won over "badekar" that I was after.

I spun up all the string distance algorithms and put the results, including the document frequency of the terms in Excel. Experimenting a bit showed that the frequency divided by 100 added to the sum of the distance results was a good metric. After ordering by that result and taking the first, my checker is pretty intelligent when providing suggestions.

Getting results

Here's a snippet from the search controller that builds up the response model:

var spellChecker = UmbracoSpellChecker.Instance; 
var checkedTerms = phrase.Split(' ').Select(t => spellChecker.Check(t));
var didYouMean = String.Join(" ", checkedTerms);
result.DidYouMean = didYouMean;

// Uses Lucene's TopScoreDocCollector
var totalHits = GetTotalHits(query);

if (result.DidYouMean != phrase)
{
    if (totalHits == 0)
    {
        query = CreateQuery(result.DidYouMean);
        totalHits = GetTotalHits(query);
        result.Modified = true;
        result.Query = result.DidYouMean;
    }
    else
    {
        result.HasAlternative = true;
    }
}

When searching for "badkear" now, I get result.Modified == true. I didn't get any hits for that word, but my spell checker found the word "badekar". I can happily go search for that and tell my user I modified his search. "Did not find any hits for 'badkear', showing results for 'badekar'.

When searching for "dujs", a misspelled version of the Norwegian word for "shower", I find that we have another typo in our data. I get result.Modified == false, but result.HasAlternative == true. My spell checker dutifully asks if I ment "dusj". "Showing 1 result. Did you mean 'dusj'?"

There's a lot more you can do with this to provide even more accurate suggestions. For my purposes however, this was all I needed. Leaning on Lucene Contrib when you need some fancy search functionality will almost always save the day.

If you want to know more about Lucene, I recommend getting the book called "Lucene in Action".