Generating documentation from NUnit tests
What's the catch?
There's a lot of benefits with unit testing. One of the less exploited ones would be documentation. I've heard it said, and I've said it myself countless times: Unit Tests can be used as documentation. They are inherently documentation by being a good place to look for sample code. But what if they could also serve as actual go-to documentation, a backlog, a technical specification and all those things that otherwise go rot in a Word document on some server?
Lately, I've been experimenting with generating good documentation from NUnit tests for our ASP.NET MVC sites. (Based on Umbraco CMS, but that's not important to this article.)
I've been baffled by the lack of good information when I've googled for NUnit result transformations. So I gave it a go, and I'll share some techniques with you in this post.
Many unit tests won't do for documentation though. I believe that testing should be done as integrated as possible, stopping just before they become slow. This allows for more user-centric, or holistic descriptions of the system. I'm doing my best to get better at naming them. They should read like stories describing the system. That way it'll be easy for other devs to find the right code to extend or debug. Namespacing and organization play a strong role, as well as good, descriptive names. It's hard, but I believe striving for this might actually also aid in the ability to use them as documentation.
An example
So for this example, let's have a look at a pretty simple case. We're integrating with a third-party event site to display relevant events on pages in our site. We have some kind of module that lets editors pick events and place them in pages. The part I'll use in this post is the part about fetching and displaying it.
The tests I'm writing are feature specific, but technical enough in nature to belong in the realm of technical documentation. For business targeted documentation, you're better off looking into BDD and all the practices that follows. If you don't have the team though, you might be able to get something working for you with the following techniques.
Here's the test output I've got so far:
I think it reads fairly well. There shouldn't be too much doubt what this feature does. Granted, it's a really simple one, and complex systems will have thousands. Even more reason to group by feature, and add areas and other useful grouping levels as namespaces.
You'll also notice I've got a test that is inconclusive saying not implemented. I've started doing this to keep a kind of backlog of things to do. It's easy to forget about technical debt or rarer cases we've ignored. Having inconclusive tests glaring at you makes it easy to remember. It may also serve as a discussion point to bring up with the customer. "Should we bother to handle this weird case? It'll increase the budget with $...".
There are still a couple of readability issues in the tests. When we add more features, it'd be nice to find all event-related tests in their own group. That's a quick fix. We'll add a namespace. (Put common tests in the same folder)
There's also a matter of order. Obviously fetching from server happens before converting to a model and then rendering it. There are concepts like ordered tests, but I believe atomic tests are faster and more flexible. I've got a solution for that, but let's live with it for now.
Having this output from the testrunner or on the build server, even in your inbox when someone commits, is useful. We have a broad overview of what the system does. The tests names are decoupled from any implementation details so we're free to change the architecture while keeping our descriptive tests.
But we could do more.
NUnit console runner output
When working on code, we use built-in test runners in our IDEs. We're able to run one or a few test in isolation and focus on just the right things. Build servers use command line runners and ususally produce some form of textual or HTML output. We can exploit the CLI tools as well. The NUnit console runner outputs an XML format by default. You can even do it from the package manager console in Visual Studio like so:
PM> .\packages\NUnit.ConsoleRunner.3.7.0\tools\nunit3-console.exe .\NUnitDocs.Tests\bin\debug\NUnitDocs.Tests.dll
If you're using Nuget to organize your dependencies, you can install the NUnit.ConsoleRunner package to get the executable in the packages folder. Otherwise, you can download it from the NUnit downloads page.
In addition to logging the results to the output, NUnit will write some metadata, and the following (more interesting) to TestOutput.xml:
<test-run id="2" testcasecount="3" result="Passed" total="3" passed="2" failed="0" inconclusive="1" >
<!-- Attributes and metadata elements stripped for clarity -->
<test-suite type="TestSuite" id="0-1005" name="NUnitDocs" fullname="NUnitDocs" runstate="Runnable" testcasecount="3" result="Passed" total="3" passed="2" failed="0" warnings="0" inconclusive="1">
<test-suite type="TestSuite" id="0-1006" name="Tests" fullname="NUnitDocs.Tests">
<test-suite type="TestSuite" id="0-1007" name="Events" fullname="NUnitDocs.Tests.Events">
<test-suite type="TestFixture" id="0-1000" name="When_Displaying_An_Event" fullname="NUnitDocs.Tests.Events.When_Displaying_An_Event">
<test-case id="0-1002" name="It_Is_Converted_To_An_Event_Model" fullname="NUnitDocs.Tests.Events.When_Displaying_An_Event.It_Is_Converted_To_An_Event_Model" methodname="It_Is_Converted_To_An_Event_Model" classname="NUnitDocs.Tests.Events.When_Displaying_An_Event" runstate="Runnable" seed="1189788506" result="Passed" start-time="2017-10-08 19:44:11Z" end-time="2017-10-08 19:44:12Z" duration="0.413085" asserts="3" />
<test-case id="0-1001" name="It_Is_Fetched_By_Id_From_The_Event_Server" fullname="NUnitDocs.Tests.Events.When_Displaying_An_Event.It_Is_Fetched_By_Id_From_The_Event_Server" methodname="It_Is_Fetched_By_Id_From_The_Event_Server" classname="NUnitDocs.Tests.Events.When_Displaying_An_Event" runstate="Runnable" seed="486335788" result="Passed" start-time="2017-10-08 19:44:12Z" end-time="2017-10-08 19:44:12Z" duration="0.001595" asserts="1" />
<test-case id="0-1003" name="It_Is_Rendered_With_A_Name_Date_And_A_Link" fullname="NUnitDocs.Tests.Events.When_Displaying_An_Event.It_Is_Rendered_With_A_Name_Date_And_A_Link" methodname="It_Is_Rendered_With_A_Name_Date_And_A_Link" classname="NUnitDocs.Tests.Events.When_Displaying_An_Event" runstate="Runnable" seed="804806648" result="Inconclusive" start-time="2017-10-08 19:44:12Z" end-time="2017-10-08 19:44:12Z" duration="0.015905" asserts="0">
<reason>
<message>
<![CDATA[Rendering has not been implemented yet.]]/>
</message>
</reason>
</test-case>
</test-suite>
</test-suite>
</test-suite>
</test-suite>
</test-run>
I know, I know. It's not cool anymore. I don't like it any more than you, but there is one durable ancient technology we could use to do something with this output. Did you guess it yet?
XSLT!
There. I said it.
Now let's look at a simple output before we dive into the more beefy things:
Now ain't that starting to look readable? There's not that much to it. The NUnit console runner has a built-in parameter with the syntax --result=SPEC, where SPEC can point to an XSLT sheet. Here's the command to generate the former:
.\packages\NUnit.ConsoleRunner.3.7.0\tools\nunit3-console.exe .\NUnitDocs.Tests\bin\debug\NUnitDocs.Tests.dll --result:"TestSummary.htm;transform=TestSummary.xslt"
The XSLT isn't much either. There's Bootstrap and (our trusty) JQuery from CDN. Some simple HTML formatting. And I wrote a one-liner JavaScript function to strip out all the underscores as well. Here's the first pass:
<?xml version="1.0" encoding="iso-8859-1"?>
<xsl:stylesheet
version="1.0"
exclude-result-prefixes="msxsl"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:msxsl="urn:schemas-microsoft-com:xslt"
>
<xsl:output method="html" indent="yes"/>
<xsl:template match="@* | node()">
<html>
<head>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous"/>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous"/>
<style type="text/css">
.Passed { color: green; }
.Inconclusive { color: #BBAA00; }
.Failed { color: red; }
ul {
margin-left: 0px;
list-style-type: none;
padding-left: 0;
}
ul ul {
margin-left: 15px;
}
.counts {
font-size: .7em;
color: gray;
}
</style>
</head>
<body>
<div class="container">
<xsl:apply-templates select="/test-run/test-suite"/>
</div>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous">// Force closing tag</script>
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js">// Force closing tag</script>
<script type="text/javascript">
$("td").each(function(i, e) {
$(e).text($(e).text().replace(/_/g, " "));
});
</script>
</body>
</html>
</xsl:template>
<xsl:template match="/test-run/test-suite">
<h1>
<xsl:value-of select="./test-run/@name"/>
</h1>
<table class="table table-striped">
<xsl:apply-templates select="./test-suite"/>
</table>
</xsl:template>
<xsl:template match="test-suite">
<tr>
<td>
<xsl:attribute name="class">
<xsl:choose>
<xsl:when test="./@failed > 0">Failed</xsl:when>
<xsl:when test="./@inconclusive > 0">Inconclusive</xsl:when>
<xsl:otherwise>Passed</xsl:otherwise>
</xsl:choose>
</xsl:attribute>
<xsl:attribute name="style">
padding-left: <xsl:value-of select="count(ancestor::test-suite)*15"/>px;
</xsl:attribute>
<xsl:value-of select="./@name"/>
</td>
<td class="counts">
<xsl:value-of select="./@passed"/> passed,
<xsl:value-of select="./@inconclusive"/> inconclusive,
<xsl:value-of select="./@failed"/> failed
</td>
</tr>
<xsl:for-each select="./test-suite">
<xsl:apply-templates select="."/>
</xsl:for-each>
<xsl:for-each select="./test-case">
<xsl:apply-templates select="."/>
</xsl:for-each>
</xsl:template>
<xsl:template match="test-case">
<tr>
<td colspan="2">
<xsl:attribute name="style">
padding-left: <xsl:value-of select="count(ancestor::test-suite)*15"/>px;
</xsl:attribute>
<xsl:attribute name="class">
<xsl:value-of select="./@result"/>
</xsl:attribute>
<xsl:value-of select="./@name"/>
</td>
</tr>
</xsl:template>
</xsl:stylesheet>
This is fairly good as a starting point, but we're stuck with the brief test names. Having some free text, at least at feature level or use-case level would be nice. And we still have this ordering problem to deal with. NUnit has the perfect tool for this:
NUnit PropertyAttribute
NUnit comes with a PropertyAttribute that we can use to decorate our tests with metadata. It's main purpose is to add properties to fixtures and tests in the test output. It's an abstract class we can use to create our own metadata, and there's a few built-in like DescriptionAttribute. There are uses of properties at runtime too, but that's way too advanced for this post.
The DescriptionAttribute is just what we want for the free text part of our documentation. It's added to the method just like the Test attribute:
[Test]
[Description(@"
Events are provided at eventsite with a REST api at the URL:
https://eventsite/api/{id}
")]
public void It_Is_Fetched_By_Id_From_The_Event_Server()
{
// ...
}
[Test]
[Description("The JSON response from the event server is deserialized as the Event type.")]
public void It_Is_Converted_To_An_Event_Model()
{
// ...
}
The Description is written to the XML output like so:
<test-case id="0-1002" name="It_Is_Converted_To_An_Event_Model" ...>
<properties>
<property name="Description" value="The JSON response from the event server is deserialized as the Event type." />
</properties>
</test-case>
By adding a few more voodoo lines to the XSLT and another one-liner JavaScript, we get the following output:
(Don't worry, complete XSLT is available at the bottom fo the post.)
Now we're really getting somewhere. We can provide that little extra that tells the reader what's really happening and stuff that might be obscured by too many, or hopefully just the right amount of abstractions.
Let's tackle the ordering problem while we're at it.
Implementing our own properties
The PropertyAttribute isn't much more than a metadata container. An ordering property is as simple as such:
public class DocumentationOrderAttribute : PropertyAttribute
{
public DocumentationOrderAttribute(int order)
: base("DocumentationOrder", order)
{
}
}
It's applied to the tests just like the descriptions:
[Test]
[DocumentationOrder(0)]
[Description(@"
Events are provided at eventsite with a REST api at the URL:
https://eventsite/api/{id}
")]
public void It_Is_Fetched_By_Id_From_The_Event_Server()
{
// ...
}
And it's output just like descriptions:
<test-case id="0-1002" name="It_Is_Converted_To_An_Event_Model" ...>
<properties>
<property name="DocumentationOrder" value="1" />
<property name="Description" value="The JSON response from the event server is deserialized as the Event type." />
</properties>
</test-case>
Now we can tackle the ordering problem with a simple XSLT sort element:
<xsl:for-each select="./test-case">
<xsl:sort select="./properties/property[@name='DocumentationOrder']/@value"/>
<xsl:apply-templates select="."/>
</xsl:for-each>
And we get a nice ordered output:
Summary
By structuring and naming our tests by feature we get nice headings for potential documentation. We also untie them completely from implementation so we're free to change our code. Applying a bit of metadata to our tests adds that little extra to make quite meaningful documentation.
The fact that the results can be tranformed means we can create rich documentation UIs with search, tags, navigation structure and a bit less than just enough prose.
Next steps
We've still got that rendering test to pass. There's also a few too many asserts in one of the tests. In an upcoming post I'll share a couple of other tools with you and how to extend those to enrich the technical documentation even more.
Complete XSLT
Here's the complete XSLT for the examples above:
<?xml version="1.0" encoding="iso-8859-1"?>
<xsl:stylesheet
version="1.0"
exclude-result-prefixes="msxsl"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:msxsl="urn:schemas-microsoft-com:xslt"
>
<xsl:output method="html" indent="yes"/>
<xsl:template match="@* | node()">
<html>
<head>
<!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous"/>
<!-- Optional theme -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous"/>
<style type="text/css">
.Passed { color: green; }
.Inconclusive { color: #BBAA00; }
.Failed { color: red; }
ul {
margin-left: 0px;
list-style-type: none;
padding-left: 0;
}
ul ul {
margin-left: 15px;
}
label {
font-weight: normal;
}
.counts {
font-size: .7em;
color: gray;
}
</style>
</head>
<body>
<div class="container">
<xsl:apply-templates select="/test-run/test-suite"/>
</div>
<!-- Latest compiled and minified JavaScript -->
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous">// Force closing tag</script>
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js">// Force closing tag</script>
<script type="text/javascript">
$("td label").each(function(i, e) {
$(e).text($(e).text().replace(/_/g, " "));
});
$(".description").each(function(i, e) {
$(e).html($(e).html().trim().replace(/\n/g, '<br/>'));
});
</script>
</body>
</html>
</xsl:template>
<xsl:template match="/test-run/test-suite">
<h1>
<xsl:value-of select="./test-run/@name"/>
</h1>
<table class="table table-striped">
<xsl:apply-templates select="./test-suite"/>
</table>
</xsl:template>
<xsl:template match="test-suite">
<tr>
<td>
<xsl:attribute name="class">
<xsl:choose>
<xsl:when test="./@failed > 0">Failed</xsl:when>
<xsl:when test="./@inconclusive > 0">Inconclusive</xsl:when>
<xsl:otherwise>Passed</xsl:otherwise>
</xsl:choose>
</xsl:attribute>
<xsl:attribute name="style">
padding-left: <xsl:value-of select="count(ancestor::test-suite)*15"/>px;
</xsl:attribute>
<xsl:value-of select="./@name"/>
</td>
<td class="counts">
<xsl:value-of select="./@passed"/> passed,
<xsl:value-of select="./@inconclusive"/> inconclusive,
<xsl:value-of select="./@failed"/> failed
</td>
</tr>
<xsl:for-each select="./test-suite">
<xsl:apply-templates select="."/>
</xsl:for-each>
<xsl:for-each select="./test-case">
<xsl:sort select="./properties/property[@name='DocumentationOrder']/@value"/>
<xsl:apply-templates select="."/>
</xsl:for-each>
</xsl:template>
<xsl:template match="test-case">
<tr>
<td colspan="2">
<xsl:attribute name="style">
padding-left: <xsl:value-of select="count(ancestor::test-suite)*15"/>px;
</xsl:attribute>
<label>
<xsl:attribute name="class">
<xsl:value-of select="./@result"/>
</xsl:attribute>
<xsl:value-of select="./@name"/>
</label>
<xsl:if test="./properties/property[@name='Description']">
<div class="description">
<xsl:value-of select="./properties/property[@name='Description']/@value"/>
<xsl:text> </xsl:text>
</div>
</xsl:if>
</td>
</tr>
</xsl:template>
</xsl:stylesheet>