Comparing Test Spec Frameworks

Up until now, I've been using a custom test spec framework geared towards helping write unit tests with a BDD style description: Given; When; Then, described in a previous post. The framework was built on top of NUnit, Moq, and Structuremap.AutoMocking nuget packages to make assertions, create mock objects and resolve the test target dependencies with mock objects respectively. There is no problem with the framework as it is, but I was seeking a way to reduce the language 'noise' that was incurred - a new method for each Given, When and Then step is cumbersome and gets tedious after a while.

This lead me to discover the Machine.Specifications (MSpec) framework. It allows you to write tests in the same BDD style, but does not require a full method for each step. Instead, you can just write a lambda function. This can reduce any one test down to a single line, by making sure that you stick to one assertion per test. It introduces some new types which enter the terminology: Establish,  Because and It. These are direct replacements for Given, When and Then, as the semantics are identical.

But what about the automocking feature of the old framework? This can be fulfilled by another package designed to work with MSpec: Machine.Fakes. This, too, automatically resolves any dependencies of the target with mocked objects, provided by your mocking framework of choice. I used the Machine.Fakes.Moq nuget package to do this, but since you also use the Fakes framework to get access to the mocked dependencies it has created for you, and also generate additional mocks you may need the fact that I'm using Moq becomes irrelevant - it is completely abstracted away. To get access to an mocked dependency you use The<T>, and to create a mock you use An<T>. I also like the nice feature that gives you a list of mocked objects: Some<T>.

That's the language hooks and mocks taken care of, but in the effort to reduce the syntax bloat as much as possible I brought in the Fluent.Assertions nuget package. This library is leaner than NUnit because it lets me make assertions using a set of extension methods, e.g. .ShouldEqual() or .ShouldBeTrue(), so I don't have to begin an assertion statement with NUnit, I can start with the object I want to make the assertion on. It also puts the final nail in the coffin for NUnit, as I have already done away with the [SetUp], [TearDown] or [Test] attributes.

Let's compare two test classes written for the same target class. The Target is shown below, followed by the before and after test classes.

Target
Before
After
As you will notice, the 'After' test class contains over 50% fewer lines. The tests are now more readable because there is less language noise to cut through when reviewing the code. The important statements jump right out, and they also take less time to write. The method bodies and calls to base methods in the Given and When steps have gone, replaced with single line statements. The MFakes framework has helped make each line that refers to a mock more concise be removing the need to call .Object. The Fluent.Assertions library has made the assertions simpler too.

There are a couple of downsides that I have discovered not shown in the example that I think are worth mentioning. Firstly, Tests (It field variables) cannot be run from a base class. They can be declared, but go undetected by the build runner. This means that you will have to duplicate any tests that are based on statements that come before a branch at an if statement, assuming you want to test both branches of the method. Whereas previously you could declare a test in an abstract class and have it executed in each text fixture class that inherits from it.

The other negative is that you cannot instantiate a mock of your own using the An<T> from MFakes at field level. You have to be inside the body of the Establish lambda/anonymous delegate before you can assign a value to the variable defined at field level. You aren't told about this error until run time which is annoying to begin with, but being forced to declare your mock variables on one line and assigning them on another doubles the amount of code needed to achieve what that the previous framework could do. So it goes against the purpose of this exercise somewhat.

All in all, I think the positives outweigh the negatives. The biggest factor being that there is less manual syntax to be typed to generate a test. Which is one less reason not to do TDD, which can only be a good thing!