Why Functional Programming?

I was in attendance at FPDays2013 to hear Bodil Stokke's talk: Programming, Only Better. I revisited this talk recently in order to put together the case for functional programming and F# to my colleagues. The first part of my presentation is largely taken from Bodil's talk, so this is basically a transcript with a couple of points from different sources. Primarily Scott Wlaschin's F Sharp for Fun and Profit.

Programming Techniques

We strive to make it simple to create programs. We use technology, tools and techniques to make programs better and to make better quality programs. The most basic tool we have is our own reasoning. Other techniques include testing, pair programming & code reviews. We'll explore 2 of things.

Reasoning - We reason about code when we are trying to infer what it will do. We reason to better understand what is going on. Debugging is the most tangible form of reasoning. Typically, we are following the code path and understanding the flow of data. In every program we write we have to reason about what the code is doing. In its most basic form, we are walking through the program in our minds.

Testing - while we can be sure that programs are always reasoned about - they are certainly not always tested. That is because testing is useful, but limited. Testing can only show you the presence of bugs, but never their absence. TDD is based around this fact. The Red; Green; Refactor; process highlights the presence of a bug as the code is not yet written to perform this task - so the test fails. Writing the code to make the test pass is the process of removing a bug. Just because a system has 100% code coverage it cannot be said to be 'bug free'.

Programming Difficulties

Programming is difficult largely because programs grow to be complex. Code volume also makes it difficult to maintain programs, but volume and complexity are not necessarily linked. Its very easy to create an overly complex program without applying thought to its design. Simplicity is much harder to achieve. We'll look at one source of complexity:

State - object oriented languages idealise encapsulated state. OO languages are useful precisely because we use more code to manipulate an object's state. So state changes with time - known as entropy in the mathematical world. Most of the time, we are intentially changing state in a useful way. But state is also permitted to be changed by something outside our control (unless we are very careful). The code example below highlights what it means to have an object enter a 'bad state' and not have that communicated back to the calling code.


State Spoils Testing - how can you test an object in all its possible states? You can only test for the states you think the object might enter into. State increases exponentially, so the more potential state you create, the lower the meaning of your tests!

State Spoils Reasoning - state allows for unpredictability. You can never be sure that you will get the same output with the same input. An objects state can change, so how can you be sure of it at any given time? This becomes even worse when you consider concurrent programs. It is much harder to describe the control flow of a program concurrent stateful program. In OO you have to use Locks to be able to protect against unwanted state change. This is cumbersome, inefficient and still imperfect.

Functional Languages

What can they do to help? They idealise Referential Transparency. RT means dealing with inputs and outputs. A function just maps an input to an output.

Predictability - is guaranteed in a functional language because you will always get the same output for the same input. It can be thought of as nothing more than a glorified switch statement!

No Side Effects - as there is not state! There are no variables in pure functional languages, only values. Once a value is created, it cannot be change. Values can be created local to functions, but that function is also immutable and cannot be changed. State is instead managed by passing values around as inputs and outputs. When you think about it, it doesn't make sense to want to change a value you have set. What makes sense is to simply set a new value. This is essentially what functions do.

State

What happens when we have no state? Reasoning becomes much simpler! Predictability makes programs easier to understand. Tests also becomes more meaningful for the same reason. However, your tests are still unable to tell you that bugs do not exist, and they are only meaningful for the inputs you are testing with.

Optimizing - once you realise that output will never change for a given input, you understand that it no longer matters what order functions are called in. This allows you to perform functions in parallel, such as map reduce. Lazy Evaluation also becomes possible, because you know it doesn't matter when you call a function - the result will always be the same for the give input. Caching is more widely permitted for the same reason. Also, as there are no side effects, it means that you can be sure nothing is going be changed from underneath you! This makes concurrent programs easier to manage, as you don't need to worry about race conditions and lock objects.

Conclusion

The techniques we use to help delivery quality software are undermined by state. Functional Languages remove state from the programming paradigm and in doing so prevent an entire class of bugs from ever entering our code. Functional programming makes our code easier to understand because of its predictability and has greater potential to optimize.

5 Reasons to Keep Software Projects Small

Thoughts of system architecture have been cropping up recently so I wanted to layout some reasons why keeping software projects small is a good thing. A small software project is one that has one main purpose and few features. A small software project can be described to a new person in one sentence: "that's the product catalogue repo, it hosts a public list of products for our clients and has an authenticated admin console." Here's why you should Think Small.

5. Code Ownership

Let's start with the obvious one, maintainability. People, developers included, generally take more care of their own stuff than others'. Software is no different. It's a side effect of keeping projects small, but a very good one: Those projects written by one or two developers will be much better looked after and cared for than projects with multiple owners. I know this because I have worked on small projects written by a few people and large projects written by many. It is less likely for a developer to feel engaged when they are 'in someone else's code'. Motivation is lacking and code craftsmanship suffers when devs are contributing to a project that they do not feel ownership of. Their best work is rarely produced and what does get committed can often contradict other areas of the system in terms of coding style or naming conventions. Over time, unless significant effort is made to enforce standards and motivate developers to take ownership then these changes turn big projects into a Big Ball of Mud. Once that happens, no one wants to work on the project and progress halts. TDD and clean code can help, but they are only preventative measures.

With smaller projects, its much harder to end up in a BBoM situation. There simply isn't that much code in the system. Projects are kept from reaching the point where no one is confident in making changes because the project responsibilities are few and its purpose well defined. You cannot accrue technical debt in a project with no room to take short cuts. And if there are, then they are easy to spot. Its easy to answer questions organisation questions as the need for a clearly named folder structure is minimized.

4. Point Person

That is the go-to-guy when questions are asked or when changes are needed. When there are multiple point-people it usually requires a meeting just to make a decision, or worse, leads to deflection of responsibility. When the one point person knows that they are the only owner, they are driven to have detailed understanding of the system's behaviour (as chances are they have written it). The knowledge should be detailed but easy to pass onto another dev, should the original author move on. Failing that, small systems permit re-writes much more readily that large ones. As for the business, its not the nuances of the particular framework that is valuable, but the role that the system plays instead. This information is more transferable with smaller projects.

3. Quality Requirements

When describing a small system, its much easier to gather the quality requirements. These are things like performance, uptime, security, accessibility, scalability etc. The audience or clients of a small system are usually more focused which lets us make design calls in the code. What is the expected throughput of the url? How soon after event x must the event data be query-able? Will we need to serve content to the other side of the world? Mobile?! When the developer knows the extent of the system's purpose they can make judgements about these kind of questions more easily.

2. (Re)Distribution

This point is about how the system can be accessed. Keeping systems small means that they can be consumed more easily in different formats. Obviously, the most common mechanism of communication is over HTTP. The reason it is so popular is because of the high number of client libraries that support the http request response model with JSON data binding, meaning language interoperability. Http also provides built in functionality for authorisation, caching and security. But the benefits of http are not exclusive to small projects. What small projects are suited to is packaging. A packaged product or library is focussed on doing one thing really well. And when a project has few responsibilities then it can be packaged easily. This becomes a virtuous circle when you consider the motivation for packaging. It makes code shareable and more re-usable. Modern systems are a composition of frameworks and libraries brought together in a particular way to provide unique functionality. Package managing tools permit rapid development by providing access to thousands of small systems that have been packaged, used and tested by others.

1. Innovation

Developers like learning new tricks and using new tools. When what the business is asking for can be delivered in small projects it benefits the business and the developer. If a developer has heard about and new framework or library they would like to try out, then they are going to be twice as motivated to take ownership of the system and enjoy the development experience when they get the chance to try it out. It might take slightly longer as learning occurs, but the development team wins as a result. New knowledge is shared and as such the team becomes more experienced. When some work arrives that has to be out the door by Friday, the team that has the most varied experience are more likely to already know the best tool for the job - and deliver. Aiming for small projects means that a team members acquire new skills more frequently as there are more opportunities to try out something new. Innovation and discovery in IT can be a driver for success in digital industries, thinking small might just give your team the advantage.

Comparing Test Spec Frameworks

Up until now, I've been using a custom test spec framework geared towards helping write unit tests with a BDD style description: Given; When; Then, described in a previous post. The framework was built on top of NUnit, Moq, and Structuremap.AutoMocking nuget packages to make assertions, create mock objects and resolve the test target dependencies with mock objects respectively. There is no problem with the framework as it is, but I was seeking a way to reduce the language 'noise' that was incurred - a new method for each Given, When and Then step is cumbersome and gets tedious after a while.

This lead me to discover the Machine.Specifications (MSpec) framework. It allows you to write tests in the same BDD style, but does not require a full method for each step. Instead, you can just write a lambda function. This can reduce any one test down to a single line, by making sure that you stick to one assertion per test. It introduces some new types which enter the terminology: Establish,  Because and It. These are direct replacements for Given, When and Then, as the semantics are identical.

But what about the automocking feature of the old framework? This can be fulfilled by another package designed to work with MSpec: Machine.Fakes. This, too, automatically resolves any dependencies of the target with mocked objects, provided by your mocking framework of choice. I used the Machine.Fakes.Moq nuget package to do this, but since you also use the Fakes framework to get access to the mocked dependencies it has created for you, and also generate additional mocks you may need the fact that I'm using Moq becomes irrelevant - it is completely abstracted away. To get access to an mocked dependency you use The<T>, and to create a mock you use An<T>. I also like the nice feature that gives you a list of mocked objects: Some<T>.

That's the language hooks and mocks taken care of, but in the effort to reduce the syntax bloat as much as possible I brought in the Fluent.Assertions nuget package. This library is leaner than NUnit because it lets me make assertions using a set of extension methods, e.g. .ShouldEqual() or .ShouldBeTrue(), so I don't have to begin an assertion statement with NUnit, I can start with the object I want to make the assertion on. It also puts the final nail in the coffin for NUnit, as I have already done away with the [SetUp], [TearDown] or [Test] attributes.

Let's compare two test classes written for the same target class. The Target is shown below, followed by the before and after test classes.

Target
Before
After
As you will notice, the 'After' test class contains over 50% fewer lines. The tests are now more readable because there is less language noise to cut through when reviewing the code. The important statements jump right out, and they also take less time to write. The method bodies and calls to base methods in the Given and When steps have gone, replaced with single line statements. The MFakes framework has helped make each line that refers to a mock more concise be removing the need to call .Object. The Fluent.Assertions library has made the assertions simpler too.

There are a couple of downsides that I have discovered not shown in the example that I think are worth mentioning. Firstly, Tests (It field variables) cannot be run from a base class. They can be declared, but go undetected by the build runner. This means that you will have to duplicate any tests that are based on statements that come before a branch at an if statement, assuming you want to test both branches of the method. Whereas previously you could declare a test in an abstract class and have it executed in each text fixture class that inherits from it.

The other negative is that you cannot instantiate a mock of your own using the An<T> from MFakes at field level. You have to be inside the body of the Establish lambda/anonymous delegate before you can assign a value to the variable defined at field level. You aren't told about this error until run time which is annoying to begin with, but being forced to declare your mock variables on one line and assigning them on another doubles the amount of code needed to achieve what that the previous framework could do. So it goes against the purpose of this exercise somewhat.

All in all, I think the positives outweigh the negatives. The biggest factor being that there is less manual syntax to be typed to generate a test. Which is one less reason not to do TDD, which can only be a good thing!

New Tricks from an Old Pattern

Its great when you can still be surprised by a pattern you think you already know inside-out. I was pleased today to think about the BDD Unit Testing Style 'pattern' in a new way. Here's my previous post on the subject that will give you the necessary background for the rest of this post.

The pattern utilises object inheritance to make it powerful and descriptive. To illustrate, we must start with an example. The code below contains a method that tests two conditions and tries to return an object. If both conditions fail then the method throws an exception. It is worth noting that the sample class is not a good example of OO design, but rather a simplified case. The important point is that there is a path through the method that will cause an exception to be thrown.

Using the BDD Style Unit Testing pattern makes it pretty easy to test the first two conditions and the object that is returned in each case. Note the use of an abstract base class which provides us with the Result variable to hold our return object and importantly encapsulates away the call to the When() method. The Class is called WhenIGetScoredPlayer, meaning we should expect to find the call to GetScoredPlayer() on the Target inside of the overriden When() method (see the preceding article if you aren't following).

This means we don't have to repeat this call to the same target method in any of the classes that inherit from this base class. It makes the classes easy to navigate and the test output becomes straight forward as coherent sentences are produced: "When I Get Scored Player, Given The Player Has a Score Greater Than 8, Then the Result is a High Score Player". This is useful information when reviewing tests results.

However, you will now notice that its not so simple in the third path of this method. That's because in the case when a player scores below 2 an exception is thrown. If I inherit from the abstract WhenIGetScoredPlayer class and setup the player's score accordingly, then the exception will be thrown too early. That's because in the base GivenA<T> class a call to the virtual When() follows a call to the virtual Given() in a method tagged with [SetUp], to make sure they happen in sequence as part of the framework - its useful for us to have things happen in this order.

I could start from scratch and perform all the setup in a new class. Making the call to GetScoredPlayer() inside of a [Test] method instead of as part of [SetUp]. But I don't want to duplicate unnecessarily. This is when I thought about the old pattern in a new way - using an override to postpone the call instead of starting all over again.

What I did was to still implement the base class with the appropriately named GivenThePlayerHasAScoreLowerThan2. The trick is to simply override the base When() and leave it empty. This prevents it from calling the GetScoredPlayer() during the [SetUp] phase of the test. It can now occur in the [Test] method, wrapped in an NUnit assertion that the correct type of Exception is thrown. Producing the desired output without unnecessary duplication: "When I Get Scored Player, Given The Player Has a Score Lower Than 2, A Score Too Low Exception Is Thrown".

Automock with StructureMap and Moq

This is a guide of how to use AutoMocking with StructureMap and Moq in conjunction with cucumber BDD style unit testing.

Taking the BDD GIVEN, WHEN & THEN Scenario approach to unit testing has a number of benefits. I find it makes tests easier to maintain as you stick to one assertion per test. The test scenarios are well partitioned making them cleaner than straight forward Arrange Act Assert style. It also produces more readable output of test results.

Automocking is a time-saving feature of StructureMap that allows you to easily instantiate the Class Under Test (the class that will be used in the GIVEN statement of the Scenario) by automatically resolving the constructor dependencies of the class with Mocked out instances. Test classes therefore become decoupled from the constructor(s) of the Class Under Test. Meaning that you'll get a test that fails if you change a constructor instead of a test that won't build. This is a debatable point but can be considered an advantage in TDD - you get feedback from your tests as opposed to the compiler, as it should be.

By installing the StructureMap.AutoMocking and Moq packages you get out-of-the-box support for AutoMocking. Given that I regularly use StructureMap and Moq as my IOC and mocking framework of choice respectively, it makes sense to take advantage.

An Example

In the example class to test there are two dependencies that will need to be automatically mocked, and a method that should be tested.


The base class provides us with the GIVEN & WHEN syntax. It uses the MoqAutoMocker provided by the StrutureMap.Automocking library to mock any dependencies on our class's constructor. It provides a handle to access any Mocked dependency that was injected when the class was instantiated. Also provided is a shortcut to Moq's Verify method as well as some syntactic sugar that lets you use [Then] instead of [Test] to round off the BDD style.


The example unit test class implements the abstract class and is named according to the method being tested. There should be one class per method being tested - i.e. one WHEN statement per scenario. If there is another public method it should be tested in a new class that also implements the base class. This practice is what makes the test output easy to digest.

References

• StructureMap Automocker Documentation
• StackOverflow question with handy Moq tip to access underlying Mock
• Scott Millet's post on BDD Unit Testing

App Settings are for Variables that Change with the Environment

My opinion is that app settings (or web.config settings) are generally overused. It is usually reasonable for developers to expect that certain values in their code will change in the future. The temptation is to capture all these variables in the app settings. Doing so however, is a mistake.

The only values that belong in application settings are things that will change with the environment, i.e. TEST, STAGE, PRODUCTION.

Everything else, including the things we think (or even know) will change belong directly in source code. I've learned this rule only over time, and I think its reinforced by the YAGNI principle.

Explanation

Sometimes business requirements are delivered in such a way that they barely need translating into a model or domain or even into some syntax. They will usually be very short user stories, and not scored very highly. Examples include:

"As a patient I want a reminder 1 hour before my appointment." 

"As a broker I want the message push interval to be 10 seconds." 

"As a user I want the warning message to read: 'Are you sure you want to delete?'"

The important thing about these stories to notice is the precision they provide. Product owners like precision. They like to know exactly what their product will or won't do. Developers however, like abstraction. They like removing precision and extracting the general rules behind the process. Developers see a rule like '... a reminder 1 hour before my appointment' and take from it the underlying purpose of the story: 'something must happen at a certain time before something else happens'.

There are clearly a number of ways in which sending the reminder could be implemented, depending on the delivery mechanism or format that I won't discuss. But what is clear is that our Product Owner wants the reminder sent 1 hour prior to the appointment. Most developers would look at this requirement and expect it to change. The one certainty about any piece of software is that it will need to change.

The temptation therefore is that the 1 hour (or 60 minutes) should not be hard-coded - if the value is going to change, then lets keep it somewhere where it's easy to change like the app settings, right? Wrong.

There are two reasons for this: Unit Tests and Continuous Delivery.

Unit Tests

The user story in question is business value waiting to be delivered. We want to be absolutely sure that we are delivering that value. If we ship code that doesn't have a passing unit test titled "A_Reminder_Is_Sent_60_minutes_before_appointment" then we haven't succeeded. (Or perhaps closer resembling conceivable objects something like "Reminder_Delivery_Time_Is_60_minutes_before_appointment_time"). We should be sure that no other changes we might make in future will interfere with this piece of logic.

You might argue that we could write a test that reads this value out of our app settings. But then what you would have is an integration test. The reason we want to capture this behaviour in a unit test is to get the answer we want in the shortest feedback loop possible. Also, we don't want to have to maintain a duplicate app settings file in our unit test project.

Continuous Delivery

The practice of automated deployment is becoming more common. As Scott Hanselman says, "If You're Using XCOPY, You're Doing it Wrong. I would take this further and say that if you are still RDC'ing onto the live server to edit a config setting (like changing 60 minute reminders to 120 minute reminders) then you're also doing it wrong. This means the ONLY way to deploy new code (or changed code) is to use an automatic task, and never tamper with the live servers. In a lot of cases developers do not even have access to the live servers.

So now that our app settings file is no longer accessible, it is certainly not easier to change. If we want to change our reminders from 1 hour to 2 hours we have to do so locally, commit and deploy.

Not forgetting our unit test of course - if we made our change in code and ran our tests we should expect a failure. This is important. We have changed the rules of the system and should expect to have to update our unit test accordingly. The new rules are then captured and protect our logic against future unexpected changes.

Environments

So when are App Settings useful? The purpose for app settings in my opinion is for values that change with the target environment. Connection strings are a perfect example of this. In a TEST environment we would expect to talk to a different database than a PRODUCTION environment.

This is where app settings and Config Transformations come in very handy. One rule to decide is this: If your app setting is the same in every target environment, then it doesn't need to be an app setting.

What about Code Reuse?

Keeping values in one place is very useful. But it doesn't mean they belong in the app settings. Globally accessible static classes with Constant field variables is a perfectly acceptable way of maintaining consistency and adhering to the DRY principle. 

When App Settings become a Feature

If we begin to update changing values on a regular basis then the overhead of updating unit tests and deploying can become a burden. At this point we have learned that the ability to update such settings needs to become part of the application. A discussion should be had with the Product Owner over the time spent redeploying versus time invested in providing a new feature to allow users to update values without requiring developer time.

Conclusion

App settings are a tempting place to put variables that we expect to change. However, it is more valuable to 'hard code' such values. This is so that we can make valid assertions against the business rules 'at present'. It is in our interest to break a test if this value changes, and know about it as soon as possible. We also know that we should be deploying changes exactly the same way - whether the changes are in the app settings or source code. This means a change to the app settings file is no more difficult or time consuming than it is to change the source code of our system. Target environment variables are the only valid use-case for app settings. This is to allow such values to vary (transform) between environments. App settings should only exist if the value changes between environments.