Accelerating Agile

I was at QCON earlier this month. I attended plenty of excellent talks. The talk I found myself nodding in agreement most often was Dan North's Accelerating Agile. In it, he describes his experience with a firm who were capable of producing valuable software in a much shorter time frame than he had previously though possible. The overall message was that the strict Agile Methodologies he was used to practising had been replaced with bare bones Agile Manifesto reasoning and very low ceremony processes. He distilled the processes the firm exhibited down to 12 steps. I pretty much agree with each one, below I have added why I do so based on experience. Presentation slides here.

Step 1: Learn the domain
Ironically, it is the first point that I have the biggest problem with! But first, I must agree it is hugely valuable to have developers become experts in the domain that they working in. However, I think the point is largely lifted from Eric Evan's Domain Driven Design Book; specifically the Crunching Knowledge chapter. Evans, too, speaks of immersion in this sense. It is the basis for building an accurate domain model and enables the tech team to speak the same ubiquitous language as the business experts. My issue with this point is that developer time spent learning the domain should be considered 'on-the-clock'. A developer who is sat at the desk of a business expert, engaged in learning the trade is obviously not designing or coding at this moment - but they have begun the project. The clock has started. And if the point of this talk is that these steps help you produce faster, then this time spent learning (1 or 2 weeks) must be factored into that.


My second issue is that this step is not always possible on every project. Learning the trade of your domain is not also as simple as taking a short course or shadowing a colleague. I have worked on a lot of e-commerce projects in which the end user is the customer - who could be anywhere in the world. This is obviously a hurdle to getting inside the mind of those using your software! Any project in which the end user is not on in the same building will have trouble with this first step and I don't think Dan did enough to address this point.

Step 2: Prioritise risky over valuable
Dan's memorable phrasing on this step was to 'Slay the Dragon's First'. And it makes absolute sense. The traditional Agile (note the capitalisation) approach is to prioritise business value and try to deliver the simplest, and most desired functionality first. However logical this sounds, its possible to run into trouble further down the line if you do not deal with the danger zones. Making sure that you have cracked the parts of the domain which you know could jeopardise the project later is a very wise decision. This point is mirrored in George Fairbanks' book Just Enough Software Architecture; A Risk-Driven Approach. In it, he stresses the importance of evaluating Risk to inform the complexity of individual components. He argues that this should prevent an over-engineered solution, whilst Dan's point is that trying to eliminate Risk enables you to fail faster (which is good!).

Step 3: Plan as far as you need
I really like this step. As much as it appeals to my lazy side, it also supports the way we have been organising ourselves at work recently. Dan backs the idea of having only a small Kanban board and planning no more that a week ahead. This immediately gives you back all the hours spent sprint planning and storyboarding. The prescription is to plan no more than a week in advance, but crucially to review the board often. 'Often' in this context is supposed to be imprecise. Just keep the board up to date and build tacit knowledge to be able to plan smarter. From experience I absolutely agree that a small board works well because it physically won't let you plan too far ahead, nor let you get distracted. It is most effective when used as a communication tool, to serve as a snap shot of what is going on at this moment. A key piece of practical advice was that if a story/piece of work has been on the board for too long is to analyse it a break it down to smaller stories, possibly to be worked on in parallel.

Step 4: Try something different
I cannot claim to be a polyglot programmer, but I agree with the sentiment here. Having more tools at your disposal enables you to solve more problems. Dan spoke explicitly about the languages and styles that you could try out for the sake of expanding your ability to solve problems. Also at QCON, Damian Conway gave an excellent keynote: Fun with Dead Languages. The moral of the story is that learning alternative languages helps us to become better problem solvers. He makes the point that a programming language is not just a tool for repeating operations, or even just communicating ideas between developers. Conway believes that a programming language's primary purpose is to provide a way to think about a problem. If we have more ways to think we have a better chance of solving a particular problem in the best way possible. I realise that not every problem is best solved with C#. But as the .net developer I am I have difficulty slicing the problem any other way. And as such I am currently learning F#!

Step 5: Fire, Aim, Ready
This isn't especially ground breaking, its an opinion that's been around for a while in fact. But nevertheless a brilliant title that communicates the message well. Don't wait to prettify, don't hesitate to show off, don't let uncertainty stop you from demonstrating. Negative possibilities are exactly what you should seek to confirm or disprove as early as you can by getting the product out there. Dan makes the case that this all about feedback from the right people. I believe it also helps maintain the right level of communication with your stakeholders and helps bake in a success story to your project. Sharing the experience of having ideas shot down or realisations occur in cross department meetings reinforces team mentality and keeps everyone's expectations aligned.

Step 6: Build small, separate pieces
This is good practical advice mostly in the name of maintainability. The benefit of having small separate pieces is that they can be replaced easily. Mapping out the geography of your project allows you to also think more clearly about the architecture. I think we generally have to think harder about how not to build monolithic objects/solutions. Its the old argument of easy versus simple. Dan points out that the DRY principle is the enemy of Decoupled. I was uncomfortable with this idea at first but Dan went on to give clear guidelines about when to think this way: by referencing Evans' DDD he used the concept of a Bounded Context. "Let this be the limit to which you should attempt to be DRY".

Step 7: Deploy small, separate pieces
This step is largely dependent on the previous step. In my experience smaller components permit smaller, lower risk, more frequent, quicker deploys. Hopefully resulting in less down time. Dan provides lots of practical DevOps advice here. He suggests never rolling backward... If you are deploying frequently then you are only ever deploying small changes. If you are only every incrementing in small steps then you can never break something so severely that you have to roll back, just fix the code and deploy again! This requires discipline that the whole team must achieve. Dan argues the case for consistent environments and consistent deployment plans that make it easy for new starters to get to grips with and less to remember when problems in deployment arise. To help with this I have recently looked at introducing Vagrant into our development work-flow.

Step 8: Prefer simple over easy
Dan offers some specific examples in this point but the main message is clear: Simple requires thought, but in the end the result is easy to communicate. A simple idea is one that you can hold in your head. 

Step 9: Make the trade-offs
Starting off with a framework might help you get some quick wins but it might be a burden in the long run. My feeling is to always rely on third party libraries to help you get started, but to revise the position on each component where necessary.

Step 10: Share the love
This is all about communication between team members. Make it common practice to pair program and to exploit those 'Aha!' moments for the team's benefit. Dan ventures that code reviews are valuable tools. My opinion is that that they can be replaced by continuous pair programming and pull-requests, but each team is different. 'On-boarding' was a process he described to initiate new starters into the process (if any!) and technologies in play. All the steps described so far I think contribute to making this journey for a beginner as easy as possible.

Step 11: Be ok with “failure”
As I mentioned earlier Failing Fast is a Good Thing! In my experience knowing what won't work can be just as valuable as knowing what will. Dan believes we should be more concerned with Product Delivery not Project Delivery. Progress, he believes, can also be defined as a series of experiments.

Step 12: There are always 12 steps
"Delivering this fast can be addictive." My final thought is that even though it is easy to jump on the SCRUM hate wagon right now, that this is could be the opposite end of the spectrum. Let's just remember that each team and each organisation is different. And as Ward Cunningham mentioned to me after the talk: "You're an engineer aren't you? Engineer your own process!" Wise words.

App Settings are for Variables that Change with the Environment

My opinion is that app settings (or web.config settings) are generally overused. It is usually reasonable for developers to expect that certain values in their code will change in the future. The temptation is to capture all these variables in the app settings. Doing so however, is a mistake.

The only values that belong in application settings are things that will change with the environment, i.e. TEST, STAGE, PRODUCTION.

Everything else, including the things we think (or even know) will change belong directly in source code. I've learned this rule only over time, and I think its reinforced by the YAGNI principle.

Explanation

Sometimes business requirements are delivered in such a way that they barely need translating into a model or domain or even into some syntax. They will usually be very short user stories, and not scored very highly. Examples include:

"As a patient I want a reminder 1 hour before my appointment." 

"As a broker I want the message push interval to be 10 seconds." 

"As a user I want the warning message to read: 'Are you sure you want to delete?'"

The important thing about these stories to notice is the precision they provide. Product owners like precision. They like to know exactly what their product will or won't do. Developers however, like abstraction. They like removing precision and extracting the general rules behind the process. Developers see a rule like '... a reminder 1 hour before my appointment' and take from it the underlying purpose of the story: 'something must happen at a certain time before something else happens'.

There are clearly a number of ways in which sending the reminder could be implemented, depending on the delivery mechanism or format that I won't discuss. But what is clear is that our Product Owner wants the reminder sent 1 hour prior to the appointment. Most developers would look at this requirement and expect it to change. The one certainty about any piece of software is that it will need to change.

The temptation therefore is that the 1 hour (or 60 minutes) should not be hard-coded - if the value is going to change, then lets keep it somewhere where it's easy to change like the app settings, right? Wrong.

There are two reasons for this: Unit Tests and Continuous Delivery.

Unit Tests

The user story in question is business value waiting to be delivered. We want to be absolutely sure that we are delivering that value. If we ship code that doesn't have a passing unit test titled "A_Reminder_Is_Sent_60_minutes_before_appointment" then we haven't succeeded. (Or perhaps closer resembling conceivable objects something like "Reminder_Delivery_Time_Is_60_minutes_before_appointment_time"). We should be sure that no other changes we might make in future will interfere with this piece of logic.

You might argue that we could write a test that reads this value out of our app settings. But then what you would have is an integration test. The reason we want to capture this behaviour in a unit test is to get the answer we want in the shortest feedback loop possible. Also, we don't want to have to maintain a duplicate app settings file in our unit test project.

Continuous Delivery

The practice of automated deployment is becoming more common. As Scott Hanselman says, "If You're Using XCOPY, You're Doing it Wrong. I would take this further and say that if you are still RDC'ing onto the live server to edit a config setting (like changing 60 minute reminders to 120 minute reminders) then you're also doing it wrong. This means the ONLY way to deploy new code (or changed code) is to use an automatic task, and never tamper with the live servers. In a lot of cases developers do not even have access to the live servers.

So now that our app settings file is no longer accessible, it is certainly not easier to change. If we want to change our reminders from 1 hour to 2 hours we have to do so locally, commit and deploy.

Not forgetting our unit test of course - if we made our change in code and ran our tests we should expect a failure. This is important. We have changed the rules of the system and should expect to have to update our unit test accordingly. The new rules are then captured and protect our logic against future unexpected changes.

Environments

So when are App Settings useful? The purpose for app settings in my opinion is for values that change with the target environment. Connection strings are a perfect example of this. In a TEST environment we would expect to talk to a different database than a PRODUCTION environment.

This is where app settings and Config Transformations come in very handy. One rule to decide is this: If your app setting is the same in every target environment, then it doesn't need to be an app setting.

What about Code Reuse?

Keeping values in one place is very useful. But it doesn't mean they belong in the app settings. Globally accessible static classes with Constant field variables is a perfectly acceptable way of maintaining consistency and adhering to the DRY principle. 

When App Settings become a Feature

If we begin to update changing values on a regular basis then the overhead of updating unit tests and deploying can become a burden. At this point we have learned that the ability to update such settings needs to become part of the application. A discussion should be had with the Product Owner over the time spent redeploying versus time invested in providing a new feature to allow users to update values without requiring developer time.

Conclusion

App settings are a tempting place to put variables that we expect to change. However, it is more valuable to 'hard code' such values. This is so that we can make valid assertions against the business rules 'at present'. It is in our interest to break a test if this value changes, and know about it as soon as possible. We also know that we should be deploying changes exactly the same way - whether the changes are in the app settings or source code. This means a change to the app settings file is no more difficult or time consuming than it is to change the source code of our system. Target environment variables are the only valid use-case for app settings. This is to allow such values to vary (transform) between environments. App settings should only exist if the value changes between environments.