Monday, 27 January 2014

Decoupling your Web Deploy Build Script from Team City with Powershell

Continuing my (inadvertent) series on stream-lining your deployment process this next instalment is about bundling your deployment script in with your deployment package and breaking dependencies between Build and Deployment Configurations.

Previous Topics:
1 - App Settings are for Variables that Change with the Environment
2 - Separate Build and Deploy Configs with Team City and MS Web Deploy
3 - Specifying Environment Variables at Deploy Time not at Build Time

When I last left this topic, environment settings were captured in Team City's Build Parameters. They were applied at deploy time using a deployment script that called the web deploy command line executable. This helps keep sensitive settings out of source control. The bad news is that this deploy script was also maintained in Team City.

The reason this is a problem becomes clear when we need to add or remove a parameter from the configuration. A new parameter name and value must obviously be added to the configuration before it can be used in the deployment. However, we are also required to update the deploy script to apply this parameter during the deployment. What we have is a dependency on the build package on the deploy script. The deploy script cannot be updated long in advance of a new parameter being added, as a deployment in the meantime that uses the new parameter where it is not yet needed will cause the build to fail. We need to move away from a shared deploy script and towards one that is specific to the package it is going to deploy.

The answer here is to bring the deploy script into source control. That way, we can manage the script like any other code artifact with version history and also, crucially, configure the script such that it will apply all the build parameters to the deployment package at the time it should be deployed. We can then add the actual parameter name and value to Team City far in advance of the actual deployment time, as we know our build script will only try to make use of the new parameter when it is executed, whilst parallel deployments can continue unaffected.

A deployment script might look something like this:

MyWebApplication.deploy.cmd /y /M:%TargetDeployUrl% /u:%Username% /p:%Password% /A:Basic -allowUntrusted "-setParam:name='LoggingEmailAddress',value='%LoggingEmailAddress%'" "-setParam:name='ServiceEndPoint',value='%ServiceEndPoint%'"

This contains only two build parameters (environment variables), but you can see that its fiddly enough to warrant version history. All we are doing is calling a command line program, so the easiest thing to do would be to create another .cmd in the same directory. I'm going to use a Powershell script to do the same because it will make it easier to work with a collection of Target Deploy Urls that we shall see later on.

The plan is to capture this call in a powershell script which is then called by our deployment build configuration in Team City. Team City will need to pass the build parameters to the powershell deploy script so that the actual values can be substituted upon execution. The simplest way to do this is with Environment Variables. Enviroment Variables are read from powershell using the $env: prefix. You'll see these in action in the script below. Warning: Do not use periods or dashes in your environment variable names as powershell has trouble escaping these characters - use underscores if necessary. Don't worry about the 'env.' that is prepended to your variable. This does not affect your deploy script.

The powershell script should be added to the root of the solution and committed to source. When the Build Configuration is run (that's build, not deploy see my previous article on separating build and deploy steps) we will need to make sure that this script is 'bundled' in with the deployment package. The deployment package contains the web deploy .cmd program that we need to call, so we should put our script in the same directory as this. To do this, configure an additional Artifact Path in the General Settings of your build configuration in Team City. We'll call our deploy script Deployment.ps1.

The deploy build configuration now has access to this script. The single build step of our deploy configuration just has to invoke Deployment.ps1. The settings shown in the image below were all I needed to enable this. The environment variables do not have to be referenced directly - this is a good thing. They are all accessible by the deploy script throughout the deployment, permitting the script to make use of whatever environment variables it needs.

Lastly, the contents of Deployment.ps1 are shown below. The main reason for using a powershell script is now clear: the ability to deploy to any number of targets. Running through the script, we see that it first defines a function to get the working directory. This is needed in order to reference the web deploy .cmd by its absolute file path. Then, we read in the TargetDeployUrls environment variable - as defined in our deploy configuration in Team City - and split it into an array on the comma character. We then loop over the array, calling the web deploy .cmd once per target. You can see that the long command is split into several lines for readability using a here-string. The line breaks are then removed so a single line operation can be performed. Note where the $env:LoggingEmailAddress and $env:ServiceEndPoint parameters are referenced. The command is executed using the Invoke-Expression alias iex.

The array loop has the advantage of allowing us to deploy to environments with any number of host machines. For example, in our staging environment we have only one server that needs to be deployed to, whilst our production environment has two. This deployment script caters for both these scenarios - all you need to when deploying to production is provide a comma separated list of deploy target urls. However, in its current configuration the script applies the same username and password to all servers. This might not be the case in every environment, but that improvement will have to wait for another day!

Friday, 10 January 2014

Why Functional Programming?

I was in attendance at FPDays2013 to hear Bodil Stokke's talk: Programming, Only Better. I revisited this talk recently in order to put together the case for functional programming and F# to my colleagues. The first part of my presentation is largely taken from Bodil's talk, so this is basically a transcript with a couple of points from different sources. Primarily Scott Wlaschin's F Sharp for Fun and Profit.

Programming Techniques

We strive to make it simple to create programs. We use technology, tools and techniques to make programs better and to make better quality programs. The most basic tool we have is our own reasoning. Other techniques include testing, pair programming & code reviews. We'll explore 2 of things.

Reasoning - We reason about code when we are trying to infer what it will do. We reason to better understand what is going on. Debugging is the most tangible form of reasoning. Typically, we are following the code path and understanding the flow of data. In every program we write we have to reason about what the code is doing. In its most basic form, we are walking through the program in our minds.

Testing - while we can be sure that programs are always reasoned about - they are certainly not always tested. That is because testing is useful, but limited. Testing can only show you the presence of bugs, but never their absence. TDD is based around this fact. The Red; Green; Refactor; process highlights the presence of a bug as the code is not yet written to perform this task - so the test fails. Writing the code to make the test pass is the process of removing a bug. Just because a system has 100% code coverage it cannot be said to be 'bug free'.

Programming Difficulties

Programming is difficult largely because programs grow to be complex. Code volume also makes it difficult to maintain programs, but volume and complexity are not necessarily linked. Its very easy to create an overly complex program without applying thought to its design. Simplicity is much harder to achieve. We'll look at one source of complexity:

State - object oriented languages idealise encapsulated state. OO languages are useful precisely because we use more code to manipulate an object's state. So state changes with time - known as entropy in the mathematical world. Most of the time, we are intentially changing state in a useful way. But state is also permitted to be changed by something outside our control (unless we are very careful). The code example below highlights what it means to have an object enter a 'bad state' and not have that communicated back to the calling code.

State Spoils Testing - how can you test an object in all its possible states? You can only test for the states you think the object might enter into. State increases exponentially, so the more potential state you create, the lower the meaning of your tests!

State Spoils Reasoning - state allows for unpredictability. You can never be sure that you will get the same output with the same input. An objects state can change, so how can you be sure of it at any given time? This becomes even worse when you consider concurrent programs. It is much harder to describe the control flow of a program concurrent stateful program. In OO you have to use Locks to be able to protect against unwanted state change. This is cumbersome, inefficient and still imperfect.

Functional Languages

What can they do to help? They idealise Referential Transparency. RT means dealing with inputs and outputs. A function just maps an input to an output.

Predictability - is guaranteed in a functional language because you will always get the same output for the same input. It can be thought of as nothing more than a glorified switch statement!

No Side Effects - as there is not state! There are no variables in pure functional languages, only values. Once a value is created, it cannot be change. Values can be created local to functions, but that function is also immutable and cannot be changed. State is instead managed by passing values around as inputs and outputs. When you think about it, it doesn't make sense to want to change a value you have set. What makes sense is to simply set a new value. This is essentially what functions do.


What happens when we have no state? Reasoning becomes much simpler! Predictability makes programs easier to understand. Tests also becomes more meaningful for the same reason. However, your tests are still unable to tell you that bugs do not exist, and they are only meaningful for the inputs you are testing with.

Optimizing - once you realise that output will never change for a given input, you understand that it no longer matters what order functions are called in. This allows you to perform functions in parallel, such as map reduce. Lazy Evaluation also becomes possible, because you know it doesn't matter when you call a function - the result will always be the same for the give input. Caching is more widely permitted for the same reason. Also, as there are no side effects, it means that you can be sure nothing is going be changed from underneath you! This makes concurrent programs easier to manage, as you don't need to worry about race conditions and lock objects.


The techniques we use to help delivery quality software are undermined by state. Functional Languages remove state from the programming paradigm and in doing so prevent an entire class of bugs from ever entering our code. Functional programming makes our code easier to understand because of its predictability and has greater potential to optimize.