Specifying Environment Variables at Deploy Time not at Build Time

Over a month ago I posted an article in which I revelled in the ability to create a publish package in one step then deploy it in another. Last week I discovered a limitation to that process, as it was, and had to find a way to reconfigure it. This post is about that challenge and the conclusions I've drawn from it.

The difficulty with the process that I described in that article was that it is largely dependent on Configuration Transformations. These have been around for a while now and do a great job of specifying the environment variables with respect to the build configuration, normally Debug or Release. But that is exactly the problem (and my realisation):  Config Transforms blur the boundary between Building and Deploying.

When I'm compiling my application I want to be thinking about the build, about optimisation, about references and dependencies. So unless I'm developing locally and debugging I usually want to build in Release mode, as its probably going to get deployed for testing or production. I might be compiling following performance enhancements - I don't want to have to concern myself with the target environment's connection strings when I'm trying to make my app go faster. I always want to build in Release mode. But with config transformations that are tightly coupled to my build configuration I'm stuck with a built package with production connections strings - I can't performance test against that!

Build configurations are necessary, but I don't want to have to create a new one just so that I can apply a different Config Transformation at Build Time. Nor do I want to specify the Publish Profile at build time either. I want to separate Build configuration and Environment configuration concerns. What I want to do is specify the Environment settings at Deploy Time, after my deploy package has been built.

Thankfully, there is a way to do this. But its much more convoluted then adding a new config transformation file to your project. It seems Microsoft have unfortunately made it easier to do things the wrong way. MS have made it easier to deploy from the desktop, but in an enterprise this is usually not the case. Perhaps the new Publish Profile is more about making it easier to deploy to Azure?

The 'Package' MSBuild target that is only available for Web projects generates a directory containing a zipped up deployable package containing all the assets and dlls required, a MyProject.deploy.cmd file that is a friendly wrapper around the msdeploy.exe and, crucially for us, a MyProject.SetParamters.xml. The xml file gives us the ability to modify the contents of the package at Deploy Time (when we call the .cmd). This is what we will be editing:


The xml file can be manually edited before deploying, but this is rarely practical. The IIS App Name and any connection strings are pulled into this file by default. The connection strings are replaced at deploy time with the actual values found in whatever the result of your configuration transformation was. It is also possible to configure the IIS App Name in your .csproj file, but then we are again dependent on the build configuration. The way to override these settings when we deploy is to call msdeploy.exe via the MyProject.deploy.cmd and make use of the -setParam flag. The syntax of which is rather cumbersome:


The second problem is this: what if you want to specify Application Settings (appSettings) in your web.config? As stated, IIS App Name and Connections strings are all that you get to override out-of-the-box. To be able to configure an App Setting, you must add a new file to the root of your web project: parameters.xml. This file needs to contain any App Settings you want to specify when you deploy. Follow this article for a more detailed explanation of the syntax of a parameter.xml. Essentially, we must specify the XPath to the setting and provide an alias to it so we can refer to it in the generated MyProject.SetParameters.xml that we will go on to edit at Deploy Time:


Now, when we create the package at Build Time there is another parameter specified in our MyProject.SetParameters.xml generated along with the package. Like so:


We then apply the same overriding "-setParam" flag when calling the MyProject.deploy.cmd at Deploy Time to configure the "My Service Url" however we want for out target environment. The final call that overrides the parameters is shown below. I've parameterised the call so that I can use the Team City UI to run a custom build and override the parameters without the chance of missing a closing quote etc.

The added bonus of this configuration is that if you are manually deploying a built package by importing it into an IIS website and using the GUI, you are then prompted to provide the settings. This is useful when only certain members of IT are permitted to know the production connection strings or other sensitive information.

However, should you not be in an enterprise with such tight restrictions, then it might be preferable to store the various environment settings in source control with the project. The benefit of this is that there is less chance of misconfiguring a deployment with a typo. This could be achieved by using the "-setParamFile" flag when calling the cmd. However, the difficulty is that if your custom SetParameters.xml is in source control then you have to go fetch it out of the deployed package, which is already zipped up by the build step! Perhaps TC's artifacts can assist here...

This feels like a much more appropriate way to configure the deployment workflow: Always compiling the code in the most suitable configuration (Release), then dealing with the deployment configuration when the time comes to Deploy (or re-deploy). The trouble is that the steps shown above are a lot more tedious and error prone in comparison to the Configuration Transforms we are already used to. Even so, I think this is the right solution for the job.

AppHarbor Environment Management

At our place we have been lucky enough to work on some greenfield projects that we chose to host in the cloud with AppHarbor. As it was our first foray into cloud hosting we hosted test and staging environments internally and used AppHarbor only to host our production environments.

Recently, we have moved our all our stage environments into AppHarbor too. So far its been a really good move and we're wondering why we didn't do it sooner.

Benefits

• AppHarbor provides instantly available environments and infrastructure.
• We get the best possible candidate for a staging environment as its an exact a replica of production (even to the extent of load/performance testing).
• Our config settings are in AppHarbor so we no longer have to manage tricky config transformations, leaving us with only one web.config.
• We have been relieved of the time spent configuring each application's continuous integration internally. Configuration time is required on Appharbor but the overhead is not nearly as much and is mostly GUI driven.
• It has made for a very clean continuous deployment workflow, with an extra 'sanity-check' step before deploying to production (instead of just git push).

The one downside we have experienced so far is that we simply have more applications to manage in AppHarbor. This was my initial reason for not wanting to duplicate applications in the name of simplicity. But, all things considered what we have now is much less complex than before. For reference there exists great deal of information from the horses mouth to get started with.

Creating a New Environment In AppHarbor

It sounds strange but AppHarbor does not provide explicit support for multiple environments for your application. It's down to how you configure the different applications in under your account i.e. making use of a convention. AppHarbor simply provides hosting (domain name and server side processing), builds & tests code before deploying and the means to apply infrastructure 'Add-ons'. You just need to create a new application for the same code base when you want a different environment.

Getting your first application up and running is only possible if you are using one of AppHarbor's supported Version Control systems. We're using Git (and GitHub) and follow the convention of having at least two branches for each repository - develop and master. There are continual commits and pushes on the develop branch and we will merge develop into master when we're happy with it. This development workflow suits our staging and production needs also.

We therefore need one application in AppHarbor per Git repository branch. We prefix the application name with the branch it builds from. So we have develop-myapplication and master-myapplication, with respective host names develop-myapplication.apphb.com and master-myapplication.apphb.com. The distinction in the url makes it clear which branch you are looking at. The develop application is our stage environment, the master is the production environment.

The basic model for deploying with AppHarbor is to add it as a remote repo and push directly to it. However, I mentioned we are using Github and we use this to our advantage. A GitHub (GH) repository can be linked to an AppHarbor (AH) application so that pushing to your GH remote origin will automatically trigger another push to your AH repo which then builds and deploys. This saves us having to push to GH and also to AH. However, this can only be configured for one of the branches. The deployment frequency will be much higher on the develop branch, so this is what we set the tracking branch to be in our AH settings.

The fact that we get this very simple 'push to deploy' feature is very beneficial in our stage environment. That it can only be used once is also of benefit because it prevents us pushing to master and triggering a master (production) build accidentally or too eagerly. When we want to deploy master - which is less often than develop - we have an extra 'sanity-check' step which is to push the master branch to our remote AppHarbor repository. This is simply just reverting back to the basic model for the master branch. This is the git command line for pushing to AH.

# only need call the line below once to add appharbor as a remote repo
git remote add AppHarbor https://apphb.com/master-myapplication-repository-url 

# push your master branch to the added appharbor remote repo
git push AppHarbor master

The diagram below shows the lines of communication between Local, GitHub and AppHarbor repositories in each environment.

Trimming Down Environments

Our stage environment has also become our first phase testing environment. In other words it is now playing the role of both TEST and PRE-PROD environments. This was also an initial deterrent but it has actually not brought any pain (yet). The classical model is great and well worth investing in if you have an appropriately complex application and domain. Having unique environments for each purpose (TEST, QA, UAT, STAGE etc) takes time to configure (not to mention getting the resources). Also considering we don't yet have any integration tests or formal QA people we reasoned that we could simply do without the extra environments.

So now having done away with TEST we just have STAGE and PROD and its working just fine. Testing is performed by developers and stakeholders on our AppHarbor hosted STAGE environment (develop-myapplication). If everyone is happy with changes we deploy to Production (master-myapplication).

Instead, this model has given us the freedom to quickly generate environments for feature branches. If we need to demonstrate, test or stage a feature branch we create a new application and prefix the application name with the name of the feature so it is available with a distinct url e.g. coolnewfeature-myapplication.apphb.com. So we have moved from a 'vertical' environment stack (TEST > QA > STAGE > PROD) to having more 'horizontal' feature driven environments.

A Proper Staging Environment

The point of a staging environment is to be identical to live, we all know why this is a good idea. We have already found bugs that we would not have found until deploying to production. A good example is that the default AuthorizeAttribute does not work on appharbor - you need to use a custom attribute. Whilst I acknowledge this is not a ringing endorsement of AppHarbor it is a good example of why a proper staging environment is important. We also get the environment made instantly available to us - no more hassling IT Ops for extra boxes or getting that internal DNS updated.

Your infrastructure add-ons don't need to be bulked up to the same scale as your production environments. E.g. we have a stage enviroment with a free 20mb database, but production has 10gb which costs a few bob. However, should you want to performance test your in stage environment just assign it the same number of web-workers make sure the both apps are hosted in the same region. You can take advantage of the Blitz Add-on that can tell you how well your app can cope with traffic surges.

Creating a stage environment in the cloud does mean that its publicly available. If this is not in your interest there is brief discussion here around how you might want to restrict access to your public staging environment.

Application Settings

AppHarbor lets you manage Configuration Values (the app settings in the web.config) from within AH itself as a simple key-value collection and straight-forward GUI. After each successful build the keys are matched to your web.config <AppSettings> and the value that exists is replaced with the value managed in AppHarbor - or added if no key is found. Then the code is deployed with the transformed config. This means you no longer need to maintain web.config transformation files like web.release.config. Whilst AppHarbor does in fact support config transformations if a web.release.config file is found, I have found it easier to take these files out of the picture altogether.

Some would argue that the transform files belong in source control but I can do without the burden of configuring the tricky XML transformation syntax, even though Appharbor kindly provides a tool to test your config tranforms. It is important to note that config transformations only works for the <AppSettings> in the web.config and not <ConnectionStrings>. So if you have a connection string that you want managed by AppHarbor put it in the App Settings instead.

What's also great is that any add-ons that require app settings or connection strings such as Databases or Queue systems auto-magically populate your AppHarbor app settings collection with the necessary key-value pair. The keys are constant between applications so you can add RavenDB to your develop and master applications and never have to touch your local web.config! We happen you use our Stage RavenDB instance to develop with locally so we do copy the connection string value into our into our web.config app settings. This could be considered bad practice as you would probably be better off having a separate DB instance to develop with but for our small team it was just the easier option.

App Settings are applied after every successful build and then deployed. If you change a config setting you need to redeploy your application for it to take affect - there is no hoping onto the box to adjust the web.config! This sounds scary but as I've blogged before - config settings should only be used for environment variables and not for things that remain constant between environments (such adhoc text or prices). With this in mind AppHarbor helps to keep what should be a constant a constant and not a config setting that isn't changing between environments. If your constants change you redeploy and not concern yourself worrying about whether your app settings have been taken affect. Generally your app settings won't change all that often if they are being used correctly. For extra reading here's good discussion about how to keep sensitive data config settings private.

Team City

We are still using good old Team City for our first line of defence by building the develop branch after each push and running all unit tests. AppHarbor does this too but we get better notifications if anything goes wrong and also unit test coverage statistics.

There will be an issue if you are using Team City 7 as your internal nuget package server and it is running on an internally hosted system. You need the nuget server of TC to be externally visible to AppHarbor so it can can download packages with your build. You can't just spin up TeamCity on AppHarbor so you're only option is put TC on a publicly visible IP and be sure to authorize all nuget access. Or open source it and get it on Code Better's TC!

Conclusion

There are plenty of reasons why we should be using cloud hosting, we had the luxury of not having to integrate with any legacy systems or databases behind a firewall. Having taken advantage of this fact I would encourage teams to break free from internal infrastructure hassles and move into instant Continuous Integration with AppHarbor, or any integrated cloud hosting provider.

App Settings are for Variables that Change with the Environment

My opinion is that app settings (or web.config settings) are generally overused. It is usually reasonable for developers to expect that certain values in their code will change in the future. The temptation is to capture all these variables in the app settings. Doing so however, is a mistake.

The only values that belong in application settings are things that will change with the environment, i.e. TEST, STAGE, PRODUCTION.

Everything else, including the things we think (or even know) will change belong directly in source code. I've learned this rule only over time, and I think its reinforced by the YAGNI principle.

Explanation

Sometimes business requirements are delivered in such a way that they barely need translating into a model or domain or even into some syntax. They will usually be very short user stories, and not scored very highly. Examples include:

"As a patient I want a reminder 1 hour before my appointment." 

"As a broker I want the message push interval to be 10 seconds." 

"As a user I want the warning message to read: 'Are you sure you want to delete?'"

The important thing about these stories to notice is the precision they provide. Product owners like precision. They like to know exactly what their product will or won't do. Developers however, like abstraction. They like removing precision and extracting the general rules behind the process. Developers see a rule like '... a reminder 1 hour before my appointment' and take from it the underlying purpose of the story: 'something must happen at a certain time before something else happens'.

There are clearly a number of ways in which sending the reminder could be implemented, depending on the delivery mechanism or format that I won't discuss. But what is clear is that our Product Owner wants the reminder sent 1 hour prior to the appointment. Most developers would look at this requirement and expect it to change. The one certainty about any piece of software is that it will need to change.

The temptation therefore is that the 1 hour (or 60 minutes) should not be hard-coded - if the value is going to change, then lets keep it somewhere where it's easy to change like the app settings, right? Wrong.

There are two reasons for this: Unit Tests and Continuous Delivery.

Unit Tests

The user story in question is business value waiting to be delivered. We want to be absolutely sure that we are delivering that value. If we ship code that doesn't have a passing unit test titled "A_Reminder_Is_Sent_60_minutes_before_appointment" then we haven't succeeded. (Or perhaps closer resembling conceivable objects something like "Reminder_Delivery_Time_Is_60_minutes_before_appointment_time"). We should be sure that no other changes we might make in future will interfere with this piece of logic.

You might argue that we could write a test that reads this value out of our app settings. But then what you would have is an integration test. The reason we want to capture this behaviour in a unit test is to get the answer we want in the shortest feedback loop possible. Also, we don't want to have to maintain a duplicate app settings file in our unit test project.

Continuous Delivery

The practice of automated deployment is becoming more common. As Scott Hanselman says, "If You're Using XCOPY, You're Doing it Wrong. I would take this further and say that if you are still RDC'ing onto the live server to edit a config setting (like changing 60 minute reminders to 120 minute reminders) then you're also doing it wrong. This means the ONLY way to deploy new code (or changed code) is to use an automatic task, and never tamper with the live servers. In a lot of cases developers do not even have access to the live servers.

So now that our app settings file is no longer accessible, it is certainly not easier to change. If we want to change our reminders from 1 hour to 2 hours we have to do so locally, commit and deploy.

Not forgetting our unit test of course - if we made our change in code and ran our tests we should expect a failure. This is important. We have changed the rules of the system and should expect to have to update our unit test accordingly. The new rules are then captured and protect our logic against future unexpected changes.

Environments

So when are App Settings useful? The purpose for app settings in my opinion is for values that change with the target environment. Connection strings are a perfect example of this. In a TEST environment we would expect to talk to a different database than a PRODUCTION environment.

This is where app settings and Config Transformations come in very handy. One rule to decide is this: If your app setting is the same in every target environment, then it doesn't need to be an app setting.

What about Code Reuse?

Keeping values in one place is very useful. But it doesn't mean they belong in the app settings. Globally accessible static classes with Constant field variables is a perfectly acceptable way of maintaining consistency and adhering to the DRY principle. 

When App Settings become a Feature

If we begin to update changing values on a regular basis then the overhead of updating unit tests and deploying can become a burden. At this point we have learned that the ability to update such settings needs to become part of the application. A discussion should be had with the Product Owner over the time spent redeploying versus time invested in providing a new feature to allow users to update values without requiring developer time.

Conclusion

App settings are a tempting place to put variables that we expect to change. However, it is more valuable to 'hard code' such values. This is so that we can make valid assertions against the business rules 'at present'. It is in our interest to break a test if this value changes, and know about it as soon as possible. We also know that we should be deploying changes exactly the same way - whether the changes are in the app settings or source code. This means a change to the app settings file is no more difficult or time consuming than it is to change the source code of our system. Target environment variables are the only valid use-case for app settings. This is to allow such values to vary (transform) between environments. App settings should only exist if the value changes between environments.