AppHarbor Environment Management

At our place we have been lucky enough to work on some greenfield projects that we chose to host in the cloud with AppHarbor. As it was our first foray into cloud hosting we hosted test and staging environments internally and used AppHarbor only to host our production environments.

Recently, we have moved our all our stage environments into AppHarbor too. So far its been a really good move and we're wondering why we didn't do it sooner.

Benefits

• AppHarbor provides instantly available environments and infrastructure.
• We get the best possible candidate for a staging environment as its an exact a replica of production (even to the extent of load/performance testing).
• Our config settings are in AppHarbor so we no longer have to manage tricky config transformations, leaving us with only one web.config.
• We have been relieved of the time spent configuring each application's continuous integration internally. Configuration time is required on Appharbor but the overhead is not nearly as much and is mostly GUI driven.
• It has made for a very clean continuous deployment workflow, with an extra 'sanity-check' step before deploying to production (instead of just git push).

The one downside we have experienced so far is that we simply have more applications to manage in AppHarbor. This was my initial reason for not wanting to duplicate applications in the name of simplicity. But, all things considered what we have now is much less complex than before. For reference there exists great deal of information from the horses mouth to get started with.

Creating a New Environment In AppHarbor

It sounds strange but AppHarbor does not provide explicit support for multiple environments for your application. It's down to how you configure the different applications in under your account i.e. making use of a convention. AppHarbor simply provides hosting (domain name and server side processing), builds & tests code before deploying and the means to apply infrastructure 'Add-ons'. You just need to create a new application for the same code base when you want a different environment.

Getting your first application up and running is only possible if you are using one of AppHarbor's supported Version Control systems. We're using Git (and GitHub) and follow the convention of having at least two branches for each repository - develop and master. There are continual commits and pushes on the develop branch and we will merge develop into master when we're happy with it. This development workflow suits our staging and production needs also.

We therefore need one application in AppHarbor per Git repository branch. We prefix the application name with the branch it builds from. So we have develop-myapplication and master-myapplication, with respective host names develop-myapplication.apphb.com and master-myapplication.apphb.com. The distinction in the url makes it clear which branch you are looking at. The develop application is our stage environment, the master is the production environment.

The basic model for deploying with AppHarbor is to add it as a remote repo and push directly to it. However, I mentioned we are using Github and we use this to our advantage. A GitHub (GH) repository can be linked to an AppHarbor (AH) application so that pushing to your GH remote origin will automatically trigger another push to your AH repo which then builds and deploys. This saves us having to push to GH and also to AH. However, this can only be configured for one of the branches. The deployment frequency will be much higher on the develop branch, so this is what we set the tracking branch to be in our AH settings.

The fact that we get this very simple 'push to deploy' feature is very beneficial in our stage environment. That it can only be used once is also of benefit because it prevents us pushing to master and triggering a master (production) build accidentally or too eagerly. When we want to deploy master - which is less often than develop - we have an extra 'sanity-check' step which is to push the master branch to our remote AppHarbor repository. This is simply just reverting back to the basic model for the master branch. This is the git command line for pushing to AH.

# only need call the line below once to add appharbor as a remote repo
git remote add AppHarbor https://apphb.com/master-myapplication-repository-url 

# push your master branch to the added appharbor remote repo
git push AppHarbor master

The diagram below shows the lines of communication between Local, GitHub and AppHarbor repositories in each environment.

Trimming Down Environments

Our stage environment has also become our first phase testing environment. In other words it is now playing the role of both TEST and PRE-PROD environments. This was also an initial deterrent but it has actually not brought any pain (yet). The classical model is great and well worth investing in if you have an appropriately complex application and domain. Having unique environments for each purpose (TEST, QA, UAT, STAGE etc) takes time to configure (not to mention getting the resources). Also considering we don't yet have any integration tests or formal QA people we reasoned that we could simply do without the extra environments.

So now having done away with TEST we just have STAGE and PROD and its working just fine. Testing is performed by developers and stakeholders on our AppHarbor hosted STAGE environment (develop-myapplication). If everyone is happy with changes we deploy to Production (master-myapplication).

Instead, this model has given us the freedom to quickly generate environments for feature branches. If we need to demonstrate, test or stage a feature branch we create a new application and prefix the application name with the name of the feature so it is available with a distinct url e.g. coolnewfeature-myapplication.apphb.com. So we have moved from a 'vertical' environment stack (TEST > QA > STAGE > PROD) to having more 'horizontal' feature driven environments.

A Proper Staging Environment

The point of a staging environment is to be identical to live, we all know why this is a good idea. We have already found bugs that we would not have found until deploying to production. A good example is that the default AuthorizeAttribute does not work on appharbor - you need to use a custom attribute. Whilst I acknowledge this is not a ringing endorsement of AppHarbor it is a good example of why a proper staging environment is important. We also get the environment made instantly available to us - no more hassling IT Ops for extra boxes or getting that internal DNS updated.

Your infrastructure add-ons don't need to be bulked up to the same scale as your production environments. E.g. we have a stage enviroment with a free 20mb database, but production has 10gb which costs a few bob. However, should you want to performance test your in stage environment just assign it the same number of web-workers make sure the both apps are hosted in the same region. You can take advantage of the Blitz Add-on that can tell you how well your app can cope with traffic surges.

Creating a stage environment in the cloud does mean that its publicly available. If this is not in your interest there is brief discussion here around how you might want to restrict access to your public staging environment.

Application Settings

AppHarbor lets you manage Configuration Values (the app settings in the web.config) from within AH itself as a simple key-value collection and straight-forward GUI. After each successful build the keys are matched to your web.config <AppSettings> and the value that exists is replaced with the value managed in AppHarbor - or added if no key is found. Then the code is deployed with the transformed config. This means you no longer need to maintain web.config transformation files like web.release.config. Whilst AppHarbor does in fact support config transformations if a web.release.config file is found, I have found it easier to take these files out of the picture altogether.

Some would argue that the transform files belong in source control but I can do without the burden of configuring the tricky XML transformation syntax, even though Appharbor kindly provides a tool to test your config tranforms. It is important to note that config transformations only works for the <AppSettings> in the web.config and not <ConnectionStrings>. So if you have a connection string that you want managed by AppHarbor put it in the App Settings instead.

What's also great is that any add-ons that require app settings or connection strings such as Databases or Queue systems auto-magically populate your AppHarbor app settings collection with the necessary key-value pair. The keys are constant between applications so you can add RavenDB to your develop and master applications and never have to touch your local web.config! We happen you use our Stage RavenDB instance to develop with locally so we do copy the connection string value into our into our web.config app settings. This could be considered bad practice as you would probably be better off having a separate DB instance to develop with but for our small team it was just the easier option.

App Settings are applied after every successful build and then deployed. If you change a config setting you need to redeploy your application for it to take affect - there is no hoping onto the box to adjust the web.config! This sounds scary but as I've blogged before - config settings should only be used for environment variables and not for things that remain constant between environments (such adhoc text or prices). With this in mind AppHarbor helps to keep what should be a constant a constant and not a config setting that isn't changing between environments. If your constants change you redeploy and not concern yourself worrying about whether your app settings have been taken affect. Generally your app settings won't change all that often if they are being used correctly. For extra reading here's good discussion about how to keep sensitive data config settings private.

Team City

We are still using good old Team City for our first line of defence by building the develop branch after each push and running all unit tests. AppHarbor does this too but we get better notifications if anything goes wrong and also unit test coverage statistics.

There will be an issue if you are using Team City 7 as your internal nuget package server and it is running on an internally hosted system. You need the nuget server of TC to be externally visible to AppHarbor so it can can download packages with your build. You can't just spin up TeamCity on AppHarbor so you're only option is put TC on a publicly visible IP and be sure to authorize all nuget access. Or open source it and get it on Code Better's TC!

Conclusion

There are plenty of reasons why we should be using cloud hosting, we had the luxury of not having to integrate with any legacy systems or databases behind a firewall. Having taken advantage of this fact I would encourage teams to break free from internal infrastructure hassles and move into instant Continuous Integration with AppHarbor, or any integrated cloud hosting provider.