There are times when ReSharper gets confused. It might be unable to locate references if you've been re-jigging your packages (oo-er) or it might be unable to perform a refactoring on a file because its in read-only mode for no good reason.
While there's always ReSharper clear caches through the Visual Studio settings...
I've been using the command line more recently, so decided to make a script to do this from the root directory of the repo. Depends on having Git installed on the command line. And expects the ReSharper version is 8.2. And that you haven't tweaked where RS keeps it caches.
Its a PowerShell script, so add the contents to your PowerShell profile and set an alias if you like.
Monday, 3 August 2015
Tuesday, 8 April 2014
ServiceStack.Redis and StructureMap
I've tripped up on this a couple of times so I've just decided to layout the right way to make use of the ServiceStack.Redis client library when using StructureMap DI Container. The mistake I used to make was to have my controller or service depend upon an IRedisClient. StructureMap would be configured to provide an instance of an IRedisClient by invoking GetClient() on the whatever singleton instance of IRedisClientsManager I was using, usually the PooledRedisClientsManager. Much was learned from this SO answer.
This was a problem because it limited the re-usability of the consuming controller/service. Such a service might provide a method that used the injected IRedisClient to interact with the Redis instance. The IRedisClient implements IDisposable and should be wrapped in a using statement in order to release the resource and close the channel when it is has completed the task. In doing so, you make the method on the service non-idempotent: it has the side-effect of changing the state of the service by disposing of its injected IRedisClient instance. You are only able to successfully call the service method once, calling it a second time throws an exception.
The solution is fairly simple. Instead of having the service depend directly on the IRedisClient, the service instead needs to depend on the IRedisClientsManager. This then allows the service to call the GetClient() method on the IRedisClientsManager itself and create an instance of IRedisClient every time the method is called. Idempotency is achieved because a client of the service can call the method any number of times and not cause any unintended side-effects. The state of the service is no longer affected by disposing of its only instance of IRedisClient. The IRedisClient instances that are created are still disposed of, but the service has the ability to create a new one for itself.
The pattern is very similar to the RavenDB IDocumentStore and IDocumentSession that I am familiar with, so I surprised it took me so long to figure out!
This was a problem because it limited the re-usability of the consuming controller/service. Such a service might provide a method that used the injected IRedisClient to interact with the Redis instance. The IRedisClient implements IDisposable and should be wrapped in a using statement in order to release the resource and close the channel when it is has completed the task. In doing so, you make the method on the service non-idempotent: it has the side-effect of changing the state of the service by disposing of its injected IRedisClient instance. You are only able to successfully call the service method once, calling it a second time throws an exception.
The solution is fairly simple. Instead of having the service depend directly on the IRedisClient, the service instead needs to depend on the IRedisClientsManager. This then allows the service to call the GetClient() method on the IRedisClientsManager itself and create an instance of IRedisClient every time the method is called. Idempotency is achieved because a client of the service can call the method any number of times and not cause any unintended side-effects. The state of the service is no longer affected by disposing of its only instance of IRedisClient. The IRedisClient instances that are created are still disposed of, but the service has the ability to create a new one for itself.
The pattern is very similar to the RavenDB IDocumentStore and IDocumentSession that I am familiar with, so I surprised it took me so long to figure out!
Saturday, 29 March 2014
F# Type Providers
Solving F# problems at a Skills Matter session earlier this week allowed me to fully appreciate the power of the Type Provider. The task can be found here for those interested: https://github.com/tpetricek/Dojo-Type-Provider-Treasure-Hunt
The aim of this post is not to provide the solution to the problem but to underline useful of the type provider - which was the ultimate goal behind the exercise. Type Providers succeed at bridging the gap between code and data. Type providers makes data easy to navigate by extracting the schema and publishing it through intellisense.
This is a gift for developers who are used to working with static types. Type Providers have been written for all the most popular data sources, so you can perform tasks against almost anything as though it had already been translated into domain objects. This is often the goal of an ORM, to abstract away the plumbing of persistence interaction. However, Type Providers are more data-driven rather than being a 'code first' affair. You specify the source of the data and the Type Provider will do the heavy lifting of examining the schema, taking the naming conventions and sample data and presenting the developer with a suitable API to explore.
If you were to try to do this with an ORM you would need to create the DTO classes yourself and plug them into the Data Context (or equivalent). Type Providers are productivity boosting in this sense as they require very little bootstrapping.
The power of intellisense:
Type Providers have done what WCF Web Services did - making strongly typed data access possible over a network - but has liberated the publisher of that data to do so in whatever format they choose. Previously, if you wanted your data to be discoverable to the developer in the IDE you would have needed to create a Web Service wrapper around your data and model all the types yourself. Now you can just provide direct access to the data source and let Type Providers do the rest!
I'm yet to make use of FunScript but this looks like a great implementation of a Type Provider - applying the same principles to the html DOM API. Meaning you can create strongly typed web pages! It even includes common js libraries so you can create rich SPAs and interact with your domain with the same workflow and familiarity across both contexts.
The aim of this post is not to provide the solution to the problem but to underline useful of the type provider - which was the ultimate goal behind the exercise. Type Providers succeed at bridging the gap between code and data. Type providers makes data easy to navigate by extracting the schema and publishing it through intellisense.
This is a gift for developers who are used to working with static types. Type Providers have been written for all the most popular data sources, so you can perform tasks against almost anything as though it had already been translated into domain objects. This is often the goal of an ORM, to abstract away the plumbing of persistence interaction. However, Type Providers are more data-driven rather than being a 'code first' affair. You specify the source of the data and the Type Provider will do the heavy lifting of examining the schema, taking the naming conventions and sample data and presenting the developer with a suitable API to explore.
If you were to try to do this with an ORM you would need to create the DTO classes yourself and plug them into the Data Context (or equivalent). Type Providers are productivity boosting in this sense as they require very little bootstrapping.
The power of intellisense:
Type Providers have done what WCF Web Services did - making strongly typed data access possible over a network - but has liberated the publisher of that data to do so in whatever format they choose. Previously, if you wanted your data to be discoverable to the developer in the IDE you would have needed to create a Web Service wrapper around your data and model all the types yourself. Now you can just provide direct access to the data source and let Type Providers do the rest!
I'm yet to make use of FunScript but this looks like a great implementation of a Type Provider - applying the same principles to the html DOM API. Meaning you can create strongly typed web pages! It even includes common js libraries so you can create rich SPAs and interact with your domain with the same workflow and familiarity across both contexts.
Monday, 27 January 2014
Decoupling your Web Deploy Build Script from Team City with Powershell
Continuing my (inadvertent) series on stream-lining your deployment process this next instalment is about bundling your deployment script in with your deployment package and breaking dependencies between Build and Deployment Configurations.
Previous Topics:
1 - App Settings are for Variables that Change with the Environment
2 - Separate Build and Deploy Configs with Team City and MS Web Deploy
3 - Specifying Environment Variables at Deploy Time not at Build Time
When I last left this topic, environment settings were captured in Team City's Build Parameters. They were applied at deploy time using a deployment script that called the web deploy command line executable. This helps keep sensitive settings out of source control. The bad news is that this deploy script was also maintained in Team City.
The reason this is a problem becomes clear when we need to add or remove a parameter from the configuration. A new parameter name and value must obviously be added to the configuration before it can be used in the deployment. However, we are also required to update the deploy script to apply this parameter during the deployment. What we have is a dependency on the build package on the deploy script. The deploy script cannot be updated long in advance of a new parameter being added, as a deployment in the meantime that uses the new parameter where it is not yet needed will cause the build to fail. We need to move away from a shared deploy script and towards one that is specific to the package it is going to deploy.
The answer here is to bring the deploy script into source control. That way, we can manage the script like any other code artifact with version history and also, crucially, configure the script such that it will apply all the build parameters to the deployment package at the time it should be deployed. We can then add the actual parameter name and value to Team City far in advance of the actual deployment time, as we know our build script will only try to make use of the new parameter when it is executed, whilst parallel deployments can continue unaffected.
A deployment script might look something like this:
MyWebApplication.deploy.cmd /y /M:%TargetDeployUrl% /u:%Username% /p:%Password% /A:Basic -allowUntrusted "-setParam:name='LoggingEmailAddress',value='%LoggingEmailAddress%'" "-setParam:name='ServiceEndPoint',value='%ServiceEndPoint%'"
This contains only two build parameters (environment variables), but you can see that its fiddly enough to warrant version history. All we are doing is calling a command line program, so the easiest thing to do would be to create another .cmd in the same directory. I'm going to use a Powershell script to do the same because it will make it easier to work with a collection of Target Deploy Urls that we shall see later on.
The plan is to capture this call in a powershell script which is then called by our deployment build configuration in Team City. Team City will need to pass the build parameters to the powershell deploy script so that the actual values can be substituted upon execution. The simplest way to do this is with Environment Variables. Enviroment Variables are read from powershell using the $env: prefix. You'll see these in action in the script below. Warning: Do not use periods or dashes in your environment variable names as powershell has trouble escaping these characters - use underscores if necessary. Don't worry about the 'env.' that is prepended to your variable. This does not affect your deploy script.
The powershell script should be added to the root of the solution and committed to source. When the Build Configuration is run (that's build, not deploy see my previous article on separating build and deploy steps) we will need to make sure that this script is 'bundled' in with the deployment package. The deployment package contains the web deploy .cmd program that we need to call, so we should put our script in the same directory as this. To do this, configure an additional Artifact Path in the General Settings of your build configuration in Team City. We'll call our deploy script Deployment.ps1.
The deploy build configuration now has access to this script. The single build step of our deploy configuration just has to invoke Deployment.ps1. The settings shown in the image below were all I needed to enable this. The environment variables do not have to be referenced directly - this is a good thing. They are all accessible by the deploy script throughout the deployment, permitting the script to make use of whatever environment variables it needs.
Lastly, the contents of Deployment.ps1 are shown below. The main reason for using a powershell script is now clear: the ability to deploy to any number of targets. Running through the script, we see that it first defines a function to get the working directory. This is needed in order to reference the web deploy .cmd by its absolute file path. Then, we read in the TargetDeployUrls environment variable - as defined in our deploy configuration in Team City - and split it into an array on the comma character. We then loop over the array, calling the web deploy .cmd once per target. You can see that the long command is split into several lines for readability using a here-string. The line breaks are then removed so a single line operation can be performed. Note where the $env:LoggingEmailAddress and $env:ServiceEndPoint parameters are referenced. The command is executed using the Invoke-Expression alias iex.
The array loop has the advantage of allowing us to deploy to environments with any number of host machines. For example, in our staging environment we have only one server that needs to be deployed to, whilst our production environment has two. This deployment script caters for both these scenarios - all you need to when deploying to production is provide a comma separated list of deploy target urls. However, in its current configuration the script applies the same username and password to all servers. This might not be the case in every environment, but that improvement will have to wait for another day!
Previous Topics:
1 - App Settings are for Variables that Change with the Environment
2 - Separate Build and Deploy Configs with Team City and MS Web Deploy
3 - Specifying Environment Variables at Deploy Time not at Build Time
When I last left this topic, environment settings were captured in Team City's Build Parameters. They were applied at deploy time using a deployment script that called the web deploy command line executable. This helps keep sensitive settings out of source control. The bad news is that this deploy script was also maintained in Team City.
The reason this is a problem becomes clear when we need to add or remove a parameter from the configuration. A new parameter name and value must obviously be added to the configuration before it can be used in the deployment. However, we are also required to update the deploy script to apply this parameter during the deployment. What we have is a dependency on the build package on the deploy script. The deploy script cannot be updated long in advance of a new parameter being added, as a deployment in the meantime that uses the new parameter where it is not yet needed will cause the build to fail. We need to move away from a shared deploy script and towards one that is specific to the package it is going to deploy.
The answer here is to bring the deploy script into source control. That way, we can manage the script like any other code artifact with version history and also, crucially, configure the script such that it will apply all the build parameters to the deployment package at the time it should be deployed. We can then add the actual parameter name and value to Team City far in advance of the actual deployment time, as we know our build script will only try to make use of the new parameter when it is executed, whilst parallel deployments can continue unaffected.
A deployment script might look something like this:
MyWebApplication.deploy.cmd /y /M:%TargetDeployUrl% /u:%Username% /p:%Password% /A:Basic -allowUntrusted "-setParam:name='LoggingEmailAddress',value='%LoggingEmailAddress%'" "-setParam:name='ServiceEndPoint',value='%ServiceEndPoint%'"
This contains only two build parameters (environment variables), but you can see that its fiddly enough to warrant version history. All we are doing is calling a command line program, so the easiest thing to do would be to create another .cmd in the same directory. I'm going to use a Powershell script to do the same because it will make it easier to work with a collection of Target Deploy Urls that we shall see later on.
The plan is to capture this call in a powershell script which is then called by our deployment build configuration in Team City. Team City will need to pass the build parameters to the powershell deploy script so that the actual values can be substituted upon execution. The simplest way to do this is with Environment Variables. Enviroment Variables are read from powershell using the $env: prefix. You'll see these in action in the script below. Warning: Do not use periods or dashes in your environment variable names as powershell has trouble escaping these characters - use underscores if necessary. Don't worry about the 'env.' that is prepended to your variable. This does not affect your deploy script.
The powershell script should be added to the root of the solution and committed to source. When the Build Configuration is run (that's build, not deploy see my previous article on separating build and deploy steps) we will need to make sure that this script is 'bundled' in with the deployment package. The deployment package contains the web deploy .cmd program that we need to call, so we should put our script in the same directory as this. To do this, configure an additional Artifact Path in the General Settings of your build configuration in Team City. We'll call our deploy script Deployment.ps1.
The deploy build configuration now has access to this script. The single build step of our deploy configuration just has to invoke Deployment.ps1. The settings shown in the image below were all I needed to enable this. The environment variables do not have to be referenced directly - this is a good thing. They are all accessible by the deploy script throughout the deployment, permitting the script to make use of whatever environment variables it needs.
Lastly, the contents of Deployment.ps1 are shown below. The main reason for using a powershell script is now clear: the ability to deploy to any number of targets. Running through the script, we see that it first defines a function to get the working directory. This is needed in order to reference the web deploy .cmd by its absolute file path. Then, we read in the TargetDeployUrls environment variable - as defined in our deploy configuration in Team City - and split it into an array on the comma character. We then loop over the array, calling the web deploy .cmd once per target. You can see that the long command is split into several lines for readability using a here-string. The line breaks are then removed so a single line operation can be performed. Note where the $env:LoggingEmailAddress and $env:ServiceEndPoint parameters are referenced. The command is executed using the Invoke-Expression alias iex.
The array loop has the advantage of allowing us to deploy to environments with any number of host machines. For example, in our staging environment we have only one server that needs to be deployed to, whilst our production environment has two. This deployment script caters for both these scenarios - all you need to when deploying to production is provide a comma separated list of deploy target urls. However, in its current configuration the script applies the same username and password to all servers. This might not be the case in every environment, but that improvement will have to wait for another day!
Friday, 10 January 2014
Why Functional Programming?
I was in attendance at FPDays2013 to hear Bodil Stokke's talk: Programming, Only Better. I revisited this talk recently in order to put together the case for functional programming and F# to my colleagues. The first part of my presentation is largely taken from Bodil's talk, so this is basically a transcript with a couple of points from different sources. Primarily Scott Wlaschin's F Sharp for Fun and Profit.
Reasoning - We reason about code when we are trying to infer what it will do. We reason to better understand what is going on. Debugging is the most tangible form of reasoning. Typically, we are following the code path and understanding the flow of data. In every program we write we have to reason about what the code is doing. In its most basic form, we are walking through the program in our minds.
Testing - while we can be sure that programs are always reasoned about - they are certainly not always tested. That is because testing is useful, but limited. Testing can only show you the presence of bugs, but never their absence. TDD is based around this fact. The Red; Green; Refactor; process highlights the presence of a bug as the code is not yet written to perform this task - so the test fails. Writing the code to make the test pass is the process of removing a bug. Just because a system has 100% code coverage it cannot be said to be 'bug free'.
State - object oriented languages idealise encapsulated state. OO languages are useful precisely because we use more code to manipulate an object's state. So state changes with time - known as entropy in the mathematical world. Most of the time, we are intentially changing state in a useful way. But state is also permitted to be changed by something outside our control (unless we are very careful). The code example below highlights what it means to have an object enter a 'bad state' and not have that communicated back to the calling code.
State Spoils Testing - how can you test an object in all its possible states? You can only test for the states you think the object might enter into. State increases exponentially, so the more potential state you create, the lower the meaning of your tests!
State Spoils Reasoning - state allows for unpredictability. You can never be sure that you will get the same output with the same input. An objects state can change, so how can you be sure of it at any given time? This becomes even worse when you consider concurrent programs. It is much harder to describe the control flow of a program concurrent stateful program. In OO you have to use Locks to be able to protect against unwanted state change. This is cumbersome, inefficient and still imperfect.
Predictability - is guaranteed in a functional language because you will always get the same output for the same input. It can be thought of as nothing more than a glorified switch statement!
No Side Effects - as there is not state! There are no variables in pure functional languages, only values. Once a value is created, it cannot be change. Values can be created local to functions, but that function is also immutable and cannot be changed. State is instead managed by passing values around as inputs and outputs. When you think about it, it doesn't make sense to want to change a value you have set. What makes sense is to simply set a new value. This is essentially what functions do.
What happens when we have no state? Reasoning becomes much simpler! Predictability makes programs easier to understand. Tests also becomes more meaningful for the same reason. However, your tests are still unable to tell you that bugs do not exist, and they are only meaningful for the inputs you are testing with.
Optimizing - once you realise that output will never change for a given input, you understand that it no longer matters what order functions are called in. This allows you to perform functions in parallel, such as map reduce. Lazy Evaluation also becomes possible, because you know it doesn't matter when you call a function - the result will always be the same for the give input. Caching is more widely permitted for the same reason. Also, as there are no side effects, it means that you can be sure nothing is going be changed from underneath you! This makes concurrent programs easier to manage, as you don't need to worry about race conditions and lock objects.
Programming Techniques
We strive to make it simple to create programs. We use technology, tools and techniques to make programs better and to make better quality programs. The most basic tool we have is our own reasoning. Other techniques include testing, pair programming & code reviews. We'll explore 2 of things.Reasoning - We reason about code when we are trying to infer what it will do. We reason to better understand what is going on. Debugging is the most tangible form of reasoning. Typically, we are following the code path and understanding the flow of data. In every program we write we have to reason about what the code is doing. In its most basic form, we are walking through the program in our minds.
Testing - while we can be sure that programs are always reasoned about - they are certainly not always tested. That is because testing is useful, but limited. Testing can only show you the presence of bugs, but never their absence. TDD is based around this fact. The Red; Green; Refactor; process highlights the presence of a bug as the code is not yet written to perform this task - so the test fails. Writing the code to make the test pass is the process of removing a bug. Just because a system has 100% code coverage it cannot be said to be 'bug free'.
Programming Difficulties
Programming is difficult largely because programs grow to be complex. Code volume also makes it difficult to maintain programs, but volume and complexity are not necessarily linked. Its very easy to create an overly complex program without applying thought to its design. Simplicity is much harder to achieve. We'll look at one source of complexity:State - object oriented languages idealise encapsulated state. OO languages are useful precisely because we use more code to manipulate an object's state. So state changes with time - known as entropy in the mathematical world. Most of the time, we are intentially changing state in a useful way. But state is also permitted to be changed by something outside our control (unless we are very careful). The code example below highlights what it means to have an object enter a 'bad state' and not have that communicated back to the calling code.
State Spoils Testing - how can you test an object in all its possible states? You can only test for the states you think the object might enter into. State increases exponentially, so the more potential state you create, the lower the meaning of your tests!
State Spoils Reasoning - state allows for unpredictability. You can never be sure that you will get the same output with the same input. An objects state can change, so how can you be sure of it at any given time? This becomes even worse when you consider concurrent programs. It is much harder to describe the control flow of a program concurrent stateful program. In OO you have to use Locks to be able to protect against unwanted state change. This is cumbersome, inefficient and still imperfect.
Functional Languages
What can they do to help? They idealise Referential Transparency. RT means dealing with inputs and outputs. A function just maps an input to an output.Predictability - is guaranteed in a functional language because you will always get the same output for the same input. It can be thought of as nothing more than a glorified switch statement!
No Side Effects - as there is not state! There are no variables in pure functional languages, only values. Once a value is created, it cannot be change. Values can be created local to functions, but that function is also immutable and cannot be changed. State is instead managed by passing values around as inputs and outputs. When you think about it, it doesn't make sense to want to change a value you have set. What makes sense is to simply set a new value. This is essentially what functions do.
State
What happens when we have no state? Reasoning becomes much simpler! Predictability makes programs easier to understand. Tests also becomes more meaningful for the same reason. However, your tests are still unable to tell you that bugs do not exist, and they are only meaningful for the inputs you are testing with.Optimizing - once you realise that output will never change for a given input, you understand that it no longer matters what order functions are called in. This allows you to perform functions in parallel, such as map reduce. Lazy Evaluation also becomes possible, because you know it doesn't matter when you call a function - the result will always be the same for the give input. Caching is more widely permitted for the same reason. Also, as there are no side effects, it means that you can be sure nothing is going be changed from underneath you! This makes concurrent programs easier to manage, as you don't need to worry about race conditions and lock objects.
Conclusion
The techniques we use to help delivery quality software are undermined by state. Functional Languages remove state from the programming paradigm and in doing so prevent an entire class of bugs from ever entering our code. Functional programming makes our code easier to understand because of its predictability and has greater potential to optimize.Wednesday, 27 November 2013
Hotfix KB2842230 for WinRM on Windows 2012 (64-bit)
This post is just to document the location of a hotfix file that Microsoft Support fails to link to. They try to provide access from here: http://support.microsoft.com/kb/2842230. But the 'Hotfix Available' link doesn't provide access to the 64-bit version needed for Windows Server 2012.
http://hotfixv4.microsoft.com/Windows%208%20RTM/nosp/Fix452763/9200/free/463941_intl_x64_zip.exe
I found the link eventually in a GitHub Repository: Packer-Windows. So thanks for that! The hotfix now allows WinRM to respect the configuration value for "MaxMemoryPerShellMB" instead of always allowing using the default value of 150Mb.
This means I can get on with provisioning MVC4 and IIS etc with Chef. Hopefully more on that to come.
http://hotfixv4.microsoft.com/Windows%208%20RTM/nosp/Fix452763/9200/free/463941_intl_x64_zip.exe
I found the link eventually in a GitHub Repository: Packer-Windows. So thanks for that! The hotfix now allows WinRM to respect the configuration value for "MaxMemoryPerShellMB" instead of always allowing using the default value of 150Mb.
This means I can get on with provisioning MVC4 and IIS etc with Chef. Hopefully more on that to come.
Tuesday, 26 November 2013
Quick How To: Entity Framework Auto Migrations
The requirement in the project is to store some tracking information in a SQL database. I have created a POCO out of basic value types to house the data. I want each instance of this object to be persisted as a row in a table: enter EF 6.0. Its as simple as ever to bring in the EF nuget package at the Package Manager console:
Install-Package EntityFramework -ProjectName <ProjectName>
Now we have the tools to set up a database context for the POCO. The database context will give us the ability to store, update, query and delete collections of objects - we just have the one for this example, so our data container, db context and tracking class all together look like this:
Then, again from the console enable migrations:
Enable-Migrations -ProjectName <ProjectName>
In the project you added EF to, find the .\Migrations\Configuration.cs class. Set AutomaticMigrationsEnabled to true in the constructor:
public Configuration()
{
AutomaticMigrationsEnabled = true;
}
Now we're all set to have EF automatically put our database together. It will create tables, stored procedures, primary and foreign keys based on the signature of our objects. Lastly, run this command from the Package Manager console:
Update-Database -Verbose -ProjectName "<Project.Name>" -ConnectionProviderName "System.Data.SqlClient" -ConnectionString "xxx"
This is same command that can be run when new entities are added to the context, or when new properties are added to an existing entity. Note that this command won't support the removal of a property by default as it will result in data loss. Column renames are supported, but beyond the scope of this blog. Watch this video tutorial for further instruction. And here for more examples of maintenance commands.
Install-Package EntityFramework -ProjectName <ProjectName>
Now we have the tools to set up a database context for the POCO. The database context will give us the ability to store, update, query and delete collections of objects - we just have the one for this example, so our data container, db context and tracking class all together look like this:
Then, again from the console enable migrations:
Enable-Migrations -ProjectName <ProjectName>
In the project you added EF to, find the .\Migrations\Configuration.cs class. Set AutomaticMigrationsEnabled to true in the constructor:
public Configuration()
{
AutomaticMigrationsEnabled = true;
}
Now we're all set to have EF automatically put our database together. It will create tables, stored procedures, primary and foreign keys based on the signature of our objects. Lastly, run this command from the Package Manager console:
Update-Database -Verbose -ProjectName "<Project.Name>" -ConnectionProviderName "System.Data.SqlClient" -ConnectionString "xxx"
This is same command that can be run when new entities are added to the context, or when new properties are added to an existing entity. Note that this command won't support the removal of a property by default as it will result in data loss. Column renames are supported, but beyond the scope of this blog. Watch this video tutorial for further instruction. And here for more examples of maintenance commands.
Wednesday, 30 October 2013
5 Reasons to Keep Software Projects Small
Thoughts of system architecture have been cropping up recently so I wanted to layout some reasons why keeping software projects small is a good thing. A small software project is one that has one main purpose and few features. A small software project can be described to a new person in one sentence: "that's the product catalogue repo, it hosts a public list of products for our clients and has an authenticated admin console." Here's why you should Think Small.
5. Code Ownership
Let's start with the obvious one, maintainability. People, developers included, generally take more care of their own stuff than others'. Software is no different. It's a side effect of keeping projects small, but a very good one: Those projects written by one or two developers will be much better looked after and cared for than projects with multiple owners. I know this because I have worked on small projects written by a few people and large projects written by many. It is less likely for a developer to feel engaged when they are 'in someone else's code'. Motivation is lacking and code craftsmanship suffers when devs are contributing to a project that they do not feel ownership of. Their best work is rarely produced and what does get committed can often contradict other areas of the system in terms of coding style or naming conventions. Over time, unless significant effort is made to enforce standards and motivate developers to take ownership then these changes turn big projects into a Big Ball of Mud. Once that happens, no one wants to work on the project and progress halts. TDD and clean code can help, but they are only preventative measures.
With smaller projects, its much harder to end up in a BBoM situation. There simply isn't that much code in the system. Projects are kept from reaching the point where no one is confident in making changes because the project responsibilities are few and its purpose well defined. You cannot accrue technical debt in a project with no room to take short cuts. And if there are, then they are easy to spot. Its easy to answer questions organisation questions as the need for a clearly named folder structure is minimized.
With smaller projects, its much harder to end up in a BBoM situation. There simply isn't that much code in the system. Projects are kept from reaching the point where no one is confident in making changes because the project responsibilities are few and its purpose well defined. You cannot accrue technical debt in a project with no room to take short cuts. And if there are, then they are easy to spot. Its easy to answer questions organisation questions as the need for a clearly named folder structure is minimized.
4. Point Person
That is the go-to-guy when questions are asked or when changes are needed. When there are multiple point-people it usually requires a meeting just to make a decision, or worse, leads to deflection of responsibility. When the one point person knows that they are the only owner, they are driven to have detailed understanding of the system's behaviour (as chances are they have written it). The knowledge should be detailed but easy to pass onto another dev, should the original author move on. Failing that, small systems permit re-writes much more readily that large ones. As for the business, its not the nuances of the particular framework that is valuable, but the role that the system plays instead. This information is more transferable with smaller projects.3. Quality Requirements
When describing a small system, its much easier to gather the quality requirements. These are things like performance, uptime, security, accessibility, scalability etc. The audience or clients of a small system are usually more focused which lets us make design calls in the code. What is the expected throughput of the url? How soon after event x must the event data be query-able? Will we need to serve content to the other side of the world? Mobile?! When the developer knows the extent of the system's purpose they can make judgements about these kind of questions more easily.2. (Re)Distribution
This point is about how the system can be accessed. Keeping systems small means that they can be consumed more easily in different formats. Obviously, the most common mechanism of communication is over HTTP. The reason it is so popular is because of the high number of client libraries that support the http request response model with JSON data binding, meaning language interoperability. Http also provides built in functionality for authorisation, caching and security. But the benefits of http are not exclusive to small projects. What small projects are suited to is packaging. A packaged product or library is focussed on doing one thing really well. And when a project has few responsibilities then it can be packaged easily. This becomes a virtuous circle when you consider the motivation for packaging. It makes code shareable and more re-usable. Modern systems are a composition of frameworks and libraries brought together in a particular way to provide unique functionality. Package managing tools permit rapid development by providing access to thousands of small systems that have been packaged, used and tested by others.1. Innovation
Developers like learning new tricks and using new tools. When what the business is asking for can be delivered in small projects it benefits the business and the developer. If a developer has heard about and new framework or library they would like to try out, then they are going to be twice as motivated to take ownership of the system and enjoy the development experience when they get the chance to try it out. It might take slightly longer as learning occurs, but the development team wins as a result. New knowledge is shared and as such the team becomes more experienced. When some work arrives that has to be out the door by Friday, the team that has the most varied experience are more likely to already know the best tool for the job - and deliver. Aiming for small projects means that a team members acquire new skills more frequently as there are more opportunities to try out something new. Innovation and discovery in IT can be a driver for success in digital industries, thinking small might just give your team the advantage.Wednesday, 9 October 2013
Heroku's Python Web Sockets Chat App Locally on Windows with Vagrant
Heroku have recently announced support for Web Sockets. They have provided a selection of code examples in various languages to illustrate the two-way communication enabled by web sockets. The focus of this article is the Python-Websockets-Chat application.
Provided is a walk-through for producing an app that will work on Heroku, taking advantage of the redis add-on that will be part of your infrastructure when your environment is created at set up. I wanted to get a better understanding of Python, Flask and the web sockets plugin, Gevent, so I needed to play with the application locally. This required some help from Vagrant. I've been over the benefits of Vagrant before in a similar article so no need to repeat myself.
The pre-requisites to having a running instance of the Python-Websockets-Chat application are numerous and complicated. Starting with a fresh Ubuntu Precise32 Vagrant box will take at least 15 minutes of install time to correctly configure the environment. Thankfully, the provisioning capabilities of Vagrant mean that we can kick off the 'Vagrant up' command in the project directory and just wait for the bootstrap shell script to do its thing.
The shell script is shown below. As you can see there are not only dependencies on python development tools and Pip - the Python plugin manager, but Ruby also. This is because the application is designed to run on Heroku which uses a Procfile to organise the processes that need deploying. Procfiles are best executed by a Ruby gem called Foreman, hence the dependency. Other than that there is the requirement to install and kick off Redis, followed by the Foreman start command to trigger the app into action. It should be available on your host machine at http://localhost:5000/.
Please see my forked repo of the chat app for all the code in one place.
Anyone with some experience in Linux will recognise a lot of this as incredibly basic instructions, understandably. This post is for the benefit of the Windows folk who have next to no knowledge of configuring Linux environments and just want to see results. That's part of Vagrant's goal, to simplify and centralize environment configuration, so I hope that this goes some way to helping that cause.
P.s. Windows users be sure that the line endings of the bootstrap.sh are in the linux format (LF) and not the Windows format (CRLF). Cloning the repo may have adjusted your line endings so if you run into a problem when the bootstrap.sh tries to be executed then please see this issue on github.
Provided is a walk-through for producing an app that will work on Heroku, taking advantage of the redis add-on that will be part of your infrastructure when your environment is created at set up. I wanted to get a better understanding of Python, Flask and the web sockets plugin, Gevent, so I needed to play with the application locally. This required some help from Vagrant. I've been over the benefits of Vagrant before in a similar article so no need to repeat myself.
The pre-requisites to having a running instance of the Python-Websockets-Chat application are numerous and complicated. Starting with a fresh Ubuntu Precise32 Vagrant box will take at least 15 minutes of install time to correctly configure the environment. Thankfully, the provisioning capabilities of Vagrant mean that we can kick off the 'Vagrant up' command in the project directory and just wait for the bootstrap shell script to do its thing.
The shell script is shown below. As you can see there are not only dependencies on python development tools and Pip - the Python plugin manager, but Ruby also. This is because the application is designed to run on Heroku which uses a Procfile to organise the processes that need deploying. Procfiles are best executed by a Ruby gem called Foreman, hence the dependency. Other than that there is the requirement to install and kick off Redis, followed by the Foreman start command to trigger the app into action. It should be available on your host machine at http://localhost:5000/.
Please see my forked repo of the chat app for all the code in one place.
Anyone with some experience in Linux will recognise a lot of this as incredibly basic instructions, understandably. This post is for the benefit of the Windows folk who have next to no knowledge of configuring Linux environments and just want to see results. That's part of Vagrant's goal, to simplify and centralize environment configuration, so I hope that this goes some way to helping that cause.
P.s. Windows users be sure that the line endings of the bootstrap.sh are in the linux format (LF) and not the Windows format (CRLF). Cloning the repo may have adjusted your line endings so if you run into a problem when the bootstrap.sh tries to be executed then please see this issue on github.
Friday, 30 August 2013
Comparing Test Spec Frameworks
Up until now, I've been using a custom test spec framework geared towards helping write unit tests with a BDD style description: Given; When; Then, described in a previous post. The framework was built on top of NUnit, Moq, and Structuremap.AutoMocking nuget packages to make assertions, create mock objects and resolve the test target dependencies with mock objects respectively. There is no problem with the framework as it is, but I was seeking a way to reduce the language 'noise' that was incurred - a new method for each Given, When and Then step is cumbersome and gets tedious after a while.
This lead me to discover the Machine.Specifications (MSpec) framework. It allows you to write tests in the same BDD style, but does not require a full method for each step. Instead, you can just write a lambda function. This can reduce any one test down to a single line, by making sure that you stick to one assertion per test. It introduces some new types which enter the terminology: Establish, Because and It. These are direct replacements for Given, When and Then, as the semantics are identical.
But what about the automocking feature of the old framework? This can be fulfilled by another package designed to work with MSpec: Machine.Fakes. This, too, automatically resolves any dependencies of the target with mocked objects, provided by your mocking framework of choice. I used the Machine.Fakes.Moq nuget package to do this, but since you also use the Fakes framework to get access to the mocked dependencies it has created for you, and also generate additional mocks you may need the fact that I'm using Moq becomes irrelevant - it is completely abstracted away. To get access to an mocked dependency you use The<T>, and to create a mock you use An<T>. I also like the nice feature that gives you a list of mocked objects: Some<T>.
That's the language hooks and mocks taken care of, but in the effort to reduce the syntax bloat as much as possible I brought in the Fluent.Assertions nuget package. This library is leaner than NUnit because it lets me make assertions using a set of extension methods, e.g. .ShouldEqual() or .ShouldBeTrue(), so I don't have to begin an assertion statement with NUnit, I can start with the object I want to make the assertion on. It also puts the final nail in the coffin for NUnit, as I have already done away with the [SetUp], [TearDown] or [Test] attributes.
Let's compare two test classes written for the same target class. The Target is shown below, followed by the before and after test classes.
Target
Before
After
As you will notice, the 'After' test class contains over 50% fewer lines. The tests are now more readable because there is less language noise to cut through when reviewing the code. The important statements jump right out, and they also take less time to write. The method bodies and calls to base methods in the Given and When steps have gone, replaced with single line statements. The MFakes framework has helped make each line that refers to a mock more concise be removing the need to call .Object. The Fluent.Assertions library has made the assertions simpler too.
There are a couple of downsides that I have discovered not shown in the example that I think are worth mentioning. Firstly, Tests (It field variables) cannot be run from a base class. They can be declared, but go undetected by the build runner. This means that you will have to duplicate any tests that are based on statements that come before a branch at an if statement, assuming you want to test both branches of the method. Whereas previously you could declare a test in an abstract class and have it executed in each text fixture class that inherits from it.
The other negative is that you cannot instantiate a mock of your own using the An<T> from MFakes at field level. You have to be inside the body of the Establish lambda/anonymous delegate before you can assign a value to the variable defined at field level. You aren't told about this error until run time which is annoying to begin with, but being forced to declare your mock variables on one line and assigning them on another doubles the amount of code needed to achieve what that the previous framework could do. So it goes against the purpose of this exercise somewhat.
All in all, I think the positives outweigh the negatives. The biggest factor being that there is less manual syntax to be typed to generate a test. Which is one less reason not to do TDD, which can only be a good thing!
This lead me to discover the Machine.Specifications (MSpec) framework. It allows you to write tests in the same BDD style, but does not require a full method for each step. Instead, you can just write a lambda function. This can reduce any one test down to a single line, by making sure that you stick to one assertion per test. It introduces some new types which enter the terminology: Establish, Because and It. These are direct replacements for Given, When and Then, as the semantics are identical.
But what about the automocking feature of the old framework? This can be fulfilled by another package designed to work with MSpec: Machine.Fakes. This, too, automatically resolves any dependencies of the target with mocked objects, provided by your mocking framework of choice. I used the Machine.Fakes.Moq nuget package to do this, but since you also use the Fakes framework to get access to the mocked dependencies it has created for you, and also generate additional mocks you may need the fact that I'm using Moq becomes irrelevant - it is completely abstracted away. To get access to an mocked dependency you use The<T>, and to create a mock you use An<T>. I also like the nice feature that gives you a list of mocked objects: Some<T>.
That's the language hooks and mocks taken care of, but in the effort to reduce the syntax bloat as much as possible I brought in the Fluent.Assertions nuget package. This library is leaner than NUnit because it lets me make assertions using a set of extension methods, e.g. .ShouldEqual() or .ShouldBeTrue(), so I don't have to begin an assertion statement with NUnit, I can start with the object I want to make the assertion on. It also puts the final nail in the coffin for NUnit, as I have already done away with the [SetUp], [TearDown] or [Test] attributes.
Let's compare two test classes written for the same target class. The Target is shown below, followed by the before and after test classes.
Target
Before
After
As you will notice, the 'After' test class contains over 50% fewer lines. The tests are now more readable because there is less language noise to cut through when reviewing the code. The important statements jump right out, and they also take less time to write. The method bodies and calls to base methods in the Given and When steps have gone, replaced with single line statements. The MFakes framework has helped make each line that refers to a mock more concise be removing the need to call .Object. The Fluent.Assertions library has made the assertions simpler too.
There are a couple of downsides that I have discovered not shown in the example that I think are worth mentioning. Firstly, Tests (It field variables) cannot be run from a base class. They can be declared, but go undetected by the build runner. This means that you will have to duplicate any tests that are based on statements that come before a branch at an if statement, assuming you want to test both branches of the method. Whereas previously you could declare a test in an abstract class and have it executed in each text fixture class that inherits from it.
The other negative is that you cannot instantiate a mock of your own using the An<T> from MFakes at field level. You have to be inside the body of the Establish lambda/anonymous delegate before you can assign a value to the variable defined at field level. You aren't told about this error until run time which is annoying to begin with, but being forced to declare your mock variables on one line and assigning them on another doubles the amount of code needed to achieve what that the previous framework could do. So it goes against the purpose of this exercise somewhat.
All in all, I think the positives outweigh the negatives. The biggest factor being that there is less manual syntax to be typed to generate a test. Which is one less reason not to do TDD, which can only be a good thing!
Wednesday, 21 August 2013
There's still a need for SQL databases
Document Databases are an important part of the NoSql movement taking place. They are appealing to developers because of their intuitive Apis and also to stakeholders as they lower the boundaries to feature development that would have once been hugely difficult a Sql-only environment. Some key benefits of document databases include:
- They permit fast development of complex objects graphs, making it easy to employ DDD. Though hand-scripting stored procedures are now largely a thing of the past thanks to modern ORMs, they are difficult to retro-fit to a legacy system and can introduce new bugs of their own. I would argue that there is more confidence in storing and accessing aggregates roots in a document database as their is 'truer' representation of the model being stored on disk, while with an ORM there will always be a layer of translation.
- Are schema-less and therefore 'backwards compatible', meaning no application downtime while the database is administered. Again, ORMs have advanced enough to support deployment tasks, see Microsoft Web Deploy and RoundhousE of the Chuck Norris Dev Ops suite, but the risk still exists of losing or corrupting data with a tool or not.
- When you need to scale, document databases have been designed from the ground up to be able to distribute the data across large numbers of machines with aim of spreading system load, increasing availability and preventing failures. There is a physical limit that Sql will reach when scaling simply by throwing more resources at it (vertical scaling). Document databases deal with sharding more effectively as they don't have relationships to maintain as in a relational SQL database. There are ways in which you can scale a sql database that are not so simple.
- When you need to perform calculations over a large dataset very quickly. Document databases lend themselves to map-reduce type queries especially well because of their distributed nature. Map reduce queries are designed to be performed in parallel, which is a great fit for a shared document database. Queries are also optimized for 'eventual consistency', meaning that result accuracy is traded for low response times.
- When you need access from JavaScript. Many document databases store their data as JSON objects, so using a JavaScript client is a great way of consuming a document database service. The increase of Single Page Applications written in JavaScript and also Node Js in the back-end has encouraged the growth of JS clients. Take the Mongoose npm package for example.
In modern applications, its hard to argue that any of these requirements can be overlooked. But if you don't have a bursty traffic model and do not need a globally accessible service then you are not yet excluded from using a Sql database...
Sql databases still have a very good use. Leaving aside the fact that they remain largely a more stable platform due to there maturity compared with the relative youthful NoSql technologies, Sql databases make a great tool for data-mining. That is, to make use of the Sql JOIN operator across multiple tables to 'slice' the data. If you are working with a normalized schema, Sql queries provide the ability to draw conclusions from a body of data with more freedom than a document database. Document databases demand that you specify the kind of report you will want up front at design-time. With a Sql database, or at least a tool that provides a Sql type language interface, you can build up reports based on the relationships built into the tables.
By reporting in this context, I primarily mean the types of back end office reports that might be needed on a daily, weekly or monthly basis. These type of reports are largely back office type reports, that tend to grow as the business rules change. The type of reports I mean reveal insight and are built up further on top of existing tables and views. This type of gradual evolution is brought about by in-house report writers, who know the schema well. These reports take time to write, as well as time to run. But that time is acceptable because of the advantages that they lend the business.
In contrast, when lots of users need to see top level summaries of data quickly, a document database would be a much better tool for the job. Dashboards headline figures can be useful debugging as well as marketing tools, but they are not the type of reports that are suited to a Sql database. When website response time is important, do not put a report generated from a Sql database on the home page (hopefully we all learned this by now, right?!)
It is easy to get caught up in the momentum and success of the NoSql movement. But it can be difficult to tell what benefits it will bring to your application, and at what cost. Hopefully this article offers you a balanced view. Its fun to be an early adopter, but I would exercise caution before binning the old Sql reporting database in favour of the latest and greatest.
- They permit fast development of complex objects graphs, making it easy to employ DDD. Though hand-scripting stored procedures are now largely a thing of the past thanks to modern ORMs, they are difficult to retro-fit to a legacy system and can introduce new bugs of their own. I would argue that there is more confidence in storing and accessing aggregates roots in a document database as their is 'truer' representation of the model being stored on disk, while with an ORM there will always be a layer of translation.
- Are schema-less and therefore 'backwards compatible', meaning no application downtime while the database is administered. Again, ORMs have advanced enough to support deployment tasks, see Microsoft Web Deploy and RoundhousE of the Chuck Norris Dev Ops suite, but the risk still exists of losing or corrupting data with a tool or not.
- When you need to scale, document databases have been designed from the ground up to be able to distribute the data across large numbers of machines with aim of spreading system load, increasing availability and preventing failures. There is a physical limit that Sql will reach when scaling simply by throwing more resources at it (vertical scaling). Document databases deal with sharding more effectively as they don't have relationships to maintain as in a relational SQL database. There are ways in which you can scale a sql database that are not so simple.
- When you need to perform calculations over a large dataset very quickly. Document databases lend themselves to map-reduce type queries especially well because of their distributed nature. Map reduce queries are designed to be performed in parallel, which is a great fit for a shared document database. Queries are also optimized for 'eventual consistency', meaning that result accuracy is traded for low response times.
- When you need access from JavaScript. Many document databases store their data as JSON objects, so using a JavaScript client is a great way of consuming a document database service. The increase of Single Page Applications written in JavaScript and also Node Js in the back-end has encouraged the growth of JS clients. Take the Mongoose npm package for example.
In modern applications, its hard to argue that any of these requirements can be overlooked. But if you don't have a bursty traffic model and do not need a globally accessible service then you are not yet excluded from using a Sql database...
Sql databases still have a very good use. Leaving aside the fact that they remain largely a more stable platform due to there maturity compared with the relative youthful NoSql technologies, Sql databases make a great tool for data-mining. That is, to make use of the Sql JOIN operator across multiple tables to 'slice' the data. If you are working with a normalized schema, Sql queries provide the ability to draw conclusions from a body of data with more freedom than a document database. Document databases demand that you specify the kind of report you will want up front at design-time. With a Sql database, or at least a tool that provides a Sql type language interface, you can build up reports based on the relationships built into the tables.
By reporting in this context, I primarily mean the types of back end office reports that might be needed on a daily, weekly or monthly basis. These type of reports are largely back office type reports, that tend to grow as the business rules change. The type of reports I mean reveal insight and are built up further on top of existing tables and views. This type of gradual evolution is brought about by in-house report writers, who know the schema well. These reports take time to write, as well as time to run. But that time is acceptable because of the advantages that they lend the business.
In contrast, when lots of users need to see top level summaries of data quickly, a document database would be a much better tool for the job. Dashboards headline figures can be useful debugging as well as marketing tools, but they are not the type of reports that are suited to a Sql database. When website response time is important, do not put a report generated from a Sql database on the home page (hopefully we all learned this by now, right?!)
It is easy to get caught up in the momentum and success of the NoSql movement. But it can be difficult to tell what benefits it will bring to your application, and at what cost. Hopefully this article offers you a balanced view. Its fun to be an early adopter, but I would exercise caution before binning the old Sql reporting database in favour of the latest and greatest.
Tuesday, 30 July 2013
New Tricks from an Old Pattern
Its great when you can still be surprised by a pattern you think you already know inside-out. I was pleased today to think about the BDD Unit Testing Style 'pattern' in a new way. Here's my previous post on the subject that will give you the necessary background for the rest of this post.
The pattern utilises object inheritance to make it powerful and descriptive. To illustrate, we must start with an example. The code below contains a method that tests two conditions and tries to return an object. If both conditions fail then the method throws an exception. It is worth noting that the sample class is not a good example of OO design, but rather a simplified case. The important point is that there is a path through the method that will cause an exception to be thrown.
Using the BDD Style Unit Testing pattern makes it pretty easy to test the first two conditions and the object that is returned in each case. Note the use of an abstract base class which provides us with the Result variable to hold our return object and importantly encapsulates away the call to the When() method. The Class is called WhenIGetScoredPlayer, meaning we should expect to find the call to GetScoredPlayer() on the Target inside of the overriden When() method (see the preceding article if you aren't following).
This means we don't have to repeat this call to the same target method in any of the classes that inherit from this base class. It makes the classes easy to navigate and the test output becomes straight forward as coherent sentences are produced: "When I Get Scored Player, Given The Player Has a Score Greater Than 8, Then the Result is a High Score Player". This is useful information when reviewing tests results.
However, you will now notice that its not so simple in the third path of this method. That's because in the case when a player scores below 2 an exception is thrown. If I inherit from the abstract WhenIGetScoredPlayer class and setup the player's score accordingly, then the exception will be thrown too early. That's because in the base GivenA<T> class a call to the virtual When() follows a call to the virtual Given() in a method tagged with [SetUp], to make sure they happen in sequence as part of the framework - its useful for us to have things happen in this order.
I could start from scratch and perform all the setup in a new class. Making the call to GetScoredPlayer() inside of a [Test] method instead of as part of [SetUp]. But I don't want to duplicate unnecessarily. This is when I thought about the old pattern in a new way - using an override to postpone the call instead of starting all over again.
What I did was to still implement the base class with the appropriately named GivenThePlayerHasAScoreLowerThan2. The trick is to simply override the base When() and leave it empty. This prevents it from calling the GetScoredPlayer() during the [SetUp] phase of the test. It can now occur in the [Test] method, wrapped in an NUnit assertion that the correct type of Exception is thrown. Producing the desired output without unnecessary duplication: "When I Get Scored Player, Given The Player Has a Score Lower Than 2, A Score Too Low Exception Is Thrown".
The pattern utilises object inheritance to make it powerful and descriptive. To illustrate, we must start with an example. The code below contains a method that tests two conditions and tries to return an object. If both conditions fail then the method throws an exception. It is worth noting that the sample class is not a good example of OO design, but rather a simplified case. The important point is that there is a path through the method that will cause an exception to be thrown.
Using the BDD Style Unit Testing pattern makes it pretty easy to test the first two conditions and the object that is returned in each case. Note the use of an abstract base class which provides us with the Result variable to hold our return object and importantly encapsulates away the call to the When() method. The Class is called WhenIGetScoredPlayer, meaning we should expect to find the call to GetScoredPlayer() on the Target inside of the overriden When() method (see the preceding article if you aren't following).
This means we don't have to repeat this call to the same target method in any of the classes that inherit from this base class. It makes the classes easy to navigate and the test output becomes straight forward as coherent sentences are produced: "When I Get Scored Player, Given The Player Has a Score Greater Than 8, Then the Result is a High Score Player". This is useful information when reviewing tests results.
However, you will now notice that its not so simple in the third path of this method. That's because in the case when a player scores below 2 an exception is thrown. If I inherit from the abstract WhenIGetScoredPlayer class and setup the player's score accordingly, then the exception will be thrown too early. That's because in the base GivenA<T> class a call to the virtual When() follows a call to the virtual Given() in a method tagged with [SetUp], to make sure they happen in sequence as part of the framework - its useful for us to have things happen in this order.
I could start from scratch and perform all the setup in a new class. Making the call to GetScoredPlayer() inside of a [Test] method instead of as part of [SetUp]. But I don't want to duplicate unnecessarily. This is when I thought about the old pattern in a new way - using an override to postpone the call instead of starting all over again.
What I did was to still implement the base class with the appropriately named GivenThePlayerHasAScoreLowerThan2. The trick is to simply override the base When() and leave it empty. This prevents it from calling the GetScoredPlayer() during the [SetUp] phase of the test. It can now occur in the [Test] method, wrapped in an NUnit assertion that the correct type of Exception is thrown. Producing the desired output without unnecessary duplication: "When I Get Scored Player, Given The Player Has a Score Lower Than 2, A Score Too Low Exception Is Thrown".
Friday, 21 June 2013
Logging with AppHarbor Web & Background Workers
Last year AppHarbor introduced Background Workers into their repertoire. You can push a .net solution with a console application project that will be compiled and executed just as you would expect if executing the task locally. You can have a Background Worker and a Web Worker in the same application. The Background Worker will have the same access and to the adds-on used by the Web Worker so you can delegate any 'offline' processes to a Background Worker to allow your web app to concentrate on just serving responses. This provides a good platform for horizontal scaling.
A source of frustration with this approach however is logging. The standard AH add-on for logging is LogEntries. LogEntries is a generic SAAS consumed by many integrated hosting providers. It provides token-based TCP logging when used as another target or appender with NLog or Log4Net respectively. Logs are collected with almost zero latency and are visible in the snappy LogEntries interface.
The frustration arises when you are using LogEntries with the Web Worker + Background Worker architecture I described. As I mentioned, LogEntries by default is token-based, meaning your application gets one token to use to identify itself. This token is shared by the Web Worker and the Background Worker.
If your Background Worker (running your Console Application) and your Web Worker (running your Web Application) are dependent on shared domain projects - meaning that they both make consume the same classes - then your logs can be ambiguous as to the source of the calling code. Has your 'FATAL' entry been encountered in the Web Worker or the Background Worker? You cannot tell by the type alone!
Thankfully, LogEntries provides a neat way of solving this issue, though it is not obvious if you are starting with the AppHarbor add-on, as the configuration is performed behind the scenes - you're fast-forwarded through the basic introduction. LogEntries' Logs live under a default Host, the AppHarbor Host, which is added upon Initialisation. A Default Log is added to the AppHarbor Host, and this is where your logs end up when using the provided token. The token is automatically injected into your web/app.config settings when the Web Worker and Background Workers are deployed.
To make our Web Worker and Background Worker write to separate Logs, we simply have to add a new one in our AppHarbor Host. Make sure that you select Token TCP transport.
In doing so we get a new token. We can then use that token in the Console Application app.config, so that all logging is directed to the Background Worker log. The configuration of the app.config le_appender in Log4Net looks like the following:
<log4net>
<appender name="LeAppender" type="log4net.Appender.LogentriesAppender, LeLog4net">
<Token value="some-thing-secret" />
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%level% %logger %m" />
</layout>
</appender>
<root>
<appender-ref ref="LeAppender" />
</root>
</log4net>
This forces the LeAppender to use the token you have provided - and not the default LOGENTRIES_TOKEN injected by AppHarbor into your app.config. Separate logs are yours!
A source of frustration with this approach however is logging. The standard AH add-on for logging is LogEntries. LogEntries is a generic SAAS consumed by many integrated hosting providers. It provides token-based TCP logging when used as another target or appender with NLog or Log4Net respectively. Logs are collected with almost zero latency and are visible in the snappy LogEntries interface.
The frustration arises when you are using LogEntries with the Web Worker + Background Worker architecture I described. As I mentioned, LogEntries by default is token-based, meaning your application gets one token to use to identify itself. This token is shared by the Web Worker and the Background Worker.
If your Background Worker (running your Console Application) and your Web Worker (running your Web Application) are dependent on shared domain projects - meaning that they both make consume the same classes - then your logs can be ambiguous as to the source of the calling code. Has your 'FATAL' entry been encountered in the Web Worker or the Background Worker? You cannot tell by the type alone!
Thankfully, LogEntries provides a neat way of solving this issue, though it is not obvious if you are starting with the AppHarbor add-on, as the configuration is performed behind the scenes - you're fast-forwarded through the basic introduction. LogEntries' Logs live under a default Host, the AppHarbor Host, which is added upon Initialisation. A Default Log is added to the AppHarbor Host, and this is where your logs end up when using the provided token. The token is automatically injected into your web/app.config settings when the Web Worker and Background Workers are deployed.
To make our Web Worker and Background Worker write to separate Logs, we simply have to add a new one in our AppHarbor Host. Make sure that you select Token TCP transport.
<log4net>
<appender name="LeAppender" type="log4net.Appender.LogentriesAppender, LeLog4net">
<Token value="some-thing-secret" />
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%level% %logger %m" />
</layout>
</appender>
<root>
<appender-ref ref="LeAppender" />
</root>
</log4net>
This forces the LeAppender to use the token you have provided - and not the default LOGENTRIES_TOKEN injected by AppHarbor into your app.config. Separate logs are yours!
Thursday, 6 June 2013
Implement a C# Interface in F#
Whilst seeking ways to learn F# and weave it into predominantly C# projects I have started to implement interfaces written in C# with an F# version. That sometimes means re-writing a C# implementation with an F# implementation. It works particularly well for a number of reasons:
Motivation - Provides direction rather than just 'hacking about' with the language.
Confidence - If you have test coverage, you can use the same tests to be sure your code is working.
Motivation - Provides direction rather than just 'hacking about' with the language.
Confidence - If you have test coverage, you can use the same tests to be sure your code is working.
Syntax - Provides a sound introduction to the indentation and parenthesis use in F# by forcing you to translate your own C# code.
Insight - Helps to reinforce that F# still compiles to IL underneath and is executed on the CLR - its all .NET after all!
Communication - Helps others to learn by examining the 'before' (C#) and 'after' (F#) implementations.
Seamless - Integrates with your existing C# solution when using dependency injection and IoC maintaining your decoupled architecture.
In F#, interfaces are implemented Explicitly as opposed to Implicitly in C#. This means that we must provide pointers to the functions we wish to fulfil the interface per method or property. The Gist below provides examples of the various signatures we would expect to implement.
Seamless - Integrates with your existing C# solution when using dependency injection and IoC maintaining your decoupled architecture.
In F#, interfaces are implemented Explicitly as opposed to Implicitly in C#. This means that we must provide pointers to the functions we wish to fulfil the interface per method or property. The Gist below provides examples of the various signatures we would expect to implement.
Wednesday, 22 May 2013
Specifying Environment Variables at Deploy Time not at Build Time
Over a month ago I posted an article in which I revelled in the ability to create a publish package in one step then deploy it in another. Last week I discovered a limitation to that process, as it was, and had to find a way to reconfigure it. This post is about that challenge and the conclusions I've drawn from it.
The difficulty with the process that I described in that article was that it is largely dependent on Configuration Transformations. These have been around for a while now and do a great job of specifying the environment variables with respect to the build configuration, normally Debug or Release. But that is exactly the problem (and my realisation): Config Transforms blur the boundary between Building and Deploying.
When I'm compiling my application I want to be thinking about the build, about optimisation, about references and dependencies. So unless I'm developing locally and debugging I usually want to build in Release mode, as its probably going to get deployed for testing or production. I might be compiling following performance enhancements - I don't want to have to concern myself with the target environment's connection strings when I'm trying to make my app go faster. I always want to build in Release mode. But with config transformations that are tightly coupled to my build configuration I'm stuck with a built package with production connections strings - I can't performance test against that!
Build configurations are necessary, but I don't want to have to create a new one just so that I can apply a different Config Transformation at Build Time. Nor do I want to specify the Publish Profile at build time either. I want to separate Build configuration and Environment configuration concerns. What I want to do is specify the Environment settings at Deploy Time, after my deploy package has been built.
Thankfully, there is a way to do this. But its much more convoluted then adding a new config transformation file to your project. It seems Microsoft have unfortunately made it easier to do things the wrong way. MS have made it easier to deploy from the desktop, but in an enterprise this is usually not the case. Perhaps the new Publish Profile is more about making it easier to deploy to Azure?
The 'Package' MSBuild target that is only available for Web projects generates a directory containing a zipped up deployable package containing all the assets and dlls required, a MyProject.deploy.cmd file that is a friendly wrapper around the msdeploy.exe and, crucially for us, a MyProject.SetParamters.xml. The xml file gives us the ability to modify the contents of the package at Deploy Time (when we call the .cmd). This is what we will be editing:
The xml file can be manually edited before deploying, but this is rarely practical. The IIS App Name and any connection strings are pulled into this file by default. The connection strings are replaced at deploy time with the actual values found in whatever the result of your configuration transformation was. It is also possible to configure the IIS App Name in your .csproj file, but then we are again dependent on the build configuration. The way to override these settings when we deploy is to call msdeploy.exe via the MyProject.deploy.cmd and make use of the -setParam flag. The syntax of which is rather cumbersome:
The second problem is this: what if you want to specify Application Settings (appSettings) in your web.config? As stated, IIS App Name and Connections strings are all that you get to override out-of-the-box. To be able to configure an App Setting, you must add a new file to the root of your web project: parameters.xml. This file needs to contain any App Settings you want to specify when you deploy. Follow this article for a more detailed explanation of the syntax of a parameter.xml. Essentially, we must specify the XPath to the setting and provide an alias to it so we can refer to it in the generated MyProject.SetParameters.xml that we will go on to edit at Deploy Time:
Now, when we create the package at Build Time there is another parameter specified in our MyProject.SetParameters.xml generated along with the package. Like so:
We then apply the same overriding "-setParam" flag when calling the MyProject.deploy.cmd at Deploy Time to configure the "My Service Url" however we want for out target environment. The final call that overrides the parameters is shown below. I've parameterised the call so that I can use the Team City UI to run a custom build and override the parameters without the chance of missing a closing quote etc.
The added bonus of this configuration is that if you are manually deploying a built package by importing it into an IIS website and using the GUI, you are then prompted to provide the settings. This is useful when only certain members of IT are permitted to know the production connection strings or other sensitive information.
However, should you not be in an enterprise with such tight restrictions, then it might be preferable to store the various environment settings in source control with the project. The benefit of this is that there is less chance of misconfiguring a deployment with a typo. This could be achieved by using the "-setParamFile" flag when calling the cmd. However, the difficulty is that if your custom SetParameters.xml is in source control then you have to go fetch it out of the deployed package, which is already zipped up by the build step! Perhaps TC's artifacts can assist here...
This feels like a much more appropriate way to configure the deployment workflow: Always compiling the code in the most suitable configuration (Release), then dealing with the deployment configuration when the time comes to Deploy (or re-deploy). The trouble is that the steps shown above are a lot more tedious and error prone in comparison to the Configuration Transforms we are already used to. Even so, I think this is the right solution for the job.
The difficulty with the process that I described in that article was that it is largely dependent on Configuration Transformations. These have been around for a while now and do a great job of specifying the environment variables with respect to the build configuration, normally Debug or Release. But that is exactly the problem (and my realisation): Config Transforms blur the boundary between Building and Deploying.
When I'm compiling my application I want to be thinking about the build, about optimisation, about references and dependencies. So unless I'm developing locally and debugging I usually want to build in Release mode, as its probably going to get deployed for testing or production. I might be compiling following performance enhancements - I don't want to have to concern myself with the target environment's connection strings when I'm trying to make my app go faster. I always want to build in Release mode. But with config transformations that are tightly coupled to my build configuration I'm stuck with a built package with production connections strings - I can't performance test against that!
Build configurations are necessary, but I don't want to have to create a new one just so that I can apply a different Config Transformation at Build Time. Nor do I want to specify the Publish Profile at build time either. I want to separate Build configuration and Environment configuration concerns. What I want to do is specify the Environment settings at Deploy Time, after my deploy package has been built.
Thankfully, there is a way to do this. But its much more convoluted then adding a new config transformation file to your project. It seems Microsoft have unfortunately made it easier to do things the wrong way. MS have made it easier to deploy from the desktop, but in an enterprise this is usually not the case. Perhaps the new Publish Profile is more about making it easier to deploy to Azure?
The 'Package' MSBuild target that is only available for Web projects generates a directory containing a zipped up deployable package containing all the assets and dlls required, a MyProject.deploy.cmd file that is a friendly wrapper around the msdeploy.exe and, crucially for us, a MyProject.SetParamters.xml. The xml file gives us the ability to modify the contents of the package at Deploy Time (when we call the .cmd). This is what we will be editing:
The xml file can be manually edited before deploying, but this is rarely practical. The IIS App Name and any connection strings are pulled into this file by default. The connection strings are replaced at deploy time with the actual values found in whatever the result of your configuration transformation was. It is also possible to configure the IIS App Name in your .csproj file, but then we are again dependent on the build configuration. The way to override these settings when we deploy is to call msdeploy.exe via the MyProject.deploy.cmd and make use of the -setParam flag. The syntax of which is rather cumbersome:
The second problem is this: what if you want to specify Application Settings (appSettings) in your web.config? As stated, IIS App Name and Connections strings are all that you get to override out-of-the-box. To be able to configure an App Setting, you must add a new file to the root of your web project: parameters.xml. This file needs to contain any App Settings you want to specify when you deploy. Follow this article for a more detailed explanation of the syntax of a parameter.xml. Essentially, we must specify the XPath to the setting and provide an alias to it so we can refer to it in the generated MyProject.SetParameters.xml that we will go on to edit at Deploy Time:
Now, when we create the package at Build Time there is another parameter specified in our MyProject.SetParameters.xml generated along with the package. Like so:
We then apply the same overriding "-setParam" flag when calling the MyProject.deploy.cmd at Deploy Time to configure the "My Service Url" however we want for out target environment. The final call that overrides the parameters is shown below. I've parameterised the call so that I can use the Team City UI to run a custom build and override the parameters without the chance of missing a closing quote etc.
The added bonus of this configuration is that if you are manually deploying a built package by importing it into an IIS website and using the GUI, you are then prompted to provide the settings. This is useful when only certain members of IT are permitted to know the production connection strings or other sensitive information.
However, should you not be in an enterprise with such tight restrictions, then it might be preferable to store the various environment settings in source control with the project. The benefit of this is that there is less chance of misconfiguring a deployment with a typo. This could be achieved by using the "-setParamFile" flag when calling the cmd. However, the difficulty is that if your custom SetParameters.xml is in source control then you have to go fetch it out of the deployed package, which is already zipped up by the build step! Perhaps TC's artifacts can assist here...
This feels like a much more appropriate way to configure the deployment workflow: Always compiling the code in the most suitable configuration (Release), then dealing with the deployment configuration when the time comes to Deploy (or re-deploy). The trouble is that the steps shown above are a lot more tedious and error prone in comparison to the Configuration Transforms we are already used to. Even so, I think this is the right solution for the job.
Monday, 29 April 2013
Developing Shopify Dashing on Windows with Vagrant
We've started using Shopify's Dashing gem internally so we can publicise any moving figures people are interested in. You can go from zero to great-looking-dashboard in next to no time once you understand the API. The Dashing Wiki has enough documentation and the example dashboards that ship with it are enough to communicate the usage pattern.
This post is about the biggest stumbling block we had when starting out. We develop on Windows machines as most of what we do is .net. Dashing is a solution written in Ruby so we wanted to develop on a Linux OS as we would be deploying to Heroku to host our production dashboards.
Enter Vagrant. This a really neat tool that lets you script the creation of a virtual machine. The Vagrant script sits in the root of your project so that it is under source control and can be shared across the team. Simply open the command line at your project directory and enter vagrant up. Within minutes a virtual machine running Ubuntu in text only mode is started.
Vagrant takes care of automatically sharing the project folder on your host machine with the guest virtual machine, so that you can continue development as normal and just execute the code inside the virtual machine. The port 3030 is forwarded so that we can view the Dashboard on our local browser. This is the content of my Vagrantfile:
Vagrant.configure("2") do |config|
config.vm.box = "precise32"
config.vm.network :forwarded_port, guest: 3030, host: 3030
end
So now we have in waiting a virtual machine running our target operating system, with a synced folder and the ability to hit any website running inside of it. However, that's not quite enough for Dashing. We need to update the version of Ruby to 1.9.1 and add some other essential apps. Here are the necessary steps that might hopefully save someone a lot of time:
From the Windows command line:
c:\MyDashing> vagrant up
c:\MyDashing> putty
Configure putty correctly and ssh into the VM.
Log in with username: vagrant
Enter commands:
$ cd ../../
$ sudo apt-get update
$ sudo apt-get install ruby1.9.1-full -y
$ exit
Back in the Windows command line:
c:\MyDashing> vagrant reload
c:\MyDashing> putty
Again, ssh into the VM:
$ cd ../../
$ sudo apt-get install build-essential -y
$ sudo apt-get install nodejs -y
$ sudo gem install bundler --no-rdoc --no-ri
$ sudo gem install dashing --no-rdoc --no-ri
$ dashing new MyDashing
$ cd MyDashing
$ bundle install
$ dashing start
Now you have initialized a dashing project. You should be able to enter dashing start at the command prompt and not be met with any error messages. Navigate to http://localhost:3030/sample on your local machine and hey presto! Development can now continue on your local machine, while serving the dashboards via your virtual box.
Gotchas:
- Make sure you update your Gemfile with any gems used in any new job you create. Then don't forget to run bundle install at the prompt!
- Any private key files or other assets required by code should reside in the root of the project.
- Keep an eye on the output when running vagrant up as Vagrant will cleverly accomodate any conflicting forwarded ports. This can affect the ports you use to ssh into the VM and view the dashboard.
P.s. I'm aware of the provisioning facilities of Vagrant, including the capability of running all of the above through a shell script during construction of the VM. However, I couldn't get it to work without including the restart so I have just left the raw commands as a the guide for now. If/when I figure it out I will update the post. However, these steps only need to be run once per machine (not every time you run vagrant up) so you might find its just about tolerable as it is.
This post is about the biggest stumbling block we had when starting out. We develop on Windows machines as most of what we do is .net. Dashing is a solution written in Ruby so we wanted to develop on a Linux OS as we would be deploying to Heroku to host our production dashboards.
Enter Vagrant. This a really neat tool that lets you script the creation of a virtual machine. The Vagrant script sits in the root of your project so that it is under source control and can be shared across the team. Simply open the command line at your project directory and enter vagrant up. Within minutes a virtual machine running Ubuntu in text only mode is started.
Vagrant takes care of automatically sharing the project folder on your host machine with the guest virtual machine, so that you can continue development as normal and just execute the code inside the virtual machine. The port 3030 is forwarded so that we can view the Dashboard on our local browser. This is the content of my Vagrantfile:
Vagrant.configure("2") do |config|
config.vm.box = "precise32"
config.vm.network :forwarded_port, guest: 3030, host: 3030
end
So now we have in waiting a virtual machine running our target operating system, with a synced folder and the ability to hit any website running inside of it. However, that's not quite enough for Dashing. We need to update the version of Ruby to 1.9.1 and add some other essential apps. Here are the necessary steps that might hopefully save someone a lot of time:
From the Windows command line:
c:\MyDashing> vagrant up
c:\MyDashing> putty
Configure putty correctly and ssh into the VM.
Log in with username: vagrant
Enter commands:
$ cd ../../
$ sudo apt-get update
$ sudo apt-get install ruby1.9.1-full -y
$ exit
Back in the Windows command line:
c:\MyDashing> vagrant reload
c:\MyDashing> putty
Again, ssh into the VM:
$ cd ../../
$ sudo apt-get install build-essential -y
$ sudo apt-get install nodejs -y
$ sudo gem install bundler --no-rdoc --no-ri
$ sudo gem install dashing --no-rdoc --no-ri
$ dashing new MyDashing
$ cd MyDashing
$ bundle install
$ dashing start
Now you have initialized a dashing project. You should be able to enter dashing start at the command prompt and not be met with any error messages. Navigate to http://localhost:3030/sample on your local machine and hey presto! Development can now continue on your local machine, while serving the dashboards via your virtual box.
Gotchas:
- Make sure you update your Gemfile with any gems used in any new job you create. Then don't forget to run bundle install at the prompt!
- Any private key files or other assets required by code should reside in the root of the project.
- Keep an eye on the output when running vagrant up as Vagrant will cleverly accomodate any conflicting forwarded ports. This can affect the ports you use to ssh into the VM and view the dashboard.
P.s. I'm aware of the provisioning facilities of Vagrant, including the capability of running all of the above through a shell script during construction of the VM. However, I couldn't get it to work without including the restart so I have just left the raw commands as a the guide for now. If/when I figure it out I will update the post. However, these steps only need to be run once per machine (not every time you run vagrant up) so you might find its just about tolerable as it is.
Subscribe to:
Posts (Atom)