Arrested Development

Synopsis

Our current build and deployment pipeline doesn't really align particularly well with our desired development practices.  Whilst the pipeline has assisted the adoption of Feature branching as it was intended, our current build and release practices are still disruptive.

What do I mean, disruptive?

They say that "Technology is at its absolute best when its invisible", this to me means it's so seamlessly integrated with a given activity that it goes unnoticed. This should be the goal of every technology and process, it should be invisible.

And this is what I mean by disruptive. Whilst as a team we're busy developing new features, we're continually disrupted by routine operations such as branching, building, deploying & cleaning up.  The kind of thing that will normally be labelled "Process", the stuff we have to do, in order to get other stuff done.

So whilst our newly adopted practices and processes have really taken us a long way from where we were, we've still got some way to go before they can be forgotten about. 

Integrated workspaces

I briefly mentioned in the synopsis, some of the routine actions that disrupt the flow of development.

• Creating new features
• Creating build definitions and managing builds
• Building environments
• Deploying packages
• Running tests

No-one would argue that these actions aren't worth the effort, they ultimately save us a lot of time, but they are disruptive. They break the flow of concentration and feature development, and it's features that drive the company forward, and pays our wages.

I've developed a plan to eradicate the disruptions I listed above, a plan that should more or less be invisible to the developers and the wider business participants.

I want to introduce the concept of integrated workspaces, that is until I can think of a better name!

Arrested development

In order to explain how I believe my plan will work, I first need to tell you why.

At the moment, we have 7 internal test environments that we refer to as Systest. The systest environments are single-host environments that are used to assist in feature development. Our SOA platform requires that platform needs hosted in order to facilitate continued development, therefore developers are continually deploying the latest versions of their features.

Compared to 12 months ago, the act of provisioning environments and deploying code is now 100% automated. And whilst these automated processes have helped to eliminate a whole raft of productivity sapping and morale crushing activities, they are still a disruptive influence on the flow of development.

Consider a typical feature development cycle:
A developer takes on a new feature,

1. A branch and build definition is created for the feature
2. The developer works on a feature
3. The feature is checked-in, and tested
4. The developer then pushes the latest build to an environment
5. The developer continues to develop the feature
6. Steps 2-5 continue until by iteration, the feature is considered complete
7. The feature is eventually completed, and re-integrated into Main
8. The testing team deploy the new trunk to UAT and begin testing
9. Steps 2-8 are repeated as bugs are found and resolved
10. Eventually the developer removes the feature branch and builds

In this typical scenario, up to 50 deployments may take place. And whilst, as I say repeatedly, the automated processes in place make this a completely dependable operation, it still a disruption, its still getting in the way of what the developer wants to achieve.

My idea for integrated workspaces is in part, to make these disruptions go away. 

However, there are also other reasons...

The human touch

Fundamentally, we still see the world in terms of “packages”, “deployments” and “hosts”, and all of these entities are accessible to anyone in the development team. Essentially, the inner workings are still laid bare, which allows developers, who are very clever people, to take advantage of.

Dirty environments

We've gotten into a situation on a number of occasions where developers have performed partial deployments (segments of the platform) with packages of varying ages. And, in some cases, even packages from different code-bases (features) have been deployed into an environment.

Meaning, that our test environments were hosting a collage of different versions of our platform.

And lastly, our ad-hoc configurations, leading to configuration drift. The code running on any environment is supposed to be entirely derived from the code base. However, everyone, from time to time, will be overcome by the temptation to make ad-hoc changes to the run-time environment, normally to "just see" if something will work. We all do it. And the problem is, each of these innocent little changes potentially increases the distance between the environments run-time and the source that is supposedly came from.

This is very damaging as it leads to an environment that is in an unknown and unrepeatable state, which we then continue to further develop against. 

Our tests validate our code; A series of actions performed against a carefully contrived application state, engineered to give us confidence that our code works. How can we have confidence that our tests are giving us accurate results when they're being deployed against an environment that no longer has the state or configuration we expected. What was the point in writing these tests if they become non-deterministic? Worse still, how much further down the development cycle will we get before false-positive results are finally discovered? Now thats a big, timely and costly disruption, that could have been so easily avoided.

Frustration

The best solution to avoiding dirty environments is to only record the state and configuration in the source code, and then deploy the changes up to the test environment. But, this to any developer will be a major frustration as their concentration and momentum is disrupted.

No-one should be surprised therefore that in order to "get things done", disruptive practices and processes will be circumvented.

Integrated workspaces

The notion of an integrated workspace was something I came up with whilst trying to work out how I could make my processes invisible to the developer. I wanted to remove all the things that disrupt the developers momentum, whilst paradoxically ensuring that all of my processes and practices couldn't be circumvented.

I want to create a flow that carries the developer along, so I began thinking about where all this begins, with a new feature. 

My thoughts have been through almost countless iterations, but I arrived at the following example, which I think is best represented in the Powershell code that ultimately it will be implemented in.

.\Create-Feature “feature-x”
.\Update-Feature “feature-x” (performed by TFS after build)
.\Complete-Feature “Feature-X”

At no point here, do I give any indication of the underlying actions. 

Create-Feature

Would be the only Powershell command the developer would need to use in order to begin work.

1. Create the branch
2. Create the build definition
3. Trigger a build
4. Provision an environment on AWS EC2 specifically for this feature
5. Update the DNS on AD to make the run-time available.
6. Deploy the new packages to the environment
7. Local hosts file updated so that the code-branch knows where to find the run-time.

In a short while, the developer would be all set to begin work with

1. A new feature branch
2. Targeted to run against feature-name.mydomain.local
3. And with  feature-name.mydomain.local provisioned and deployed

Powershell	Workspace
Create-Feature “feature-x”	feature-x.dev.mydomain.com
Create-Feature “feature-u”	feature-u.dev.mydomain.com

Update-Feature (TFS only)

Each time the developer performs a check-in, TFS will invisibly build, compile, package and re-deploy to the new code to the provisioned host.

A check-in is something the developer has to do, but the rest is invisible. 

When the build is complete, the developer will be able to simply resume development. The code being developed in Visual Studio will already been running against the latest version. 

Almost entirely free of disruption for the developer.

There could be arguments that the "build" stage would be disruptive, and this is true, but unless we can introduce real-time compilation there's not an awful lot I can do about that.

All I can do is continue with projects that will drive down the build times, such as distributed unit testing and alike.

Complete-Feature

Is the final Powershell command the developer will need to know.

This step will be a tidy-up exercise, removing the now redundant builds from TFS, removing the build definition and the branch. The environment that was provisioned specifically for this feature can be marked for deletion, and torn-down later by scheduled task.

In summary

The integrated workspaces project, isn't going to be terribly difficult to implement, it will entirely reuse the current build and deployment pipelines. The major difference is that the entire process will be more or less invisible to the developer. 

The most significant chunk of work will be AWS provisioning, and configuring the code in the branch to run against the new environment. 

The proposal for this project is currently circulating around the development team via Sharepoint, so with a few adjustments I'm sure, work will hopefully commence shortly.

Measuring success

Our Kanban process already provides us with very detailed metrics for cycle-times and a cumulative state-flow. It should therefore be very easy to compare both overall cycle times, but also the state cycle times too.