Project A

Project A

A high availability financial platform

I've been working on the build and delivery aspects of 'Project A' for about 9 months. It's been quite a challenge as there are so many inter-dependent components in this high-availability platform.

This platform delivers an online gaming experience with over 4000 transactions per minute. Most of the transactions have a financial aspect which places extra emphasis on stability.

A high-level overview of this platform, from a perspective of build and release breaks down as follows:

Delivery stack

• Microsoft Windows Server 2008 R2
• Microsoft AppFabric 1
• Microsoft Web Farm Framework 2
• ASP.NET Framework 3.5 & 4
• Apache + Tomcat
• RabbitMQ
• Microsoft MSMQ
• Microsoft SQL Server 2008 Enterprise
• Citrix Xen Server
• Windows Scheduler
• Microsoft Web Deploy 2
• Powershell 2

Development stack

• Visual Studio 2010
• ASP.NET Framework 3.5 & 4
• MSBuild
• NCover
• MSTest
• SpecFlow
• Microsoft Web Deploy 2
• Powershell 2

Platform composition

• 100 x WCF applications
• 10 x Web applications
• 1 x Apache Tomcat application
• 14 x MSMQs
• 1x RabbitMQs 
• 6 x Windows Services
• 10 x Scheduled Tasks
• 1 x HTML5 client

Continuous delivery pipeline

This has to be the most complex project that I've had to deliver an automated build and deployment pipeline for. They say that necessity is the mother of invention and this project has certainly demanded a lot of innovation.

Parallel build pipeline

The build pipeline essentially produces the packaged artefacts for deployment. As part of the pipeline code is compiled, tested and analysed. The product of successful builds are uploaded to an artefacts repository for later deployment. 

The platform consists of over 80 solutions. The solutions are built in a specific sequence and parallelised where possible to minimise compilation times. Running sequentially, the platform would take 45 minutes to compile, in the paralleled build pipeline, this time is a mere 4 minutes.

Unit tests and code coverage analysis is also performed in parallel. 

Parallel deployment pipeline

Artefacts generated by the build pipeline need to be deployed to an environment. 

The build artefacts are packages for Websites, WCF applications, Windows Services, Scheduled Tasks and Database.

There are 5 environments for Project A, each different in composition and optimised for a specific purpose. 

• Developer testing
• System testing
• User acceptance testing
• Operations testings
• Live

Developer, System and UAT test environments are single host solutions provided by Citrix Xen Server. Small, cheap, disposable and automatically provisioned. Their purpose is primarily to aid the software development process across the development teams. At any given time, we can have over 10 of these environments running different versions of the platform.

Operations testing is a scaled back version of the live environment. OAT as we call it, is a multi-host solution that mimics the live environment in terms of composition, and its purpose is primarily for DevOps and other specialist teams to perform tests in a "like-live" environment.

Live, is the production environment, a monstrous array load-balanced and high-availability servers serving many thousands of transactions per minute.

The deployment pipeline takes packages from the build pipeline, and pushes them to any one of the above environments. The packages and the deployment mechanisms are platform agnostic, what varies are "service maps" that define the specifics of each target environment.

The service maps allow the deployment mechanism to deploy, configure and test each package.

As with the build pipeline, this many packages would take a long time to deploy if performed sequentially. When deployed sequentially, the entire platform takes about 30 minutes to deploy, but using a parellel solution, deployment times can be as low as 3 minutes.

Dev Ops toolset

Supporting the platform outside of the routine deployment deserves automation just as much as the build and deployment processes.

These tools use the same service maps used by the deployment mechanism. 

• Automatic environment provisioning
• Web farm framework (WFF) configuration and application request (ARR) routing rules
• App Fabric configuration
• Database restores and patch upgrades

Continuous Delivery, Continuous Integration, Continuous Improvement

This platform is continually evolving, each week new features are introduced. Occasionally, these new features will require additional support from the build and deployment pipeline, and I'm hoping to journal these extensions through this blog.

The approach

Initially most of my posts will be a catch-up of how Project A has already evolved, and then I'll move into less frequent posts as and when the pipeline evolves. 

Happy reading!