Friday, 26 July 2013

Keeping your build agents clean

To keep our TFS Build Agents working smoothly, with as little maintenance as possible, we run a number of scheduled tasks each evening.

This script is run nightly

  • Clears out the local workspace left behind by every build definition
  • Removes the build folder and all the output's


[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.TeamFoundation.Client")
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.TeamFoundation.VersionControl.Client")

$tfsUri = "http://_______:8080/tfs/defaultcollection";
$tfsCollection = [Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory]::GetTeamProjectCollection($tfsUri);

$vc = $tfsCollection.GetService([type] "Microsoft.TeamFoundation.VersionControl.Client.VersionControlServer");

$localPC= $(hostname).ToUpper()

$workspaces = $vc.GetType().InvokeMember('QueryWorkspaces', 'InvokeMethod', $null, $vc, @($null, $null, $localPC))

foreach ($workspace in $workspaces)
{
    if ($workspace.Folders.length -gt 0)
    {
        $vc.GetType().InvokeMember('DeleteWorkspace', 'InvokeMethod', $null, $vc, @($workspace.name, "NTDomain\YourAccount"))
    }
}

Remove-Item "e:\builds" -recurse -force

Thursday, 25 July 2013

Using Hard Links in your builds

Our build engine performs as much work as possible concurrently, to reduce feedback times as much as possible.

But sometimes, it feels like MSBuild wasn't really deigned for this type of activity. We'd previously had issues with locked files which we'd managed to resolve, but even with no over-lapping project compilations, we were still hitting a locked file during copy operations from time to time.

We would see an MSB3021 sporadically, in different branches, on different computers. Ther was no pattern of behaviour would could discern.

Symptoms

MSBuild will complain that it couldn't copy a file because it was locked.
It is normally, an assembly being copied into BIN

E.g. "The process cannot access the file ______.___ because it is being used by another process. msbuild"

Also, the problem can manifest within Windows Explorer:
  • The file can not be deleted
  • You can not take ownership of the file
The only cure, appears to be rebooting the PC, then the file is gone. 

The cause

Absolutely no idea, it seems to strike randomly, and without reason.

The fix

In our case, we made three modifications, based upon other peoples experiences.

1) Enabled the "Application Experience" service.

2) Allowed MSBuild to use Hard Links (aka. Symbolic Links) 
The Hard Link effectively creates an alias to the original file, and doesn't attempt to copy it.
  • /p:UseHardlinksIfPossible=true
  • /p:CreateHardLinksForCopyFilesToOutputDirectoryIfPossible=true
  • /p:CreateHardLinksForCopyAdditionalFilesIfPossible=true
  • /p:CreateHardLinksForCopyLocalIfPossible=true
  • /p:CreateHardLinksForPublishFilesIfPossible=true
3) Allows MSBuild's copy task(s) to try again without failing on the first encounter.
  • /p:CopyRetryCount=10
  • /p:CopyRetryDelayMilliseconds=1000
  • /p:OverwriteReadOnlyFiles=true



Tuesday, 9 July 2013

Considering Microsoft Azure as an IAAS hosted development platform

In recent months our internal development infrastructure has become wearisomely unstable. We have simply exceeded by some way our initial growth expectations, and as a result the two year old infrastructure is not capable of performing under its current workload.

After discussions with the MD and Operations, we had three choices:
  1. Cloud based IAAS
  2. In-House provisioning
  3. A combination of both

Background

Our current development platform is provided by an array of virtual servers running on a single appliance. 
The specification of this hardware is enormous, but even so, this is still somewhat exceeded by our demands.

In addition to the primary issue of being over contended, our current development platform also lacks any form of resilience and has very limited disaster recovery. Given we are a software development organisation who's very existence is owed to the development of new features, this is a bit of a concern that needs to be addressed.

The other concern is one of capacity, we need a 1:1 ratio of features to development environments. Each feature is developed in isolation. We have reached the point where we no longer have any capacity to maintain this 1:1 ratio, which has a considerable impact on the organisations ability to grow from here on.

Objectives

We'll use the following criteria to compare any proposed solutions.

Scalability

The ability of the proposed solution to expand and contract with our needs.

Cost

How much will the solution cost:
  • Upfront investment
  • Annual cost

Performance

Performance is measured in different ways:
  • Dependability
  • Speed
  • Scaling

Our requirements

  • We require three hosts per each development environment
  • We need the ability to expose services running on each environment to the outside world
  • We need the new environments to be interoperable with our current infrastructure

Azure IAAS

First contender is Azure services from Microsoft
  • Our MSDN premium subscriptions give us £100 entitlement
  • Our Operations team are already experienced with administrating an Azure hosted platform

Specific information

  • Three hosts per environment 
    • 1 x XS
    • 1 x S
    • 1 x M
    • Total cost of $0.56c per hour
  • We need them to be running for 9 hours a day, 5 days a week (operational hours)
  • Our platform is 1GB in size
  • We deploy 7 times a week (developer survey)
  • Azure's pricing model is simple
  • We get £100 free entitlement per MSDN Premium Subscription
  • There are 3 hosts which are required to be on 24x7
    • VPN network link
    • Storage host (XS)
    • Domain controller (XS)

Azure pricing


  1. With our MSDN licenses we get the first 8 environments for free
  2. PAYG remains viable until 12 units
    1.  Gives us flexibility to scale up by an extra 4 environments without any commitment

Self-Hosted IAAS

The alternative is to expand our own infrastructure, to give us an IAAS in-house.
  • We've estimated that we can host 4 environments per server node.
  • That the annual operating costs are £938 (power, cooling, maintenance) per server.
  • When comparing with Azure, we chose 20 development environments.
  • Each server was priced at £8938. (Dell Computers)
  • The accumulative cost over 5 years is £67,008.00.
  • No upgrade to the WAN infrastructure is required.

Comparison

For our particular requirements, and using a 20 environment solution as the comparison, Azure is compellingly competitive over self-hosting.



  • The annual maintenance costs of self-hosting is only marginally lower than an annual Azure subscription 
  • The initial hardware purchase is never recouped. 
  • After 5 years, we would have to refresh our hardware which is likely to be just as much again. 
    • We could lessen the impact by upgrading one node each year, but the accumulative cost would remain the same.

Conclusions

Comparing both solutions on Price, Scalability and Performance, Azure is the 2:1 winner.
  • Price -> the case for Azure is compelling, a saving of £29,601 over 5 years.
  • Scalability -> Azure is the clear winner, internal hosting can not compete unless we have spare capacity installed, prepared and going unused. With Azure, we can scale up in a matter of hours and at relatively lower increments.
  • Performance -> Self-Hosting is superior by and large due to the limitations of our WAN connection to Azure. At 20 Mbit it takes almost 9 minutes, compared with 35 seconds over the internal LAN.

Recommendations

If the cost wasn't an issue, then a solution that resides on our LAN would be the superior solution. 
However, cost is a significant factor, and on that basis, the case for Azure IAAS is compelling.

Split hosting

During our investigation we identified several development environments that would need to be left running 24/7. The 7 identified environments could be left running on the existing virtual host server, whilst the rest are re-provisioned up to the Azure infrastructure. 

Running these 7 victuals in-house is more in-line with the original expectations for our host appliance, and would see CPU contention return to a more realistic 1:1 mapping. 

Further investigation

A member of our operations team has just informed me that our hardware vendor offers an IAAS service with a bit of a twist. The pricing model is definitely IAAS, but the hardware is installed locally.

We are now investigating this as a potential "best-of-both-worlds" solution. The flexibility of Azure, coupled with LAN performance. I'll come back and update this post after I learn more.