Faster builds using NuGet

on

It. Takes. Too. Long! :@

Continuous Improvement, like history, is just one thing after another, and whilst we've made great strides along our process maturity road-map, there's still inevitably always room for more improvement. 

It takes ages to create a new feature branch.

Our repository is huge, it has a disk foot print of approximately 9GB. Each branch is therefore also 9GB. It doesn't take TFS particularly long to create the new branch, the delay is what happens next:

  • Downloading the repository takes about of 10 minutes.
  • A full get has a detrimental impact on the TFS (virtual) server affecting other developers

The process of creating new features is entirely scripted in Powershell, so the procedural aspect has been automated; The developers are however still inconvenienced by the lengthy 'wait' for completion.

It takes ages to build our new feature (building dependencies)

Once the "Get All" is complete, there is another wait of about 20 minutes for the entire product to be built locally so that all the referenced assemblies are available within the workspace.

The architecture of the product demands that its built in this way. We can't decouple as this is by design, it's meant to work this way. Its a fine design conceptually, operationally and architecturally. It is however, a nightmarish web of inter-dependency to build.

It takes ages to publish to a test environment

Once our new feature is branched and built it needs to be published so that its web services can be consumed and the messaging system can function. This takes a further 20 minutes as its not a trivial task to deploy over 100 websites and databases. 

Let's have a go at making things better?

As a first pass at improving this situation we attempted to split the team along the traditional divide of skill-set, a separation of the platform and UI.

The two distinct skill-sets with a clear division of labor presented us with an opportunity to try and split the product.

The UI team's solutions are typically consumers of the platform, so in the build pipeline they are the last to be built and packaged. Meaning that the UI devs have to wait for the platform to be built first.  Using NuGet we were able to directly embed these assemblies into the UI solutions.

The platform builds published the platforms API to our own hosted Nuget repository which was then immediately available to the UI team for consumption.

The results were pretty pleasing:
  1. UI tier builds were reduced to under 1 minute. 
    From a typical wait of 8 minutes.
  2. Platform builds, free of the UI, were reduced by a couple of minutes.
    From a typical wait of 8 minutes to 6.
The division along these lines was working reasonably well for the UI team. But platform developers and so-called "full-stack" developers were not feeling the love. 

Let's try again...

The biggest issue with the division we'd created was the associated changes in working practices.

Each feature now required two branches, one for each team. The NuGet packages provided the link between Platform and UI branches.

This kind of worked but for..
  1. Two lots of "Feature Branch Creation" were needed
  2. "Full-Stack" developers now had two sets of branches to manage, with an un-welcome partition between the two.
  3. Reverse integrations from both API and UI branches required a degree of coordination and communication. 
  4. We had some solutions that had a foot in both camps e.g. back-office admin sites.
  5. Deployments to Production required a rebuild of the UI branch to ensure it was using the new target Platform assemblies.
But we had learnt much from our first attempt...
  • How to perform on-the-fly Nuget substitution using Powershell.
  • How to reverse the NuGet substitutions back into relative file references.
  • How to upgrade and downgrade Nuget versions.
  • How to host our own Nuget Repository.
  • How to programatically cloak and un-cloak sections of the local workspace.
  • How to programatically cloak sections of the build definition.
  • Improved dependency discovery.
During the discussion an idea was born of the nascent techniques developed during our first attempt. Instead of separating the teams as we had done, let's try and separate the feature instead. Let's only take what we need from the repository and use NuGet to plug the remaining dependency gaps in the workspace.

Partial builds and Isolating the feature

We've used NuGet to facilitate a situation where the developers can be select which domains they intend to change, and use NuGet to reference the domains they leave behind. When I say leave behind, I mean simply absent from the local workspace.

By actively managing the local workspace mappings we can hide-away the domains we're not changing.  The end result of this is that developers of any specialisation, or generalisation, can work on a feature in a workspace tailored to their exact needs. 

Here's an extract from our internal training documentation for new developers.  This visual aid hopefully illustrates the difference between a partial build using selective workspace mapping,  and a full-output build.




The benefits of partial builds

When we look at the list of benefits, it's embarrassing that we didn't think of this sooner. 
  • A branch with selective mapping can take as little as 10 seconds to download
    • An improvement on 10 minutes
    • Workspaces are refreshed multiple times during the lifetime of a feature, so the savings are compounded over time.
  • Only one feature branch is required
    • Negates the need for the creation and management of multiple branches
    • Coordinated reverse integration is not required
  • Build & feedback times are closer to 30 seconds
    • Feedback is obtained far quicker
    • Quicker builds improves wait-times on the TFS build agents for the entire team
  • The NuGet provides a stable code-base for feature development
    • Features can be based upon solid foundations
    • Features can be based upon previous code releases, current production, or the latest green build. Or, if necessary, another feature branch.
  • Lighter infrastructure footprint
    • Less network traffic and disk usage (<500MB versus 10GB)
    • Faster builds, shorter queues, and a significantly smaller artefacts repository
    • Reduced TFS contention.
    • Deployments of features over the WAN link to Azure are much quicker.
  • Less management
    • Simpler and smaller branches are easier to reverse integrate with the main.
    • The smaller changesets and workspaces vastly improve the performance of TFS and Visual Studio Team Explorer.
    • Check-In Policy is working with smaller changesets and is nimbler.
  • Promiscuous 'tweaks' branch gone
    • We had to use an promiscuous branch for releasing quick-fixes between release periods. Mostly UI and other visual tweaks. This was a bit of an administrative nuisance, which is now, thankfully, gone. 

Revised lifecycle management

In order to make this work the developers have been provided with a set of Powershell based scripts that perform all the heavy lifting. Crucially the scripts follow a Q&A approach to ascertaining the developers requirements before acting.

Starting a new feature

Defining a new feature, entails created an isolated feature branch on the developers local workspace, with only the solutions of interest present.
    1. Gather requirements from the developer
    2. Create the new branch on TFS
    3. Selective workspace mapping in the local workspace
      1. Perform Get after mappings declared
    4. Create a new CI build definition on the TFS build service
      1. Source cloaking to match developers workspace
    5. Perform NuGet substitution on absent solutions
      1. Download the required packages from NuGet
      2. Replace references to absent solutions with NuGet
      3. Manage the workspaces NuGet configurations
At the end of this process, which takes about 30s to complete, the developers have a new branch within their workspace which contains only the domains requested. All the other domains swapped for NuGet references from the 'green build' selected.
New Feature >  Confirm your choices

You would like to create a new feature as described?:
 Create a new branch called 'FeatureX'
 Reference all excluded domains with packages from build 23070
 The TFS path '$/Platform/Sprints/FeatureX' is mapped locally as 'E:\Projects\Platform\Sprints\FeatureX'

 You have chosen to add the following into your local workspace:

Core
Core.ACL
Core.Content
Ui.Content
Ui.Core
Ui.Models
Ui.Sites.Primary

Mapping $/Org/Platform/Sprints/Core
Mapping $/Org/Platform/Sprints/Core/ACL
Mapping $/Org/Platform/Sprints/Core.Content
Mapping $/Org/Platform/Sprints/UI/Content
Mapping $/Org/Platform/Sprints/UI/Core
Mapping $/Org/Platform/Sprints/UI/Models
Mapping $/Org/Platform/Sprints/UI/Sites/Primary

Reference all excluded domains with packages from build 23070

Are you sure?
[Y] Yes  [N] No  [?] Help (default is "N"): y

Build Platform

Performed more frequently than any other action, builds will be performed on the solutions isolated in the feature branch. Dependency discovery is still important, as there is still a build order to be observed for those solutions which are present.
    1. Gather requirements from the developer
    2. Offer opportunity to update NuGet references to latest 'green build'
      1. Dependency discovery
    3. Start build

Get Feature

Occasionally, another developer may wish to collaborate on a feature already under development. In this instance the new developers need to recreate the same workspace configuration within their own environment. The TFS build definition has its own set of source mappings that recreate the same level of isolation on the CI server, so this can be queried to extrapolate the solutions of interest.
  1. Gather requirements from the developer
  2. Inspect the identified TFS build definition
    1. Identify which domains the feature is using
    2. Recreate the matching TFS mappings into the developers own local workspace
    3. Perform a Get
  3. Start build

Reverse Integration

Reverse integration is very straight forward. The heavily cloaked feature branch only contains a small isolated group of solutions, so it takes TFS far less time to identify changes during the merge process.

Once the feature branch has been successfully merged back into Main, the isolated solutions are reunited with the rest of the platform. And without any missing dependencies,
  1. Perform TFS merge from feature to main
  2. Switch back all the NuGet references for actual relative path references
  3. Remove all redundant "packages.config"
  4. Remove all redundant "packages.config" references in "repository.config"
  5. Dependency discovery
  6. Start build

Hot Fixes

During the creation of a new branch, the Q&A will work out whether you're taking the 'Head' of the repository, or going back in time to a previous build. 

If you select an older build label, you are presumed to be performing a hot-fix on the current production version. The Q&A script will present a selection of 'QA approved' builds from TFS which will serve as the base of the hot-fix branch. 

The NuGet packages used to substitute absent domains are fixed to the same version of platform as the hot-fix branch source. This ensures that complete compatibility of the domains affected by the hot-fix with the current production version.

We typically release weekly, and so hot-fix branches are used frequently by the UI developers for getting quick updates out to production ahead of the next release.

The results

Full platform (Before)
This is how the world looked before we undertook the isolated feature work.

  • Disk footprint: 9GB
  • Time to first line of code: 50m



 Partial build (isolated feature)
This is a bit finger-in-the-air, as the code affected by a feature varies greatly depending on the scale o the feature.

  • Disk footprint: <500MB
  • Time to first line of code:  5m


Headline benefits
The things that really matter to the people I support:

  • We didn't have to buy another build agent after all
  • Work on a feature can start 45 minutes sooner
  • Our CI builds are over 6 minutes quicker
  • Build agent queue lengths are practically gone.
  • Publishing is over 20 minutes quicker
  • You can work on any part of the platform you need too
  • Back to a single manageable branch! 
Development time
Took approximately 4 weeks.

*Typically, up to 3 solutions are affected by a feature.

Comments

Post a Comment