Monday, 14 July 2014

Implementing DotCover from Powershell

We've just made the jump to JetBrains DotCover, and I'm reasonably impressed by the command line implementation. I think its worth mentioning that the Developers's are very happy with the Visual Studio integration, but that's not a part of the world I inhabit so I couldn't comment.

Supporting DotCover in our build pipeline was almost painless. The only part where I struggled was on the DotCover command line documentation. It took me a good while to realise that TargetArguments is where I specify the command line arguments used by the test-runner. In our case, the actual MSTest command line arguments.

Excuses aside, here's my Powershell invocation of JetBrains DotCover.

At present, we do not use the XML configuration files. But, when Developers start producing their own XML configurations within each visual studio solution, I shall implement that functionality later.
Instrumentation & Test

We automatically exclude the test assembly namespace from the coverage analysis.
  • We're not interested in how well our tests cover our tests.
  • i.e. com.product.domain.project.tests 
The scope of the coverage analysis is limited to only the project under examination.
  • Any unintentional coverage of other projects is discarded.
  • i.e. com.product.domain.project 
Individual coverage reports are merged into a domain report last
  • i.e. com.product.domain

$testRunner = "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe"
$testContainers = "D:\org\branch\namespace\domain\project.tests\bin\Output\Org.Namespace.Domain.Project.Tests.dll","D:\org\branch\namespace\domain\project.tests\bin\output\Org.Namespace.Domain.Project.Tests.dll"
foreach($test in $testContainers)
{
   $testAssembly = Get-Item $test
   $namespace = $testAssembly.BaseName -replace ".tests", ""
   $testName = $testAssembly.Name
   $testDirectory = (Split - Path $test - Parent)
   $testReport = "C:\temp\$testName.trx" & $coverageTool cover /TargetExecutable = "$testRunner" / Filters = "-:$namespace.Tests;+:class=$namespace*" /TargetArguments = "/testContainer:$test /resultsFile:$testReport" / Output = "C:\temp\$testName.dcvr" / LogFile ="C:\DotCover.log.txt"
}


Report Merging

The next steps merge up all the individual coverage reports generated by the unit test runs. This gives us our test coverage for the domain : com.product.domain.project.
$testReports = $testContainers | % {
   $name = Split-Path $_ -Leaf
   return ("c:\temp\{0}.dcvr" -f $name)
}

$testReportArgument = [String]::Join(";", $testReports)
& $coverageTool merge /Source="$testReportArgument" /Output="C:\temp\mergedSnapshots.dcvr"

Report Generation

The unified domain coverage reports can now be transformed into useful reports
HTML Report
Handy for quick review of coverage issues. We archive these reports in the build artefacts for the purposes of providing quick lookups. The reports persist until removed by the TFS retention policies.

& $coverageTool report /Source="C:\temp\mergedSnapshots.dcvr" /Output="C:\temp\mergedReport.html" /ReportType="HTML"

XML Report

Ideal format for the next stage where I extract the coverage metrics to complete the code quality assessment in the build pipeline.

& $coverageTool report /Source="C:\temp\mergedSnapshots.dcvr" /Output="C:\temp\mergedReport.xml" /ReportType="XML"

& $coverageTool report /Source="C:\temp\mergedSnapshots.dcvr" /Output="C:\temp\mergedReport.xml" /ReportType="XML"

Summary Extraction

The XML report provides a lot of information, but for the purposes of "Pass or Fail", I just need the combined coverage percentage. If it's falls below our agreed threshold, then the build must fail.

[xml]$coverageAnalysis = Get-Content "C:\temp\$mergedReport.xml"

$blocksCovered = $coverageAnalysis.Root.CoveredStatements;

$totalBlocks = $coverageAnalysis.Root.TotalStatements;

$totalCoverage = $coverageAnalysis.Root.CoveragePercent;




Thursday, 10 July 2014

Enriching our build tweets

We have several Team Foundation projects now running concurrently, and switching between these projects and work-spaces in Visual Studio 2013 can be time consuming and frustrating chore.

For ease of use, I prefer to aggregate my build messages into a single location. In the past I've used the SonarQube build management to aggregate information about builds in a single repository, and I probably will again at some point in the future. But, at the moment we're using a hidden twitter account to push all status messages into.



The Tweet gives us a quick over-view of the build outcome.

We've added a BitLy link to each tweet that references an internal HTTP server that gives us quick access to the build transcript generated by TFS.

How we used Powershell to talk to Twitter is covered in an older post, but adding a Bit.Ly link is demonstrated below:

$username = "-----------"
$apiKey = "-------------------------------" # Legacy API Key

Function Get-ShortURL {
Param($longURL)
$url = "http://api.bit.ly/shorten?version=2.0.1&format=xml&longUrl=$longURL&login=$username&apiKey=$apiKey"
$request = [net.webrequest]::Create($url)
$responseStream = new-object System.IO.StreamReader($request.GetResponse().GetResponseStream())
$response = $responseStream.ReadToEnd()
$responseStream.Close()

$result = [xml]$response
Write-Output $result.bitly.results.nodeKeyVal.shortUrl
}


Get-ShortURL "http://server.local/Build26098.txt" 


Finding satellite and indirect assembly references during the build

Over the past few years, I've posted on several occasions about missing implicit references.

The solutions I came up with for each of these posts were simply managing the symptoms of a hidden problem. But, we couldn't quite put our finger on the real underlying problem. 

After we switched over to NuGet packages to improve our feedback cycles, the problem presented itself more frequently. Everything came to a head when a key facet of our framework assembly was unable to function on application start-up. At this point, we had to find out what the real problem was and we had a new symptom to work with. 

It seemed that core functionality that was supplied by satellite (indirectly referenced) assemblies was unable to function, as during the build process MSBuild was not able to recognize the need for and/or locate the required satellite assemblies and so didn't copy them into the output folder.

After scrabbling around on the internet for a good half a day, we found this genius MSBuild Custom Target, that used .NET reflection during the AfterBuild stage to grab any and all indirect references.

<Target Name="AfterBuild">
    <!-- Here's the call to the custom task to get the list of dependencies -->
    <ScanIndirectDependencies StartFolder="$(MSBuildProjectDirectory)" StartProjectReferences="@(ProjectReference)" Configuration="$(Configuration)">
        <Output TaskParameter="IndirectDependencies" ItemName="IndirectDependenciesToCopy" />
    </ScanIndirectDependencies>

    <!-- Only copy the file in if we won't stomp something already there -->
    <Copy SourceFiles="%(IndirectDependenciesToCopy.FullPath)" DestinationFolder="$(OutputPath)" Condition="!Exists('$(OutputPath)\%(IndirectDependenciesToCopy.Filename)%(IndirectDependenciesToCopy.Extension)')" />
</Target>


<!-- THE CUSTOM TASK! -->
<UsingTask TaskName="ScanIndirectDependencies" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v$(MSBuildToolsVersion).dll">
    <ParameterGroup>
        <StartFolder Required="true" />
        <StartProjectReferences ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true" />
        <Configuration Required="true" />
        <IndirectDependencies ParameterType="Microsoft.Build.Framework.ITaskItem[]" Output="true" />
    </ParameterGroup>
    <Task>
        <Reference Include="System.Xml" />
        <Using Namespace="Microsoft.Build.Framework" />
        <Using Namespace="Microsoft.Build.Utilities" />
        <Using Namespace="System" />
        <Using Namespace="System.Collections.Generic" />
        <Using Namespace="System.IO" />
        <Using Namespace="System.Linq" />
        <Using Namespace="System.Xml" />
        <Code Type="Fragment" Language="cs">
          <![CDATA[
var projectReferences = new List<string>();
var toScan = new List<string>(StartProjectReferences.Select(p => Path.GetFullPath(Path.Combine(StartFolder, p.ItemSpec))));
var indirectDependencies = new List<string>();

bool rescan;
do{
  rescan = false;
  foreach(var projectReference in toScan.ToArray())
  {
    if(projectReferences.Contains(projectReference))
    {
      toScan.Remove(projectReference);
      continue;
    }

    Log.LogMessage(MessageImportance.Low, "Scanning project reference for other project references: {0}", projectReference);

    var doc = new XmlDocument();
    doc.Load(projectReference);
    var nsmgr = new XmlNamespaceManager(doc.NameTable);
    nsmgr.AddNamespace("msb", "http://schemas.microsoft.com/developer/msbuild/2003");
    var projectDirectory = Path.GetDirectoryName(projectReference);

    // Find all project references we haven't already seen
    var newReferences = doc
          .SelectNodes("/msb:Project/msb:ItemGroup/msb:ProjectReference/@Include", nsmgr)
          .Cast<XmlAttribute>()
          .Select(a => Path.GetFullPath(Path.Combine(projectDirectory, a.Value)));

    if(newReferences.Count() > 0)
    {
      Log.LogMessage(MessageImportance.Low, "Found new referenced projects: {0}", String.Join(", ", newReferences));
    }

    toScan.Remove(projectReference);
    projectReferences.Add(projectReference);

    // Add any new references to the list to scan and mark the flag
    // so we run through the scanning loop again.
    toScan.AddRange(newReferences);
    rescan = true;

    // Include the assembly that the project reference generates.
    var outputLocation = Path.Combine(Path.Combine(projectDirectory, "bin"), Configuration);
    var localAsm = Path.GetFullPath(Path.Combine(outputLocation, doc.SelectSingleNode("/msb:Project/msb:PropertyGroup/msb:AssemblyName", nsmgr).InnerText + ".dll"));
    if(!indirectDependencies.Contains(localAsm) && File.Exists(localAsm))
    {
      Log.LogMessage(MessageImportance.Low, "Added project assembly: {0}", localAsm);
      indirectDependencies.Add(localAsm);
    }

    // Include third-party assemblies referenced by file location.
    var externalReferences = doc
          .SelectNodes("/msb:Project/msb:ItemGroup/msb:Reference/msb:HintPath", nsmgr)
          .Cast<XmlElement>()
          .Select(a => Path.GetFullPath(Path.Combine(projectDirectory, a.InnerText.Trim())))
          .Where(e => !indirectDependencies.Contains(e));

    Log.LogMessage(MessageImportance.Low, "Found new indirect references: {0}", String.Join(", ", externalReferences));
    indirectDependencies.AddRange(externalReferences);
  }
} while(rescan);

// Expand to include pdb and xml.
var xml = indirectDependencies.Select(f => Path.Combine(Path.GetDirectoryName(f), Path.GetFileNameWithoutExtension(f) + ".xml")).Where(f => File.Exists(f)).ToArray();
var pdb = indirectDependencies.Select(f => Path.Combine(Path.GetDirectoryName(f), Path.GetFileNameWithoutExtension(f) + ".pdb")).Where(f => File.Exists(f)).ToArray();
indirectDependencies.AddRange(xml);
indirectDependencies.AddRange(pdb);
Log.LogMessage("Located indirect references:\n{0}", String.Join(Environment.NewLine, indirectDependencies));

// Finally, assign the output parameter.
IndirectDependencies = indirectDependencies.Select(i => new TaskItem(i)).ToArray();
      ]]>
        </Code>
    </Task>
</UsingTask>


With this custom target in place, our output folders became much larger, containing all the indirect references that had previously been lost.

There was another blog post we found that takes the original idea and expands upon the satellite reference discovery to go even deeper - but this wasn't needed in our case.

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="12.0" DefaultTargets="Build">
<!-- All the stuff normally found in the project, then in the AfterBuild event... -->
<Target Name="AfterBuild">
    <!-- Here's the call to the custom task to get the list of dependencies -->
    <ScanIndirectDependencies StartFolder="$(MSBuildProjectDirectory)" StartProjectReferences="@(ProjectReference)" Configuration="$(Configuration)">
        <Output TaskParameter="IndirectDependencies" ItemName="IndirectDependenciesToCopy"/>
    </ScanIndirectDependencies>
    <!-- Only copy the file in if we won't stomp something already there -->
    <Copy SourceFiles="%(IndirectDependenciesToCopy.FullPath)" DestinationFolder="$(OutputPath)" Condition="!Exists('$(OutputPath)\%(IndirectDependenciesToCopy.Filename)%(IndirectDependenciesToCopy.Extension)')"/>
</Target>
<!-- THE CUSTOM TASK! -->
<UsingTask TaskName="ScanIndirectDependencies" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v12.0.dll">
    <ParameterGroup>
        <StartFolder Required="true"/>
        <StartProjectReferences ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true"/>
        <Configuration Required="true"/>
        <IndirectDependencies ParameterType="Microsoft.Build.Framework.ITaskItem[]" Output="true"/>
    </ParameterGroup>
    <Task>
        <Reference Include="System.Xml" />
<Using Namespace="Microsoft.Build.Framework" />
<Using Namespace="Microsoft.Build.Utilities" />
<Using Namespace="System" />
<Using Namespace="System.Collections.Generic" />
<Using Namespace="System.IO" />
<Using Namespace="System.Linq" />
<Using Namespace="System.Xml" />
<Code Type="Fragment" Language="cs">
      <![CDATA[
var projectReferences = new List<string>();
var toScan = new List<string>(StartProjectReferences.Select(p => Path.GetFullPath(Path.Combine(StartFolder, p.ItemSpec))));
var indirectDependencies = new List<string>();

bool rescan;
do{
  rescan = false;
  foreach(var projectReference in toScan.ToArray())
  {
    if(projectReferences.Contains(projectReference))
    {
      toScan.Remove(projectReference);
      continue;
    }

    Log.LogMessage(MessageImportance.Low, "Scanning project reference for other project references: {0}", projectReference);

    var doc = new XmlDocument();
    doc.Load(projectReference);
    var nsmgr = new XmlNamespaceManager(doc.NameTable);
    nsmgr.AddNamespace("msb", "http://schemas.microsoft.com/developer/msbuild/2003");
    var projectDirectory = Path.GetDirectoryName(projectReference);

    // Find all project references we haven't already seen
    var newReferences = doc
          .SelectNodes("/msb:Project/msb:ItemGroup/msb:ProjectReference/@Include", nsmgr)
          .Cast<XmlAttribute>()
          .Select(a => Path.GetFullPath(Path.Combine(projectDirectory, a.Value)));

    if(newReferences.Count() > 0)
    {
      Log.LogMessage(MessageImportance.Low, "Found new referenced projects: {0}", String.Join(", ", newReferences));
    }

    toScan.Remove(projectReference);
    projectReferences.Add(projectReference);

    // Add any new references to the list to scan and mark the flag
    // so we run through the scanning loop again.
    toScan.AddRange(newReferences);
    rescan = true;

    // Include the assembly that the project reference generates.
    var outputLocation = Path.Combine(Path.Combine(projectDirectory, "bin"), Configuration);
    var localAsm = Path.GetFullPath(Path.Combine(outputLocation, doc.SelectSingleNode("/msb:Project/msb:PropertyGroup/msb:AssemblyName", nsmgr).InnerText + ".dll"));
    if(!indirectDependencies.Contains(localAsm) && File.Exists(localAsm))
    {
      Log.LogMessage(MessageImportance.Low, "Added project assembly: {0}", localAsm);
      indirectDependencies.Add(localAsm);
    }

    // Include third-party assemblies referenced by file location.
    var externalReferences = doc
          .SelectNodes("/msb:Project/msb:ItemGroup/msb:Reference/msb:HintPath", nsmgr)
          .Cast<XmlElement>()
          .Select(a => Path.GetFullPath(Path.Combine(projectDirectory, a.InnerText.Trim())))
          .Where(e => !indirectDependencies.Contains(e));

    Log.LogMessage(MessageImportance.Low, "Found new indirect references: {0}", String.Join(", ", externalReferences));
    indirectDependencies.AddRange(externalReferences);
  }
} while(rescan);

// Expand to include pdb and xml.
var xml = indirectDependencies.Select(f => Path.Combine(Path.GetDirectoryName(f), Path.GetFileNameWithoutExtension(f) + ".xml")).Where(f => File.Exists(f)).ToArray();
var pdb = indirectDependencies.Select(f => Path.Combine(Path.GetDirectoryName(f), Path.GetFileNameWithoutExtension(f) + ".pdb")).Where(f => File.Exists(f)).ToArray();
indirectDependencies.AddRange(xml);
indirectDependencies.AddRange(pdb);
Log.LogMessage("Located indirect references:\n{0}", String.Join(Environment.NewLine, indirectDependencies));

// Finally, assign the output parameter.
IndirectDependencies = indirectDependencies.Select(i => new TaskItem(i)).ToArray();
]]>
        </Code>
    </Task>
</UsingTask>
</Project>


All of the missing assembly reference issues we'd previously had were gone. At this point, we could remove all the explicit test assembly references and the run-time discovery framework worked entirely as it should.