TFS Build agents not picking up the next job in the queue

We have two build agents, and from time to time a situation occurs where only one build agent collects jobs from the queue. The other build agent, just sits idle.

In essence, normal priority builds are ignored in TFS.

What didn't work?

  • Restarted the Build Controller
  • Restarted both build controllers

What was it?

There was a job that had never completed in TFS, and was still marked as "In Progress".
This job was assigned to the build agent that wasn't picking up any new jobs.

Why TFS was no longer showing this job Visual Studio is another matter entirely, but we were confident that this job was dead and buried. It was over a month old, and the build agents had been rebooted many times since then. It was definitely not still running.

Query [tbl_BuildQueue] in our TFS database we found the old job with the status of "1"

0  – None
1  – In progress
2  – Queued
4  - Postponed
8  – Completed
16 – Cancelled

So, we used SQL to manually mark this job as cancelled.

Taking a note of the QueueId, we set the status of the stuck job to "16" and this released the build agent to start picking up new jobs from the queue.

Update tbl_BuildQueue set status=16 where QueueId = ______

This is an action I need to perform once every few months.