Ian's Blog

Build serialization using a mutex from NAnt

with 3 comments

A frequent topic on the CruiseControl.NET (CCNet) mailing list is how to serialize builds. Lots of people have multiple projects that build on one server, and sometimes dependency relationships between those projects are fairly complex. Solving the problem to everyone’s satisfaction without burdening everyone with its complexity is quite a challenge, which is probably why there’s no functionality in CCNet to accomodate this to date.

We don’t have to deal with dependencies for our builds at SourceGear, but we did have to resolve some serialization issues when we first set it up.

We have five projects currently built by CruiseControl.NET:

  • Dragnet 1.0.x
  • Dragnet Trunk
  • Vault 3.1.x
  • Vault 3.5.x
  • Vault Trunk

For people whose build times run into hours, it’s important to build as little as possible to fully reap the rewards of Continuous Integration. Including all unit tests, Vault builds in about 45 minutes, and Dragnet less than that. We have a smaller set of tests that runs on every checkin to get Vault’s time down around 20 minutes. We essentially build all the code for every build on each of these projects, sparing us the dependency issues. Still, there are a couple of reasons we couldn’t just allow anything to build at any time:

  1. The unit tests require that the Vault server be installed and you can only have one instance of a Vault server installed on a machine. We can’t have multiple Vault builds trying to install the server willy-nilly.
  2. We use Wise for our installers, and strange things happen when multiple instances of it are running.

Initially, I cobbled up some modifications to CCNet to only allow one build at a time. This worked, but it sometimes made the wait for a build much longer than it had to be. In our configuration, when CCNet does a build it’s actually doing all these things:

  1. Clean out the source directory
  2. Get the latest source
  3. Build
  4. Create installers
  5. Install server
  6. Unit tests
  7. Uninstall server
  8. Label source

My modified version of CCNet essentially made the entire process one big critical section. It was effective, but far from optimal. At one point last fall someone on the mailing list mentioned that they had created a mutex script in NAnt that gave them finer-grained locking. John Hardin at CRS Retail Systems was kind enough to save me the effort of rolling my own and provided his script.

Adding this block to your NAnt script gives you the ability to add named mutexes as NAnt tasks. In our configuration, steps 3 through 7 are all performed within a NAnt script that CCNet kicks off. For the Vault builds, steps 5 through 7 are encapsulated by a mutex. Each build waits for it’s turn to install and test a server:

<mutex mutexName="VAULT_SERVER_MUTEX" />

<!--
Install Server...
Do Unit Tests...
Uninstall Server...
-->

<mutex mutexName="VAULT_SERVER_MUTEX" />

We did the same thing with a “Wise” mutex for the parts of the script that create the Wise installer.

The NAnt mutex method is just as effective as one huge lock from CCNet, but because we’re locking only when necessary, builds spend a lot less time waiting for others to finish. And we no longer need to run a customized version of CCNet.

Even if CCNet includes serialization features some day, I can’t imagine that this level of locking would be possible. Once you’re running NAnt (or MSBuild, or whatever) you’re outside CCNet’s control: locking within NAnt itself ought to be a useful trick for some time.

Advertisements

Written by Ian Olsen

October 23, 2006 at 2:45 pm

3 Responses

Subscribe to comments with RSS.

  1. Another useful mutex application:
    We have a fast continuous integration build project in CCNet for each software system (application). We also have a test deployment project that does a complete release build and deploys to all test servers. Both builds use the same NAnt script and thus the same build output directory and source repository. It’s important not to have the two running at the same time so we use a mutex to keep a given system’s builds separate.

    Peter Lanoie

    November 7, 2006 at 12:28 pm

  2. The link to John Hardin’s script returns an error:

    — 403: Access Denied —
    This file requires authorization:
    You must both be a user of this blog as well as be currently logged into WordPress.com

    :sadface:

    boinst

    February 20, 2012 at 5:36 pm

  3. The link to John Hardin’s script returns an error: still returns error, even as logged in wordpress user

    lbenhart

    May 2, 2014 at 9:44 am


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: