Thursday, 21 June 2007

LazyWeb: spawning maven builds via VMWare in your CI tool

We're using the cute Bamboo tool from those nice Atlassian chaps to build projects like ActiveMQ, Camel and ServiceMix. Its got some lovely eye candy and reporting stuff.

The problem is, lots of the test cases of SOA projects like ActiveMQ often end up hard coding a port number to do some networking tests. The ActiveMQ test suite is probably one of the worst offenders :). (Yes I know, one day all those test cases should be refactored to use zero to create dynamic ports etc).

So we kinda have to run all of the test cases in a single build queue to avoid the test cases conflicting with each other on ports.

It'd be great if rather than just running mvn clean install or whatever, we could boot up maven in a separate VMWare image, with its own private ports and so forth - so the build runs in a separate virtual box and can't conflict with other tests.

e.g. using vmware-mvn clean install

Even better then, would be the ability to run different kinds of vmware images of maven (java 1.4, 1.5, 1.6 etc, then windows, solaris, linux etc).
mvn-win-1.5 clean install
mvn-linux-1.6 clean install
etc

If the VMWare ninja could be wrapped up in a command line that looked like the mvn command, this featre would just drop into all the various maven-supporting CI tools (of which there is a fair few; Hudson, Continuum, TemCity, LuntBuild etc). So the caller of vmware-mvn would get the incremental output via stdout; and after the build, the build files would be synced with the caller, so from the CI tool's perspective its just like calling mvn.

6 comments:

Ken Horn said...

as a middle ground, how about binding each test to a different IP address? then the ports can conflict and you can have, even, a dynamic hosts file with logical hostnames per test..

Hiram Chirino said...

Perhpas if the VMWare box is setup to use a remote file system like NFS or SMB that maps to the Bamboo file system then all Bamboo would have to do is to use ssh to issue a remote maven build command.

Mike Cannon-Brookes said...

Mate - isn't there a simpler solution here?

Bamboo can pass the build number into your build (and a number of other parameters too), so just do that as a system property.

Then, inside your build just bind to a port that's some function of the build number. For example, bind to 10000 + the last 3 digits of the build number?

Everything in your build and tests could simply look at that system property (or else use a default) to then determine what ports to run against.

Voila - individual, parallel builds. Or did I miss something?

m

James Strachan said...

Thanks for the comments guys.

Yeah - ideally I'd refactor all those hundreds of tests in all the various different projects to always take a system property / maven property to define the port to use. One of these days I'll get there & do that... its just gonna take a lot of time.

I'd still love to have a bunch of VMWare images of different JDKs / OSes which I can use on a single Bamboo server though! :)

BTW Thanks Mike - the build number idea is cool - I"ll definitely use that one of these days. Whats the property its available as?

Mark said...

James I've added a page about adding variables to your Bamboo build here

Basically, you should be able use the Maven goal

clean package -DbambooBuildNumber=${bamboo.buildNumber}

and then access the passed property in your pom.xml by using:

${env.bambooBuildNumber}

which you could then use to calculate a port number.

Give us a yell if you have any problems here.

James Strachan said...

Awesome, thanks Mark!