Some Thoughts on Build Servers

Continuing from last week’s thoughts about build systems, let’s talk about build servers.

Say you’ve gotten a build system, like CMake, up and running in your project. Now the developers on your project are doing consistent builds across all your different platforms, and people are hopefully not missing important build steps anymore. However, there can still be differences between systems, and it’s hard for any one developer to try building on all platforms. Additionally, they could have some left over crud from old builds that throws things off.

The next step in sanity for your project workflow is to set up a build server. This is a box (or boxes) that sits there and pulls down clean copies of your codebase and does full builds, from scratch, all the time. It is kept in a pristine condition so that you don’t e.g. introduce a new dependency in your production binaries by updating Visual Studio. To do these builds, it runs some continuous integration package that lets developers trigger builds, check up on their status, and view build logs to find out why things broke. CI packages can also do fancier tricks like run unit tests, upload builds to QA or even the public, tag releases, and so on. (We’ll get into advantageous use of these in a later post.)

There are a lot of great build server options out there, and I’ve worked with or evaluated many of them. Let’s walk through them in order:

Tinderbox Mozilla’s Tinderbox was one of the first public continuous integration packages. Much like Bugzilla, it was the first place many people looked – and most people moved on to look at something better suited to their needs right away. This is not necessarily a knock on Bugzilla or Tinderbox, as they and their derivatives have continued to serve Mozilla just fine over the years.

PMEase QuickBuild QuickBuild was my first experience running a build server. We used QuickBuild 1.0 for C++ Torque builds on multiple platforms, which was weird and new – at the time, most build server packages – including this one – were very Java oriented. Luckily, we could call out to MSVC and XCode from Ant! PMEase has kept with it, and now they’re at version 4.0. I found them to have very responsive support, and QB itself was nice to configure and work with. It’s more expensive than some of the other options ($3k/site), but if you have the budget it’s worth a look.

Bamboo From Atlassian, I had high hopes for Bamboo, as JIRA is a powerful and reliable tool. However, when I last evaluated it (in late 2010 or so), I found its paradigm hard to understand. I just couldn’t figure out how I was supposed to use it – it had a lot of proprietary terminology that confused me. Additionally, it did not have good support for building all topic branches, which was a big part of the workflow I wanted my team to use. Looking it over again, I’m not sure it has improved on either front. However, I hold Atlassian in fairly high regard, so I am hopeful that someone else has figured it out and can enlighten me. 🙂

Jenkins/Hudson After a sale to Oracle, Jenkins was forked from Hudson to continue open source development. It runs on a wide variety of platforms, and it’s easy to set up. It has a large set of plugins of varying levels of maturity, and it’s not that hard to write your own. However, key plugins (for instance, the Git plugin) can have frustrating gaps and holes, and because it’s community driven, bugs can linger for a long time. The REST API is also weak in places, making it hard to extend with custom tools/scripts. My experience is that Jenkins is a solid choice for simpler projects but if you want to push your build server it can fall apart on you. We used it for several smaller projects, where it worked great, then we took it into a project with a large, 50+ member team of artists and developers. In that scenario, we ended up having to extend it heavily with custom scripts, mostly to add functionality it should have had to begin with.

JetBrains TeamCity TeamCity is free for lighter use, although heavier usage requires purchasing licenses from JetBrains. TeamCity has very solid Git/JIRA integration, and a well thought out UI. Setup isn’t hard and it has good support for adding distributed build agents. We’ve been very happy with it for our C++ and Ruby projects. It has good support for building topic branches, too.

In the end, what is crucial for continuous integration software? It should be reliable, and especially robust in the face of broken builds. It should be easy for the team to understand and use, especially when they are debugging build issues. It should be able to scale to build across all your platforms, quickly – it shouldn’t take more than 10 minutes or so for a full build across all platforms to complete.