Babysitting your Continuous Integration System

Question

One of my roles in my team is the build person. I am responsible for maintaining/updating our build scripts and making sure we are building 'smoothly' on the continuous integration server. I usually do not mind this job, though often it feels like I am constantly babysitting the CI server.

This job can be annoying at times because if the build breaks I have to drop the story I am working on and investigate the build failure. Build failures happen daily on our team. Sometimes developers simply do not build locally before committing so tests fail on the CI server. In this situation I like to quickly get to the person who had the 'bad commit' so the build does not stay broken too long. Sometimes (a lot less frequently) a strange condition exists on the CI server that needs to be debugged.

I know that many mature teams use Continuous Integration but there is not a lot of material out there about good practices.

Do my problems point out that our continuous integration is not very mature or is this simply part of the job?

What are some good practices to follow? What are the characteristics of mature continuous integration?

Update

Instead of answering some comments I am going to make an update instead. We have a single, simple command that does exactly what the build server will do when building the app. It will compile, run all unit/integration and some quick UI based tests.

Reading everyone's answers, it feels we might have two major problems.

1. CI Server not complaining loud enough when a build fails.
2. Developers do not feel like its everyone's responsibility to make sure their commit goes through successfully.

What makes things harder in my team is that we have a large team (10+ developers) AND we have a couple of off-shore team members committing when we are not even at work. Because the team is large and we established that frequent, small commits are preferred, we sometimes have really a lot of activity in a day.

Answer 1

First and foremost: each person is responsible for the build process. It sounds like members in your team are not mature... No one gets away with writing code and fobbing it off to the CI server hoping that it works. Before committing code, it should be tested on their local machine. You should be sure that the code you're checking in isn't going to break the build. Of course, there are cases when the build break unintentionally (e.g. if a config file has been changed or sloppy commit was inadvertently made).

Most CI servers (I've only used Hudson) will send an automated email detailing the commits made that caused the build to break. The only part of your role is to stand behind them looking tough until the suspect fixes whatever they broke.

Answer 2

Your team got one thing wrong:

Being responsible for the build server is not the same as being responsible for the build.

It is the responsibility of the person checking in code to make it "work" (for some value of work). The sole reason for having a build server is to catch oversights in this process. Any build server worth its salt can notify the person(s) who have checked in code since the last build (as well as anyone else who is interested) with the following information:

• Build broke!
• What went wrong when building!
• What has changed since the last build!

This very frequently happen by email. It is important that this happens quite quickly after each check-in.

The person can then see what went wrong, and fix it, and the build server then notify anyone interested that the build is back to normal. This should happen all by itself without any involvement of others but the check-in culprits.

So to answer your question: A mature CI environment do NOT require the involvement of a janitor in its normal operation.

Also, if "strange conditions exists" happen very frequently then find out what cause it, and make the system more robust.

Answer 3

Change the process. If a commit breaks the build, automatically rollback that commit and notify the developer who broke it. It's silly to allow an error by one team member to slow down the rest of the team. Or instead of having integration builds done automatically, have the developers check out the integration machine, and if the build succeeds, then they can commit. Continuous integration does not mean "check in whatever junk you like and someone will fix it for you".

The "golden branch" strategy doesn't work unless there is a gatekeeper for the golden branch.

A DVCS like Git might help; instead of committing, the developer can just submit a changeset for integration to the CI server, and the server can then attempt to integrate the changeset. If the integration succeeds, the changeset is merged, and if not it is rejected.

Answer 4

One problem I often see is that developers cannot perform a local build that has exactly the same steps as a CI build. That is, the CI server is configured to include extra steps such as unit/integration tests, coverage, etc that cannot be performed locally. Inevitably, developers will be bitten by one of the steps that they cannot perform locally and will begin to question why they're bothering to build a release locally at all prior to check-in.

I keep my entire build self-contained and have the CI server simply kick off a release build with no extraneous configuration/steps defined. Developers can run a release build locally prior to check-in which includes all the steps that will be performed by the CI build and be far more confident that nothing will break when they do check-in.

Added benefits to this approach include:

• it's easy to switch between CI servers because you've not invested a bunch of time configuring extraneous steps
• all build-time tooling is under source control, which means all developers are using exactly the same tool set to build your system
• in addition to the above point, it is simple to alter build tooling in a controlled manner

PS. the whole build babysitter concept is ridiculous, but others have covered that.

Answer 5

First off, developers should not be breaking builds regularly - they should be building and running the tests locally before they commit to the CI branch. It should be a mark of shame to break the build, and it's important to enforce that. I've done it through posting statistics, and I've seen other teams have a "build bag" where you put in a dollar every time you break the build. At the end of the project that money goes toward beer.

If shame/personal pride doesn't work, you may have to move to heavier stuff (e.g. threatening termination). Breaking the build before leaving for the day should be a major offense. And every developer should have a build status notification on their desktop. The best part of all this is that it encourages smaller commits, which are preferred anyway.

That said, the build will sometimes break (CI configuration reasons, for instance). And sometimes people will screw up and leave for the day with the build broken. That's why you should aim for a process which allows for quick and easy rollbacks to known good versions. If you can always roll back to the last good build (and have the rolled-back version get deployed to all the necessary places), then in the worst-case scenario of a broken build where the culprit has left for the evening you can roll back to the last good version and yell at him/her in the morning.

I can't recommend the Continuous Delivery book enough. If you're looking for a guide on how to mature your CI process, give it a try.

Answer 6

I heard about a thing Microsoft (maybe?) do, which is to have the role of build babysitter move around the team. The way they do this is that when someone breaks the build (which should probably include checking in something which fails its tests), they assume the role. This makes people responsible for the consequences of their actions in a very direct way. And given that it's a somewhat annoying job to do, it encourages them not to break the build again.

The person who is currently responsible for the build could have a special hat. There could be a ceremony for handing it over.

Note that as Thorbjørn says, being responsible for the build is not the same as being responsible for the build server. Responsibility for the server could rest permanently with one or more more infrastructurally inclined members of the team while responsibility for the build moves around.

Now, details of process aside, i am going to join the chorus of people expressing dismay at developers checking in without having run a build and test. Unacceptable!

If you have some members of the team who are more likely to break the build (and based on my own experience, i'm mostly thinking of the members in another country), and if you're using some nice modern source control like Mercurial or Git, you could have them check in to a different branch to the rest of the team, run a separate CI process on that, and automatically merge changes from that branch to the trunk after a successful build (note that you'd have to run a second build and test after the merge before checking the merge in!). Since automatic merging is not always successful, this would eventually leave the branch requiring manual attention, though, which could be a real pain. Might be less painful than having broken code checked in for the rest of the team, though.

Answer 7

As Jonathan Khoo said you should all be responsible for the build server and the build script. There are three reasons:

1. At the moment you have a case of a "Bus of 1" which means that if you got run over by a bus then the whole knowledge of the build server and build scripts is lost.
2. The scripts that have been written (rightly or wrongly) by you have only had your input. Just like any code the more people that are involved the broader the knowledge base that can be applied to it.
3. Finally when something goes wrong only you are feeling the pain. Pain is a good thing but not when it is isolated. At the moment you are dealing with the pain but if everyone else was having the pain then you would find they would eventually be more stringent in testing code before committing.

I am very involved with CI myself and fall into the trap of being the one that maintains the scripts but here are a couple of things you can do to mitigate that.

1. Build scripts should not just run on the CI servers but on local development machines as well. They should produce the same outputs, run the same tests and fail for the same reasons. This allows developers to run the script before committing their code.
2. If any one does break the build make it public via the use of task tray popups, emails, flashing lights, noises etc. It serves to not only embarrass the team members but it provides an opportunity for everyone else in the team to jump up and assist.
3. For a while avoid fixing the build. Get someone else to do it. If no one else jumps up wait for bad things to happen and use it as a learning point for the whole team to understand why the CI server is important.
4. Try and keep your build server and development machines as devoid of installed third party components as much as possible, especially keep the GAC clean. Rely on third party components being in the projects library folder. This helps identify missing components more quickly.

Answer 8

Visibility will help you out. All team members need to know that there is an active problem that they should fix. Email notifications are useful, but if a developer is busy and does not react to it immediately, he will probably forget about it and the email will end up in a huge notifications pile.

You can try using tools like Catlight or BuildNotify. They show the current status of the important builds in tray area. Every time developer looks at the clock he will see that there is a broken build that need to be fixed.

Catlight will also show who broke the build first, putting social pressure on this person, as the whole team will see it on each consecutive build failure.

Answer 9

One strategy is to use lots of small branches for lots of small projects. Then when someone breaks the build, they are just breaking the build for themselves. So they get the annoyed email from the build server and it is up to them to worry about it.

Another is to improve people's responsibility level. For example if you use something like Rietveld then people can't commit without passing peer review. (The process in practice is much lighter weight than you think it is. But it does force people to "pipeline" and work on multiple things at once.) Preserving the build is the responsibility of both the committer and reviewers. If anyone is either regularly breaking the build or approving things that break the build, then don't let them give final approvals for commits. Combine with a process where anyone can easily roll back any change, and the build won't break as often, and won't stay broken once the change is made.

Answer 10

I am going to break with the chorus here, and say that actually, breaking the build occasionally is not such a bad thing, as it shows that actual work is being done. Yes, developers should build and test before they commit, but the main burden of testing should be borne by the development automation tools, including the continuous integration server. The tools are there to be used, and unless the build breaks now and again, then it is not clear that you are pushing as hard as you could. I do, however, think that the build should never ever remain broken for any significant length of time, and would even favour automatic rollbacks or a multi stage commit process if that helps support the twin objectives of quick feedback from centralised, automated test facilities, plus a "green" trunk.

Answer 11

Couples things to keep in mind:

1. Your team needs a set of standards to follow
2. You need to get management on board with the idea of clean code and strive for improved code practices.
3. Test test test! Always test before you check in.
4. Although I do agree it is ok to break build, that's a rare and I mean extremely rare occasion that would be acceptable. If this is daily, I'd be looking for the door or talking to my boss about standards.

Overall you guys need to set down, write up standards based on best practices and not your practices individually (obviously those are not working out). Once everyone agrees on standards, start code review process and enforcing standards. It almost sounds like management went on vacation and never came back. These are things that honestly are pretty basic for any shop. the foundation of good code starts with good standards to dictate the teams code and how its written. Just my thoughts. I recently walked into a similar situation at my new job and I talked to my boss. Explained to him we need XYZ to be completed because it affects ABC. Two weeks later I wrote a list of code standards to follow and presented it. Followed by my co workers giving there input and about 2 months later we had standards in place which resolved tons of issues.