One of my roles in my team is the build person. I am responsible for maintaining/updating our build scripts and making sure we are building 'smoothly' on the continuous integration server. I usually do not mind this job, though often it feels like I am constantly babysitting the CI server.

This job can be annoying at times because if the build breaks I have to drop the story I am working on and investigate the build failure. Build failures happen daily on our team. Sometimes developers simply do not build locally before committing so tests fail on the CI server. In this situation I like to quickly get to the person who had the 'bad commit' so the build does not stay broken too long. Sometimes (a lot less frequently) a strange condition exists on the CI server that needs to be debugged.

I know that many mature teams use Continuous Integration but there is not a lot of material out there about good practices.

Do my problems point out that our continuous integration is not very mature or is this simply part of the job?

What are some good practices to follow? What are the characteristics of mature continuous integration?


Instead of answering some comments I am going to make an update instead. We have a single, simple command that does exactly what the build server will do when building the app. It will compile, run all unit/integration and some quick UI based tests.

Reading everyone's answers, it feels we might have two major problems.

  1. CI Server not complaining loud enough when a build fails.
  2. Developers do not feel like its everyone's responsibility to make sure their commit goes through successfully.

What makes things harder in my team is that we have a large team (10+ developers) AND we have a couple of off-shore team members committing when we are not even at work. Because the team is large and we established that frequent, small commits are preferred, we sometimes have really a lot of activity in a day.

  • 17
    Wow, I've heard of developers not testing code on their local machine before committing, but not building it? That's practically criminal.
    – Aaronaught
    Commented Jun 9, 2011 at 1:30
  • 2
    @Alison: Maybe that's so, although "breaking the build" to me means committing code that doesn't build. A failed test is a much less critical issue; it's usually not going to prevent other developers from getting their work done.
    – Aaronaught
    Commented Jun 9, 2011 at 13:55
  • 1
    @Aaronaught personally I'd see it as the opposite - code that compiles might still fail automated testing and therefore "break the build".
    – Armand
    Commented Jun 10, 2011 at 10:07
  • 2
    To quote a soundbite from Martin Fowler: if it hurts, do it more often.
    – rwong
    Commented Oct 19, 2015 at 19:56
  • 1
    Someone (if I remember correctly, it was Ward Cunningham) during a lecture told me the practice of his team: the one who broke the build had to wear a T shirt for the day, with the words "I broke the build" on it. He also mentioned that T shirt never got washed.
    – Doc Brown
    Commented Apr 12, 2016 at 19:17

11 Answers 11


First and foremost: each person is responsible for the build process. It sounds like members in your team are not mature... No one gets away with writing code and fobbing it off to the CI server hoping that it works. Before committing code, it should be tested on their local machine. You should be sure that the code you're checking in isn't going to break the build. Of course, there are cases when the build break unintentionally (e.g. if a config file has been changed or sloppy commit was inadvertently made).

Most CI servers (I've only used Hudson) will send an automated email detailing the commits made that caused the build to break. The only part of your role is to stand behind them looking tough until the suspect fixes whatever they broke.

  • 3
    What happens if its late in the day and they leave work? Should there be a rule that you cannot commit unless you can make sure it resulted a successful build?
    – c_maker
    Commented Jun 9, 2011 at 0:52
  • 4
    I am afraid that threatening will encourage large overarching commits instead of small focused ones which are preferred.
    – c_maker
    Commented Jun 9, 2011 at 1:41
  • 3
    @c_maker: Wouldn't small, focused commits be less likely to break the build? It sounds like a discipline problem to me, not a process problem.
    – Aaronaught
    Commented Jun 9, 2011 at 1:53
  • 6
    Anyone who breaks the build is to get coffee / cake / whatever your team's preferred candy for everybody. Or get coffee for everybody all day long, or... There are many measures you can come up with that make breaking the build unwelcome, while not being so threatening as to get people to avoid submitting. The latter can also be somewhat addressed by requiring everybody to at least submit their changes once a week. (Once a day is too often when working on something more substantial.) Commented Jun 9, 2011 at 6:12
  • 5
    Who cares about who breaks the build, as long as they FIX it?
    – user1249
    Commented Jun 9, 2011 at 14:00

Your team got one thing wrong:

Being responsible for the build server is not the same as being responsible for the build.

It is the responsibility of the person checking in code to make it "work" (for some value of work). The sole reason for having a build server is to catch oversights in this process. Any build server worth its salt can notify the person(s) who have checked in code since the last build (as well as anyone else who is interested) with the following information:

  • Build broke!
  • What went wrong when building!
  • What has changed since the last build!

This very frequently happen by email. It is important that this happens quite quickly after each check-in.

The person can then see what went wrong, and fix it, and the build server then notify anyone interested that the build is back to normal. This should happen all by itself without any involvement of others but the check-in culprits.

So to answer your question: A mature CI environment do NOT require the involvement of a janitor in its normal operation.

Also, if "strange conditions exists" happen very frequently then find out what cause it, and make the system more robust.


Change the process. If a commit breaks the build, automatically rollback that commit and notify the developer who broke it. It's silly to allow an error by one team member to slow down the rest of the team. Or instead of having integration builds done automatically, have the developers check out the integration machine, and if the build succeeds, then they can commit. Continuous integration does not mean "check in whatever junk you like and someone will fix it for you".

The "golden branch" strategy doesn't work unless there is a gatekeeper for the golden branch.

A DVCS like Git might help; instead of committing, the developer can just submit a changeset for integration to the CI server, and the server can then attempt to integrate the changeset. If the integration succeeds, the changeset is merged, and if not it is rejected.


One problem I often see is that developers cannot perform a local build that has exactly the same steps as a CI build. That is, the CI server is configured to include extra steps such as unit/integration tests, coverage, etc that cannot be performed locally. Inevitably, developers will be bitten by one of the steps that they cannot perform locally and will begin to question why they're bothering to build a release locally at all prior to check-in.

I keep my entire build self-contained and have the CI server simply kick off a release build with no extraneous configuration/steps defined. Developers can run a release build locally prior to check-in which includes all the steps that will be performed by the CI build and be far more confident that nothing will break when they do check-in.

Added benefits to this approach include:

  • it's easy to switch between CI servers because you've not invested a bunch of time configuring extraneous steps
  • all build-time tooling is under source control, which means all developers are using exactly the same tool set to build your system
  • in addition to the above point, it is simple to alter build tooling in a controlled manner

PS. the whole build babysitter concept is ridiculous, but others have covered that.

  • amen. just because something can be done, doesn't mean it always should be done.
    – gbjbaanb
    Commented Jun 9, 2011 at 12:00

First off, developers should not be breaking builds regularly - they should be building and running the tests locally before they commit to the CI branch. It should be a mark of shame to break the build, and it's important to enforce that. I've done it through posting statistics, and I've seen other teams have a "build bag" where you put in a dollar every time you break the build. At the end of the project that money goes toward beer.

If shame/personal pride doesn't work, you may have to move to heavier stuff (e.g. threatening termination). Breaking the build before leaving for the day should be a major offense. And every developer should have a build status notification on their desktop. The best part of all this is that it encourages smaller commits, which are preferred anyway.

That said, the build will sometimes break (CI configuration reasons, for instance). And sometimes people will screw up and leave for the day with the build broken. That's why you should aim for a process which allows for quick and easy rollbacks to known good versions. If you can always roll back to the last good build (and have the rolled-back version get deployed to all the necessary places), then in the worst-case scenario of a broken build where the culprit has left for the evening you can roll back to the last good version and yell at him/her in the morning.

I can't recommend the Continuous Delivery book enough. If you're looking for a guide on how to mature your CI process, give it a try.

  • 2
    Agreed about breaking the build before leaving. You commit a change, you wait for the build to finish so you know it worked. It didn't work? Rollback your change or fix it, before you leave. Don't want to do that? Don't commit changes last thing in the day. Commented Jun 9, 2011 at 6:32
  • 1
    "should" is nice but any human factor is a non-zero chance of happening. If the build is crucial then have one or more staging build servers.
    – user1249
    Commented Jun 30, 2011 at 12:14

I heard about a thing Microsoft (maybe?) do, which is to have the role of build babysitter move around the team. The way they do this is that when someone breaks the build (which should probably include checking in something which fails its tests), they assume the role. This makes people responsible for the consequences of their actions in a very direct way. And given that it's a somewhat annoying job to do, it encourages them not to break the build again.

The person who is currently responsible for the build could have a special hat. There could be a ceremony for handing it over.

Note that as Thorbjørn says, being responsible for the build is not the same as being responsible for the build server. Responsibility for the server could rest permanently with one or more more infrastructurally inclined members of the team while responsibility for the build moves around.

Now, details of process aside, i am going to join the chorus of people expressing dismay at developers checking in without having run a build and test. Unacceptable!

If you have some members of the team who are more likely to break the build (and based on my own experience, i'm mostly thinking of the members in another country), and if you're using some nice modern source control like Mercurial or Git, you could have them check in to a different branch to the rest of the team, run a separate CI process on that, and automatically merge changes from that branch to the trunk after a successful build (note that you'd have to run a second build and test after the merge before checking the merge in!). Since automatic merging is not always successful, this would eventually leave the branch requiring manual attention, though, which could be a real pain. Might be less painful than having broken code checked in for the rest of the team, though.


As Jonathan Khoo said you should all be responsible for the build server and the build script. There are three reasons:

  1. At the moment you have a case of a "Bus of 1" which means that if you got run over by a bus then the whole knowledge of the build server and build scripts is lost.
  2. The scripts that have been written (rightly or wrongly) by you have only had your input. Just like any code the more people that are involved the broader the knowledge base that can be applied to it.
  3. Finally when something goes wrong only you are feeling the pain. Pain is a good thing but not when it is isolated. At the moment you are dealing with the pain but if everyone else was having the pain then you would find they would eventually be more stringent in testing code before committing.

I am very involved with CI myself and fall into the trap of being the one that maintains the scripts but here are a couple of things you can do to mitigate that.

  1. Build scripts should not just run on the CI servers but on local development machines as well. They should produce the same outputs, run the same tests and fail for the same reasons. This allows developers to run the script before committing their code.
  2. If any one does break the build make it public via the use of task tray popups, emails, flashing lights, noises etc. It serves to not only embarrass the team members but it provides an opportunity for everyone else in the team to jump up and assist.
  3. For a while avoid fixing the build. Get someone else to do it. If no one else jumps up wait for bad things to happen and use it as a learning point for the whole team to understand why the CI server is important.
  4. Try and keep your build server and development machines as devoid of installed third party components as much as possible, especially keep the GAC clean. Rely on third party components being in the projects library folder. This helps identify missing components more quickly.
  • I strongly disagree with embarrasing anybody. Would you like your compiler to flash an alarm when you have a syntax error?
    – user1249
    Commented Jun 30, 2011 at 12:17
  • @Thorbjørn This is a CI server not your local development box. The point is the team should do everything in their power to prevent checking in code that breaks the build. Hopefully people work in a fun friendly environment and the embarrassment I am talking about is not mean spirited but it makes people think next time before they commit. But we do have a funny sound that plays when the build server breaks.
    – Bronumski
    Commented Jun 30, 2011 at 18:32
  • I still disagree. A building server is just a building server and everybody can make mistakes. Just notify the culprit and let him fix it. If he doesn't fix it, then we can begin considering whether anybody else should know.
    – user1249
    Commented Jun 30, 2011 at 19:28
  • @Thorbjørn Perfectly entitled to disagree and disagreement to certain extent is good as it allows us to discuss different ideas. Looking forward to disagreeing with you again, now I must get back to belittling junior developers :)
    – Bronumski
    Commented Jul 1, 2011 at 10:28

Visibility will help you out. All team members need to know that there is an active problem that they should fix. Email notifications are useful, but if a developer is busy and does not react to it immediately, he will probably forget about it and the email will end up in a huge notifications pile.

You can try using tools like Catlight or BuildNotify. They show the current status of the important builds in tray area. Every time developer looks at the clock he will see that there is a broken build that need to be fixed.

Catlight build warning status in tray

Catlight will also show who broke the build first, putting social pressure on this person, as the whole team will see it on each consecutive build failure.

enter image description here


One strategy is to use lots of small branches for lots of small projects. Then when someone breaks the build, they are just breaking the build for themselves. So they get the annoyed email from the build server and it is up to them to worry about it.

Another is to improve people's responsibility level. For example if you use something like Rietveld then people can't commit without passing peer review. (The process in practice is much lighter weight than you think it is. But it does force people to "pipeline" and work on multiple things at once.) Preserving the build is the responsibility of both the committer and reviewers. If anyone is either regularly breaking the build or approving things that break the build, then don't let them give final approvals for commits. Combine with a process where anyone can easily roll back any change, and the build won't break as often, and won't stay broken once the change is made.

  • The 'lots of small branches' does not work well when you have one large app that everyone contributes to.
    – c_maker
    Commented Jun 9, 2011 at 11:11
  • 2
    no, it does work well. However you shift the pain from development time to merge time. If you merge small work packages regularly, then this doesn't hurt much at all.
    – gbjbaanb
    Commented Jun 9, 2011 at 12:14
  • @gbjbaanb: Right, that's why I specified small branches and small projects. As a further bonus, if the main build is broken for an hour, the odds are that other people are likely to be able to continue working because their builds haven't been broken.
    – btilly
    Commented Jun 9, 2011 at 14:21
  • @c_maker: Every strategy comes with a set of trade-offs, and none is right for all situations. However both that I gave you are being used right now with considerable success in multiple organizations.
    – btilly
    Commented Jun 9, 2011 at 14:22

I am going to break with the chorus here, and say that actually, breaking the build occasionally is not such a bad thing, as it shows that actual work is being done. Yes, developers should build and test before they commit, but the main burden of testing should be borne by the development automation tools, including the continuous integration server. The tools are there to be used, and unless the build breaks now and again, then it is not clear that you are pushing as hard as you could. I do, however, think that the build should never ever remain broken for any significant length of time, and would even favour automatic rollbacks or a multi stage commit process if that helps support the twin objectives of quick feedback from centralised, automated test facilities, plus a "green" trunk.


Couples things to keep in mind:

  1. Your team needs a set of standards to follow
  2. You need to get management on board with the idea of clean code and strive for improved code practices.
  3. Test test test! Always test before you check in.
  4. Although I do agree it is ok to break build, that's a rare and I mean extremely rare occasion that would be acceptable. If this is daily, I'd be looking for the door or talking to my boss about standards.

Overall you guys need to set down, write up standards based on best practices and not your practices individually (obviously those are not working out). Once everyone agrees on standards, start code review process and enforcing standards. It almost sounds like management went on vacation and never came back. These are things that honestly are pretty basic for any shop. the foundation of good code starts with good standards to dictate the teams code and how its written. Just my thoughts. I recently walked into a similar situation at my new job and I talked to my boss. Explained to him we need XYZ to be completed because it affects ABC. Two weeks later I wrote a list of code standards to follow and presented it. Followed by my co workers giving there input and about 2 months later we had standards in place which resolved tons of issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.