1

Breaking the build is a bad thing. Different teams take different measures to encourage their members to avoid breaking the build at all costs.

  • Some measures are moderate. For example Joel Spolsky tells that his Excel team had a rule that “whoever broke the build had to babysit the builds until someone else broke it.” (Joel Spolsky. Joel on Software, page 20).

  • Others are more drastic. For example, Steve McConnell reports that “[s]ome projects have developers wear beepers and require them to fix the build day or night if they are responsible for breaking it.” (Steve McConnell. Rapid Development, page 384)

Forcing the developers to wake up and come to work at the middle of the night to fix the build they've broken looks like an excellent opportunity to focus their attention on the reliability of the code they commit and encourage them to check it twice.

At the same time, programmer analysts usually favor personal life (fourth top priority motivation according to Table 11-1 in Rapid Development page 252, fifteenth for managers and general population) and may not appreciate the risk of hearing their phone ringing at 3 AM.

Personally, it doesn't annoy me to go and fix my mistakes at 3 AM, but only if I work for a company with flexible working hours. Most colleagues I know will refuse to come to work at 3 AM, no matter the reason and no matter how important their presence is for the company.

What is the way of setting up severe measures like night and day beepers while keeping motivation high? Or such drastic measures should be avoided at all costs, and be used only when there are no other choices, to the detriment of the motivation of team members?

closed as primarily opinion-based by gnat, thorsten müller, Kilian Foth, Bart van Ingen Schenau, user22815 Oct 24 '14 at 14:50

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    Consider potential legal issue with local labor laws. Where I work it would be a callback - minimum 3 hours, @ 3AM, double time. SOme devs would break the build for the cash...... – mattnz Oct 24 '14 at 6:23
  • 2
    A broken build is not a production issue. Unless you have teams spread across timezones who are held up by broken builds, why can't it wait until 9am? – Julia Hayward Oct 24 '14 at 9:17
  • 1
    Frankly, this sounds like a crazy idea. I don't think I'd like to work anywhere that even considers this. – sleske Oct 24 '14 at 11:36
  • This explains why a sane user of CI would configure most of the stuff to run during office hours? (Alternatively, you can make your office hour begin at 3 AM.) – rwong Oct 24 '14 at 11:58
  • 2
    Why is the broken build discovered at 3 AM? If the developer checks in just before leaving at 6, that means a 9 hour build time. If this is a real question rather than a troll, perhaps you should focus on that. – kdgregory Oct 24 '14 at 12:11
6

Broken builds happen. It is impossible to completely avoid them. Developers may be distracted while merging changes, or may simply not have run all of the tests in a very large system where a complete test run may take a significant length of time. Whatever the reason, it is important to realise you cannot always ensure that they never happen.

Developers don't want to be called back outside of office hours. I think this is pretty much a given. Quite aside from any effect on morale, they're likely to do whatever they can to avoid this happening. As it isn't really possible to 100% ensure that they never break the build, I suspect the effect that this kind of system will have is to make at least some of them reluctant to check in their changes close enough to the end of regular hours that they risk being called back. If your full integration test suite takes an hour to run, that means people will be reluctant to check in after 4pm, meaning there's a full eighth of the day that they're not checking in for. This will result in many large check-ins early in the morning, with an increased risk of the build breaking (because the risk is approximately proportional to the size of the check-in).

So, no, I don't think this is a good strategy. I suspect it will actually have the opposite effect to the intended one. Joel's suggestion you quote in your question is a much better idea.

  • As discussed, the effect of preventing check-ins late in the office hours might actually be the intended effect. Basically, a developer would be forced to review stuff written the day before. Assuming good sleep, a developer may be able to see new defects in their own code. Some large companies can buy stability however they want, even if they have to pay the productivity cost of discouraging check-ins for one-eighth of a day. – rwong Oct 24 '14 at 12:04
7

I suspect this may be a subjective question, but it's unlikely that forcing people to fix builds late at night/early morning would do anything to improve morale or motivation.

Further, it's likely that work done at 3am is not going to be of the same quality as work done during normal hours, even factoring in programmers' odd schedules. There will also be fewer (if any) people around to do code reviews, or whatever else is part of the bug fix process.

Finally, it seems like a solution to a problem you shouldn't really encounter, with proper environment setup. We use Git along side a CI system that allows branch builds, so that developers are running a build on a CI agent every time they make a commit on their branch (including unit tests). This ensures that when a feature branch is merged into trunk, the code has already been merged (trunk -> feature branch), built and tested on an agnostic system. There are very few legitimate reasons for a build on trunk to fail in this case.

So, in the end, work smarter, not harder.

  • "it seems like a solution to a problem you shouldn't really encounter": there are two possible cases. One of them is when it takes a very long time (several hours) to do the build of a large project. Another one is when builds are stacked and frequent commits push successive builds to be built late at night. Moreover, if we consider that failing regression tests is the same as breaking the build, then with a bunch of system and functional tests, it's not that difficult to have build+tests running for hours even for a medium-scale project. – Arseni Mourzenko Oct 24 '14 at 4:33
  • @MainMa Fair enough points, but in such situations, you could disable auto-builds on branches, and run it only immediately prior to merging to the trunk. As for long-running test suites, IMO, if they really take hours, I wouldn't necessarily queue them up for each commit. It might make more sense just to run them nightly (let the machines work at 3am), then address any issues in the morning. – Dan1701 Oct 24 '14 at 4:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.