I've always launched builds after each commit, but on this new project, the architects just asked me to change the frequency to "one build every 15 minutes", and I just can't understand why that would be a good reason vs "building on each commit".

First off, some details :

  • Objective-C (iOS 5) project
  • 10 developpers
  • each build actually takes ~1 min, and includes build and unit testing.
  • the integration server is a Mac Mini, so computing power isn't really a problem here
    • we use Jenkins with the XCode plugin

My arguments were that if you build at each commit, you can see right now what went wrong, and correct directly your errors, without bothering the other devs too often. Plus our tester is less bothered by UT errors this way. His arguments were that devs will be flooded by "build error" mails (which is not completely true, as Jenkins can be configured to send a mail only for the first broken build), and that metrics can't be done properly if the frequency of builds is too high.

So, what's your opinion on this ?

  • Sure your build time will be ~1min in 2 or 3 months, with 10 devs continuously adding more code including unit tests to your project? – Doc Brown Feb 23 '12 at 12:00
  • It would be interesting to explore the architects' reasoning for requesting the change; your points are good, but do they address the actual issue? – user34530 Feb 24 '12 at 16:54

Fail fast is a good principle - the sooner you know the build is broken, the sooner the offending commit can be identified and the build fixed.

Build on every commit is the right thing to do.

Building every 15 minutes can be pointless if the project has a high volume of commits within such a timeframe - tracking down the bad commit would take longer and may be difficult to determine (one might also be in a situation where multiple commits have different things that break the build). There is also the possiblity that in quiet times (night time?) you end up rebuilding though no changes have been made.

If the build breaks so often that it is a problem, the answer it to re-educate the team on the importance of not breaking the build and in techniques that ensure this does not happen (frequent fetches, checkin dance, compiling and running unit tests locally etc...).

| improve this answer | |
  • 16
    +1. The answer to annoyingly frequent "build failed" messages is to not break the build annoyingly frequently. – suszterpatt Feb 23 '12 at 15:16
  • 3
    On quiet times - Jenkins's third option, "Poll SCM", is meant just for that. It will only update/run tests when changes are found in the repository. For example, we have one job set to run every 5 minutes if there are any changes (unit tests), and a second set to run every 3 hours if there are any changes (integration tests). Both are quiet at night/weekends because no one is committing anything. – Izkata Feb 23 '12 at 19:00

There is literally no point in doing a build every 15 minutes if nothing has changed. But equally there is no downside, afaik, jenkins will only e-mail on fail and then success and not everything in between (eg 10 fails).

We do it on every commit. However we do poll the repository every fifteen minutes and check for changes, maybe that is what yr colleagues are referring to.

You expect your 10 dev's to be committing more than once every fifteen minutes ? That sounds like rather a high estimation. 10 dev's means that after every 150 minutes the same person is committing again, so thats 2.5 hrs. So in your average day each dev commits 3 times. Personally I do one commit ~ 2 days ... sometimes more sometimes less.

| improve this answer | |
  • 1
    Actually, commits go pretty fast around here, so this is entirely possible, but yes, I see what you mean. – Valentin Rocher Feb 23 '12 at 9:59
  • 3
    @NimChimpsky: you do one commit every 3 days? If that's true, I suggest you should seriously reconsider your commit strategy. Whenever you are going to reset something to a previous state, you will loose up to 3 days of work! How do you describe changes of 3 full days in a few words in your change log? That sound very absurd. Personally, I do a commit whenever I have added a working feature slice to my program, typically several times a day. – Doc Brown Feb 23 '12 at 11:54
  • 2
    @DocBrown its far from absurd. I might commit to different projects and different repositories three times in a minute. On the other hand I might not write any code at all for a whole week. I suggest you seriously consider your commenting strategy. – NimChimpsky Feb 23 '12 at 12:18
  • 1
    @NumChimpsky: I was assuming you were talking about a situation comparable to the one described by the OP. We are talking about 10 devs working on the same project. If the median time between commits per dev is 3 days, then something goes very, very wrong in that project. – Doc Brown Feb 23 '12 at 12:22
  • 2
    @DocBrown wtf ? You don't know what you are talking about ... I take it you don't work on multiple projects simultaneously. – NimChimpsky Feb 23 '12 at 12:24

It's going to flood developers with mail more if it's every 15 minutes only. That's because it won't know for certain who broke the build and thus mail more people.

As for metrics, if it's really a problem—which I can't tell because I don't know what metrics they think there is a problem with—, you can always make another job for collecting the metrics.

| improve this answer | |

Perhaps the requirement shold be "build at most once per 15 minutes". This could make sense for projects with very frequent commit activity (i.e. multiple commits within few minutes) and perhaps long build times. Of course it depends also on how the builds are used. For testers it might be somewhat confusing to get multiple builds within 15 minutes ...

But I agree that it makes no sens to build if nothing has changed.

| improve this answer | |

Some devs want to be allowed to do commits in a way where the files belonging to one functional change are not committed in one single, atomic procedure. They need two or three minutes to do a "logical commit", which consists of some "physical commits". This is typically the case when your devs directly commit to a central repository, not using a DVCS.

For this cases, it may be a good idea to let your CI server wait some time after the last commit before starting a new build. But 15 minutes seem to be a very high number, 5 minutes or less should be enough.

Or, better(!), try to guide your devs to work only in small portions, only one thing at a time, making it much easier to do only "functional complete" physical commits. Then a build after every commit will work seemlessly.

| improve this answer | |

Even if you have Jenkins set up to build on a source control commit to a project or any of its dependencies, this does not prevent a developer from deploying to the artifact repository without first committing to source control. If they deploy an uncoordinated API change or a bug in a dependency to the artifact repository, this can break your build without triggering the Jenkins job. I personally would want to know about this ASAP.

Ideally you would build for every commit and also on a schedule to check for situations like I just described. This would conceptually mean you build at least once every 15 minutes.

If you have your Jenkins jobs set up to run on dependency artifact deploys (and if you do...Bravo), you can prob axe the scheduled build if your unit tests are solid (meaning they don't test the dependencies) and your integration/functional tests are not part of the Jenkins job.

As far as the "flood with email" problem, I say getting flooded with email on a broken build is a good thing. Make sure you force developers to respond to the email with a description and you will see your broken builds go way down, trust me.

| improve this answer | |

You said you can't understand their reasoning, so you have to ask them. Don't just guess at what they want and try to implement that, and certainly don't just implement what they asked for without understanding why they asked for it.

That said, to answer the general question, it depends on the development philosophy you use. If every commit is expected to work, then every commit should be tested. If you use a DVCS with the philosophy that every push should work, then test every push.

In general, it's better to tell people something they don't know, then to repeat what they do know. For example, if they want to reduce the number of redundant emails they get, tweak the email settings, not the build frequency.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.