2

I have seen this question and do not believe this to be a duplicate What software models are appropriate for daily builds and continuous integration? .

I don't fully understand what advantage automatic daily complete builds may have over building on every commit in practice (continuous integration being the term I believe). I commonly hear companies mention they build on every commit when asked about daily builds.

The 2nd article by Joel below mentions "It’s tempting to do continuous builds, but you probably can’t, because of source control issues which I’ll talk about in a minute. " but it's still not fully clear to me, despite searching.

https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/

https://www.joelonsoftware.com/2001/01/27/daily-builds-are-your-friend/

In these 2 articles it is alluded to that a daily build is more practical/advantageous as opposed to continuous integration (which to my understanding would be building on every commit).

However it still isn't clear to me. I currently only thought of 2 specific examples perhaps

1) With many of (git) pushes/merges constantly coming in, with a sufficiently long build time to do a full build.

Joel's description of a complete build is as follows: "Complete – chances are, your code has multiple versions. Multiple language versions, multiple operating systems, or a high-end/low-end version. The daily build needs to build all of them. And it needs to build every file from scratch, not relying on the compiler’s possibly imperfect incremental rebuild capabilities. "

Perhaps you could save resources but just doing 1 full build around the middle of the work day, and settle for running all relevant quicker unit/integration tests on every push.

The constant queue possibly of complete builds non-stop could build a big enough queue defeating the relevancy and purpose of your builds perhaps because they'd always be behind.

This plays into a full build on every push being perhaps the ideal but in practice you won't have hardware capable of dealing with your push thoroughput.

2) it might be more convenient to sweep through daily builds rather than builds by every push (reason 5 in his daily builds are your friend article).

  • 6
    Worth noting: Joel's article on daily builds was written in 2001, at a time when continuous integration was probably an unaffordable luxury. – Robert Harvey May 29 '18 at 18:51
  • 5
    For many systems, a build can take an hour or two (depending on how you define "build"). If you have half a dozen programmers committing a few times a day, that's going to end up creating a huge backlog of builds. – Bryan Oakley May 29 '18 at 20:23
  • Thank you @BryanOakley that's a clearer and more succinct way of expressing the example 1 in my question. – user1821961 May 29 '18 at 22:49
  • 1
    Downvoters I am happy to edit this question to make it clearer or close it if it is doesn't belong on this site. – user1821961 May 30 '18 at 1:49
5

I work on a very large project with dozens of software engineers, dozens of QA engineers, etc. Our codebase is several million lines of code. We have nightly builds and we have continuous builds. They serve different purposes:

  1. Nightly builds are builds from scratch. The build server checks out the code and builds it. This ensures that we can build from scratch and aren't relying on build artifacts from previous builds laying around in the build directory or whatever. We give these builds to QA every morning and they run tests on them.
  2. During the day, as we're writing the software, the first commit of the day kicks off a build. After that, commits are queued up, and when the previous build finishes, the next build starts with all commits that happened in between. We actually do 2 continuous builds. A quick, incremental build that only builds things that changed in the new commits, and another full build from scratch. The full builds take ~90 minutes with full unit tests. Incremental builds only take a few minutes to build, and I believe run a smaller set of unit tests for quick turnaround. If a build or tests fail we know it was caused by one of the commits between the previous successful build and the current build. That narrows down which commit caused the problem pretty well. We might have to do a little detective work to figure out which of the 5-8 commits it was, but usually it's pretty obvious since developers aren't usually working in the same areas.

When QA does find a problem with a nightly build, we can go back through the continuous builds we built that day and figure out that the problem was in a particular build. (In other words, we regress the builds to see where it occurred.) We can see which commits were in that build, narrow it down to the exact one if necessary, and either revert the commit or fix the problem.

1

I used to have a similar view. Now I think that there should be a daily build in addition to per-commit builds.

  1. Extra Tasks: you can add extra tasks onto your cron build that you would not want on your per-commit build. For example, your cron build can scan dependencies for upgrades and auto-magically create a pull request regarding them.
  2. Sanity: If your cron build fails for some reason (it should not but it could), then it is logical to expect subsequent per-commit builds to fail for the same reason. Rather than pulling your hair out figuring out how your code changes break the build, solve the underlying problem.
  3. Status Meetings: If you have status meeting at 10:00, then you should have a cron build at 9:00. It should just succeed, but if for some reason it does not then your status should include fixing the build.
  4. Cost: It is probably not needed, but how much does it cost. I think the benefits justify the minimal cost.

Cron builds are a supplement to per-commit builds and should never be considered as a replacement.

0

There really aren't any advantages of doing daily (or any other periodic) builds as long as you have enough resources to do the same builds for every commit.

The periodic builds are just a compromise if you don't have enough such resources: you give up the ability to immediately identify a faulty commit as, if such build fails after picking up multiple new commits since the previous successful build, you need to perform additional work to exactly identify the faulty one(s) and fix the problem(s). And that may not always be simple, there is room for all kinds of potential complications, for example:

  • multiple faulty changesets picked up simultaneously in the new build, making identification more difficult
  • clean backout/revert of a faulty changeset may not be possible due to subsequent changesets committed on top of the faulty one, the fix doesn't backtrack to a known good repository state but moves it to a new state which may or may not be good, requiring one or more additional builds for confirmation
  • even clean backout of a faulty changeset may not lead to a good build due to subsequent changesets depending on the backed-out one or being also faulty

Personally I favour CI systems doing pre-commit/gating verifications: they can be a lot more cost effective, especially in large scale projects, prevent faulty changesets from even reaching the repository than to detect and fix them after they already impacted the entire team working on the project.

Depending on the orchestration algorithm used, such systems can guarantee 100% prevention of breakages/regressions even without having enough resources for doing a build for each and every candidate changeset.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.