I'm part of a development group with 5 teams, total of about 40 developers. We're following the Scrum methodology, with 3-week sprints. We have a continuous integration setup (Jenkins), with a build pipeline taking several hours (due to extensive automated tests). Basically, the development process works well.

However, we observe that after a few days into a new sprint, our build often becomes unstable, and remains shaky until the sprint-end "commit stop". The adverse effect of this is that build steps far down the pipeline, especially UI-/Webtests are not executed for several days (because only triggered on a 'green' build). Consequently, newly introduced bugs are often only detected very late in the sprint.

  • Each commit is verified against a basic set of tests. Once verified, the change is pushed to master after a code review (Gerrit)
  • Basic unit tests run every 30min, duration less than 10min
  • Integration tests run every 2h, duration 1h
  • UI-/Webtests run on successful integration tests, duration several hours

Depending on who's responsible for build stability during the sprint (that responsibility is passed around per sprint), there might be intermediate, ad-hoc "commit stops" to get the build back to stable.

So, we want:

  1. Our dev teams to develop and commit changes during a sprint unhindered
  2. Our build process to abandon if a build step fails, as subsequent build results have little meaning
  3. Our build process to give the developers quality feedback on a timely basis

Given (2), points (1) and (3) seem to contradict each other. Does anyone have a good practice how to deal with this?

(We are currently loosening point (2), and are allowing build continuation even on failed build steps. I don't have any feedback yet how that influences our quality)

Thanks, Simon

  • 3
    I'd say that if a build is taking several hours then that is the real problem. it signifies that the combined solution is too big and too broad. You should look to componentize the solution and then have small chunks of code as packages (available in one way or another in all major languages on all platforms). Thus any changes would go into the components only and will be detected much faster. The full build will essentially just put already combined components together, and will also be faster. You would then only possibly run some tests to ensure the right components were resolved. – zaitsman Mar 12 '17 at 8:32
  • Is your build environment on-premise or cloud-based? – Lauri Laanti Jun 12 '17 at 22:53
  • @LauriLaanti, our build environment is on-premise, 1 Jenkins instance with 3 slaves. – Simon Jun 15 '17 at 13:22

A couple of basic principles first: - Major changes should always be on a feature branch in your VCS - Feature branches should pass all tests before merge into trunk. Added - Commits should always build - A broken build requires immediate action from the committer &/or the rest of the team. - A failed test should only abort the remaining tests if it is a critical test.

If you, as a team, follow these practices and enforce them, e.g.: "name & shame" when the build is broken then you should be good to go as any commits that might break the build will be on a feature branch. Other commits that break the build will have to be addressed immediately and then you will get your downstream test results.

You can also add an automatic test of the latest "successful" build, (not necessarily one that passes the integration tests), for the UI/Web tests as an overnight run reporting first thing in the morning.

| improve this answer | |
  • 3
    A good practice to add here is that feature branches should pass all tests before they are merged to mainline – Bart van Ingen Schenau Mar 11 '17 at 9:12
  • @BartvanIngenSchenau - good point added! – Steve Barnes Mar 11 '17 at 10:20
  • @SteveBarnes, thanks for the input. A commit into Gerrit is always on a branch, and is only merged on success (my first bullet point in the build process). It's after that where the problem starts. With 30 devs commiting changes mutliple times per day, we need to merge early, and then verify. There is immediate action after a broken build, but as the time between commit and build feedback is 2 hours, there will be several other commits in the meantime. Possibly breaking the next build. – Simon Mar 12 '17 at 13:55
  • @Simon the point of the "name & shame" is to get your developers to stop committing broken code! On most systems it is possible to perform a test build in a short time using tools like ant, make, scons, etc. If your project is well structured most languages allow partial re-builds to test if things will build, (full/clean builds still need to be done of course). – Steve Barnes Mar 12 '17 at 16:31

Has nothing to do with Scrum. Your build should be continuously stable, regardless.

Nobody should check anything in unless they've performed a local build and run the unit tests locally (and both passed, of course). Your local build and test process should be sensitive to modifications, and can skip tests for code that hasn't changed.

Anyone that introduces something that causes the build to fail or a unit test to fail should be publicly shamed. If the build is broken it must be fixed immediately.

| improve this answer | |
  • 2
    On one hand, it should be emphasized that build stability is everyone's responsibility. On the other hand, I'd advise against public shaming, because (1) more experienced team members have bigger responsibility in helping junior members achieve build stability (by code review, pair programming, or just working closely together before a commit, or by fixing a broken build together), (2) shaming takes away the team's psychological safety. – rwong Mar 11 '17 at 16:11
  • 1
    If people don't want to be shamed then they shouldn't break the build. It's not like that is an unreasonably high standard. If you have devs that can't hack it, let them have their own branch to play in until they figure out how not to break the team's critical Commons. (That being said, any actual shaming should be in good taste). – John Wu Mar 11 '17 at 21:56
  • In our process, any commit is branched (in Gerrit), and has to pass a basic set of tests before merging to master. Those basic tests cannot run for an hour, as we want to code review and merge quickly. It's after the merge where the problem starts, see my comment to @SteveBarnes – Simon Mar 12 '17 at 14:01

Your problem seems to be that tests take too long to run. Fortunately, Moore's law has provided us with a solution to that problem. Today, high-end server CPUs may easily have over 10 cores (and over 10 HyperThreads). There may be multiple such CPUs in a single computer.

If I had tests that take this long, I would solve the problem with more hardware. I would purchase a high-end server and then parallelize the tests so that the tests take fully advantage of all CPU cores. If your tests are today single-threaded, taking advantage of 10 cores and 10 HyperThreds probably makes the tests run 15 times faster. Of course, this means they also use 15 times the memory, so the computer has to have enough RAM.

So, the several hours will turn into 10-30 minutes.

You didn't say how much time the build takes, but standard build tools such as make allow parallelizing also the build. If you parallelize your unit tests and typical developer computer has 4 cores and 4 HyperThreads, the less than 10 minutes of unit tests will turn into less than 2 minutes. So, perhaps you could enforce a policy that everyone should run the unit tests prior to committing?

About test failure stopping further tests: don't do that on the build server! You want as much information as possible about the failure, and further tests might reveal something important. Of course, if the build itself failed, you cannot run unit tests. If the developer runs unit tests on his own machine, then you may want to abort at the first failure.

I do not see any connection between Scrum and your problems. The problems could really occur with any development process.

| improve this answer | |
  • I agree, with a faster build things would be much easier. Our TechTeam has spent days improving the speed of our build process, otherwise we'd be waiting days instead of hours. As for now, that feedback duration is given at approx. 2 hours. So I'm looking for an approach which takes that as a "given". (Of course, we're continuously trying to speed up the build. But for the near future, it'll be 2 hours) – Simon Mar 12 '17 at 13:58
  • Some tests can conflict with parallel run – deFreitas Mar 17 '17 at 0:15
  • It is only possible to throw more hardware at it if the tests are written in a way where they can run independently from each other without introducing side effects.. The further you get from the hardware the harder this gets... most devs do not get this right, so while I agree with you I would first focus on structuring the tests the right way. – c_maker Jun 12 '17 at 23:13

Is it not possible to have more Jenkins installations and having the developers check on a separate Jenkins instance ?

I would think the best solution here is to have the code pass all the tests before it gets checked into the master branch and compiled/tested by the main Jenkins instance. Don't let people check in code that breaks the build.

I check my code into the development branch, see if it passes the tests and create a pull request. But you could obviously have jenkins pull a feature branch and test that one.

| improve this answer | |

Point (2) seems to be the most painful point, so I will focus on that.

It might be time to break the project into multiple modules.


A. High-level modules should not depend on low-level modules. Both should depend on abstractions.

B. Abstractions should not depend on details. Details should depend on abstractions.

If one module fails to build, the build for other modules will be able to continue, as long as those other modules can depend on an interface, and the code that makes up that interface was built successfully.

This will give you feedback on what other build failures might occur, so that you have time to fix more than one broken modules before the next build happens.

In general, the SOLID principles are conceived to deal with libraries and build issues. In other words - this set of principles is conceived to solve the exact kind of problems you are facing.

As a side note (see juhist's answer), you can't make the build run faster (by parallelization) if you don't partition the build into separate modules.

| improve this answer | |

I think your team is missing one of the key tenets of scrum: done, working software. A PBI should not be marked as done until it passes the Definition of Done your team has established. Definition of Done is different for every team, but likely would include things like:

  • Code has unit tests
  • Unit test pass as part of an automated build
  • Code has been merged into main (and conflicts have been resolved)
  • etc.

So, essentially, what's happening is that your team is marking stuff done that is in fact not done. That's a problem in an of itself.

Other than that, it boils down to proper version control management. If you're using Git, then all work is done in the dev's local repository and they should not be pushing anything to remote unless it's "done" and potentially releasable. Incomplete work should not ever be pushed to your main repository. If the dev needs to work on a longer lived feature branch and wants to push to remote to ensure that the they don't lose their work, then they should be working off a fork, and you would then merge that fork into main when the feature is "done" and potentially releaseable - not before.

For TFVC, it's slightly more complicated because everything happens on "remote". That means that devs should therefore always be working out of branches, unless it's a quick fix. The same rules apply here as with Git, though: incomplete software does not get committed. Period. In properly managed version control, "main" should always be releasable. If a commit has been made that borks "main", then you're not doing it right.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.