Are scrum and a stable development build a contradiction?

Question

I'm part of a development group with 5 teams, total of about 40 developers. We're following the Scrum methodology, with 3-week sprints. We have a continuous integration setup (Jenkins), with a build pipeline taking several hours (due to extensive automated tests). Basically, the development process works well.

However, we observe that after a few days into a new sprint, our build often becomes unstable, and remains shaky until the sprint-end "commit stop". The adverse effect of this is that build steps far down the pipeline, especially UI-/Webtests are not executed for several days (because only triggered on a 'green' build). Consequently, newly introduced bugs are often only detected very late in the sprint.

• Each commit is verified against a basic set of tests. Once verified, the change is pushed to master after a code review (Gerrit)
• Basic unit tests run every 30min, duration less than 10min
• Integration tests run every 2h, duration 1h
• UI-/Webtests run on successful integration tests, duration several hours

Depending on who's responsible for build stability during the sprint (that responsibility is passed around per sprint), there might be intermediate, ad-hoc "commit stops" to get the build back to stable.

So, we want:

1. Our dev teams to develop and commit changes during a sprint unhindered
2. Our build process to abandon if a build step fails, as subsequent build results have little meaning
3. Our build process to give the developers quality feedback on a timely basis

Given (2), points (1) and (3) seem to contradict each other. Does anyone have a good practice how to deal with this?

(We are currently loosening point (2), and are allowing build continuation even on failed build steps. I don't have any feedback yet how that influences our quality)

Thanks, Simon

Answer 1

A couple of basic principles first: - Major changes should always be on a feature branch in your VCS - Feature branches should pass all tests before merge into trunk. Added - Commits should always build - A broken build requires immediate action from the committer &/or the rest of the team. - A failed test should only abort the remaining tests if it is a critical test.

If you, as a team, follow these practices and enforce them, e.g.: "name & shame" when the build is broken then you should be good to go as any commits that might break the build will be on a feature branch. Other commits that break the build will have to be addressed immediately and then you will get your downstream test results.

You can also add an automatic test of the latest "successful" build, (not necessarily one that passes the integration tests), for the UI/Web tests as an overnight run reporting first thing in the morning.

Answer 2

Has nothing to do with Scrum. Your build should be continuously stable, regardless.

Nobody should check anything in unless they've performed a local build and run the unit tests locally (and both passed, of course). Your local build and test process should be sensitive to modifications, and can skip tests for code that hasn't changed.

Anyone that introduces something that causes the build to fail or a unit test to fail should be publicly shamed. If the build is broken it must be fixed immediately.

Answer 3

Your problem seems to be that tests take too long to run. Fortunately, Moore's law has provided us with a solution to that problem. Today, high-end server CPUs may easily have over 10 cores (and over 10 HyperThreads). There may be multiple such CPUs in a single computer.

If I had tests that take this long, I would solve the problem with more hardware. I would purchase a high-end server and then parallelize the tests so that the tests take fully advantage of all CPU cores. If your tests are today single-threaded, taking advantage of 10 cores and 10 HyperThreds probably makes the tests run 15 times faster. Of course, this means they also use 15 times the memory, so the computer has to have enough RAM.

So, the several hours will turn into 10-30 minutes.

You didn't say how much time the build takes, but standard build tools such as make allow parallelizing also the build. If you parallelize your unit tests and typical developer computer has 4 cores and 4 HyperThreads, the less than 10 minutes of unit tests will turn into less than 2 minutes. So, perhaps you could enforce a policy that everyone should run the unit tests prior to committing?

About test failure stopping further tests: don't do that on the build server! You want as much information as possible about the failure, and further tests might reveal something important. Of course, if the build itself failed, you cannot run unit tests. If the developer runs unit tests on his own machine, then you may want to abort at the first failure.

I do not see any connection between Scrum and your problems. The problems could really occur with any development process.

Answer 4

Is it not possible to have more Jenkins installations and having the developers check on a separate Jenkins instance ?

I would think the best solution here is to have the code pass all the tests before it gets checked into the master branch and compiled/tested by the main Jenkins instance. Don't let people check in code that breaks the build.

I check my code into the development branch, see if it passes the tests and create a pull request. But you could obviously have jenkins pull a feature branch and test that one.

Answer 5

Point (2) seems to be the most painful point, so I will focus on that.

It might be time to break the project into multiple modules.

https://en.wikipedia.org/wiki/Dependency_inversion_principle

A. High-level modules should not depend on low-level modules. Both should depend on abstractions.

B. Abstractions should not depend on details. Details should depend on abstractions.

If one module fails to build, the build for other modules will be able to continue, as long as those other modules can depend on an interface, and the code that makes up that interface was built successfully.

This will give you feedback on what other build failures might occur, so that you have time to fix more than one broken modules before the next build happens.

In general, the SOLID principles are conceived to deal with libraries and build issues. In other words - this set of principles is conceived to solve the exact kind of problems you are facing.

---

As a side note (see juhist's answer), you can't make the build run faster (by parallelization) if you don't partition the build into separate modules.

Answer 6

I think your team is missing one of the key tenets of scrum: done, working software. A PBI should not be marked as done until it passes the Definition of Done your team has established. Definition of Done is different for every team, but likely would include things like:

• Code has unit tests
• Unit test pass as part of an automated build
• Code has been merged into main (and conflicts have been resolved)
• etc.

So, essentially, what's happening is that your team is marking stuff done that is in fact not done. That's a problem in an of itself.

Other than that, it boils down to proper version control management. If you're using Git, then all work is done in the dev's local repository and they should not be pushing anything to remote unless it's "done" and potentially releasable. Incomplete work should not ever be pushed to your main repository. If the dev needs to work on a longer lived feature branch and wants to push to remote to ensure that the they don't lose their work, then they should be working off a fork, and you would then merge that fork into main when the feature is "done" and potentially releaseable - not before.

For TFVC, it's slightly more complicated because everything happens on "remote". That means that devs should therefore always be working out of branches, unless it's a quick fix. The same rules apply here as with Git, though: incomplete software does not get committed. Period. In properly managed version control, "main" should always be releasable. If a commit has been made that borks "main", then you're not doing it right.