I'm part of a development group with 5 teams, total of about 40 developers. We're following the Scrum methodology, with 3-week sprints. We have a continuous integration setup (Jenkins), with a build pipeline taking several hours (due to extensive automated tests). Basically, the development process works well.
However, we observe that after a few days into a new sprint, our build often becomes unstable, and remains shaky until the sprint-end "commit stop". The adverse effect of this is that build steps far down the pipeline, especially UI-/Webtests are not executed for several days (because only triggered on a 'green' build). Consequently, newly introduced bugs are often only detected very late in the sprint.
- Each commit is verified against a basic set of tests. Once verified, the change is pushed to master after a code review (Gerrit)
- Basic unit tests run every 30min, duration less than 10min
- Integration tests run every 2h, duration 1h
- UI-/Webtests run on successful integration tests, duration several hours
Depending on who's responsible for build stability during the sprint (that responsibility is passed around per sprint), there might be intermediate, ad-hoc "commit stops" to get the build back to stable.
So, we want:
- Our dev teams to develop and commit changes during a sprint unhindered
- Our build process to abandon if a build step fails, as subsequent build results have little meaning
- Our build process to give the developers quality feedback on a timely basis
Given (2), points (1) and (3) seem to contradict each other. Does anyone have a good practice how to deal with this?
(We are currently loosening point (2), and are allowing build continuation even on failed build steps. I don't have any feedback yet how that influences our quality)