3

We are working on a software project, a team of 5-10 developers. The code base is continuously integrated using Bamboo. We have a build plan running also unit and integration tests and then a functional test plan. We get automatic emails about failures but sometimes days pass before a failed plan is fixed.

Question: How can we improve our processes/tools to make people fix the failures faster? What tools/processes do you have?

Edit: We are working with feature branches but the Bamboo jobs are run only on the main branch. There is a Git hook in place that would allow a person to disable pushing until Bamboo was green. It can be a sollution to automate this although we have some security constraints in Ops part and it might not be possible.

Edit: the build with unit & integration tests takes 20 minutes, the functional tests plan is scheduled twice a day and lasts about 2h

  • 1
    Assuming the developers responsible for fixing the build are receiving the e-mail immediately, it becomes more of a HR issue. That's not to say they should be punished, just that if these developers don't have the time to dedicate towards fixing the problems, the problem that needs to be addressed is this. – Neil Jun 23 '17 at 12:46
  • How long does it take to run the full build plan? A minute, an hour, a day? – Doc Brown Jun 23 '17 at 14:07
  • @Doc Brown - the build with unit & integration tests takes 20 minutes, the functional tests plan is scheduled twice a day and lasts about 2h – GabiM Jun 23 '17 at 14:16
  • Wrong solution: the person to blame has to wear a T-Shirt with "I broke the build" on it (and noone will ever wash that shirt). – Doc Brown Jun 23 '17 at 14:51
  • Does it stay broken because it's that hard to fix or because it takes days before anyone looks at it? – RubberDuck Jun 23 '17 at 16:08
7

As you already have notifications of failed builds, getting them fixed is primarily a people problem.

You should get agreement among the team members that a broken build is a serious problem that needs to be fixed before everything else.
As long as the build remain broken, you should agree that

  • no work can be marked as completed, unless it can be proven that all commits have been included in the last successful build.
  • no new merges to the main branch can be made unless it is to fix the problem that causes the build to fail.

If the build breaks frequently, then you should investigate why that happens and what counter measures you can take.
One possibility here would be to see if you can put a process in place where the feature branch gets built on Bamboo before being merged to the main branch. Or even better, the projected result of the merge should be built. Only if this build of the branch/merge is green should the actual merge be done.

  • Note that if you prevent anyone to commit for one failure, using different branches for different developers may become necessary. – Walfrat Jun 23 '17 at 14:22
  • I edited the question: we are working with feature branches but the Bamboo jobs are run only on the main branch. There is a Git hook in place that would allow a person to disable pushing until Bamboo was green. It can be a sollution to automate this although we have some security constraints in Ops part and it might not be possible. – GabiM Jun 23 '17 at 14:22
4

There's a couple things I would change to the build process, because the sooner problems are detected the easier they are to fix:

  • Have the build run against every branch and pull request. That includes feature branches and all unit tests should be run against that build.
  • Keep the long running functional tests against the main branch only
  • Prevent merging a pull request until both the feature branch and the pull request are green.

The last bit is very handy to both prevent broken code from being merged as well as forcing the developer to deal with breaks before they are merged. The way forward would be for the developer to pull from the main branch, fix the parts where they are out of step, and then create a new pull request.

The first time I had this automated was with Github and Appveyor CI. It's useful to know if your merge is going to break tests before it's actually merged.

If you still have issues with frequent build breaks, you can look at the following:

  • Is the problem sloppy development, or is your code architecture brittle?
  • Maybe introduce a penalty. Last company I worked for you paid $1 if you broke the build. That money went toward team building events like happy hours or things like that.
  • Find ways of making the code architecturally resilient.
  • Thanks for the answer. Our biggest problem is with functional tests (a failed unit test would prevent a deploy in the testing env and so would prevent the tester closing the developer's story). Good idea to run the build on each PR. I wonder how I could do that with GitHub & Bamboo. Maybe a hook in the remote Git repo when a new branch is created/merged? – GabiM Jun 25 '17 at 10:41
  • I believe that's how Appveyor does it. I would recommend even the functional tests to be run against everything, although it would delay every pull request by at least 2 hours. You might want it run against every commit to the develop branch as well. – Berin Loritsch Jun 26 '17 at 12:14
  • There is a problem with running the integration tests per PR because they are kind of heavy, we would need to have a database schema per PR, for instance, to have Hibernate working – GabiM Jul 11 '17 at 12:42
  • 1
    Understood. It's a trade-off between assurance and testing. Is is possible that the database be an "in memory" instance? Another possibility is having a base database that is copied and mounted (like in SQL Server) then discarded. An in memory database could speed up the integration tests by several orders of magnitude if it is possible at all. – Berin Loritsch Jul 11 '17 at 12:53
  • That would be nice, we didn't find a way to get it working in memory because datatypes and even sql syntax differs from our Oracle DB so the flyway scripts we have won't work. Seems like there's a bunch of stuff that would be nice to have but they weren't top priority so we didn't get them yet – GabiM Jul 11 '17 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.