0

Say you finished a sprint, successfully deployed your application to the production environment and started another 2-week-long sprint. But then you find a critical bug in the version you just shipped, so critical that there's no way you can wait until the end of the sprint to deploy the fix.

What are the guidelines for managing such situations in scrum? A couple of specific questions:

  • Do you fix these bugs right away or do an emergency rollback of your system to the previous stable state?
  • If you decide to fix them right away, how do you keep track of them (they obviously can't go to the scrum board as they don't belong to the sprint)?
  • How do you measure the impact such bugs make on the team's velocity?
  • Don't such things go into the next backlog (albeit at the front of the stack), or in the current backlog if it's not completed? You affect velocity the same way anything else in the backlog affects velocity. – Robert Harvey Mar 31 '16 at 15:13
  • related: Scrum in combination with ad hoc bug fixes – gnat Mar 31 '16 at 15:22
  • 1
    Possible duplicate of How to account for a bug fixing iteration? – Robert Harvey Mar 31 '16 at 16:20
  • Why do people think you can only release at the end of a sprint? Release as often as you're ready. – RubberDuck Apr 1 '16 at 1:36
  • @RubberDuck "ready" is usually something the product owner announces. Therefor releases are usually end of sprint, it's when the team demos their work to the product owner. For hotfixes this is different of course. – Joppe Apr 3 '16 at 15:24
6

One of the core tenets of Agile is that it is more important to figure out what works for the team than to blindly follow the rules. Yes, the "rule" is that you never take in work mid-sprint.

The reality is sometimes messier. If a critical defect comes in, and maybe a flaw is costing you money, or opening your company up to liability, or leaving an opening for hackers, or disabling a fundamental feature, you damn well fix it now, rules be damned.

In all of these cases, though, you should always consider "rollback" as the preferred option. This is not just because of the Agile process. It is also because rushed hotfixes are a common way to create even more disastrous defect. (I could tell stories...)

There are then two important things to consider.

1) You have to make sure this only happens for critical defects, not "the senior VP doesn't like the color of the background". This of course gets a little harder if you are in the habit of patching critical issues. So obviously the solution is to never get into this situation in the first place. This is why Agile puts such a strong emphasis on automated testing. If you have a truly robust CI and testing infrastructure, you should only see these sort of hotfix situations once in a blue moon.

2) How do you account for the work? That's easier. You did work, so you have a story. You point the story. It's included in the velocity. But of course as mentioned above, this should be uncommon.

  • I wouldn't say that Agile puts a strong emphasis on automated testing. It does promote visibility, which can mean writing tests that make any issues more visible. But Agile itself is just a framework which doesn't actually say anything about testing. – Bernard Apr 3 '16 at 16:50
  • @Bernard Strangely enough, I've heard the exact opposite. My understanding was that agile requires constant refactoring, and refactoring can only be done efficiently/safely/confidently when you have plenty of automated tests, therefore agile methodologies need to emphasize automated testing more than non-agile ones. – Ixrec Apr 3 '16 at 21:01
  • @Ixrec agile says nothing whatsoever about the detail of development. It is about delivering products in a flexible and efficient way. All that stuff about refactoring and testing is cargo-cultist speak. – gbjbaanb Apr 5 '16 at 9:03
  • Rollback is the worst option, I think. Really, you don't wanna rollback your entire version, and it's tricky to find specific parts of software to revert from your source control. You should always have some extra time in your current sprint ready for this. See my answer softwareengineering.stackexchange.com/a/381041/196667 – Ricardo Vigatti Nov 6 '18 at 4:01
  • There isn't any rule against taking in work during the sprint in the Scrum guide. What it says is : "No changes are made that would endanger the Sprint Goal; Quality goals do not decrease; and, Scope may be clarified and re-negotiated between the Product Owner and Development Team as more is learned." – bdsl Nov 6 '18 at 8:21
3

When an issue is found, you first do a triage of the issue to determine if it is a really critical "fix now" problem or if it can wait and be planned along with the other work.

If the issue really has to be fixed immediately, then you should pull it into the current sprint as unplanned work and track how much time the team spent on it.
At the end of the sprint, you will probably find that you couldn't complete all the work that you originally planned. A lot of this "left over" work will be due to the unplanned work that had to be done. During the sprint review, it is also good to mention how much time the team 'lost' on unplanned work, because it can be an indication for problems within the organisation (or the team) if a significant portion of the sprint is lost on unplanned work.

2

Whether you start fixing it immediately or roll the update back shouldn't be based on the status of any sprint. This item should go on the backlog and you may need to do some sort of mini sprint for a few days or just extend the previous sprint. Fix the problem and don't worry about Scrum too much.

The whole point is to be agile and not to arbitrarily stick to 2 week planning blocks. They're used as a guide to have some consistency to make planning easier (So how much do we typically get done?). Some times you may start a sprint and have to throw the whole thing away and start over because of some unforeseen problem. The key is to let those in charge know that interruptions are always painful. There's no getting around the setbacks they create in all projects. This bug fix isn't going to happen without something else not getting done, so make sure everyone knows you shouldn't even try it.

2

Let the issue upset/break into your current sprint

This is usually not a popular opinion but: fix the issue. You dropped the ball last sprint. Let it cut into your velocity for this sprint. This will create incentive to not drop the ball again and make your velocity more realistic due to including fixing of issues that do happen.

However if it is possible to postpone the fix to next sprint, due to it not being critical or a rollback not being an issue. Include the fix as a bug story without assigning story points. I usually go this route if it's an issue that has been around for while, undetected, not necessarily created by the previous sprint.

0

After the sprint is done, your version must stay closed, so it's impossible to add or remove stories, but you still can edit they.

  1. Find the story that originated this bug and reopen it. Name it status something like "Rejected", or "Test Failed", or whatever do you think is reasonable.
  2. Allocate work for this story again, I call it "Tasks", during your current sprint. Every sprint should have extra time planned for those kind of issues. It's easy to be prepared for that after revision's and delivery's meetings.
  3. Do it, test it, report it and deliver it as a "hotfix".
  4. Close the story again.

If the shit is too big to fit in your calendar, create a new story in your product backlog, assemble it high priority, and fix it fastest as possible. Keep your old story opened and flagged to not forget about it, you can even do a relation like "this story is blocked by this another one". The thing is, just close it after the new story is also done.

  • I don't think stories should be re-opened. The story was done, and it met all of the acceptance criteria. You shouldn't be trying to change history. You are right though that the bug should be fixed in the current sprint. If it prevents a story from getting completed, it simply needs to be noted that the team had to work on a hotfix, and that prevented work from happening on one of the stories in the sprint. – Bryan Oakley Nov 6 '18 at 16:05
  • @BryanOakley you're right. – Ricardo Vigatti Nov 7 '18 at 16:14
-1

Roll back the previous deployment and queue the work in the backlog.

If you get 'critical' bugs like this its points to a failure in testing, rushing in more changes only makes things worse. You push back the next release, which is in turn rushed through, which leads to more bugs, which pushes back the next release.. etc

Roll back the release, take time to work out why the bug was missed, ensure you put processes in place to prevent it happening again.

-2

Assuming it can't wait and a rollback is not necessary, impossible or undesirable, you split off into two parallel sprints. Of course you need to split your resources to do this but hopefully it's just one dev and some QA time.

One sprint takes the current production branch and develops, tests & deploys the hot-fix and only the hot-fix. Because of the branch, no "in progress" work is included and you can release independently ads soon as the fix is ready.

The other sprint continues with the rest of the team and current backlog knowing that they're going to be short on the resources. Do what makes sense to the product owner (complete using a bit more time or hit the date but fall short of a few items) and deploy.

You took one or more team members out of the main sprint. Effects on velocity and whatnot can be evaluated and justified/reconciled as makes sense. The key thing is that you can manage to rollback, fix, plough thorough or schedule as you wish. Don't have a process that does not give you these choices.

  • I don't think you need the overhead of an entire parallel sprint. Just have the team fix the bug and continue as usual. If it prevents work from the current sprint from being finished, that simply needs to be noted and discussed during the demo and retrospective. It doesn't need to be any more complicated than that. – Bryan Oakley Nov 6 '18 at 16:07

protected by gnat Nov 6 '18 at 5:45

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.