How should support/bugfixes work in a larger organisation?

Question

We have a team of about 40 engineers who work on a large SaaS platform. As with any organisation, we have an enormous backlog of things we want to deliver from our roadmap. But of course, we also have a backlog of bugs, of all priorities, as well as occasional production incidents to react to.

We're aware of a few models of doing support/maintenance work, but are not sure which one is right (either axiomatically, or just "right" for us).

The models we're aware of are:

• Total ownership: each self-contained scrum team deals with bugs and incidents relating to areas of the application that they own. This theoretically goes all the way to being woken up in the night if the sysadmin team can't figure out what's going on.

• Dedicated support team: a team of maintenance developers who are solely responsible for cleaning up the bug backlog, writing hotfixes for production incidents, etc. This leaves other developers free to focus on roadmap work.

• Rotating support team: each scrum team takes a shift of several sprints where they take up maintenance duties as described above; but not permanently.

At the moment we're going with the "total ownership" model, but PO/PMs do complain, with some justification, that sprint velocity is sometimes adversely affected when larger-than-expected bugs need to be tackled.

Apart from writing a bug-free system with no incidents, how do other organisations tackle this problem?

Answer 1

I am another supporter of "Total Ownership".

I think that helps drive quality. I've worked in an organization where there were "feature teams" and "bug teams". It seems like the bug teams were burning hard and still losing ground on their backlog. Eventually they realized perhaps there needed to be just be teams and each team was responsible for both the features and the bugs in it's area.

The last thing you want is developers "on call". There's no way the right answer is a code change/role out in the middle of the night with no QA or adult supervision. It's just not going to end well.

As other folks have suggested, there should be some sort of DevOps or support staff that has a small set of buttons they can push to get the customer back online. Exactly what and how depends on the nature of your software/service. They can write up the incident and forward that to the appropriate scrum team.

In my organization we support multiple products and each team has a weekly bug triage meeting with the support lead, the product owner and the tech lead where an initial disposition can be made.

We also plan our sprints to only 80% capacity so we have a buffer for unexpected customer service escalations (and other surprises). This means that you aren't impacting the planned work from the moment something happens. And, if you have a quiet sprint support-wise, you can always pull in a story or two to fill the excess capacity.

Answer 2

In my experience, bugs come in two variants:

• Needs to be fixed like... yesterday. Red blinking lights. Sirens. Bosses demanding a status.
• Oh. Yeah. We are gonna fix it. At some point. Maybe.

In the first case, it is what it is. This does not care for "Sprints" or planning or velocity or points. It needs to be done ASAP and it will ruin your plan. There is nothing you can do. You don't want whoever is on some kind of rotation for this, you want the person that can do the job best on this. And you want it now. There is nothing you can do. Most likely it is the very PO/PMs that complain about velocity that want that. Let them figure out how to recover their plan after such an incident.

In the second case, you want stability for planning. But obviously you don't know what will be uncovered if the developers really dive into the bug report. So use a tested Scrum tool: Timeboxing. Allocate a timebox for your first sprint to uncover what needs to be fixed. For example: one person takes one business day to figure out what's wrong. If fixing that bug fits in as well, great. Just do it. If not, your story is "done" in so far as that you figured it out and can now write a story for the next sprint that can be properly estimated. If that timebox is over and you haven't figured out the bug yet, then that story is not completed at the end of the sprint and maybe needs to be discussed. Whatever happens, you have stability for this sprint through the timebox and you have a way better estimation for the next sprint because you have hard data and not just a bug report.

---

I think it goes without saying that Scrum needs ownership of the product. You cannot have proper Scrum if another team can make arbitrary breaking changes in your product behind your back.

Answer 3

The best approach I've seen is a mix of "total ownership" and "dedicated support team"

The dev team for the app owns all the bugs and has to prioritise and write the fixes in their sprints.

The support team deploys the app and reacts to problems in live. But only by rolling back releases, rebooting servers and raising bugs etc. Not changing code.

So in the case of a 'train smash' bug in live you have a predetermined response. Roll back to the previous version to get the customers back online, put the bug on the back log for the team to prioritise for the next sprint.

Now the pressure on the dev team is no longer 'the site is down!!' but the less worrying 'the last release has been rolled back! we need those features!'. Giving the project manager more flexibility in how to react to the problem.

Smashing out a code fix and putting it on live servers in the middle of the night really isn't what anyone wants to be doing. Planned and phased deployments, separate from the development process, smooth over these problems by limiting the affected users and giving a rehearsed immediate response