Quality Assurance (Testing) in Scrum

Question

I just had a Release Retrospective for my scrum team. We talked a lot about our release process.

I pointed out that because our company is not able to tolerate bugs in our production environment we are not be able to adhere to the traditional scrum mantra of releasing often.

In short, we are a medical company. Bugs in production can cause issues with patient care. (A rapid fix release does not help the patient that was negatively impacted by the bug.)

I pointed out that scrum does not have a formal quality assurance process. (It is assumed that testing will be done during development.)

I then stated that scrum has an implied expectation of bugs in production. (Based on the process of releasing early and often.) The Scrum Process people in the room said that scrum is not that way. They said that properly done scrum can be bug free in production.

So here is my question:

How does testing and quality assurance work for scrum? (So that there is very low occurrence of bugs in production.)

OR

Is there documentation that bugs are expected to a small degree in Scrum (along with quick follow up releases)?

NOTE: This is for full enterprise level development. We have 6+ WCF services, several Service Buses, 4 databases, a WPF front end application and a Web Interface all written by two separate scrum teams of about 6-8 people each. This means answers that include just coding it right the first time are not realistic.

NOTE II: I know that no software product is going to be bug free. But our release process (non-agile) catches the few that get past our development process and brings our software fairly close to the "No Bugs" level.

Answer 1

How does testing and quality assurance work for scrum? (So that there is very low occurrence of bugs in production.)

Actually, I don't think SCRUM addresses this. SCRUM does not cover all stages of a product/project. SCRUM is mainly about organizing the development itself - the part from "we have an idea for a feature" to "the dev team thinks this is production ready". It does not cover the very first part of a project (basic idea, project vision, finding stakeholders, getting a budget...), and it does not cover the final delivery (deployment, customer feedback etc.).

So if you feel you need an additional stage after the SCRUM team is done, by all means have a separate stage of QA testing, regression testing, certification, whathever, before going into production.

You still get the value of frequent releases because there is something to test and get feedback from stakeholders - but just because you want feedback does not mean you must ship to customers (though this is often helpful). Note that SCRUM speaks of a "potentially shippable product", because you might not actually want to ship the product for various reasons.

Note: Obviously, the disadvantage of a separate QA/testing stage is that it will take longer to actually ship a feature to customers. However, if quality is more important than quickly shipping new features, then that may be the right compromise for you - that is up to the stakeholders and developers to decide.

Answer 2

in Scrum each history has a Definition of Done, if one history its considered "done" its because fulfills this definition and its prepared to go to production. If you have productions bugs frequently you must work in have a better definition of done and ensure that the history fulfills this definition.

Scrum its only a lightweight project management technique, when the difficult question like yours arise normally scrum don't have concrete answers. You need to find those answers in other agile methods (like TDD or pair programming from extreme programming). For example a generic definition of done that all your user histories must pass can be something like that:

• All unit test pass and coverage its revised
• The code is revised (we prefer pair programming, its only an example)
• The feature its deployed in test server
• The feature its revised by QA people.
• Load testing its ok
• ...

You can add whatever you want, the key its that you must ensure that a user history that fulfills this definition its at the level of quality your project needs and can go into production.

Only two more things about QA

• QA people cannot be a separate team, QA people its part of the team. A scrum team needs all the skills to fulfills the DoD.
• If your automatic testing (TDD or not) its not enough to achieve the level of quality you need think in ways to do a better job with your automatic testing, probably the QA people can help with this automation. Rapid release is not possible if you need to perform long (days perhaps) manual regression testing, scrum cannot give you rapid releases if you don't have an automated (or semi-automated, but very quick) way to ensure that something can go to production with the needed quality level.

Answer 3

If you are practicing Test-Driven Development, scrum can be bug free, insofar as the software that you release to your V&V team fulfills all of the expectations set forth by your unit tests.

To put it another way, you may not be able to certify that your software is bug free, but you can certify that it passes all of its unit tests. The unit tests are something that a supervisor or manager can sign off on. It's not a medical-grade certification (you still need QA for that), but it does relieve your team of the burden of demonstrating a state of being bug free.

So instead of claiming bug-free software on each release (an assertion that is probably untrue), you demonstrate that the approved Unit Tests prove reliability within their agreed-upon scope.

You can also perform functional tests on the software before releasing it to QA. QA can help you develop some or all of those tests.

Further Reading
http://en.wikipedia.org/wiki/Therac-25

The Therac 25 is an excellent case study because problems with the development process (and not several specific software bugs that were found) was cited as the primary reason for the failure.

Answer 4

In my experience, it all depends on how flexible you are able to handle the whole SCRUM-thing. If you need a bug-free release, why don't you add a QA Stage before actually releasing the product to the customer?

The QA Team decides there, if you can ship it to the customer - if it's bug free enough. If not, they'll create bug-reports for you that you can tackle during your next sprint.