We have implemented Scrum quite successfully for the past 5 months. Though, we are 3 weeks away from PROD without ever doing any end-to-end integration test. OUCH! I need help. Without tackling the causes of this (at THIS point), we need now to plan the current iteration, which consists of minor improvements and MANY still unknown bug fixes. How do you account for this scenario? How do you plan your iteration to fix bugs yet to be found?
Scrum or not, bugfixing is basically impossible to predict. The best I believe you can do is:
- Start testing immediately, without initial estimate when it will be done.
- When you discover each bug, do initial analysis to the point you can estimate it.
- Estimate the bug and decide whether it has to be fixed and whether it has to be fixed for the initial relese.
- If it has to be fixed, add it to the iteration.
- Plot a burn-down chart. At some point it will start to decrease, meaning you are no longer finding bugs faster than you manage to fix them. At that point you will be able to give out rough estimation (and progressively more precise) when the release can be made.
Than you should make sure next time you start testing early and fix bugs as you go. All sensible methodologies, agile or not, call for fixing known bugs before progressing with new features. Also you should account how much time was spent bugfixing each feature, so you can improve your estimate for implementing feature to debugged state in future.
The estimation and bugfixing are nicely covered by Joel Spolsky in Evidence Based Scheduling and Hard-assed Bug Fixin'. It's not Scrum-related, but I think it's general enough that much of it applies.
How to account for a bug fixing iteration? How do you plan your iteration to fix bugs yet to be found?
Regarding a "bug fixing iteration". Bugs found should be treated no differently from stories. Work with the team to estimate the effort (story points) to fix each bug and work with the product owner/customer to decide if the bug should go into the next iteration.
Regarding "bugs yet to be found". Preferably, the team is finding and fixing issues each iteration. If your not, then discuss this in your next retrospective. If the product quality is so low that release is not possible, then immediately move your best "bug finders" to finding bugs (not fixing). If quality is high enough to provide a beta release to select users - do it. If you can't, then at minimum provide live user demos discussing weak areas you recommend needs improvement.
We don't plan 'bug fix iterations', but we do plan system test iterations before each release. System test is integration, regression and realese tests on all parts of the product. Testers test the product (a fairly large legacy system) and developers fix any bugs that are found. If no bugs are found we either start investigating features schedules for next project or work on internal improvements.
Currently, we plan for six weeks of system tests after code freeze (for a five month project, system test included), to make sure everything works. This is on top of all the testing that is done during the implementation iterations.
You need to define a set of "release" criteria. These could include:
- Mean time between failure
- Number of defects found per day
- Severity of defects found per day
- Number of defects outstanding
Then at the end of each iteration where you have some people testing (either manually or by writing automated tests) and others fixing check to see if you've met your criteria. If you have then release, if not then go for another iteration.
There should be the possibility of an override on this as well as often the raw numbers don't present a realistic picture of the application. You may have a couple of really serious defects, but they only manifest under rare conditions which you can live with in the short term.
One way to do it is to write stories for your integration testing, during which you write new stories for any bugs you find, then fix the bug stories in the next iteration.
Another way to do it is to just make a story that says "Fix bugs found in integration testing." From previous releases you should have an idea of how many issues are usually found, and how difficult they are to fix, so you can assign story points based on that knowledge. You could maybe split it up into components if that makes it more manageable. There's always unavoidable uncertainty in this. Add some extra story points to account for it.
You've probably realized belatedly that the best way is to incorporate a little integration testing into every iteration if possible. Congratulations on recognizing that and improving your process just a little bit for your next release.