Can defect containment metrics be readily applied at an organizational level when there is only a consistant organizational process framework?

Question

Defect containment metrics, such as total defect containment effectiveness (TDCE) and phase containment effectiveness (PCE), can be used to give a good indicator of the quality of the process. TDCE captures the defects that are captured at some point between requirements and the release of a product into the field, indicating the overall effectiveness of the entire process to find and remove defects. PCE provides more detail at each phase of the software development life cycle and how the defect detection and removal techniques are working.

Applying these metrics makes sense at a level where you have a well-defined process and methodology for product development, often a project. However, some organizations provide a process framework that is tailored at the project level. This process framework would include the necessary guidance for meeting certifications (ISO9001, CMMI), practices for incorporating known good techniques (agile methods, Lean, Six Sigma), and requirements for legal or regulatory reasons. However, the specific details of how to gather requirements, design the system, produce the software, conduct test, and release are left to the product development teams.

Is there any effective way to apply defect containment metrics at an organizational level when only a process framework exists at the organizational level? If not, what might be some ideas for metrics that can be distilled from each project (each using a tailored process that fits into the organizational process framework) that captures defect containment metrics to discuss the ability of the process to find and remove defects?

The end goal of such a metric would be to consolidate the defect containment practices of a large number of ongoing projects and report to management. The target audience would be people in roles such as the chief software engineer and the chief engineer (of all engineering disciplines) for the organization. Although project specific data would be available, the idea is to produce something that quantifies the general effectiveness of all tailored processes across all ongoing projects. I would suspect that this data would also be presented as part of CMMI, ISO, or similar audits to demonstrate process quality.

Answer 1

Consolidation of defect management sounds like a good idea unless the projects are so orthogonal as to require separate defect managing systems. For instance, a solo developer doing his own testing could get by with bullet list, while a distributed organization with customers reporting bugs to a support team requires a much more heavy-weight process. This is obvious, but it is important to consider how your consolidation will effect all the developers involved and whether they will be motivated to buy into it.

The problem with bug metrics is if developers feel that bug counts are a way of measuring their performance, then they may get fudge numbers (grouping multiple bugs into one, underestimating bug impact, saying they caught the bug further upstream). This is especially a risk if you present the reports to management. Even you get support from all the developers there can be large variances simply from how they individually report bugs.

Defect containment metics, in particular seem to hold the assumption of an exponentially increasing cost curve downstream. While this may be the case, agilists make a convincing arguments for changing the cost-curve rather than stopping defects as early as possible. I would be skeptical of any metric that assumes a fixed cost-curve, since each project and type of bug may have a dramatically different cost-curves.

If it is dramatically more expensive to catch a bug downstream, then lowering the cost curve by adopting better tools, adopting continuous integration, automated testing, or other agile practices may be a better solution. Of course, in some projects there may be large unavoidable downstream costs (e.g. space shuttle code) in which case defect containment metrics make more sense.

Otherwise, useful 'metrics' in my opinion are categorizing bugs by type to help analyze any systemic flaws. Tracking where the bug was caught (development, integration, or in the field) is also important for analysis, but as I said before I would be hesitant in plugging them into a fixed cost curve to get a single number. Instead cost to fix bug can be estimated by tracking the hours a developer spends on it using a tool such as Mylyn, although this requires some discipline on the developer's part.

Answer 2

IMHO opinion (downvote if you disagree) this is square peg, round hole.

Process frameworks specifically like Agile exist purely because process is not as important as people (See the Agile Manifesto). That is not to say that metrics are not as important. The important ones like Velocity for instance have to do with the rate of user story completion. Defects should be associated with a user story and short lived, otherwise they simply should become yet another user story. But I digress.

Metrics like TDCE and PCE don't make sense in process frameworks like Agile because Agile specifically runs counter to viewing software on the macro project level, especially bug resolution which inherently implies a rigid set of requirements before the development phase.

It is like trying to measure the Dustiness of Bacon. Sure you could probably measure it if you tried, but why would you want to?

With that being said, a process framework encourages the development of a custom framework that specifically meets the project or organizational needs. I don't see why a Mini-PCE metric couldn't be devised as long as you feel that it is relevant and beneficial to the organization, the product, and the customer.

Answer 3

I found an article from CrossTalk magazine titled Advancing Defect Containment to Quantitative Defect Management, written by two engineers at Raytheon. This article describes a technique to do exactly what I want to do - bring software defect containment metrics and try to make them meaningful at an organizational level. They call this technique Quantitative Defect Management (QDM).

It centers around historical defect data and comparing current project data against historical data. By determining the number of defects generated in a given phase in similar past projects, it's possible to estimate the number of defects that you expect to generate in your current project in that same phase. Of course, in an environment that embraces continuous improvement, you would expect to see a downward trend in defects over time, but you can account for this.

Once you know your historical data, you can compare your found in-phase defects to that projected number. This allows you to allocate time and resources to various defect detection strategies for each phase of development, along with reviewing your defect removal and containment data to improve the process to prevent defects in future projects. By budgeting time appropriately for each phase, and continually monitoring and improving the techniques used to detect defects, one can expect an improvement in both process and product quality.

The authors also talk about this metric in relationship to Six Sigma and CMMI (especially Level 4 and Level 5).

Although this doesn't correspond exactly to defect containment, I feel that this is appropriate for management. It will provide data that can be used across projects to compare the current process quality to previous projects, provide additional data for monitoring product quality, and (over time) demonstrate the effectiveness of process improvement efforts.