I am working on a project and my task is to evaluate quality of that project by quality metrics. I found three types of metrics namely product, process & project. On some websites I fount other types like Defect Detection Percentage, Defect Removal Efficiency, Defects Find Rate, Escaped Defects Found, Function Points, Lines of Code, Mean Age of Unresolved Defects, Mean Time to Fix, Resolved Defects metrics, etc .

So my question is which are the right types of metrics and how do I get that I should use X type of metrics for calculation(My project is in .NET). Also help with the names of freely available tool.

  • 2
    doesn't matter that much - evaluation will end up being highly subjective anyways. – Arnis Lapsa Mar 5 '11 at 10:27
  • 1
    how about user satisfaction and customer satisfaction (not necessarily the same, the ones who pay are not always the ones who use it) – Mörre Mar 5 '11 at 10:29
  • Yes Morre, It is one of the criteria which we can consider. But my client asked me to send detailed report with calculations or using metrics to calculate the quality. – Ujj Mar 5 '11 at 10:31
  • Possibly more answers can be found here: stackoverflow.com/questions/1060548/software-quality-metrics – CarneyCode Mar 5 '11 at 10:49
  • 1
    I know this is a few months old but to Pascal.. software project quality metrics are not relevant to programming??? If we go that route in the literal sense, we might as well remove anything in software engineering that is not directly related to coding for loops and if statements. Last time I checked, this is not a programming venue, but a programmers venue. – luis.espinal Aug 29 '11 at 20:26

One important aspect is left out in the question. Are you trying to measure on individuals, on a team or on the entire organization producing the software? This makes a huge difference for which measurements to use.

Process measurements

In complex software systems, measuring test coverage and commits breaking tests helps focus developer minds.

On the subject of measuring bugs

From experience, I find that using quantitative measures of bugs to measure software development performance tends to have a big impact on how and whether bugs are both reported and resolved. In essence, attaching prestige or loss thereof to bug reporting and/or bug resolution tends to reduce both bug report quality and software development performance.

I think that the main reason for this is that the severity of most bugs are highly subjective.

Instead, measure on the performance (of the bits of user process directly affected by the software) and satisfaction of the users of the software. While this is equally subjective, it will focus your development team on trying to understand the problem domain and producing satisfied customers.

| improve this answer | |

Before specific metrics and graphs, think about 5-10 questions you want to answer with those metrics in simple words. After that plan the information you need to answer those questions. From there you can derive graphs and reports.

For example: - How many significant defects are still unresolved for the project? --> can be as simple as defining correct filter (that represents "significant" issues in your defect tracking system), and getting total number of such defects - How often testers find significant issues? --> can be presented as a trend graph over time for the filter similar to question one - How many defects were found externally by users, rather than internally by testers? etc.

As for the tools, it mainly depends what you use for your defect/testing management software, and what you have in your company. Those tools may provide you with all necessary reports, or it can be as simple as setting MS Excel or OpenOffice Calc document, or you can go to luxury options, e.g. setting BI reporting system.

| improve this answer | |

I should think it is the client's onus to indicate your team what quality metrics he/she wants to see. If that's not the case, one should clarify back with the client if what he/she really wants is evidence (in the form of metrics) that your software shop produces work of quality.

For the later, defects per release (or a delta per release such as changed SLOCs and/or FPs or implemented requirements) would be one to consider. This alongside

  1. customer satisfaction,
  2. rate of defect removal per release/release delta,
  3. average age of defect (from detection to removal), both as a average for the last X months, for the last Y number of releases, and annual since the beginning of time (hopefully to show defect age is not increasing.)
  4. source-code level metrics such as cyclomatic complexity and LCOM (to show the quality of the code structure itself.)

There are tools such as Sonar that can make this type of metric gathering a lot easier. Once you have such QA infrastructure plumbing in place and integrated with your build-release cycles, it is just a matter of gathering the data over time.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.