How do you know if software is good or bad based upon empirical metrics?

Question

I'm currently being asked to look at a project that has finished core development five months ago, but still has a high level of defects. What transpires is for around every 10 defects fixed, we raise at least 4 and in some cases 8 defects.

I believe coding practice at the vendor is poor and there is general agreement around this. However, I am wondering if there is a structural issue with the software? Defect density is a useful measure, but more if the core software is badly written, then all the vendor is doing is shifting the problem about.

In infrastructure it is more defined if something is poorly built, what measurements can you use for software beside defects per LOC?

The product has been in defect fixing phase for 4 months and still has not resolved enough critical defects. We are not injecting new functionality, just fixing regression issues.

This indicates a development quality problem, that is not contented. However, if the product itself is fundamentally flawed, that is a different problem. Concern being is the core code base has been badly written and has limited documentation, all the external developers are doing is shifting the problem from A to B. Once the internal development teams take over I am concerned that they will have to fundamentally rewrite code to get it functional.

So when you accept a product from a third party and are asked to support it, what acceptance criteria would you use to define standards?

Besides getting our lead developer to do peer review of code per release, not sure what else can be done?

Answer 1

You don't.

Software quality is really hard to measure objectively. Hard enough that there isn't a solution. I'm refraining in this answer to dabble on the question whether there can be a solution at all, but simply point out why defining one would be really hard.

Reasoning by status quo

As Kilian Foth pointed out, if there was a simple measure for "good" software, we'd all be using it and everyone would demand it.

There are projects in which managers decided to enforce certain metrics. Sometimes it worked, sometimes it didn't. I am not aware of any significant correlations. Especially critical systems software (think airplanes, cars, etc.) have a lot of metrics requirements to "ensure" SW quality - I am not aware of any studies showing that these requirements actually result in higher quality, and I have personal experiences to the contrary.

Reasoning by counter-intelligence

Also hinted at by Kilian already, and more generally phrased as "every metric can and will be played".

What does it mean to play a metric? It's a fun game for developers: you ensure the metric values look really good, while doing really shitty stuff.

Let's say you measure defects per LOC. How am I going to play that? Easy - just add more code! Make stupid code that results in a no-operation over 100 lines and suddenly you have less defects per LOC. Best of all: you actually decreased the software quality that way.

Tool shortcomings are abused, definitions are stretched to their max, completely new ways are invented.. basically, developers are really smart people and should you have just one developer on your team that has fun playing metrics, then your metrics will be questionable.

This is not to say that metrics are always bad - but the attitude of the team towards these metrics is crucial. In particular, this implies it's not going to work well for any subcontractor/3rd party vendor relationship.

Reasoning by wrong targeting

What you want to measure is software quality. What you do measure is one or more metrics.

There is a gap between what you measure and what you believe it'll tell you. This gap is sort of huge even.

It happens all the time in all sorts of businesses all around us. Ever seen decisions based on KPIs (key performance indicators) ? It's just the same problem - you want a company to do well, but you measure something else.

Reasoning by quantifiability

Metrics can be measured. Which is the only reason we deal with them at all. Software quality, however, extends way beyond these measurable entities and has a lot to it that is very tough to quantify: How readable is the source code? How extensible is your design? How hard is it for new team members to get onboarded? etc. etc.

Judging software quality only by metrics and turning a blind eye to the parts of quality that you can't quantify is certainly not going to work out well.

edit:

Summary

Let me point out that the above is all about objectively judging whether software is good or bad based on metrics. This means, it is not saying anything about whether and when you should apply metrics.

In fact, this is a unidirectional implication: bad metrics imply bad code. Unidirectional means that bad code does not guarantee bad metrics, nor do good metrics guarantee good code. On the other hand, this in itself means that you can apply metrics to judge a piece of software - when you keep this implication in mind.

You measure software A, and the metrics turn out really bad. Then you can be certain that the quality of the code is bad. You measure software B and the metrics are ok, then you have no clue whatsoever about the code quality. Don't be fooled into thinking "metrics good = code good" when it's really just "code good => metrics good".

In essence, you can use metrics to find quality problems, but not quality itself.

Answer 2

Yes, you can tell the code has quality problems by looking at metrics to some degree.

More specifically, run a complexity analysis tool on the code base and you will get an idea of whether the code is good or bad.

For example, you could run source monitor.

What this will tell you is how complex the code is. I can tell you every problematic system that I have experienced had bad numbers. Complexity on 10s to 100s of methods well over acceptable limits. Terrible numbers. Terrible complexity, nesting, depth, etc. This will lead to lots of problems, constant high defect rate, hard to make changes, without breaking something else, etc.

Another thing is defects. Over time the system should stabilize. Ideally new defects should trend toward zero or flatten out to a low number, which means new and current defects should decrease over time.

The plot should look something like this:

Defects Over Time

If they stay constant or increase, then the software is not good. Your only 4 months in, so I would give it a few more months to a year. After 6 months to a year, if you had a constant stream of defects, then it's bad quality. Your team developed another ball of mud.

Next up tests. Do you have them? If no then less quality, more bugs, more churn. If you have them, metrics like code coverage are good to get an idea of how much code is being covered, but it will not measure quality. I've seen great code coverage numbers but the actual tests were crap. They weren't testing any behavior or functionality of the system. Developers were just writing them to improve the metric numbers for management. So you have to have tests and they have to be good. Code coverage metrics in of themselves are not an indicator of quality.

Code reviews, are you performing them? If not, less quality. This is especially important with junior developers. If your doing agile, just add a code review task to your story called "code review" . If management wants to track numbers you will need a tool that track reviews like Crucible. I think the code review numbers or metrics are not that important here other than the fact that they should be a part of your process. Not every check in needs a review. But, reviews can help make sure people are not re-inventing the wheel or writing code that others can't understand and/or maintain.

Finally, your just going to have to assess the code, no metric will help outthere. Every problematic code project had these qualities:

• Poor data structures. Everything is a string, or XML is passed everywhere and parsed everywhere, god objects, or needlessly complex or generic data structures, no domain model.
• Lack of organization or structure, any non-trivial project should have some division or layering that separates out the logic. Have a look here...If you don't see this type of organization or separation (mixing of logic everywhere) then the system will be harder to maintain and understand.
• Over abstractions. Sometimes the reverse is true, the system is over abstracted. If everything is a interface and abstract class, or you have to navigate through a ton of class "wrapper" type classes, the quality will be bad because new developers will not be able to navigate through the object bloat.
• Hard to understand code. If your a seasoned developer and if you are looking at code that hard to understand, it will have quality problems. My personal metric is that if I am looking at code and it is hard follow or makes me feel dumb, or I feel like I am wasting a lot of brain cycles to figure out the logic, then it's bad code. If seasoned developers have a hard time following, then just imagine what it will be like for newer developers. They will introduce problems.

My advice would be run an complexity analysis on the code. Don't share the results, instead follow up on the results do some independent investigation (look at the code) and make a determination of the overall health of the code base. From this, form an action plan and try remediate some of the complex areas of the code.

Answer 3

The sad thing with metrics is that you may end up improving the resulting values of your metrics, but not the quality intended to be measured by them...

In Visual Studio, there is a setting for treating compiler warnings as errors. Now some people do not understand the warnings, and in order to get the code compiled, will use simple tricks (like ´pragma disable warning´ or adding a ´new´ keyword to a function/property hiding a non-virtual member of a base class).

If you have access to the source code, you can run static code analysis. For .Net projects, you can use e.g. FxCop or ReSharper InspectCode. When used by the developing team correctly, the tools will help improve quality. But of course, some "fixes" for removing warnings without understanding them are possible.

You could look at automated tests / UnitTests: how good is the code coverage? But coverage alone won't tell you if the tests actually check the code, or just had it executing once.

Striving for high quality is an attitude, which can be supported by many tools and their metrics, but metrics without the attitude of the developers do not help.

Answer 4

One thing you should look at in addition to collecting a metric like defect injection is figuring out the source of the defects. Often times it is related to the specification.

Basically, is is an error in the specification, an ambiguity in the speification, left for the implants to decide or is it a bug in the implementation.

A more qualitative approach is to ask is the software useful? If you look hard enough you can find defects in any piece of software. However, if it works well enough to earn money, then it might be not be so bad.

Answer 5

Bottom, there isn't a way to know.

For the original question (before philosophical answer): What is the product supposed to do and does it do it? Measuring by defect count / density isn't sufficient. I couldn't tell if this was a library, or an application, how large the code base, how large the problem domain is, nor what the severity of the defects is. For example, not handling one of 123 input formats could be a trivial defect or a show stopper, depending on the importance of the format not properly handled. And better than nothing is a high standard.

Assumption I make for this question: There is a difference between Code and Software. I define software as what a client/user uses to solve a problem, whereas code is the building material of software.

Software can only be measured subjectively. That is, the metric that matters for software is whether people use it to solve a problem. This metric depends on other's behavior, hence its subjectively. Note: For some problems a piece of software may be quite useful, and thus considered high quality (Excel for calculations), but not quality software for a different problem (Excel for playing MP3 files).

Code can (usually) be measured with empirical metrics. But the interpretation isn't 'yes/no' for quality, or even really on a scale of '0 to N'. Metrics measure against a standard. So, metrics can find areas of concern defined by the standard, but the absence of areas of concern is not proof that this is quality code. For example, useful metrics: Does it Compile? No -> Not quality. Yes -> ???. Does it pass Unit Test? No? Maybe? (because, Is the Unit Test Quality Code?), Yes -> ???.

So, like Godel's Incompleteness Proof showed for axioms of mathematics (that is, there exist mathematical statements that can't be proven true or false for any finite set of axioms), I don't think we could ever actually answer 'is this quality code?' for every piece of code. Intuitively, there is probably a mapping in there between software metrics to answer quality and mathematical axioms to answer provably true or false.

Another way to make this argument, is to step into natural language. William Shakespeare, Lewis Carroll and Mark Twain were all successful writers, and beloved of many for the quality of their writings. Yet what standard of grammar, vocabulary, style or voice could we apply that would consistently rate them higher than random 12th graders? And, while it may be possible to create some synthetic measure for those three, how would it rate the Book of John (KJV), J.R.R. Tolkien, Homer, Cervantes, etc? Then throw in Burroughs, Faulkner, Hemingway, Sylvia Plath, and so on. The metric won't work.

Answer 6

I would measure this by auditing (and looking for deviations in) their process.

I would be looking for evidence of a process to deliver that involved central source control, central build system and a process that ensured code is tested before integration into the released branch.

I would also be looking for evidence of how they have modified their processes in response to situations where defects have passed through their release process.

If they are unable to pass this level of audit, then you can not expect them to deliver consistent reliable releases.

If they pass this audit, and are continually improving their process then their consistency of output is likely to improve over time.

If this does not fix it, then it is likely they have a code architectural problem which makes extending and testing their current code base problematic, in which case there are no good options.

Answer 7

If you're looking for completely automated measurements, then I recommend these guys: Software Improvement Group

It's essentially an aggregate of various metrics that can be automatically extracted from the source code (like, unit test coverage, function size, class entanglement, duplication, LOC, etc.). Those values are then converted into 1-5 star rating.

They also have a decent book describing all their metrics in practice that's worth a read: 'Building Maintainable software'.