12

There are various types of quality that can be measured in software products, e.g. fitness for purpose (e.g. end use), maintainability, efficiency. Some of these are somewhat subjective or domain specific (e.g. good GUI design principles may be different across cultures or dependent on usage context, think military versus consumer usage).

What I'm interested in is a deeper form of quality related to the network (or graph) of types and their inter-relatedness, that is, what types does each type refer to, are there clearly identifiable clusters of interconnectivity relating to a properly tiered architecture, or conversely is there a big 'ball' of type references ('monolithic' code). Also the size of each type and/or method (e.g. measured in quantity of Java byte code or .Net IL) should give some indication of where large complex algorithms have been implemented as monolithic blocks of code instead of being decomposed into more manageable/maintainable chunks.

An analyis based on such ideas may be able to calculate metrics that are at least a proxy for quality. The exact threshold/decision points between high and low quality would I suspect be subjective, e.g. since by maintainability we mean maintainability by human programmers and thus the functional decomposition must be compatible with how human minds work. As such I wonder if there can ever be a mathematically pure definition of software quality that transcends all possible software in all possible scenarios.

I also wonder if this a dangerous idea, that if objective proxies for quality become popular that business pressures will cause developers to pursue these metrics at the expense of the overall quality (those aspects of quality not measured by the proxies).

Another way of thinking about quality is from the point of view of entropy. Entropy is the tendency of systems to revert from ordered to disordered states. Anyone that has ever worked on a real world, medium to large scale software project will appreciate the degree to which quality of the code base tends to degrade over time. Business pressures generally result in changes that focus on new functionality (except where quality itself is the principle selling point, e.g. in avionics software), and the eroding of quality through regression issues and 'shoe-horning' functionaility where it does not fit well from a quality and maintenance perspective. So, can we measure the entropy of software? And if so, how?

Improve this question
1
  • I agree with S. Lott. In life there is frequently a difference between 'how it ought to be' and 'how it is'. Boy do I wish more people on this planet had overcome their 'good intentions' approach and looked hard at 'how it is'. In addition to wrong incentives, there will be a dangerous false sense of security. Combine that with people trying to game the system (which is only natural because they ALWAYS try to better their conditions (whether monetary or other)), and you get a crappy situation. It should come as no surprise that 'once in a millennium' market crashes occur once every 2 decades.
    – Job
    Aug 3, 2011 at 17:51

7 Answers 7

Reset to default
20

This a dangerous idea. "Objective" proxies for quality lead directly to management rewards and developers will pursue these metrics at the expense of the actual quality.

This is the law of unintended consequences.

Quality -- while important -- is only one small aspect of software. Functionality and value created by the software are far, far more important than quality.

All metrics lead to activity to optimize the metric. That, in turn, has consequences that you might not really like.

Software is very complex. It's hard to understand how truly complex it is.

Even such "obvious" things as unit test code coverage can waste time. Getting to 100% may require creating tests that are actually more complex than the trivial code being tested. Getting to 100% coverage may involve an unacceptable cost. [The alternative for trivial, small, rarely-used code is test-by-inspection. But that doesn't fit the metrics game of 100%.]

Another example is Cyclomatic Complexity. It is one of the best measures of code quality. But it's can be gamed by creating many small functions that may be harder to read (and harder to maintain) than one larger function. You wind up in code reviews where you agree that it may not be very readable but it meets the complexity threshold.

Improve this answer
23
  • 3
    "All metrics lead to activity to optimize the metric." I think that is too often true. However, it shouldn't be. Metrics should, like I alluded to in my last paragraphs, guide management. Too often, though, decisions are made exclusively because and for the numbers, without an understanding of the meaning of the numbers and the risks and trade-offs associated with the decision.
    – Thomas Owens ♦
    Jul 29, 2011 at 10:10
  • 3
    "However, it shouldn't be." Explain some way in which people can be told not to optimize their rewards. Find a single example of human behavior where cultural rewards (based on all kinds of crazy social structures) are not primary, paramount and the most important thing people will pursue. Anything that involves "should" or "should not" has to be gauged against what people really do. They really optimize their rewards. If metrics are part of the rewards, people optimize the metrics. Please don't use "should" to describe people's behaviors.
    – S.Lott
    Jul 29, 2011 at 10:13
  • 2
    @Thomas Owens: "You simply don't have any rewards to optimize based on metrics". That's funny. How will you keep them so secret? Once I find out that your code was accepted sooner than mine, I'll want to know how management decided that your module was done and mine was not done. Once I find the metric that "guides" that decision, I'll totally game the metrics to get done as early as you. If there's no metric that I can game, then I'll see that the decision was arbitrary, management likes you better than me, and I'll quit because there's no performance standard that I can perceive.
    – S.Lott
    Jul 29, 2011 at 11:01
  • 2
    @Thomas Owens: "I've never seen metrics lead to rewards". Cultural incentives exist in all situations where two or more people work together. "Individuals are recognized for their performance". A metric for cyclomatic complexity becomes a goal. If you meet your cyclomatic complexity goal quicker than me, then there are cultural rewards: you're more "productive" than me. I need to game my complexity metric to appear as "productive" as you.
    – S.Lott
    Jul 29, 2011 at 12:29
  • 2
    @Thomas Owens: "It's a matter of personal pride". That's a great example of a cultural reward system. Metrics can skew this because of the unintended consequences of being able to create a good-looking metric that doesn't match good code. You've provided an excellent example of cultural rewards that can be skewed by metrics.
    – S.Lott
    Jul 29, 2011 at 12:57
4

I also wonder if this a dangerous idea, that if objective proxies for quality become popular that business pressures will cause developers to pursue these metrics at the expense of the overall quality (those aspects of quality not measured by the proxies).

Bingo, and no "if" about it. This is called "Measurement Dysfunction" and has been observed and written about many times Joel wrote an article on it in 2002 referring a book by a Harvard Professor.

That doesn't mean such metrics are useless, just that one should never base incentives or policies explicitly on such proxy measurements. If you want to improve quality, a proxy metric with a very bad value is probably a good point to start. But you cannot conclude that quality is good just because all your metrics have great values.

Improve this answer
3

What I'm interested in is a deeper form of quality related to the network (or graph) of types and their inter-relatedness, that is, what types does each type refer to, are there clearly identifiable clusters of interconnectivity relating to a properly tiered architecture, or conversely is there a big 'ball' of type references ('monolithic' code).

This sounds like fan-in and fan-out. Fan-in counts the number of modules that call a given module and fan-out counts the number of modules called by a given module. A warning sign to use would be modules that have a large fan-in and a large fan-out, as this might indicate poor design and a key target for refactoring or redesign.

Also the size of each type and/or method (e.g. measured in quantity of Java byte code or .Net IL) should give some indication of where large complex algorithms have been implemented as monolithic blocks of code instead of being decomposed into more manageable/maintainable chunks.

A simple measurement would be lines of code. You could break it down into total lines of code across the entire project and lines of code per module (perhaps using different size modules). You can use this as a warning indicator indicating that you should review particular modules. A book on software quality measurements and metrics discusses some work which indicates that the relationship between defect rates and module size is curvilinear, where the average defect per KSLOC comes with modules with a size between 175 and 350 SLOC.

Something a little more complex would be cyclomatic complexity, which is designed to indicate the testability, understandability, and maintainability of a system. Cyclomatic complexity counts the number of independent paths through an application or module. The number of tests, and therefore the effort needed to produce and execute the tests, is strongly related to cyclomatic complexity.

The exact threshold/decision points between high and low quality would I suspect be subjective, e.g. since by maintainability we mean maintainability by human programmers and thus the functional decomposition must be compatible with how human minds work.

I'm not sure that is the case.

For example, there's been research that suggests that a human's working memory can only hold 7 plus/minus 2 objects. This is probably of interest for measuring fan-in and fan-out - if I'm working in a module, and it is connected to more than ~7 other modules, I probably won't be able to keep track of exactly what those other modules are in my head.

There has also been work on relating defects to metrics such as cyclomatic complexity. Since you want to minimize defects in your system, you can identify points that either need more effort testing or refactoring, as identified by high cyclomatic complexity.

I also wonder if this a dangerous idea, that if objective proxies for quality become popular that business pressures will cause developers to pursue these metrics at the expense of the overall quality (those aspects of quality not measured by the proxies).

This is the case with any measurements or metrics. They need to be used to understand the system and make informed decisions. The phrase "you can't manage what you can't measure" comes to mind. If you want high quality software, you need some measurements and metrics to assess that quality. However, there is a flip side to this. You can't manage exclusively by the numbers. You can use the numbers to make informed decisions, but you can't make a decision only because the numbers say so.

Improve this answer
2
  • The thing with fan-in/out is that it gives two numbers per module/class (or whatever) and therefore ignores some of the deeper organizational structure of how modules are connected. E.g. you could have a small cluster of highly connected modules related to a logical tier, and you would expect the connections between tiers to be minimal (in comparison), representing a well defined interface/contract between tiers. I figure we are happy that some modules are heavily connected (e.g. commonly used helper methods/classes), but depending on the 'structure' of the connectivity (that's my hypothesis).
    – redcalx
    Jul 29, 2011 at 12:29
  • @locster You probably want to expand on it and note, for example, which packages the classes you are connected to are in. Don't just look at the raw numbers, but break it down into things like X classes within my package, Y classes outside my package, or Z classes in this particular package. If you see fan-out between modules in your data model and modules in your UI, that could be an indicator of a problem. You need to dig a little deeper than just the raw numbers.
    – Thomas Owens ♦
    Jul 29, 2011 at 12:39
3

There are metrics or proxies for many of the qualities you are interested in:

  1. Lines of code
  2. Fan in, fan out
  3. Error rate per 1000 lines of code
  4. Cyclomatic complexity
  5. Code coverage
  6. Decision point coverage
  7. Ratio of errors fixed/introduced by maintenance activities
  8. Function point analysis

There are some issues with all these items:

  1. Work being done to optimise the metric - a universal trend; massively exacerbated if any of the metrics are used as the basis for assessment or reward for teams or individuals.
  2. I am not aware of any metric that is context free. This implies that no comparison is possible across shops - only within shops, over time. Metrics arising from such comparisons are still valuable - "are we producing code more correctly now than a year ago".

The total effect of these issues is that metrics such as these are likely to be valuable only within a wider culture - such as TQM, quality assurance (not control), continual improvement, kaizan etc. It is necessary to define elements of both the culture, and how it needs to change. If you have definition of these, then metrics such as these become essential tools in helping to improve the quality of the code, working practices, productivity, etc. Without this wider context, metrics will generate work to optimise the metric; will become the tool of the beancounter to increase productivity, and decrease costs; and will become an obstacle to be gamed by the development staff.

Improve this answer
2

You could be obsessed with metrics, or you could be obsessed with the best people, tools, practices for engineering and QA that you can afford. I would be much happier having several paranoid QA geniuses who have read 'Fooled by Randomness' and who like to automate than a bunch of nicely formatted reports with numbers.

Improve this answer
2
  • +1 for Nassim Taleb book reference. Flawed reasoning/epistemology being on the chain of causality for low quality.
    – redcalx
    Aug 14, 2011 at 10:20
  • @locster, your comment made me think of the F# pipeline operator :). You start with 'Nassim Taleb book reference' but end with 'chain of causality for low quality' (instead of 'low quality causality chain'). Just as in English we sometimes like to have two ways of saying things, we might prefer that in a programming language too.
    – Job
    Aug 14, 2011 at 17:08
1

There's this fundamental problem with metrics.

Pretty much all of the metrics proposed have been shown, in the real world on real code, to be strongly or very strongly correlated with raw SLOC (source lines of code).

This is what killed Halstead's metrics, back in the 1970s. (By accident one day, ca. 1978, I sat in on a talk by a new PhD about Halstead's metrics, in which he pointed this out.)

More recently, McCabe's cyclomatic complexity has been shown to be very strongly correlated with raw SLOC, to the point that the guy who did the study wondered out loud if McCabe's metric told us anything useful at all.

We've known for decades that big programs were more likely to have problems than small ones. We've known for decades that big subroutines were likelier to have bugs than small ones. Why do we need arcane metrics to tell us this, when looking at four printer pages spread out over a table should be convincing enough?

Improve this answer
4
  • To be maintainable we need the code to be in human 'chunks, hence a SLOC metric looks pretty good from that perspective. Howeverm, for a given size you could have varying numbers of unique paths (as per cyclomatic complexity) and I would argue that more paths is a proxy for less easily understandable. Hence I would argue that CC probably does add /some/ additional value to SLOC, so long as you allow for some flexibility, exceptions to the rule, etc. That is, don't strictly enforce CC.limits/goals.
    – redcalx
    Feb 4, 2013 at 17:31
  • 1
    @locster: Given two 100 SLOC modules, one with a CC of 47 is likely to have more problems than one with a CC of 3. HOWEVER, for real-world code, in large quantity, one finds that short modules tend to have low CC and long modules tend to have high CC, to the point that knowing the SLOC gives you a very good guess at the CC, and vice versa. This is what is meant by "very strongly correlated." AS SUCH, on real code, any benefit you get from noticing CC = 47 is MORE EASILY gotten from noticing SLOC = 1500. (Numbers pulled at random, principle is the same.)
    – John R. Strohm
    Feb 5, 2013 at 22:22
  • Yes I agree that they will tend to be strongly correlated, although the relationship is generally non-linear. e.g. A CC score is roughly LOC raised to some power. So from a psychological point of view the CC score can be seen to become very large very fast, whereas the associated SLOC score just seems 'only a bit higher'. Yes I know I'm clutching at straws here :)
    – redcalx
    Feb 5, 2013 at 22:54
  • @locster: I've been doing this for something over 30 years. These days, I routinely see stream-of-consciousness run-on routines, that go on and on for a few hundred SLOC, for no reason. In all those years, I have seen exactly one (1) routine that actually NEEDED to be more than one printer page of code (about 60 lines). All the rest could have been quite profitably factored down, and the readability and reliablity increased significantly. (That doesn't count big state machines. They can be a problem in this area, but they are RARE.)
    – John R. Strohm
    Feb 6, 2013 at 10:36
-2

Given all the other answers here, I feel kind of silly with this small one. Take a look at Crap4j, which tries to rank methods in java by how much they stink. (The project looks abandoned.)

It uses a combination of cyclomatic complexity and test coverage. Like every other metric, it's gameable.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.