35

What are useful metrics to capture for source code?

How can metrics, like for example (Executable?) Lines of Code or Cyclomatic Complexity help with quality assurance or how are they beneficial in general for the software development process?

Improve this question
3

17 Answers 17

Reset to default
30

"Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs."- Bill Gates

Improve this answer
6
  • 3
    Please don't update non-answers.
    – Eric Wilson
    Jan 12, 2011 at 14:04
  • 3
    While an amusing anecdote, this answer does little to contribute to the answer of this question.
    – Chris Knight
    Feb 28, 2011 at 10:31
  • 8
    @Chris This answer got a lot of up-votes (or "updates" as FarmBoy wants to call it) because many developers believe that software metrics are useless. If you disagree or feel that you have a better response to the question, then post your own answer. Commenting like you've done here isn't productive; you've contributed nothing yourself.
    – chrisaycock
    Feb 28, 2011 at 15:30
  • 7
    My downvote and comment are intended to discourage answers which lack depth and don't directly address the OP's question. This could be a much better answer if you went into more detail about why you believe software metrics to be useless with regards to software development and quality assurance and focused on more than just LOC.
    – Chris Knight
    Feb 28, 2011 at 23:27
  • Software metrics are actually very useful if you use them properly. That is, the more the LoC -> the more the bugs -> the worse the quality. I've never seen it fail as a measure for quality. And an airplane is definitively better if it does the same travel at the same speed but requiring much less weight. Obviously Bill Gates didn't know much about airplanes when he said that, nor did know enough about software either it seems.
    – Pablo Ariel
    Dec 14, 2018 at 18:42
12

Take a look on Jeff's posts on the subject:

A Visit from the Metrics Maid

Software Engineering: Dead?

There is an old, but good, post from Joel too, closely related to software metrics, and I strongly recommend its reading: The Econ 101 Management Method

The key point, for me, is this, quoting Jeff: "Responsible use of the metrics is just as important as collecting them in the first place."

Improve this answer
1
  • +1 for quoting that Jeff's one-liner. Pure, battle-hardened wisdom right there.
    – luis.espinal
    Feb 8, 2011 at 17:10
12

What confuses me about code metrics is that it isn't done more. Most companies report on the efficiency of their employees, suppliers, and systems in place, but nobody seems to want to report on code. I will definitely agree with answers that state that more lines of code is a liability but what your code does is more important.

Lines Of Code: As 'Ive mentioned this is a vital measurement and should be taken the most seriously, but on each level. Functions, classes, files and interfaces can indicate do-everything code that is hard to maintain and costly in the long term. It's infinitely hard to compare the total lines of code versus what a system does. It could be something that does many things and in that case there will be many lines of code!

Complexity: This measurement is good to do on code bases you haven't worked on, and can give you a good indication of where problem areas lie. As a useful anecdote I measured complexity on one of my own code bases, and the highest complexity area was the one that I was spending the most time when I needed to change it. Working towards reducing the complexity resulted in a massive reduction in maintenance time. If management had these measurements at hand they could plan refactoring iterations or redesigns of specific areas of a system.

Code duplication: This is a very important measurement as far as I'm concerned. Code duplication is a very bad sign and could point to either deep problems in low levels of a system's design or developers that are copy pasting, causing massive problems in the long term and systems that are unmaintainable.

Dependency Graphs Finding bad dependencies and circular dependencies are an important measurement in code. This almost always points to an incorrect high level design that needs revising. Sometimes one dependency can suck in a whole lot of unnecessary other ones, because someone is using addNumber inside an e-mail library to do their finance calculations. Everyone is shocked when the e-mail library is changed and finance breaks. If everything is dependent on one thing it can also point to do-everything libraries that are hard to maintain and badly designed.

A good measurement will always tell you that every feature of a system has a small footprint. Less dependencies, less complexities, less duplication. This points to loose coupling and high cohesion.

Improve this answer
8

Won't this "source code metrics" crap EVER die?

Raw source lines of code (SLOC) is the oldest, easiest, most basic metric there is.

Halstead originally proposed a whole bunch of metrics. Lots of people were having lots of fun writing measurement programs until some spoilsport did the obvious study, and demonstrated that each and every single Halstead metric was strongly directly correlated with SLOC.

At that point, Halstead's metrics were abandoned, because SLOC is always easier to measure.

Improve this answer
5
  • 1
    Any links to the study?
    – Jon Hopkins
    Jan 12, 2011 at 14:08
  • Google is your FRIEND, but here's one to get you started. ecs.csun.edu/~rlingard/comp589/HoffmanArticle.pdf
    – John R. Strohm
    Jan 12, 2011 at 14:42
  • 6
    Interesting study, though their study only looked at programs generally between 50 and 100 lines of code. With such a small well-defined problem to solve, the end result doesn't seem so surprising.
    – Chris Knight
    Feb 28, 2011 at 23:48
  • I'd say that in the real world, all these studies turn to mud.
    – Warren P
    Nov 14, 2012 at 14:33
  • This is true. The more the lines of code, the sh1ttiest the quality.
    – Pablo Ariel
    Dec 10, 2018 at 16:11
8

Source code metrics for quality assurance aim at two objectives:

  • writing code with less bugs inside
  • writing code for easy maintenance

Both lead to writing code as simple as possible. This means:

  • short units of code (functions, methods)
  • few elements in each unit (arguments, local variables, statements, paths)
  • and many other criteria more or less complex (see Software metric in Wikipedia).
Improve this answer
8

Metrics are only useful if you know what to do with the answers you get. In essence a software metric is like a thermometer. The fact that you measure something at 98.6° F doesn't mean anything until you know what the normal temperature is. The above temperature is good for body temperature but really bad for ice cream.

Common metrics that can be useful are:

  • Bugs discovered/week
  • Bugs resolved/week
  • # Requirements defined/release
  • # Requirements implemented/release

The first two measure trends. Are you finding bugs faster than you can fix them? Two possible outcomes: maybe we need more resources fixing bugs, maybe we need to stop implementing new features until we catch up. The second two provide a picture of how close you are to being done. Agile teams call it a "burn down" chart.

Cyclomatic Complexity is an interesting metric. At it's base concept it's the number of unique execution paths in a function/method. In a unit-test heavy environment this corresponds to the number of tests needed to verify every execution path. Nevertheless, just because you have a method that has a Cyclomatic Complexity of 96 doesn't mean it is necessarily buggy code--or that you have to write 96 tests to provide reasonable confidence. It's not uncommon for generated code (through WPF or parser generators) to create something this complex. It can provide a rough idea of the level of effort needed to debug a method.

Bottom Line

Every measurement you take needs to have the following defined or it is useless:

  • An understanding of what "normal" is. This can be adjusted over the life of the project.
  • A threshold outside of "normal" where you need to take some sort of action.
  • A plan for dealing with the code when the threshold is exceeded.

The metrics you take may vary widely from project to project. You may have a few metrics that you use accross projects, but the definition of "normal" will be different. For example, if one project discovered an average of 5 bugs/week and the new project is discovering 10 bugs/week it doesn't necessarily mean something is wrong. It just might be the testing team is more meticulous this time around. Also, the definition of "normal" may change over the life of the project.

The metric is just a thermometer, what you do with it is up to you.

Improve this answer
2
  • Another bug related on that can be useful in some cases is bugs per lines of code. In general, mature code bases should have a fairly low number of bugs per lines of code as opposed to applications that are still under development.
    – rjzii
    Jan 17, 2011 at 13:01
  • @Rob Z, with any metric, people will do just enough to optimize that metric. At bugs per line of code, you might have a developer introduce an unused variable that they increment just to increase the number of bug-free LOC (since SLOC counters can detect multiple semicolons). Of course, that also artificially increases the amount of code to wade through.
    – Berin Loritsch
    Feb 28, 2011 at 17:04
7

To the best of my knowledge, the number of bugs found directly correlates with lines of code (probably churn), modulo language, programmer, and domain.

I don't know of any other straightforward and practical metric well-correlated with bugs.

One thing I'd like to do is start running the numbers for different projects I'm on - Test Coverage :: kLOC, and then discuss "perceived quality" to see if there is a correlation.

Improve this answer
7
  • 1
    So the more code there is the more bugs there are in it?
    – user1249
    Dec 18, 2010 at 5:12
  • 3
    @Thor:l yep yep. shocker, huh?
    – Paul Nathan
    Dec 18, 2010 at 7:13
  • As far as I remember typical industry numbers are around 2-3 errors per 1000 lines of code for average projects, approaching something like 0.5 errors per 1000 lines of code for nuclear plant control software or NASA projects where they put down an enourmous amount of effort, control, testing, review, etc because failures can have very severe consequenses. Anyone that have some reference to numbers supporting this?
    – hlovdal
    Dec 20, 2010 at 23:41
  • 2
    @hlovdal: 2-3 errors per KSLOC is already a very low figure. The lowest figures I know from aerospace and security domains are of the order of 0.1 errors per KSLOC. Typical figures seem to be 20 to 50 errors per KSLOC. For reference, Google for Andy German's paper titled "Software Static Code Analysis - Lessons Learnt".
    – Schedler
    Jan 12, 2011 at 11:03
  • 1
    I'd dispute these figures - it entirely depends on the language, compiler and executable environment. Typos in JavaScript code can take years to find, but a typo in a compiled language would be found on the first compile.
    – JBRWilkinson
    Feb 27, 2011 at 22:48
7

Source code is a liability, not an asset. With that in mind, measuring lines of code is analogous to tracking dollars spent while building a house. It needs to be done if you want to stay under budget, but you wouldn't necessarily think that spending $1000 a day is better than spending $50 a day; you'd want to know how much of the house got built for that money. It's the same with lines of code in a software project.

In short, there are no useful metrics for source code because measuring source code by itself isn't useful.

Improve this answer
4

Since source code is simply a combination of sequence, selection, and repetition. If I were to describe the most optimal piece of software that we could ever reasonably expect to produce it would be as follows. Software with nearly 100% testing code coverage using the least amount of lines of code necessary to do the job and yet flexible enough to withstand changes.

Improve this answer
6
  • 2
    100% coverage is only 100% if it covers all paths, not just all lines. In any realistic piece of software 100% path coverage is a bad goal to set, because it will be very expensive to reach, and still will only tell you that your code behaves as designed, not that the design itself is sound. You could have gaping security holes, and have 100% path coverage.
    – Joeri Sebrechts
    Jan 12, 2011 at 10:58
  • +1 More source code is not necessarily better.
    – Larry Coleman
    Jan 12, 2011 at 13:53
  • Only very simple applications are amenable to 100% test coverage (making the coverage redundant). It is computationally expensive (if not infeasible) to achieve 100% test coverage for complex software. We have known that fact on firm grounds for like 6 decades now. Secondly, testing only tells you that you have not found a bug - it doesn't guarantee you that there are no bugs not about structural quality, size or complexity (something also known for a pretty long time.) Not knowing these facts when working in software is akin to a physicist not knowing the laws of thermodynamics, really.
    – luis.espinal
    Feb 8, 2011 at 17:06
  • @luis.espinal Software so big that is too computationally expensive to test is incredibly poorly written software. It is close to not having a clue on how to make working software.
    – Pablo Ariel
    Dec 10, 2018 at 16:22
  • @PabloAriel - "Software so big that is too computationally expensive to test" << That is not what I said. Read the comment (perhaps two or three times) to make sure you are actually reading what you think you are reading.
    – luis.espinal
    Dec 10, 2018 at 18:43
4

An anecdote to show why KLOC counts are useless (and even harmful) to gauge performance.

Years ago I worked on a large project (70+ people in our company, another 30+ at our customer) which used KLOC counts as the sole measure of performance of teams and individuals.

For our Y2K effort (tells you how long ago it was :) ) we did a large cleanup of the section of the code my team was responsible for. We ended up for the release writing about 30.000 lines of code, not a bad 3 months of work for 5 people. We also ended up scrapping another 70.000 lines of code, a very good job for 3 months of work, especially combined with the new code.

End result for the quarter: -40.000 lines of code. During the performance review following the quarter we got an official reprimand from the company for failing to meet our productivity requirements of 20.000 lines of code produced per quarter (after all, the tools had shows we'd produced -40.000 lines of code), which would have resulted in all of us being listed as underperforming and bypassed for promotions, training, pay increase, etc. etc. had not the project manager and QA team intervened and gotten the reprimand overturned and replaced by a commendation.

A few months later (such things take time) we were told the company was reviewing their productivity standards and had hired a team of experts to create a new system based on function point analysis.

Improve this answer
5
  • Why didn't you just show the diffs?!
    – reinierpost
    Feb 28, 2011 at 8:49
  • I think that's what was done. But if a system is so rigid it doesn't even ring alarm bells when such a blatantly false datapoint appears it won't do much good.
    – jwenting
    Feb 28, 2011 at 10:41
  • 2
    Your answer doesn't show that KLOC are useless, it shows how to not use them.
    – Neil N
    Nov 17, 2011 at 18:19
  • 2
    it shows that relying on them as a measure of productivity is shortsighted, relying on them as the only measure is idiotic. In other projects using KLOC as a measure of productivity and even quality we easily inflated the numbers by creating coding standards that caused loads of lines (C++ bracing practices, extra empty lines with just a short comment everywhere, splitting the conditions in an if statement over 3 lines, etc.).
    – jwenting
    Nov 21, 2011 at 6:28
  • 1
    Using SLOC as a productivity metric is just dumb and will probably never give good results. Using SLOC as a quality metric indicating maintainability and number of defects is more sane, with all the caveats already discussed on this question.
    – redcalx
    Nov 7, 2013 at 14:26
2

I'm surprised no-one's mentioned Statement/Decision Coverage of Unit tests (percentage of code exercised by unit tests) yet.

Code coverage is useful in that you know what percentage of the application doesn't fail catastrophrically; with the rest of its usefulness depends on the quality of the unit tests.

Improve this answer
2
  • code coverage is a false metric (though can have some use) as well. It invites writing nonsense tests just to get higher coverage. And of course there's stuff that'll never be covered, and people will start to avoid writing that stuff. e.g. I've seen code coverage tools that flagged JavaDoc as code and of course it wouldn't be covered. another tool flagged all empty lines as not being covered by tests. You'd agree that doing away with comments and whitespace in your code is worse than missing out on unit tests for some setters I hope?
    – jwenting
    Feb 28, 2011 at 10:44
  • Absolutely, bad unit tests hurt more than they help in many ways. For example, you could get 100% code coverage for a suite of tests that didn't have single assert.
    – StuperUser
    Feb 28, 2011 at 10:50
1

The smaller the commits the better, usually. This is about SCM tools, not code per-se, but it's a very measurable metric. The smaller the commit the easier it is to see each change as an atomic unit; the easier it is to revert specific changes and pin-point when things broke.

As long as no commit breaks the build...

Improve this answer
1

These are not very useful absolute metrics in terms of progress, but can be used to give a general idea of the state of the code.

Notably Cyclomatic Complexity I have found to be useful in terms of visualizing how modularized a given code base is. You generally want a low complexity as this means that the number of sources per module is low and there are many modules.

Improve this answer
1

I often work on a giant C++ package, and when looking for problematic code worth refactoring the Cyclomatic Complexity or horrible FanIn/FanOut are usually pretty good red flags to look for. Fixing problems there will usually lead to improvements in the whole codebase.

Of course these numbers can only serve as a hint on what would be worth looking at. Making this some hard threshold after which to fail a build or to refuse a commit would be ridiculous.

Improve this answer
1

There are many situations at my work where I use code metrics:

While writing code

The biggest and perhaps most important use in my daily job is in Checkstyle, a tool for java developers which continually checks the metrics (among other things) of my code against a set of rules we've defined and flags places where my code does not comply to those rules. As I develop code, it tells me in real time if my methods become to long, to complex or to coupled allowing me to step back and think about refactoring it to something better.

Developers are completely free to break all the rules since they will never apply to all situations. The "rules" are there to stimulate thought and say "Hey, is this the best way to do this?"

During QA/Code Reviews

The first thing I generally do when I perform a code review is to check the code coverage of the code I am reviewing in conjunction with a code coverage tool which highlights which lines of code have been covered. This gives me a general idea of how thorough the test code is. I don't really care if the coverage is 20% or 100% so long as the important code is well tested. Thus the percent covered is somewhat meaningless, but 0% sure stands out like a sore thumb as something I want to look carefully at.

I also check which metrics agreed by the team have been 'broken', if any, to see if I agree with the developer that it was OK or if I can suggest ways to improve it. Having these development metrics agreed upon in our team for writing new code has made big in-roads into improving our code. We write a lot less monolithic methods and are much better at the single responsibility principle now.

Trending improvements to legacy code We have a lot of legacy code around that we'd like to improve. The metrics at any point in time are fairly useless, but what's important to us is that over time code coverage goes up and things like complexity and coupling go down. Therefore, our metrics are plugged into our Continuous Integration server allowing us to look over time to ensure we are on the right track.

Getting to grips with a new code base About the only time I ever use lines of source code metric is when looking at a code base I'm not familiar with. It allows me to quickly gauge the rough size of the project compared to others I've worked with. Using other metrics I can get a further rough idea of the quality of the project too.

The key things are to use metrics as starting points for trending, discussions or ways forward and not to religiously manage them to exact figures. But I do strongly believe that they can help you improve the code you right when used properly.

Improve this answer
0
0

Q: What are useful metrics to capture for source code?

For business:

A: Number of man-hours

For coder's supervisor:

A: Doesn't matter. Let's do everything today

For coder's self-esteem:

A: Number of SLOC (Source Lines of code)

For coder's mother:

A: Eat more of these soft French rolls and drink tea

continued in comments below...

Improve this answer
-1

Remember: All code can be reduced by at least 1 instruction. All code has at least 1 bug. Therefore, all code can be reduced to a single instruction which does not work. Hope that helps!

Improve this answer
2

Not the answer you're looking for? Browse other questions tagged or ask your own question.