What are useful metrics to capture for source code?
How can metrics, like for example (Executable?) Lines of Code or Cyclomatic Complexity help with quality assurance or how are they beneficial in general for the software development process?
Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.Sign up to join this community
Take a look on Jeff's posts on the subject:
There is an old, but good, post from Joel too, closely related to software metrics, and I strongly recommend its reading: The Econ 101 Management Method
The key point, for me, is this, quoting Jeff: "Responsible use of the metrics is just as important as collecting them in the first place."
What confuses me about code metrics is that it isn't done more. Most companies report on the efficiency of their employees, suppliers, and systems in place, but nobody seems to want to report on code. I will definitely agree with answers that state that more lines of code is a liability but what your code does is more important.
Lines Of Code: As 'Ive mentioned this is a vital measurement and should be taken the most seriously, but on each level. Functions, classes, files and interfaces can indicate do-everything code that is hard to maintain and costly in the long term. It's infinitely hard to compare the total lines of code versus what a system does. It could be something that does many things and in that case there will be many lines of code!
Complexity: This measurement is good to do on code bases you haven't worked on, and can give you a good indication of where problem areas lie. As a useful anecdote I measured complexity on one of my own code bases, and the highest complexity area was the one that I was spending the most time when I needed to change it. Working towards reducing the complexity resulted in a massive reduction in maintenance time. If management had these measurements at hand they could plan refactoring iterations or redesigns of specific areas of a system.
Code duplication: This is a very important measurement as far as I'm concerned. Code duplication is a very bad sign and could point to either deep problems in low levels of a system's design or developers that are copy pasting, causing massive problems in the long term and systems that are unmaintainable.
Dependency Graphs Finding bad dependencies and circular dependencies are an important measurement in code. This almost always points to an incorrect high level design that needs revising. Sometimes one dependency can suck in a whole lot of unnecessary other ones, because someone is using addNumber inside an e-mail library to do their finance calculations. Everyone is shocked when the e-mail library is changed and finance breaks. If everything is dependent on one thing it can also point to do-everything libraries that are hard to maintain and badly designed.
A good measurement will always tell you that every feature of a system has a small footprint. Less dependencies, less complexities, less duplication. This points to loose coupling and high cohesion.
Won't this "source code metrics" crap EVER die?
Raw source lines of code (SLOC) is the oldest, easiest, most basic metric there is.
Halstead originally proposed a whole bunch of metrics. Lots of people were having lots of fun writing measurement programs until some spoilsport did the obvious study, and demonstrated that each and every single Halstead metric was strongly directly correlated with SLOC.
At that point, Halstead's metrics were abandoned, because SLOC is always easier to measure.
Source code metrics for quality assurance aim at two objectives:
Both lead to writing code as simple as possible. This means:
Metrics are only useful if you know what to do with the answers you get. In essence a software metric is like a thermometer. The fact that you measure something at 98.6° F doesn't mean anything until you know what the normal temperature is. The above temperature is good for body temperature but really bad for ice cream.
Common metrics that can be useful are:
The first two measure trends. Are you finding bugs faster than you can fix them? Two possible outcomes: maybe we need more resources fixing bugs, maybe we need to stop implementing new features until we catch up. The second two provide a picture of how close you are to being done. Agile teams call it a "burn down" chart.
Cyclomatic Complexity is an interesting metric. At it's base concept it's the number of unique execution paths in a function/method. In a unit-test heavy environment this corresponds to the number of tests needed to verify every execution path. Nevertheless, just because you have a method that has a Cyclomatic Complexity of 96 doesn't mean it is necessarily buggy code--or that you have to write 96 tests to provide reasonable confidence. It's not uncommon for generated code (through WPF or parser generators) to create something this complex. It can provide a rough idea of the level of effort needed to debug a method.
Every measurement you take needs to have the following defined or it is useless:
The metrics you take may vary widely from project to project. You may have a few metrics that you use accross projects, but the definition of "normal" will be different. For example, if one project discovered an average of 5 bugs/week and the new project is discovering 10 bugs/week it doesn't necessarily mean something is wrong. It just might be the testing team is more meticulous this time around. Also, the definition of "normal" may change over the life of the project.
The metric is just a thermometer, what you do with it is up to you.
To the best of my knowledge, the number of bugs found directly correlates with lines of code (probably churn), modulo language, programmer, and domain.
I don't know of any other straightforward and practical metric well-correlated with bugs.
One thing I'd like to do is start running the numbers for different projects I'm on - Test Coverage :: kLOC, and then discuss "perceived quality" to see if there is a correlation.
Source code is a liability, not an asset. With that in mind, measuring lines of code is analogous to tracking dollars spent while building a house. It needs to be done if you want to stay under budget, but you wouldn't necessarily think that spending $1000 a day is better than spending $50 a day; you'd want to know how much of the house got built for that money. It's the same with lines of code in a software project.
In short, there are no useful metrics for source code because measuring source code by itself isn't useful.
Since source code is simply a combination of sequence, selection, and repetition. If I were to describe the most optimal piece of software that we could ever reasonably expect to produce it would be as follows. Software with nearly 100% testing code coverage using the least amount of lines of code necessary to do the job and yet flexible enough to withstand changes.
An anecdote to show why KLOC counts are useless (and even harmful) to gauge performance.
Years ago I worked on a large project (70+ people in our company, another 30+ at our customer) which used KLOC counts as the sole measure of performance of teams and individuals.
For our Y2K effort (tells you how long ago it was :) ) we did a large cleanup of the section of the code my team was responsible for. We ended up for the release writing about 30.000 lines of code, not a bad 3 months of work for 5 people. We also ended up scrapping another 70.000 lines of code, a very good job for 3 months of work, especially combined with the new code.
End result for the quarter: -40.000 lines of code. During the performance review following the quarter we got an official reprimand from the company for failing to meet our productivity requirements of 20.000 lines of code produced per quarter (after all, the tools had shows we'd produced -40.000 lines of code), which would have resulted in all of us being listed as underperforming and bypassed for promotions, training, pay increase, etc. etc. had not the project manager and QA team intervened and gotten the reprimand overturned and replaced by a commendation.
A few months later (such things take time) we were told the company was reviewing their productivity standards and had hired a team of experts to create a new system based on function point analysis.
I'm surprised no-one's mentioned Statement/Decision Coverage of Unit tests (percentage of code exercised by unit tests) yet.
Code coverage is useful in that you know what percentage of the application doesn't fail catastrophrically; with the rest of its usefulness depends on the quality of the unit tests.
The smaller the commits the better, usually. This is about SCM tools, not code per-se, but it's a very measurable metric. The smaller the commit the easier it is to see each change as an atomic unit; the easier it is to revert specific changes and pin-point when things broke.
As long as no commit breaks the build...
These are not very useful absolute metrics in terms of progress, but can be used to give a general idea of the state of the code.
Notably Cyclomatic Complexity I have found to be useful in terms of visualizing how modularized a given code base is. You generally want a low complexity as this means that the number of sources per module is low and there are many modules.
I often work on a giant C++ package, and when looking for problematic code worth refactoring the Cyclomatic Complexity or horrible FanIn/FanOut are usually pretty good red flags to look for. Fixing problems there will usually lead to improvements in the whole codebase.
Of course these numbers can only serve as a hint on what would be worth looking at. Making this some hard threshold after which to fail a build or to refuse a commit would be ridiculous.
There are many situations at my work where I use code metrics:
While writing code
The biggest and perhaps most important use in my daily job is in Checkstyle, a tool for java developers which continually checks the metrics (among other things) of my code against a set of rules we've defined and flags places where my code does not comply to those rules. As I develop code, it tells me in real time if my methods become to long, to complex or to coupled allowing me to step back and think about refactoring it to something better.
Developers are completely free to break all the rules since they will never apply to all situations. The "rules" are there to stimulate thought and say "Hey, is this the best way to do this?"
During QA/Code Reviews
The first thing I generally do when I perform a code review is to check the code coverage of the code I am reviewing in conjunction with a code coverage tool which highlights which lines of code have been covered. This gives me a general idea of how thorough the test code is. I don't really care if the coverage is 20% or 100% so long as the important code is well tested. Thus the percent covered is somewhat meaningless, but 0% sure stands out like a sore thumb as something I want to look carefully at.
I also check which metrics agreed by the team have been 'broken', if any, to see if I agree with the developer that it was OK or if I can suggest ways to improve it. Having these development metrics agreed upon in our team for writing new code has made big in-roads into improving our code. We write a lot less monolithic methods and are much better at the single responsibility principle now.
Trending improvements to legacy code We have a lot of legacy code around that we'd like to improve. The metrics at any point in time are fairly useless, but what's important to us is that over time code coverage goes up and things like complexity and coupling go down. Therefore, our metrics are plugged into our Continuous Integration server allowing us to look over time to ensure we are on the right track.
Getting to grips with a new code base About the only time I ever use lines of source code metric is when looking at a code base I'm not familiar with. It allows me to quickly gauge the rough size of the project compared to others I've worked with. Using other metrics I can get a further rough idea of the quality of the project too.
The key things are to use metrics as starting points for trending, discussions or ways forward and not to religiously manage them to exact figures. But I do strongly believe that they can help you improve the code you right when used properly.
Q: What are useful metrics to capture for source code?
A: Number of man-hours
For coder's supervisor:
A: Doesn't matter. Let's do everything today
For coder's self-esteem:
A: Number of SLOC (Source Lines of code)
For coder's mother:
A: Eat more of these soft French rolls and drink tea
continued in comments below...