Are there known valid uses of SLOC to measure productivity?

Question

I had an unusual, brief conversation with a very senior architect about dynamic and static languages. He said that company data shows that there is evidence for higher productivity when static languages are used. Note, it's a large company with long history. To my (and others) surprise, the metric he used was lines of code added.

He quickly dismissed objections regarding the metric saying that within a same company, with similar culture, line of business and with enough data, differences (as to unique situations and capabilities of individuals) blend enough so that the SLOC metric is useful for comparing productivity of tools and languages.

While I don't think that this claim is backed by rigorious statistical analysis, is there some evidence in the industry that would support this line of thinking?

Answer 1

The argument of the senior architect could mean two things.

1. It may mean that an average developer in the company produces more lines of code when using static languages than when using dynamic ones. For instance, if fifteen developers work with Java for six months, they will write 100 KLOC, and if the same fifteen developers work with Python for six months, they will write only 50 KLOC.

   There is no correlation between LOC and productivity here. What if it takes four times more lines of code in Java to produce the same feature than in Python? If that is true, using Python would result in twice the productivity, based on the KLOC metrics above.

2. He may also mean that an average developer in the company produces fewer lines of code when using static languages than when using dynamic ones: fifteen developers would write in six months 100 KLOC in Java, or 200 KLOC in Python.

   While less lines of code is usually better (less code to write, to read and to maintain), it's still unclear how much features did Java developers produce compared to Python ones. Maybe they wrote half lines of code compared to Python developers, but also produced half the number of features?

In both cases, LOC is not a valuable metric, because the same feature wouldn't translate in the same amount of lines of code in different languages. Some languages tend to be more verbose; others—more compact. While in some cases, compactness is valuable, there is no general rule for that. An extreme example would be Brainfuck language which has extreme compactness, but which is not popular for its readability. Comparing even similar languages could be tricky: for instance, when it comes to curly braces, Java follows K&R style, while in C#, the opening curly brace is on its own line in most cases when following the official style, which leads to an artificial increase of LOCs for C#. And what happens when one compares a procedural language with an object-oriented one, or with a functional language?

Instead of using a error prone metric, the senior architect could rely on a group of metrics which do measure productivity when used together: the number of features developed per month, the number of bugs introduced in the code base and the time spent solving those bugs, the evolution of the technical debt, etc. This comparison could be tricky at the beginning, since one has to take in account the unfamiliarity of the team with the new language. Once the team becomes familiar enough with it, the choice should be based on the stable metrics, as well as in most part on the preference of the members of the team themselves.

LOC has a value in some narrow situations. For instance, it could give a hint about the size of the project and parts of the project (and in average correlates with function points, while being often easier to measure), or it could indicate the methods and classes which may need further attention because of their large size. However, LOC should be used with care, since it is misused too often by persons who imagine some correlation between unrelated things. The most humanly disastrous usage of LOCs was in the past the attempt to measure the productivity of an individual developer based on the LOCs written per month.

Answer 2

About productivity and SLOC

The problem with SLOC

The problem with the SLOC metric is that it measures an approximation of the quantity of code written, without taking into account:

• the quality of the code (i.e. what if for every 100 SLOC you have to add another 90 SLOC because of bugs, but that you don't know at the moment your code is delivered ?)
• the goals reached with the code (i.e. do the 10K SLOC handle all the expected use cases or user stories ? or only a tiny subset ?)
• the maintainability of the code (i.e. will you have to add 1% or 50% more code for adjusting the code to expectable evolving requirements ?).

Otherwise stated, the production of error prone unmaintainable spaghetti code with lots of copy-pasted parts will be considered as more productive than carefully engineered reusable code.

So SLOC is definitively not the best way to measure productivity.

What productivity are we considering ?

Productivity is measured for a process. So SLOC could be a perfectly valid indicator for the coding process alone.

If for instance, you misunderstand poor requirements, spend five month to produce the software, show it to the user, discover that it's plain wrong and spend another 5 month to rewrite it for good from the scratch, you would have the same productivity in SLOC/month, that a team writing the code right at the first time, for example because they used an agile process that reduces misunderstandings through frequent feed-back. This apparent equal productivity hides huge problems.

So, measuring software development productivity needs to take into account the whole process, including analysing requirements, designing what to code, coding, testing, debugging, and verifying that user expectations are met. As all these activities are very different, the best thing is to measure the only think that matters: working software, i.e. what the software produced means to the user.

How to measure software deliverables ?

Several approaches exist:

• The typical approach in classical software engineering is Function Points (FP). Function points are measured based on the requirements to fulfill (e.g. number of forms, number of fields in each forms, etc...). Productivity is then measured in FP per unit of time and per person. Some companies even have data that telling how many function points a developper can produce per unit of time in a given language for a given domain. The problem with FP is that it requires very detailed requirements upfront and it's time consuming.
• A more modern and pragmatic approach is story points(SP). These are used to evaluate the complexity of the code to be produced, and are routinely used for evaluating velocity of development teams. However, SP is an estimation measure for work performed before all the details are known. It's not a final measure of what actually happened. So some care must be taken when using it as a productivity measure because it could backfire on the estimation process.

---

About productivity of static vs. dynamic typing

I have to confess that I'm personally a fan of statically typed languages, because in my inner self I know that it's more reliable (years of coding proved me that).

So one thing that I take for sure is that statically typed language are able to prevent much more errors/bugs at compile time (e.g. typos, mismatch in the expected types, etc...) than non statically typed languages. But in all objectivity, I wouldn't dare to abusively generalize this as a higher productivity.

Is your architect right ?

Maybe, maybe not.

But his arguments do not seem valid: the productivity gain of statically typed language comes from a significant number of errors that are caught upfront by the compiler.

Consequently it is not possible to find out this "higher" productivity gain by looking at SLOC alone without looking at the rework required for dynamically typed languages. So his comparison can't be fair.

The argument of comparable circumstances does not hold either. Some dynamically typed languages allow some higher level constructs that require less code than doing the same in one of the classic statically typed languages. So you could need less time, write less code, but add the same analysis, testing and verification overhead. So measuring the productivity by the SLOC would dilute the potential productivity gains, thus creating a bias against dynamically typed language.

Any study to support that claim ?

Several recent academic studies exist on the topic. Although some of them see an advantage of static typing, it's in general limited to a specific purpose (documentation, reuse of poorly documented code or API, etc..). Prudent wording is also used because modern IDE have significantly reduced the risks related to dynamic typing:

• An empirical comparison of static and dynamic type systems on API usage in the presence of an IDE: Java vs. groovy with eclipse, 2014
• Considering Productivity Effects of Explicit Type Declarations, 2014, but non conclusive
• An empirical study of the influence of static type systems on the usability of undocumented software

Answer 3

Here is a counterexample for your senior architect: Suppose I want to write a hierarchy of three classes, two of which derive from the third, implementing some virtual functions that the base class defines.

If I write these three classes in C++, that's pretty straight forward. I declare the classes, use virtual at the correct places, and be done.

If I write these three classes in C, I'll need to add quite a bit of code: I need to define structs for the v-tables, I need to add a v-table pointer to the base class, I need to add code to the constructors to actually set the v-table pointers, I need to add code the constructors to actually call the base class constructor, I need to add code to perform memory allocation explicitly before calling a constructor (which C++ new does in a single step), likewise, I need to separate destruction from the subsequent free() call, and so on, and so forth.

The point is, all of these additional things are pretty mindless. I can do them very quickly. So, I won't take much longer to write the C version than I need to write the C++ version. Nevertheless, I've produces many more lines of C code than C++ code. So much so, that I'll appear to have been more productive in C in terms of SLOCs.

Any language that requires some amount of boilerplate code will appear more productive in terms of SLOCs than a language that does not require the same amount of boilerplate code.

You see, the SLOC-argument is so fundamentally flawed, that I'd actually see it the other way round: I would take the statement "programmers tend to produce more SLOCs in static languages" to mean: "static languages appear to require more boilerplate code, and thus reduce productivity".

Answer 4

I'll be the contrarian.

We track SLoC at our job (although we don't use it directly in staffing decisions), and I've had people argue what most people are saying in their answers. In effect, "LoC doesn't matter because X technology lets us do more with less code" or "Better developers write better, shorter code, and so they don't write more than anyone else". In my experience (although I don't have any hard numbers to back these things up), these objections are simply not correct. In my own time I've seen a clear correlation in both the rate and quality of code production for our developers, when compared with all other meaningful measurements of their overall "competency" as an engineer. To give some counter-examples to the sorts of arguments made above:

1. Yes, some languages can do more with less code. In fact, we've got an entire framework that we have built that "automates" large portions of the development for our particular business problems (back-end only). The result of all of this is not that people write less code, but simply that we have more time to write code. As a result, in our company, the overall rate of code writing is fairly constant across technologies and depends primarily upon the competency level of the engineer.
2. The idea that a better developer will produce less code because they are writing smarter is definitely not true. Yes, a better designed program might take up less lines of code. However, I have personally found that the "better" developers writing more efficient code don't take any longer to plan it out than a more junior developer writing stuff the long way. As a result, the more senior developer will get through their coding tasks quicker, and move on to writing different code at the same high rate.

That lat part is my overall summary, BTW. What I have found is that regardless of the tech stack or kind of project, most developers have their own pace, which is the pace that they operate at. If a language has a lot of features that makes the developers code more effective, then that is a big boon for the business, but that doesn't mean that they will write less code as a result. Instead, they get features done quicker, and quickly move on to new code. Again, the end result is that they rate at which they code depends primarily on their skill, and less on their tech stack. In fact, because of this, I would generally expect the tech stack to make more of a difference on the rate at which tickets and features get developed than the rate at which people code. So I would probably be included to side more with your senior architect than the other answers here.

That being said, neither code-writing-rate nor ticket-closing-rate is a perfect measure of productivity, which is why we don't directly make staffing decisions on the basis of SLoC. Instead it is a part of the process, and employee evaluations are done using as many data points as possible. I would say though that your Architect certainly isn't crazy.

One exception

The one exception I agree with is the possibility of boiler-plate code. If there is a lot of copy-and-paste going on from one class (or whatever) to another to get it up and running, then that is obviously going to skew the metrics. This is also true if you have tooling that can auto-generate large amounts of code for you. However, I think that these will often be the exception rather than the rule. If your developers spend any amount of time copying around boiler-plate code to get started, then you are using the wrong tech set. If they are actually writing the code, even if it is fairly repetitive, then I expect this to skew any measurements only a small amount: when writing code, most of the time we're limited by how fast we can think through the problem rather than how fast we can type. Even when writing relatively repetitive code, you still have your brain engaged and are thinking through the process.

Obviously, everything above is based on my own personal experience. Your mileage may vary, and obviously I'm in the minority. Feel free to disagree. In summary though:

I find that rate of coding depends more on how fast you can think through your problems then anything else. As a result, I have found that coding rate is a decent measure of productivity, even across tech sets, with only a few possible exceptions.

Answer 5

Though I’m jumping on the bandwagon. I think the impact on the behavior of programmers needs to be highlighted.

Using SLOC as a measure for productive has a toxic effect on programmer morale. The moment any engineer in your team/company realize they are measured on SLOC several things happen:

1. They start writing much longer code to do the same function
2. they will care less about the quality of their code
3. they will stop doing other things that help your team(recruiting, debugging, helping juniors)
4. they will hate their work and likely leave

I can’t stress strongly enough how corrosive it is to engineer morale as I’ve seen it happen twice at 2 different companies. Whatever seemingly valid use cases you have for it, I argue that its unlikely to be worth the impact on your team/company even if there is only a small chance that its use will be discovered. Even though there in some cases maybe a correlation between number of lines written and amount of useful features it encourages all the wrong behaviors in your programmers and sends the message the quality isn’t important.

Answer 6

It is generally not considered a valid way to measure productivity. Smaller code is usually better than larger code, so a more productive developer usually produces less code. Productivity takes its biggest hit in debugging; efficient developers spend little time debugging.

Statically typed languages are more productive (if you control for all other differences between languages), because when used wisely, they reduce debugging time, catching errors in the compile phase, where they are faster to fix.

Answer 7

The only metric that you can use to compare productivity for developers between languages is a metric that doesn't compare the code between the languages. Some languages are notoriously verbose (COBOL for the legacy win), and others require several steps to do something you can do in one line of code (assembly vs. just about everything else). Even if you compare only active lines of code (i.e. don't count declarations and only count code that has some action involved) you can still skew your results.

You might be able to make an argument for rates of change. I.e. lines of code added, comparing the slope of productivity over the same time period. However, that doesn't account for negative changes in lines of code. For example, you inherit a project that has copy and paste code everywhere. You perform some quick and easy refactorings to reduce the number of repeated code blocks--by definition you have a negative slope.

In all seriousness, comparing the productivity of teams/languages is meaningless because there are so many additional factors that affect a team's productivity you can't draw meaningful conclusions from it.

I worked on a project where the infrastructure was very fragile and the tools were outdated. The project was built on Java with a Single Page App slapped on to it, but hosted in a portlet container for no apparent benefit. The time it took to do even simple changes was ridiculously long. If you were to base all your conclusions on that particular project you might conclude that Java was bad, or Single Page Apps were bad. Neither are true. The system that the ugly project was supposed to replace was built on C# and WebForms. When we made the business case to extend the existing application to handle the customers needs, our productivity skyrocketed. Does that mean that a tightly coupled WebForms app is superior? You can only make that conclusion for this particular case and it doesn't extend to the world at large. And it only makes sense because there was an existing application with sufficient maturity to extend.

Even comparing rates of resolving items in the issue tracking system is flawed in the sense that you are comparing complete project infrastructures against each other. The libraries and frameworks used can either speed up or slow down progress. You might be in the start up phase with very little inertia to overcome, where the project you are "better than" is in a maintenance phase where the number of new tickets is relatively low. It's never a case of comparing like things.