55

I had an unusual, brief conversation with a very senior architect about dynamic and static languages. He said that company data shows that there is evidence for higher productivity when static languages are used. Note, it's a large company with long history. To my (and others) surprise, the metric he used was lines of code added.

He quickly dismissed objections regarding the metric saying that within a same company, with similar culture, line of business and with enough data, differences (as to unique situations and capabilities of individuals) blend enough so that the SLOC metric is useful for comparing productivity of tools and languages.

While I don't think that this claim is backed by rigorious statistical analysis, is there some evidence in the industry that would support this line of thinking?

  • 25
    Productivity is the wrong term. That term is defined as the amount of work accomplished in a period of time, which is unrelated to code produced. – Frank Hileman Nov 20 '17 at 5:18
  • 25
    A wise person said we should consider lines of code not as "built" but as "spent"; in physical engineering when we consider the parts count and length of the BOM, smaller is better. – pjc50 Nov 20 '17 at 11:08
  • 23
    Comparing different languages (no matter if static or dynamic) defeats the assumption of "within a same company, with similar culture, line of business": the differences in the languages make SLOC comparisons meaningless. – rob Nov 20 '17 at 11:51
  • 4
    This method is tragically flawed. Even two different developers in same company using the same development environment will often produce drastically different SLOC to implement the same feature set. – 17 of 26 Nov 20 '17 at 13:48
  • 8
    Using SLOC to measure productivity makes as much sense as using pollution emitted to measure distance traveled when what you should care about is fuel efficiency. The ways in which this is right are still wrong. Use this. – candied_orange Nov 20 '17 at 17:12
66

The argument of the senior architect could mean two things.

  1. It may mean that an average developer in the company produces more lines of code when using static languages than when using dynamic ones. For instance, if fifteen developers work with Java for six months, they will write 100 KLOC, and if the same fifteen developers work with Python for six months, they will write only 50 KLOC.

    There is no correlation between LOC and productivity here. What if it takes four times more lines of code in Java to produce the same feature than in Python? If that is true, using Python would result in twice the productivity, based on the KLOC metrics above.

  2. He may also mean that an average developer in the company produces fewer lines of code when using static languages than when using dynamic ones: fifteen developers would write in six months 100 KLOC in Java, or 200 KLOC in Python.

    While less lines of code is usually better (less code to write, to read and to maintain), it's still unclear how much features did Java developers produce compared to Python ones. Maybe they wrote half lines of code compared to Python developers, but also produced half the number of features?

In both cases, LOC is not a valuable metric, because the same feature wouldn't translate in the same amount of lines of code in different languages. Some languages tend to be more verbose; others—more compact. While in some cases, compactness is valuable, there is no general rule for that. An extreme example would be Brainfuck language which has extreme compactness, but which is not popular for its readability. Comparing even similar languages could be tricky: for instance, when it comes to curly braces, Java follows K&R style, while in C#, the opening curly brace is on its own line in most cases when following the official style, which leads to an artificial increase of LOCs for C#. And what happens when one compares a procedural language with an object-oriented one, or with a functional language?

Instead of using a error prone metric, the senior architect could rely on a group of metrics which do measure productivity when used together: the number of features developed per month, the number of bugs introduced in the code base and the time spent solving those bugs, the evolution of the technical debt, etc. This comparison could be tricky at the beginning, since one has to take in account the unfamiliarity of the team with the new language. Once the team becomes familiar enough with it, the choice should be based on the stable metrics, as well as in most part on the preference of the members of the team themselves.

LOC has a value in some narrow situations. For instance, it could give a hint about the size of the project and parts of the project (and in average correlates with function points, while being often easier to measure), or it could indicate the methods and classes which may need further attention because of their large size. However, LOC should be used with care, since it is misused too often by persons who imagine some correlation between unrelated things. The most humanly disastrous usage of LOCs was in the past the attempt to measure the productivity of an individual developer based on the LOCs written per month.

  • 8
    Yup. The only metric I trust is the number of tickets (features, bugs, research etc.) completed per unit time. It varies by team (different team break down tickets with different granularity) but within the same team or group of teams a culture will emerge to make the ticket sizes fairly accurate (as long as you don't compare them from outside that culture) – slebetman Nov 20 '17 at 2:35
  • 10
    The thing I like more: "Never rely on only one metric" – Chococroc Nov 20 '17 at 8:52
  • 30
    @slebetman I am jealous of the precision/consistency of the person creating your tickets, but I have to resolve issues ranging from "Fix writing of 2 words" to "Add feature X". The metric of tickets is even less useful to me than LOC. Reduced class code by 20 LOC at least gives me an idea of the work done. Resolving 5 tickets can be an hour of work, but could just as well be a week. – R. Schmitz Nov 20 '17 at 10:52
  • 3
    @R.Schmitz This is the same at my company, but each ticket also has a size associated, so summing over ticket sizes would work. – Nico Burns Nov 20 '17 at 12:38
  • 1
    Even trying to use those metrics has issues. What if the added features are complex and hard to implement? Or it might even be a situation where specific features are particularly easy or difficult to implement for a language, but in general the language is easier/harder to work with. The lack of productivity might also be due to current employees not being familiar with a language at first. One shouldn't primarily rely on metrics to determine what language to use. – John Smith Nov 20 '17 at 17:47
26

About productivity and SLOC

The problem with SLOC

The problem with the SLOC metric is that it measures an approximation of the quantity of code written, without taking into account:

  • the quality of the code (i.e. what if for every 100 SLOC you have to add another 90 SLOC because of bugs, but that you don't know at the moment your code is delivered ?)
  • the goals reached with the code (i.e. do the 10K SLOC handle all the expected use cases or user stories ? or only a tiny subset ?)
  • the maintainability of the code (i.e. will you have to add 1% or 50% more code for adjusting the code to expectable evolving requirements ?).

Otherwise stated, the production of error prone unmaintainable spaghetti code with lots of copy-pasted parts will be considered as more productive than carefully engineered reusable code.

So SLOC is definitively not the best way to measure productivity.

What productivity are we considering ?

Productivity is measured for a process. So SLOC could be a perfectly valid indicator for the coding process alone.

If for instance, you misunderstand poor requirements, spend five month to produce the software, show it to the user, discover that it's plain wrong and spend another 5 month to rewrite it for good from the scratch, you would have the same productivity in SLOC/month, that a team writing the code right at the first time, for example because they used an agile process that reduces misunderstandings through frequent feed-back. This apparent equal productivity hides huge problems.

So, measuring software development productivity needs to take into account the whole process, including analysing requirements, designing what to code, coding, testing, debugging, and verifying that user expectations are met. As all these activities are very different, the best thing is to measure the only think that matters: working software, i.e. what the software produced means to the user.

How to measure software deliverables ?

Several approaches exist:

  • The typical approach in classical software engineering is Function Points (FP). Function points are measured based on the requirements to fulfill (e.g. number of forms, number of fields in each forms, etc...). Productivity is then measured in FP per unit of time and per person. Some companies even have data that telling how many function points a developper can produce per unit of time in a given language for a given domain. The problem with FP is that it requires very detailed requirements upfront and it's time consuming.
  • A more modern and pragmatic approach is story points(SP). These are used to evaluate the complexity of the code to be produced, and are routinely used for evaluating velocity of development teams. However, SP is an estimation measure for work performed before all the details are known. It's not a final measure of what actually happened. So some care must be taken when using it as a productivity measure because it could backfire on the estimation process.

About productivity of static vs. dynamic typing

I have to confess that I'm personally a fan of statically typed languages, because in my inner self I know that it's more reliable (years of coding proved me that).

So one thing that I take for sure is that statically typed language are able to prevent much more errors/bugs at compile time (e.g. typos, mismatch in the expected types, etc...) than non statically typed languages. But in all objectivity, I wouldn't dare to abusively generalize this as a higher productivity.

Is your architect right ?

Maybe, maybe not.

But his arguments do not seem valid: the productivity gain of statically typed language comes from a significant number of errors that are caught upfront by the compiler.

Consequently it is not possible to find out this "higher" productivity gain by looking at SLOC alone without looking at the rework required for dynamically typed languages. So his comparison can't be fair.

The argument of comparable circumstances does not hold either. Some dynamically typed languages allow some higher level constructs that require less code than doing the same in one of the classic statically typed languages. So you could need less time, write less code, but add the same analysis, testing and verification overhead. So measuring the productivity by the SLOC would dilute the potential productivity gains, thus creating a bias against dynamically typed language.

Any study to support that claim ?

Several recent academic studies exist on the topic. Although some of them see an advantage of static typing, it's in general limited to a specific purpose (documentation, reuse of poorly documented code or API, etc..). Prudent wording is also used because modern IDE have significantly reduced the risks related to dynamic typing:

  • 3
    Your criticism points were already addressed in the question: “within a same company, with similar culture, line of business and with enough data, differences (as to unique situations and capabilities of individuals) blend enough so that the SLOC metric is useful”. I.e. the argument was that at this scale, all code bases would be of comparable quality. Though personally, I highly doubt that's true. – amon Nov 19 '17 at 22:56
  • We use gitprime (gitprime.com) for concrete measurements, and one of the things it does is to track how many times a developer re-writes the same lines of code. So if you write some code, get a bug report, and re-write the code, it actually measures your efficiency and reports your net productivity. In short, I don't think your comments are inherent problems in using SLoC as a measure of productivity. Rather, I think your complaints are with systems that don't measure SLoC "properly". – Conor Mancone Nov 20 '17 at 17:40
  • 8
    @ConorMancone No one gets paid to write code. They get paid to create solutions. An analogy would be measuring a carpenter by how many nails and boards he uses up. A clown who cuts boards to short and bends up more nails that he drives home will be more productive than a master carpenter by this metric. – JimmyJames Nov 20 '17 at 20:06
  • 1
    @Christophe I've been experimenting with measuring deliverables to production as the main productivity metric. The only tricky part is that some things might be more work than others but from what I can tell, over time, things trend towards pretty (statistically) consistent throughput based on team size and composition. Of course, a lot goes into that so attribution can be a challenge but it's the anchor for any other development productivity measurement. – JimmyJames Nov 20 '17 at 20:13
  • 2
    Years ago, in at least one programming shop some people wrote logic diagrams and other people converted those logic diagrams into compilable code. In essence that shop's compiler had human preprocessors. It would be fair to use SLoC/month to measure the productivity of one of those human preprocessors; that's analogous to how many screws an assembly-line worker can install in the holes where the engineers said they should go. The engineer who specifies 100 screws when 15 are what the job requires is lowering the firm's productivity. (Likewise if they specify 5 screws!) – David K Nov 20 '17 at 21:01
7

Here is a counterexample for your senior architect: Suppose I want to write a hierarchy of three classes, two of which derive from the third, implementing some virtual functions that the base class defines.

If I write these three classes in C++, that's pretty straight forward. I declare the classes, use virtual at the correct places, and be done.

If I write these three classes in C, I'll need to add quite a bit of code: I need to define structs for the v-tables, I need to add a v-table pointer to the base class, I need to add code to the constructors to actually set the v-table pointers, I need to add code the constructors to actually call the base class constructor, I need to add code to perform memory allocation explicitly before calling a constructor (which C++ new does in a single step), likewise, I need to separate destruction from the subsequent free() call, and so on, and so forth.

The point is, all of these additional things are pretty mindless. I can do them very quickly. So, I won't take much longer to write the C version than I need to write the C++ version. Nevertheless, I've produces many more lines of C code than C++ code. So much so, that I'll appear to have been more productive in C in terms of SLOCs.

Any language that requires some amount of boilerplate code will appear more productive in terms of SLOCs than a language that does not require the same amount of boilerplate code.

You see, the SLOC-argument is so fundamentally flawed, that I'd actually see it the other way round: I would take the statement "programmers tend to produce more SLOCs in static languages" to mean: "static languages appear to require more boilerplate code, and thus reduce productivity".

  • 1
    I like your last sentence. – Peter A. Schneider Nov 20 '17 at 13:27
  • 1
    "static languages appear to require more boilerplate code, and thus reduce productivity": This again shows how flawed the SLOC metric is. The final number of lines does not consider (1) how many times one needed to rewrite the code before getting the final solution (2) how many additional lines of code in the form of unit tests are required (dynamically typed languages require on average more unit tests to have a comparable confidence in the correctness of the production code). The SLOC metric is definitely flawed. – Giorgio Nov 21 '17 at 7:09
6

I'll be the contrarian.

We track SLoC at our job (although we don't use it directly in staffing decisions), and I've had people argue what most people are saying in their answers. In effect, "LoC doesn't matter because X technology lets us do more with less code" or "Better developers write better, shorter code, and so they don't write more than anyone else". In my experience (although I don't have any hard numbers to back these things up), these objections are simply not correct. In my own time I've seen a clear correlation in both the rate and quality of code production for our developers, when compared with all other meaningful measurements of their overall "competency" as an engineer. To give some counter-examples to the sorts of arguments made above:

  1. Yes, some languages can do more with less code. In fact, we've got an entire framework that we have built that "automates" large portions of the development for our particular business problems (back-end only). The result of all of this is not that people write less code, but simply that we have more time to write code. As a result, in our company, the overall rate of code writing is fairly constant across technologies and depends primarily upon the competency level of the engineer.
  2. The idea that a better developer will produce less code because they are writing smarter is definitely not true. Yes, a better designed program might take up less lines of code. However, I have personally found that the "better" developers writing more efficient code don't take any longer to plan it out than a more junior developer writing stuff the long way. As a result, the more senior developer will get through their coding tasks quicker, and move on to writing different code at the same high rate.

That lat part is my overall summary, BTW. What I have found is that regardless of the tech stack or kind of project, most developers have their own pace, which is the pace that they operate at. If a language has a lot of features that makes the developers code more effective, then that is a big boon for the business, but that doesn't mean that they will write less code as a result. Instead, they get features done quicker, and quickly move on to new code. Again, the end result is that they rate at which they code depends primarily on their skill, and less on their tech stack. In fact, because of this, I would generally expect the tech stack to make more of a difference on the rate at which tickets and features get developed than the rate at which people code. So I would probably be included to side more with your senior architect than the other answers here.

That being said, neither code-writing-rate nor ticket-closing-rate is a perfect measure of productivity, which is why we don't directly make staffing decisions on the basis of SLoC. Instead it is a part of the process, and employee evaluations are done using as many data points as possible. I would say though that your Architect certainly isn't crazy.

One exception

The one exception I agree with is the possibility of boiler-plate code. If there is a lot of copy-and-paste going on from one class (or whatever) to another to get it up and running, then that is obviously going to skew the metrics. This is also true if you have tooling that can auto-generate large amounts of code for you. However, I think that these will often be the exception rather than the rule. If your developers spend any amount of time copying around boiler-plate code to get started, then you are using the wrong tech set. If they are actually writing the code, even if it is fairly repetitive, then I expect this to skew any measurements only a small amount: when writing code, most of the time we're limited by how fast we can think through the problem rather than how fast we can type. Even when writing relatively repetitive code, you still have your brain engaged and are thinking through the process.

Obviously, everything above is based on my own personal experience. Your mileage may vary, and obviously I'm in the minority. Feel free to disagree. In summary though:

I find that rate of coding depends more on how fast you can think through your problems then anything else. As a result, I have found that coding rate is a decent measure of productivity, even across tech sets, with only a few possible exceptions.

  • 4
    There's also one other exception: bug hunting. Bug hunting for especially nasty bugs can take a long time, but usually result in a single line of code change. – Nathan Merrill Nov 20 '17 at 17:14
  • @NathanMerrill That's a good point to, although less relevant to the OP: debugging is debugging in all languages and (off the top of my head), I see no reason why it would be substantially easier or harder from one techstack to another. That being said, that is a reason why, overall, you can't judge productivity exclusively on code written, anymore than you can on any other metric. – Conor Mancone Nov 20 '17 at 17:26
  • We use gitprime (gitprime.com) in our company, and as both a manager and engineer, I think it's about the best thing in the world. Again, it's only part of the picture for us, but it has been extremely helpful in identifying potential problems with engineers long before there was an actual problem. The transparency is awesome, and everything they do ultimately boils down to SLoC. Given the amount of value and insight it adds, I'm always very dubious about the tendency of some engineers to dismiss SLoC out-of-hand. Anyone is welcome to their opinion but it definitely works – Conor Mancone Nov 20 '17 at 17:37
  • The question is asking if LoC can be used to compare tools and languages, in the context of the senior developer saying it shows higher productivity in "static" languages. You seem to be answering a different question - LoC can be used to compare developers, but you still agree it cannot be used to compare languages because a given developer writes the same number of LoC regardless of tool/language? You say you are contrarian against the other answers here, but it seems you are in agreement? – TessellatingHeckler Nov 20 '17 at 19:52
  • As a developer I can think of many times I took a bunch of non-DRY code and replaced it with a small set of reusable functionality. I then added a significant amount of new functionality. Reducing the amount of code while adding a multiple of real value is a good thing in my book. In my experience, the best engineers write the least lines of code and the worst write the most. – JimmyJames Nov 20 '17 at 20:19
6

Though I’m jumping on the bandwagon. I think the impact on the behavior of programmers needs to be highlighted.

Using SLOC as a measure for productive has a toxic effect on programmer morale. The moment any engineer in your team/company realize they are measured on SLOC several things happen:

  1. They start writing much longer code to do the same function
  2. they will care less about the quality of their code
  3. they will stop doing other things that help your team(recruiting, debugging, helping juniors)
  4. they will hate their work and likely leave

I can’t stress strongly enough how corrosive it is to engineer morale as I’ve seen it happen twice at 2 different companies. Whatever seemingly valid use cases you have for it, I argue that its unlikely to be worth the impact on your team/company even if there is only a small chance that its use will be discovered. Even though there in some cases maybe a correlation between number of lines written and amount of useful features it encourages all the wrong behaviors in your programmers and sends the message the quality isn’t important.

  • Indeed... any metric that disincentives someone from removing redundant code ("you've had a negative SLoC metric this week!" is wrong, plain wrong! – Andrew Nov 1 '18 at 11:18
1

It is generally not considered a valid way to measure productivity. Smaller code is usually better than larger code, so a more productive developer usually produces less code. Productivity takes its biggest hit in debugging; efficient developers spend little time debugging.

Statically typed languages are more productive (if you control for all other differences between languages), because when used wisely, they reduce debugging time, catching errors in the compile phase, where they are faster to fix.

  • 1
    This could be a valid point if we were comparing the productivity of individual developers. The question is however about the comparison between languages, so the context is very different. This also means, for instance, that smaller code is not better or worse than larger code; compare LOC of code written in Brainfuck with code written in, say, Ruby. – Arseni Mourzenko Nov 19 '17 at 23:06
  • 1
    @ArseniMourzenko Aside from jokes like Brainfuck, well designed languages are actually compared on the basis of the amount of code needed to solve a task. Usually such a comparison is called expressiveness. It is true though, I was talking about LOC within a single language, not across languages. Productivity is generally defined as how long it takes to perform a task; that is not specific to programming. – Frank Hileman Nov 20 '17 at 5:16
0

The only metric that you can use to compare productivity for developers between languages is a metric that doesn't compare the code between the languages. Some languages are notoriously verbose (COBOL for the legacy win), and others require several steps to do something you can do in one line of code (assembly vs. just about everything else). Even if you compare only active lines of code (i.e. don't count declarations and only count code that has some action involved) you can still skew your results.

You might be able to make an argument for rates of change. I.e. lines of code added, comparing the slope of productivity over the same time period. However, that doesn't account for negative changes in lines of code. For example, you inherit a project that has copy and paste code everywhere. You perform some quick and easy refactorings to reduce the number of repeated code blocks--by definition you have a negative slope.

In all seriousness, comparing the productivity of teams/languages is meaningless because there are so many additional factors that affect a team's productivity you can't draw meaningful conclusions from it.

I worked on a project where the infrastructure was very fragile and the tools were outdated. The project was built on Java with a Single Page App slapped on to it, but hosted in a portlet container for no apparent benefit. The time it took to do even simple changes was ridiculously long. If you were to base all your conclusions on that particular project you might conclude that Java was bad, or Single Page Apps were bad. Neither are true. The system that the ugly project was supposed to replace was built on C# and WebForms. When we made the business case to extend the existing application to handle the customers needs, our productivity skyrocketed. Does that mean that a tightly coupled WebForms app is superior? You can only make that conclusion for this particular case and it doesn't extend to the world at large. And it only makes sense because there was an existing application with sufficient maturity to extend.

Even comparing rates of resolving items in the issue tracking system is flawed in the sense that you are comparing complete project infrastructures against each other. The libraries and frameworks used can either speed up or slow down progress. You might be in the start up phase with very little inertia to overcome, where the project you are "better than" is in a maintenance phase where the number of new tickets is relatively low. It's never a case of comparing like things.

protected by gnat Nov 21 '17 at 4:51

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.