3

As the title says, why do developers (especially, but not only, new developers) habitually underestimate the work involved in 'greenfield' projects or 'total rewrites'?

We all know that software estimation is not a science, but most problems to be solved are not new, and many elementary problems have been getting solved again and again for several decades, so collectively there is a fair amount of accumulated experience of how long things take (certainly to a granularity of man-years or man-decades).

Very often a developer will look at something that has taken a 3-man team 5 years to deliver, and insist that it is a total mess. But that's 15 man-years to do it badly. Redoing it well may take even more, especially if it hopes to achieve even more in terms of complexity or integration, but even if it takes less, one would be braced for it to still take on the order of several years to redo from scratch.

The obvious answer to the question is ignorance, but I'm looking for a more structural explanation for why ignorance in this area of estimation appears so often to crop up. We have academic courses and we have professional forums where developers exchange knowledge and experience.

Why do we fail to systematically reproduce at least gross rules-of-thumb about how much time various kinds of project tend to consume?

12
  • 1
    "most problems to be solved are not new" - maybe, but what happens is that the problem you think you're supposed to be solving turns out not to be the problem that actually needs solving, so if you're not paying attention, you build something based on wrong assumptions, and then end up trying to shoehorn the "changed requirements" into that. And then you have a new problem on your hands, a hell of your own making - which is, in the detail, novel every time: how to put things that are at odds with each other together and handle all the accidental complexity. Which is why agile came to be. – Filip Milovanović Jan 7 at 8:32
  • 1
    Developers are bad at ad hoc estimating. They aren't so bad at structured estimating. There's also the question of scale: they're bad at estimating entire projects, but not so bad estimating for a one-week iteration. Especially if they have the benefit of like experience. Which requires repeatable processes. "We all know that software estimation is not a science" Actually it is :) – John Wu Jan 7 at 8:52
  • 2
    @rwong: "Current university-level CS curriculum teaches too little of software engineering" – To be fair, though, CS and SE are two very different things. I wouldn't expect a CS curriculum to teach SE in-depth, just like I wouldn't expect an SE curriculum to teach CS in-depth. The problem is, of course, that the entire industry as a whole pretends that the two are the same, keeps hiring (sometimes even preferring) CS graduates for SE jobs, keeps telling high school graduates that a CS degree will land them programming jobs, etc. – Jörg W Mittag Jan 7 at 9:55
  • 2
    The amount of ignorance in this question seems to be greater than the amount of ignorance involved in incorrect estimations having to be done in a moments notice. Project Manager pops in the head "Hey, how long will feature X and Y take?" "Can't say now" "Just give me a rough number" "Two weeks at the very least". 1 month later, objective still not completed "You said it'll take 2 weeks". Literally happened. – marstato Jan 7 at 14:14
  • 1
    I've changed the title of your question as it states a false premise, much like the question "have you stopped beating your wife?" – Robert Harvey Jan 7 at 15:38
7

Estimation typically happen at a stage where the project is not specified in detail. The less detail in the specification, the simpler it looks.

When you start specifying in detail, you discover it was much more complex than you first thought. All the special cases, error conditions etc.

This is even the case when rewriting existing software. When looking at a software system from the outside, it looks a lot simpler than it really is. Most complexity is hidden. Unless the existing system have a complete and very detailed specification of its behavior, you will underestimate its complexity.

The solution might seem to be to specify everything in detail up front - the infamous waterfall methods. But this has other problems. Most importantly, the development process is a process of discovery. When the system is presented to the customer, they might realize that it doesn't actually solve their problem. The later this happens, the more costly it is. Therefore agile process uses short iterations with continuous feedback from the product owner. But this also means the project scope is changing all the time. And the initial estimate will be useless.

In addition to these fundamental problems of estimates there are some common pitfalls:

  • Conflating development-time with calendar time. If the estimate is 5 days development, this does not mean the task can be finished in five days! There will be overhead like meetings, installing the development environment, hardware trouble, having to assist with other projects etc. People can be sick, holidays etc.

  • Thinking developers are interchangeable. The person making the estimate might estimate how long would it take for me to solve this task. But this will of course vary from developer to developer depending on experience, familiarity with the system and so on.

  • New framework/platform. The new framework always looks amazing in the proof-of-concept prototype, and seem like it could cut down on development time. But when it is used in a large real-world project, it turns out there are some scenarios it doesn't handle well and require complex workarounds.


most problems to be solved are not new, and many elementary problems have been getting solved again and again for several decades, so collectively there is a fair amount of accumulated experience of how long things take

Solutions to recurring problems becomes standardized in libraries and tools. Using these standard solution saves a lot of time, but this just means the majority of the development time will be spent on the problems that does not have an out-of-box solution available. So estimates does not become easier even though productivity is improved.

3
  • "Estimation typically happen at a stage where the project is not specified in detail. The less detail in the specification, the simpler it looks." - agreed. But then one principle of estimating would be not to estimate at that stage - to lay it down as a cardinal law that an estimate based on a cursory glance is one that will be a worthless under-estimate. – Steve Jan 7 at 19:26
  • 1
    "Thinking developers are interchangeable" - I was just reading this again and I think there is another important nugget of truth here. A lot of developer effort is not spent on code, but on constructing or absorbing a mental model of the problem and searching for some approach that solves it. Even where an analyst does most of that work and records it as effectively as possible, a developer still needs to absorb that understanding, and then go further in order to translate it effectively into code. The labour cost and capital value of such embodied understanding, is quite underappreciated. – Steve Jan 20 at 13:18
  • @Steve: This is a very good point, and indeed underappreciated. – JacquesB May 23 at 18:15
4

What is a piece of software?

No seriously, answer that and we might be able to get around to estimating changes to it.

What most people will tell you is that its the code and digital artefacts used by that code. Fair enough.

But... It needs something to run on. Say an OS, or some other platform.

But... The platform needs more libraries, platforms, frameworks, and a expertly crafted grain of sand.

But... The expertly crafted grain of sand needs many more grains of sand interlinked by wires.

But... The wires need electrical current, so we need a power planet and a power distribution system.

... I could keep on going down, where does the software end exactly?

How about up. That software talks to other things. Like Humans, and robots.

... Those humans need training, and those robots need software to actually be able to use this software.

.. And those humans need organising into businesses, and the like to really guide that software into being useful.

... and we can keep going up. Again where does it end?

Leaky Abstractions

My point here is that each of these levels is an abstraction of sorts, and like all abstractions details leak.

Even if we take the original definition that software is just the code and digital artefacts (as good a definition as any), it has to deal with those leaky abstractions going up, and the leaky abstractions going down.

What leaks in is messy reality:

  • Bugs, Flaws, Glitches, and power outages from below
  • Common Sense Stupidity, Accidents, and Environmental Systems with positive and negative feedback loops running in size and power from tiny, through to large, quickly through to slowly.

System Types

So realising that software is one aspect of a System it makes sense to look over at what sorts of systems we have: S, P, and E

S Systems are our nice well-behaved problems like sorting. We can define the problem, and we have many ways of solving that problem that are well described and researched.

P Systems are ones where its clear what we want to solve, we have good problem definitions, we might even know an algorithm that will eventually solve the problem provided: good inputs, enough time, or a machine actually capable of running it. Weather prediction fits here:

  • We don't have perfect scanners that can detect the state of every atom on earth at once, so we have poor/approximate input.
  • If physics is correct then we have an algorithm that will solve it, its just not runnable on any machine that exists.
  • The algorithms that are runnable are approximations and can't solve the problem, but can give reasonable guesses.

But we have a very simple way to check if the answer was right, we observe the real weather.

E Systems are the worst. Solving the problem, changes the problem. Even partially solving the problem, changes the problem. Its easier to describe this kind of system using an example: Air traffic control.

There are five planes wanting to land. The software tracks this and assigns them runways. queueing two of them up because there are three runways. Except now a plane wants to take off. So it directs it to the third runway. Except on landing the plane crashes blocking runway three. So the plan has to be rerouted back to runway 2 except one of the landing planes is currently using the taxi way. And fire crews need to get to runway 3.So an operator needs to stop the next plane from landing on runway 2 to clear it, to clear the third runway. And we still have to operate landings and takeoffs without runway 3 for a while. Maybe some planes need to be redirected to another airport, or need to be told to not take off from those other airports because they won't be able to land.

See how quickly the problem went from assigning planes to runways, to stopping planes at other airports from taking off.

Now try estimating for that.

Novelty and Estimation

The core problem is this. If something already existed that solved the problem at hand, we would just copy it and be done. We do this everyday with applications like Word. The estimation would be just for the time to locate the solution and copy some files - something language, library, framework, and application authors have been working on for over half a century.

Unfortunately its not always possible to find the solution to the solved problem in a reasonable amount of time (say the time to just make it again). Sometimes when we find the solution something else gets in the way like: a business wanting to keep hold of its trade secrets, a developer who wants to be paid, or the solution requires a no longer obtainable component.

But copying only works for S and P type systems that have been explored and are no longer novel. You can't copy E systems because that means copying the world around them, so every E system is by definition novel.

Novel means we haven't explored it - Which means learning.

Now estimate how much learning you have to do to understand X.

You can't. The best we can do is find some examples that look similar and approximate.

A P System algorithm. Which means we don't have a usable exact algorithm, we only know how to verify the answer.


To boot we developers are also human, and humans are generally speaking optimists. You can go and review the psychology research to verify that.

4
  • "If something already existed that solved the problem at hand, we would just copy it and be done." - actually this doesn't happen, not least because of intellectual property rights. Take e-commerce for example - there is almost certainly existing software conceived for the entire operation, yet the same cake is produced again and again from ingredients. Perhaps because those who have the existing software would not want to share it with other users and lose the competitive advantage they have over the new users. So it is certainly false to say that we simply copy existing systems. – Steve Jan 7 at 19:34
  • 1
    @Steve Yes, your point is a good one. I did have a point in there that we can't always find, or gain access to copy, but my own editing has dropped it out. I'll put that back in. – Kain0_0 Jan 7 at 22:55
  • @steve Also your deduction is false, we do simply copy systems. Its the backbone of the entire software industry. When I want to do desktop publishing I install (aka copy) Word. When I want to browse a website I install (aka copy) Firefox. When I want to operate my computer I install (aka copy) an OS: Mac, Windows, Unix, Linux... If we can't gain access to a solution to copy, then it is as if the solution did not exist, not that we don't copy. If we aren't copying we are learning. Which leads me back to why we can't estimate well. – Kain0_0 Jan 7 at 23:29
  • there is that case, but that is only copying the program as-is, not the source code, nor the business analysis. Obviously, I use that kind of copy all the time in daily life. And someone on here recently was talking about rewriting Excel VBA calculators in a "proper" language - I didn't previously have this example in mind, but one of the points I did make in response to that question was how expensive in time and effort it would be to reproduce even a subset of the Excel UI functionality, which that person did not appear to factor (as Excel has millions of man-hours invested in it). – Steve Jan 8 at 7:17
3

Human nature. People in general suffer from biases and reasoning fallacies. The bigger the project, the more unknown unknowns there are. Combine that with a culture where is not, strictly speaking, acceptable to not give a delivery date (be it for a feature, or project in general) and you end up with a rough estimate. The more unknowns there are, the worse the estimate lives up to reality.

What I'd argue, though, is that the actual problem here is in the definition of "bad code". We tend to give that label to everything we don't understand (immediately or after some effort), but sometimes somehow "bad code" becomes "ok code" when someone explains it. A 100-foot view helps a lot but that is rarely available. Trying to understand how the systems works by only looking at the code is like looking at a chair through a microscope and trying to determine what it is. So a failure to properly estimate a project delivery date reflects the failure to understand the project in the first place.

1
  • There are sound reasons for bad estimates, and you've touched on some of them but came to the wrong conclusions. Yes, there are unknowns. No, that's not because of biases and reasoning fallacies. – Robert Harvey Jan 7 at 15:40
3

They give the estimation they think will allow the project to be started.

Businesses are reluctant to refactor, programmers see a need for refactoring, but know that any estimates not considered "so short that it wont delay anything" will kill any refactoring before it starts.

Or maybe in a less cynical way, you could say they have a bias for being optimistic about projects they see as necessary.

I would add that there is probably a confirmation bias in your assumption. When estimates are too short its a problem, if they are too long you never notice.

7
  • It's interesting to see the contrast between this, and another answer that argues for "trusting developers". Here, the lesson appears to be that developers may have an interest in misrepresenting the estimate so as to achieve their own goals. Those goals may be aligned with the best interests of the business, such as to engage in necessary refactoring, but it seems to me that they could as easily be aligned with other goals, such as the developer's desire to engage in difficult work as a form of professional development for themselves, rather than acting as a maintenance programmer. – Steve Jan 7 at 19:59
  • certainly. but don't forget this applies to everyone. Perhaps even more so to sales guys who are directly incentivised to add features over stability. Trust and openness are the keys to resolving this, but they are rare commodities in most businesses – Ewan Jan 8 at 9:43
  • I definitely agree with that aspect of the problem. I'm just thinking what the picture would look like if developers' honest estimates were taken at face value. Yes, necessary refactoring would get done, but potentially they would also be more free to embark on ruinously difficult work by accident, and (I suggest) would often wrongly estimate the effort of a total rewrite against engaging in difficult maintenance of an existing codebase. That is apart from considering the potential existence of a developer whose interests are not aligned with that of the business. – Steve Jan 8 at 20:09
  • whos opinion is better for the business, the dev who cares about code quality or the sales guy who wants the feature added in time for their monthly bonus calculation? – Ewan Jan 9 at 2:12
  • I'm playing devil's advocate here, but consider the alternative interpretation: whose opinion is better for business? The sales guy whose pay depends on bringing in money to the business now, or the developer on a fixed salary concerned about the aesthetics of code or it's long-term viability? If the business folds in the long term, the director and the sales guy can just set up another phoenix company, starting with a fresh code base. And what does the customer want? Good code next year, or tolerable, usually-working code today? – Steve Jan 9 at 2:33
2

There is always a part of unknown

Even if you could exactly estimate how to do something it's only valid with the information at hand at the time. Often the things we want to do change with time, or we missed some details. Also people are people, not machines. You can't have the same work output every day, all year long and that's not even accounting that people in a team can come and go. Talking about man-days and man-years seems like a good abstraction to size a team and forecast delivery, until you find out that not two people are the same and work the same.

You're not applying a cooking recipe but inventing the recipe

No two software are the same, you can't really say "ok we did this once we can redo it exactly the same way" that's because if it was already done, why would you want to redo it ? If you want to do something it's because it's not already done.

Hierarchical pressure

Project management and managers will always want the lowest cost possible, that's understandable. However they will often pressure developers or bargain for lower estimations in place of trusting developers.

1
  • "However they will often pressure developers or bargain for lower estimations in place of trusting developers." - yes, this is definitely a problem. But I'm thinking of cases where developers themselves, without any management pressure, place unreasonably low estimates on what they are about to embark upon. It's not uncommon for a developer to be arguing for a rewrite, or some significant home-grown system, and (technically proficient) managers are saying "no, no, you're factoring far too little time and effort for that proposal". – Steve Jan 7 at 19:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.