Why is software schedule estimation so hard? [duplicate]

Question

Possible Duplicate:
Why are software schedules so hard to define?

I often need to explain to senior management why software estimation is so hard, and why our preliminary estimates are often so far out. I suspect some want to know why it is not mathematically precise engineering, like building a bridge.

You can help me by listing a few dot points relevant to this subject. Many thanks!

Answer 1

Software estimation isn't actually more difficult than estimating other types of work. It just seems so because the CONDITIONS under which it's estimated are more difficult. Say a software company was tasked with something similar to what a car company is tasked with. Build the same thing over and over again, which has existed for decades, and with only minor variations. Furthermore, you will work from a complete and detailed spec from the beginning of the project and it will be frozen once development begins. Under those circumstances software estimation would be cake.

Answer 2

I'm stealing this answer from somewhere on SO. I can't seem to find the original author but the answer stuck with me.

Pointy Haired Manager: how long will this take?
Employee: hard to say, a month? maybe a month and a half?
PHM: We need a better estimate
E: Look, how long does it take you to drive to work?
PHM: Huh? 30 minutes why?
E: 30 minutes plus or minus what?
PHM: +- 5 minutes depending on traffic
E: So you can estimate a task you have done hundreds of times before with a single bounded unknown variable to within 17% and when I estimate how long a task will take that is so complex as to need to hire a [certified-professional/contractor/whatever-qualification-you-have], to do something that's never been done before, that has thousands of unknowns, to better than 20%, you say that's not good enough!?

PMH: Oh...

Answer 3

it's because the last 20% of the project usually take 80% of the time. The initial outline is probably because people often base the total time based on the core mechanics, without realizing the finer details of the project. It's less of a programming question and more of a psychological question in my opinion. You ask a person how a helicopter or a lock works and they think of the maybe the one or two key features leaving out the majority of the finer mechanics. Programming is similar, since its a good chance the programmer is doing something new he probably doesn't have the exact finer understanding of the system will go together, just the core components that will make it work.

Also, with programming rarely does something work at a first go, debug takes a long time and you rarely know the problems you will run into (or you wouldn't run into them :P)

some links that may help: http://www.apa.org/monitor/feb03/overestimate.aspx

there is also this: http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect but I don't completely agree with it since programming is a fairly creative task

EDIT: just as a side not this study in psychology is a fairly interesting and new, it has a lot to do with motivation and depression, where our entire culture (western) views themselves from the perspective of the future and tend over estimate our abilities, (USA although rated fairly low in education standards is rated the highest in confidence, so even though we're performing poorly we think were performing amazingly). There's also this great quote: "Socialism never took root in America because the poor see themselves not as an exploited proletariat but as temporarily embarrassed millionaires." John Steinbeck. Anyways, sorry for the tangent, but this perspective can have a variance in time estimate between different cultures (east vs west).

Answer 4

Run, do not walk, to get Software Estimation: Demystifying the Black Art by Steve McConnell. The first few chapters are full of answers to the exact question that you are asking.

If you come away with nothing else, come away with an appreciation of Tom DeMarco's observation that in most organizations the definition of an estimate is, the most optimistic prediction that has a non-zero chance of coming true. One of the biggest drivers of this is the desire to tell people what they want to hear. Furthermore many managers attempt to use schedule as a motivator. (It has the opposite effect, but that doesn't stop managers from trying.)

As for your presentation, the key lessons that I think you should aim to have the managers understand are as follows:

1. To produce accurate estimates you need to
   1. Break the project down into all necessary tasks of just a few hours. Larger tasks than that cannot be accurately estimated.
   2. Build into your estimates overhead for inevitable delays due to people being sick, etc.
   3. Have past data on how your organizations estimates and reality have correlated to correct those estimates.
   4. Accept that feature requests will invalidate prior estimates.
   5. The larger the project, the easier it is to miss stuff and so the worse estimates become. Typically estimates are off by over 100%.
   6. Any project of a type that you do not have organizational experience with is harder to estimate because you don't know what the likely pitfalls are.
2. Estimates serve several possible purposes. You need to be clear on what you are using the estimates for. Here are the major possibilities.
   1. Figuring out when a particular feature is likely to be done.
   2. Aiming to have something to show for a given date (eg so you can announce it at a conference).
   3. As a tool to motivate developers to work harder. (Note, estimates are very often used this way, and this is usually counter-productive.)
3. Getting accurate estimates for a running project is notoriously difficult. In most organizations, the news that the project is not on track arrives about 2 weeks before the due date, no matter how long the overrun is. Why?
   1. People always want to give the most optimistic spin on reality as they know it.
   2. People always want to hear what they have been told in the most optimistic possible way.
   3. After a few layers, there is a complete disconnect between reality and what is being communicated. How Shit Happens is an amusing and surprisingly realistic demonstration of how quickly this disconnect develops.
   4. Nobody wants to bear the bad news that the schedule will not be met. This results in a state of collective denial until denial cannot possibly work.
4. Things can be done to improve tracking projects.
   1. Whenever possible, do multiple, small, projects rather than single big ones. Smaller projects are inherently easier to keep track of.
   2. Have "100% complete" milestones with clear, unambiguous definitions. An example of what not to do is, "Coding 90% complete." People often think they are 90% done when they are only halfway through. An example of what to do is, "Module X has been written and passes its unit tests."
   3. Use an iterative development process. Each iteration should have concrete, measurable deliverables. As iterations finish, compare how long it took with your initial plan and adjust accordingly. Assume that discrepancies between estimation and measurement will be proportional.. So if 3 iterations were supposed to take 6 weeks and take 8 weeks, don't assume that you can "catch up". Instead immediately announce that you are on track to finishing the final project in 1/3 more time than originally estimated.
   4. Have a constantly updated list of "top risks" to your project, and proactively address them. This reduces the odds of your being blindsided later in the process.

Answer 5

Because you have potentially infinite variables to estimate against and you have users that don't really know what they want (or can't accurately express what they want).

"Design a car for me". You design a car, but they really meant a truck. Well, it DOES have wheels, right...

When we estimate we try to get the first estimate within 25% and the second one, after a detailed design, within 10%.

We don't even call it an estimate right out of requirements... we call it a SWAG (silly wild ass guess).

Answer 6

The problem is that customers don't know what they want. They only have a vague idea.
However to find out, you must start building a software, which you will then change.

What you would want software development to be like is Analysis > Design > Implementation.
This is what for example construction works like. Nobody would dare to say 'ok, just start building and we'll give you feedback as the construction progresses', because you can't just decide half way through, that you want your 60 floor building to be a cylinder instead of a box.

Thus software development is Analysis > Design > Implementation > Evaluation > Redesign > Reimplementation > Evaluation > .... If you don't know, where you're are going, it's hard to say, when you'll be there.

Answer 7

Building a bridge isn't so precise. In fact, I would bet they usually run over budget as well. I know a planned tunnel b/w NJ and NY just recently got canceled b/c of overruns. And think of all the (American) football stadiums built recently that all go over original estimates. Just grab some of those examples, and drive home that building things we can actually see, and that are pretty similar to other things we can see, is still dicey. How much more when its not really similar, and something that can't be seen?

Answer 8

Construx has a really good explanation of this phenomenon that they call The Cone of Uncertainty. Essentially the business problem is that estimates and often commitments have to be made earlier in the process than it's possible to know the outcome for large projects. There are various techniques to help deal with this, which is why the industry has moved to smaller deliverables with shorter iterations (a.k.a., "Agile") in order to more effectively manage risk.

Answer 9

Software schedule is hard but not impossible. Give enough time to understand the requirement and break the whole system into small parts, a schedule can be concluded.

What makes a schedule unrealistic is usually due to:

• requirement not confirmed
• certain technical barrier are not fully studied
• lack of experience on similar project type

In order to overcome the issues, a project manage should:

• make sure the requirements are final, or at least let upper management to understand any change will incur addition cost (time / resource)
• technically capable of foresee the complexity of a software
• don't guess, break down and break down, until both the full scope and smaller parts are fully studied

If you have ever read a tender of a bridge-building project, you'll see why software estimations (usually) are so unrealistic.

Answer 10

There are a number of factors which apply to software estimation which often don't apply to estimating other projects:

• Custom (new) requirements.
• Changing requirements.
• Unclear project scope.
• New tools and techniques.
• Lack of consistent methodology.
• Interchanging skilled resources into areas outside their expertise.
• High variance in productivity by individuals.
• Starting the project without required tools.

While most projects will have some degree of these factors, software projects often have a high degree of many of these factors.

Most organizations require more time to do anything which is custom. Those that can handle custom requirements easily still limit the customization within a limited scope. Customizations outside the acceptable scope will significantly increase time required to do the customization.

Changing requirements require rework. In some cases work already done needs to be undone. Consider the cost of rearranging rooms in a house at various points in construction.

Project scope may be unclear or changing. Changes in scope will significantly alter the cost. If I ask you to estimate the cost of a truck you might suggest 20 to 30 thousand dollars. What happens to the the estimate if what I want is a semi truck and trailer.

Software tools, languages, and techniques are still evolving rapidly. The development team will likely have to learn as they go. This learning curve will alter the speed at which work is done. Some estimates suggest it takes 10,000 hours to become fully proficient. Estimates for Python suggest that it takes months to become fully proficient.

Many projects have little in the way of methodology. Even where this is a methodology, it may be poorly implemented or changing. As a result the team may be continuously learning how to do things. This will reduce performance. Recent reports indicate that using any methodology will improve productivity.

Software developers have skill sets which are not interchangeable. Many organizations consider one developer is a the same as another. Treating developers as interchangeable will result in developers working on task outside their skill set with resulting performance variations. Consider the impact on building a house if all you have are plumber and electricians.

Research on programmer productivity shows some programmers may be as much as 26 times more productive as others. This level of productivity variance can have a significant impact on estimates.

Many projects start without some of the basic tools required. Both hardware and software are often required to get a project started. Selection and setup of these tools may require skills not available with the project team. Organizations with a standard set of tools, and project setup processes are likely to see less variance in estimates.

Answer 11

It's because some of our assumptions on how software estimation works are flawed. For example, the assumption that if you break down the software into tasks and then estimate the tasks you can add up the total and there's your project estimate. There are two major problems with this assumption.

The first glaring one, is that all tasks are known ahead of time, which is patently false. Anyone who tells you that they can know all tasks ahead of time is lying.

Let's say we can accommodate for this, let's say by using some kind of stories known metric and expand our estimate. Now we're faced with another fallacy. That is the assumption that all story estimates will even out in the end. We'll get roughly the same amount that were under as were over and by a similar amount. However, that's not what typically happens.

First of all, a lot of the time, you get Parkinson's law taking effect. Second of all, even if you estimate in points or some other nebulous non day estimate to counteract Parkinson's law. There are only so many ways you can come in under the estimate. However, there is almost no limit to the ways you can come in over.

Think of it this way. Let's say your daily commute takes 30 minutes. If you hit all the lights, or transit lines up just right, you might be able to shave off 5 maybe 10 minutes from that commute. However, if there's an accident, or a bus breaks down, your commute's now double or maybe triple what the average is.

Answer 12

We predict the future by looking at the past. So I think there are two main problems with estimation:

1. If we’re asked to do something we’ve never done before, we can’t know how long it’ll take, because we’ve never done it before.
2. If we don’t keep a record of how long things took, even when we’re asked to do something similar again, we won’t have the data upon which to estimate.
3. It's hard to produce and maintain accurate lists of all the tasks that need to be done to implement a feature request.

You can’t control what people ask you to do, but you can control how you respond to their requests.

Honest and clear communication is really important. Although it’s hard, especially if someone’s pushing for a precise figure, you must stick to an estimate you believe in. So, if someone’s given you a very vague request (e.g. “we need a brochureware website, roughly how long will it take?”), give an estimate with a wide range, based on actual past projects — e.g. “Between 5 and 25 man-days, depending on design complexity and size.”

‘Rapid Development’ by Steve McConnell is really good on this.

Answer 13

There are many methods to estimate the size of software projects. Converting the size to effort depends on some kind of productivity factor. Also you may have to adjust for certain technical complexity factors like you need high maintainability or high scalability. You may also need to adjust for environmental factors like the experience of the developers in the technology or the process or even their motivation. So as you see there are many variables that need experience to correctly chose the values that will give you more accurate estimates. So estimation relies somehow on the experience of the estimator let alone if the estimator does not use any method and relies only on his experience. This is the main problem. We should rely more on documented historical data from various projects in the organization. Measures should be collected regularly from projects and stored in organizational repository to support estimation of later projects.

Another problem is that we tend to think only about coding time, we sometimes forget or under-estimate other activities like analysis, design , testing, configuration, and so on.

Answer 14

Software schedule estimation isn't really very hard at all once you have some experience and can base your best guesses on tasks that appear to be similar to something you have done before. So if (as I'm suggesting) it isn't hard to estimate project schedules, why does it seem so difficult?

I'll start by answering the second part of the question first. Why isn't writing software more like bridge building? It's because of the level of input by each of the stake holders, and because of the perception by all involved at the ease with which software can change mid project. Change the design of a bridge, and you need to factor in significant material costs and the scheduling chaos that it can create organizing many individuals with a number of different skill sets. Software by comparison seems like it should be easier because all of the software developers are meant to be similarly skilled, and there isn't usually much in the way of materials or scheduling that can't be generally overcome (IE: delay is relatively cheap compared to delayed construction projects). The reality however is that software is all about dealing with people, and creating solutions to problems for people. When the project is so people-focused, it becomes less precise, and more affected by the problems that occur to and between people.

I believe it comes down to several factors, and the most obvious ones that come to my mind at the moment are as follows:

• The developer doesn't fully understand the problem domain.

  You can blame the customer all you want for "not knowing their own mind", but the real underlying problem is that the customer is generally and genuinely not able to put into words what it is that they NEED. Wanting and needing are two very different things, and it is up to a skilled developer to draw out the customers needs, rather than merely responding to what the customer initially states as wants. When you are able to talk to the customer in their own language and have nailed what it is that the customer actually needs, then you are able to claim to fully understand the problem domain.

• Limited input from team members of varying levels of experience.

  Let's face facts. Nobody works well when driven by someone else's estimates. Every developer's experience is somewhat unique or at the very least different, and each will view a particular task from a different personal experience. If you only have one person estimating the entire project, they will only really get about 25-40% of their estimates correct, and it will be the difficult stuff that they have either no experience with, or which has been misunderstood that will end up causing the most delays. You can bet your auntie that the greatest delays will come from something a single estimator thought of as relatively trivial. This is where group estimation is very important. You get all of the people who will actually work on the project to provide all of the estimates (A planning poker session can really help here), and basically use this process to effectively hedge your bets against possible scheduling failures in the future.

• Developer hasn't tuned their Software Development Method

  There has been a lot of stuff written about software development methodology over the last 20 years. While I can't claim to have read everything, I can like most experienced people out there claim to have sampled a bit of everything, and have tried much of it. One of the reasons that I have become quite avid about being Agile is because of the emphasis it places on tuning your processes, and about being adaptable in both how you apply your practices, and in your thinking. Sometimes the method needs to be tuned to match the needs of the customer, but more often the greatest method tuning needs to be with the team itself. Whenever a new team member joins, it is an opportunity to tweak your processes, and when the team doesn't gel, you need to tune even further. Most serious failures happen because of teams that work ineffectively as a unit, and where the method doesn't (or can't) account for the unique situation of a given team. Everything from personal issues, to office politics, fatigue, experience, all of it can have an effect, but if the process doesn't make sense to the team, if it makes them feel uninspired or if it doesn't make the team feel in control or empowered by their efforts, then it's going to fail the team, and the team will end up failing the project. Processes that help team members feel like they are achieving good results in small manageable stages, and where team members feel rewarded by the effort itself, are processes that will ultimately deliver the best results to the customer.

• Plans are derailed by the most unexpected things

  As any military planner will tell you, you can have the best strategy all worked out with every minute detail planned, but once you start the battle the plan will be shot all to hell. Sure, you can load your estimates with likely delays, such as people getting sick, planning their holidays, or having that massive trade fair pop up right in the middle of your planned development period, but you can't always plan for things like massive incompetence, your entire server farm going down, your entire development team simultaneously contracting the Bengal Flu, or your suppliers suddenly going bankrupt. Things can, and most certainly will go wrong, and finger pointing won't help anybody. You can however plan to handle the unforeseen by negotiating alternative release schedules, and by ensuring that every release to the customer provides them with working software that delivers business value with every release.

So by this definition, it's systemic failures that will derail projects, and not always problems with scheduling. Scheduling itself is relatively easy compared to all the other stuff that a team needs to get right in order to deliver a successful outcome.

Answer 15

pfff its easy.

build 1 billion bridges and you will know how long it takes to build one.

build 1 billion software , and you will know how long it takes to build one.

a year ago i did a small research on the subject, and found out an article that says, that for mega projects (those who cost more then half a billion dollars) most of them fail , and the half that deliver are over cost and over due.

its a miracle that anything happens in this world.

here is a link to a nice presentation

http://www.google.com/url?sa=t&source=web&cd=1&ved=0CBcQFjAA&url=http%3A%2F%2Fwww.pmisac.com%2Fdata%2Fuploads%2FPresentations%2F2008_05_28_CostOverrunsOnMegaProjects.ppt&rct=j&q=mega%20project%20fail&ei=3h2KTbzyII3P4gaJ7tmjDg&usg=AFQjCNHHDLmJ_vFdfpR2jPQWOp0qCTNjxw&sig2=Ng2oat_bnTErpU5riw7RkQ&cad=rja