How can you estimate time for tasks which primarily consist of figuring out a problem?

Question

While it is relatively possible for an experienced developer to estimate how long it will take to implement code when the pattern and problem the code is solving is well understood, how can you make a good estimate when, while the end goal is well understood, the implementation is 95% theoretical/problem solving and has very small amounts of implementation?

My work often consists of tasks to accomplish well defined goals, however I have to find the way to reach that goal and until I understand the solution, it i6sn't clear what additional barriers there may be. More specifically, I'm often working on code generation or automated code manipulation tools. Once the solution is fully solved and the tool perfected, it will directly do 95% of the actual changes very quickly. I don't, however, have any way to estimate how many additional issues may have to be resolved to make the generation or analysis tool deal with unforeseeable edge cases.

For planning purposes, my company wants a better idea of how long it will take, but since I don't know how many additional problems may come up while working through solving each step of the solution. I'm not sure how I can approach giving a better estimate.

Answer 1

Before I get too far, let me say that Software Estimation: Demystifying the Black Art is an excellent resource for people looking at and thinking about estimations. Both of the images below are from that book as are the core if the ideas presented following.

As you've noted, estimations are an important part in being able to accurately predict and plan work. Not having estimations makes the business blind about how long something will take. It is not uncommon for business to have a completely mistaken idea on how long things will take - what they think is easy takes six to eight weeks and what is thought to be hard is a friday afternoon hack.

The first thing is to give an estimate. An estimate itself is not a single number - thats a commitment. "How long will ABC take" --> "About 5 days" means that its about 5 days. However, a good estimate is a range where you are 90% confident that you will have it in that range. If you mean to say "I am 90% confident that it will take betwene 1 and 5 days" then say that. Don't work from "I think it will take between 1 and 10 days, so 5 days is probably about average" - thats not an estimate and you'll be wrong 50% of the time.

Well, 50% or more of the time, programmers are notorious underestimators for task times.

Consider the Cone of Uncertainty:

^{_{Image from http://www.construx.com - full article at http://www.construx.com/Thought_Leadership/Books/The_Cone_of_Uncertainty/}}

Realize that the first estimate in that range is 16x. This is akin to saying "I think it will take between an afternoon and two weeks" - but you don't know yet. As you go forward with the design a bit, the range narrows down to 4x. This does not mean that it will take one week, it means that you would instead be saying "after looking at this a bit, it will take between three weeks" - yes, the estimate went up, but also the range of the estimate went down.

With each estimate you give, you need to be 90% sure that the estimate is within that range. You can be wrong - 10% of the time it will fall out of that range.

There are many ways to estimate the size of projects. Comparing it to past projects, using a proxy (I think it would take 1000 lines of code which would take this long to write), using function points (to convert into LOC...), getting estimates from a number of people and then refining it iteratively... some work for some projects, some work for other projects.

A very important chapter in this book that I mentioned at the top is #23 that deals with the politics of estimation and dealing with managers and executives.

The key to an estimation is the iterative process of refining it after working on it a bit.

Giving a too precise of an estimate too early in the process can be very error prone. If you aren't sure about it, give the wide estimate and then come back with another estimate after some period of time for more introspection into the problem and possibly sketching out how you will do it, looking at how much code you wrote it for the last similar problem and other factors that will impact the estimate.

---

Estimates require some thought - don't give off the cuff estimates. These often have huge errors associated with them compared to what it takes when you think about it a bit.

From How to respond when you are asked for an estimate?

What to Say When Asked for an Estimate

You say "I'll get back to you."

You almost always get better results if you slow the process down and spend some time going through the steps we describe in this section. Estimates given at the coffee machine will (like the coffee) come back to haunt you.

From Chapter 4 of Software Estimation:

Note that in this, the estimates after a bit of review are systematically less wild and error prone than the off the cuff estimates. Don't do off the cuff estimates. Sit down and think about the task and estimate it after a bit of thought on it.

Answer 2

Boss: AJ, We have 3 dogs, 2 rabbits, a catapult, and a nun. We need to find a way to get all 7 (yes, the catapult too) over a 20-foot wall and into the lake on the other side without the dogs eating any rabbits, and without drowning the nun. How long will it take you to come up with the solution?

See, the problem estimating how long it will take to solve a problem is that it takes different people a different amount of time. If you have a history of solving similar problems, you can estimate based on how long it has taken you in times before. If you don't, then you're not estimating, you're just guessing.

Furthermore, the problem may not even have an acceptable solution. Or perhaps the solution will require further authorization which could throw off the whole project. Or perhaps the solution changes the whole perceived nature of the problem such that the solution becomes entirely unnecessary.

The moral of the story is, if you don't have enough information to make a reasonable estimation, then don't. Not yet. Get more information. Research more. Typically it's perfectly OK to say, "I'll get back to you in 2 days with some more solid numbers."

When designing a solution for I client, I won't sign a contract till I have enough of the general design complete that I know what the solution will look like and how long the project will take. This means that I'm at risk of having done initial design work that I don't get paid for (if the project doesn't go through), but that's better than being at risk of significantly under-billing for the work being done.

Answer 3

I would suggest you to try something midway between the answers of tylerl and MichaelT with the following:

• split the work to be done in 3 or 4 phases. The phases should be:
  1. Problem analysis
  2. Solution prototyping
  3. Real world solution
  4. Output assessment (test)
• provide an estimate only for phase 1 (analysis) based on your experience, or on phases 1+2 (analysis + prototype) to your management. Then, provide them with an estimate for phases 3+4 when the problem phases 1 and 2 are done (or at least advanced enough so that you can have confidence in your estimate).

The rationale behind is that you know by experience that you need X days to analyze a given codebase (probably depending on its size) and have a suit of basic tools or scripts running (and maybe failing). Then, the number of errors should provide you some information about the actual difficulty of the task at hand.

This may not be exactly what the management wants, but I believe it's always better to come up with estimates that you will actually meet.

Answer 4

As this question is primarily about research types of work, asking software developers is a brave approach, a common metric is that software developer that takes twice as long as their estimate is probably a good developer. However, that said, research (and architecture design) tasks are very much a part of programming, and are often skipped / minimised. They are also often hard to estimate.

The first question I would ask myself, is this a problem that can be solved? This is not an intellect or a brain power issue, but one of practical reality. Unless you are in the world of Google moon shots, where failure is an expected outcome, the hard reality is that I will be expected to deliver this, whatever this turns out to be. A rough rule of thumb seems to be do we already know what 90% of the solution needs to be?

The second question I would ask, what else would be useful to know in thinking about the solution? This is really a way of double checking on do we really know enough to come up with a solution that will be acceptable. It may generate a series of fact finding tasks that help to better define what the solution needs to be, each of which are usually quite easy to define and estimate.

The third question is, who is best suited on the team for this kind of problem? Whoever gets this task will flavour the outcome with their own style. Giving this kind of a problem to a programmer that has 10 million questions at the start of a task, and then goes away and delivers something first time (albeit slowly) may well be a better choice than giving it the programmer that bangs out implementation quickly, but when there's a problem, its only discovered at the end of the process.

Then, the actual task would be to think about possible solutions, implementations and approaches, and to have a fixed time scale in which they need to report back.

When they report back, you then have a choice about getting a wider set of possible solutions, giving the go ahead on implementing a solution, or reflecting, as the solution is still not defined clearly enough

Answer 5

With researchy questions where it's unclear that there is an answer at all, let alone a clear idea of what needs to be done, I usually propose to spend x amount of time on it as a start.

"I have no idea if this is even possible, but I could spend two days researching it. That probably won't give us a solution, but maybe I'll be able to rule out some things and I'll probably have an idea what some concrete next steps could be and what kind of time investment they would mean. Then we can decide whether it makes sense to take another step."

So put the uncertainty in the other direction -- the estimate is nicely precise (I'll spend two days), it's just very unspecified what will be achieved by then.

Timeboxing, basically.