How do I predict the time it will take to complete a task?

Question

There are three scenarios where I am asked to predict how long it will take:

1. I know I can solve the problem without further consultation of anything (that is, there is no unknown in the problem. I already have all the knowledge to solve the problem)
2. I know the problem can be solved but I need to do a lot of experimentation to see how things work (for example, you may know a car tire is easy to change, but if you have never changed it, you may run into issues that you will avoid/correct the next time around, thus making your prediction of how long it will take, better... yes, this I know is one of the answers to my question here... experience)
3. I am told the problem can be solved, but I have no idea where to begin so I start breaking it down into small pieces and go to Step 2 (where I know it can be solved, but needs experimentation)

I am generally bad at predicting how long something will take even if I have done something similar before. I always run into a snag I did not foresee. My question is mostly directed towards Point 1 and somewhat toward Point 2 (in Point 3, I know my predictions will always be wrong). I know that it requires experience, which will allow me to do good predictions but there has to be some basic rules I can follow to at least start making somewhat accurate predictions. Right now I feel like I will never be able to make good predictions, no matter how much experience I gain. One rule that I have heard is something along the lines of predict how long it will take, then multiply it by 2, then add two more weeks (more of a joke... or maybe not?)

Now I know that predicting how long a task will take is a general problem in the industry, but people still predict to some extent and can't always be wrong (specifically on smaller tasks). In my case, if somebody tells me How long do you think a linked list with these features will take? I will give a wrong prediction even though I know how to make one without consulting any body/book.

So my request is to SO (especially veterans in the industry) is to give me (us) pointers on how to get better at predicting how long a task will take.

Background

I am currently a Masters student working at a company. I am very young in terms of industry experience and experience in general. I am reading lots of books on better programming, design etc. (most of them recommended here at SO) and my knowledge of programming is ever increasing thanks to work/school and SO; but of course, there is literally tons more that I need to absorb.

Answer 1

I've had the most success with Evidence-Based Scheduling.

Long story short, you split up the task, write down how long you think it'll take to complete each task (including interruptions like coffee breaks, bosses nagging you, StarCraft, etc.), and then time how long it actually takes you to complete some of those tasks. It's important to be blunt, and to include all your distractions -- because that's part of life as a developer, and it's a realistic factor that influences the time it takes to finish your work.

Then you can either multiply the average estimated-to-actual time against the estimated time of the remaining tasks, or you can take the error factor of your estimations to run a Monte Carlo simulation and determine the probability that you'll be done your tasks on any given date.

The end result is a graph that looks like the first one in that link, where the X axis is the finish date and the Y axis is the probability that it's finished:

(Taken from joelonsoftware.com)

Answer 2

. Learn your own error factor. If your feeling tells you “2 hours“ but it takes six, next time tripple your official estimate.

. Always expect that interruptions etc. will take away a part of your worktime, but nobody lets you charge another project. Include a markup for that in your estimate.

. Remember that the gap between “done“ and “really done“ can be huge. Cleanup, debugging for corner cases, proper error handling etc. can take a lot of time and must be included in your estimate.

Answer 3

The answer requires another information : did YOU already did what's asked before?

1. if yes, then estimate on the real time you needed to do it before;
2. if no, you can't

Here, 2. means that, if you can, you should prototype before being able to give an estimate. If you can't prototype (because you're not allowed to spend your time prototyping for example) then you just can't accurately estimate.

Note also that even with 1., you rarely need to write something again with the same software environnement or programming language, meaning that even 1. estimation will be wrong. But it's certainly the best you can do.

Also, read the chapter of "The Pragmatic Programmer" on the subject.

Answer 4

One of the best mechanisms I've found -- although it does involve a lot of work! -- is building up a database of your past performance. One option there is CMU Software Engineering Institute's Personal Software Process. PSP is usually found in a bundle with the Team Software Process; PSP is the individual part, TSP the whole.

PSP involves excruciatingly detailed tracking of the time you spend designing, coding, compiling, and testing (all separately), as well as any defects you encounter (even just compiler errors and warnings). It seems to slightly favor indivisible design / code / compile / test phases, versus rapid iteration through three or four of these -- although it's somewhat adaptable over time, it's worth going with the spirit of the thing at least initially for a month or two.

PSP and TSP are usually taught by certified instructors (who cost a pretty penny to hire, I might mention -- lucky enough to work at a company that paid for the training despite being small). Since you're a student you can get a lot of the material for free through Self-Study PSP Material.

The Software Process Dashboard is some free software that supports the process as a whole.

People at this company who used it reported definite improvements in both their ability to estimate and their ability to code without creating defects (either compiler errors or runtime bugs).

That having been said, the mental overhead was high enough that no-one is using it anymore. Some of the principles have definitely carried forward, though -- I'd still be using it if I wasn't more of a "director" than a programmer these days.