What can I do to get better at estimating how long projects are going to take? [duplicate]

Question

I don't want to make life hard for management. I really don't. They're nice enough guys, but every time I am assigned a new project or task and get asked "how long do you think it will take to do this" I end up sputtering off ridiculous timeframes; "between a day and three weeks".

I would like to believe that it's not entirely my fault: I am the company's sole programmer, I am relatively new to proper development (Was it only six months ago that I wrote my first unit test? sigh...), and I am working with a code base that at times is downright absurd.

So I would like some advice. Obviously, experience is the biggest thing I lack but anything that would make me better would be greatly appreciated. I am looking for reading material, methodologies, maybe even actual tools. Any way that I can give my boss more accurate information without having to sit down and design the darn thing first is appreciated.

Ok magic stackoverflow genie, what have you got for me?

EDIT:

@Vaibhav and others suggesting I take time to research and sketch out the system

I agree with you in principle, but how do you balance that with real-world constraints? When you're a one man show or even a part of a small team "I will need 2 days to build an estimate" is a real deterrent when you can fight 4 fires in the time it takes to get a simple estimate.

Answer 1

You need to ask for some time before you give an answer back to management. Go back and study the task:

• Write down an overall approach
• Maybe do some exploration of the code for a few hours
• Break down the task in steps in an excel (I would suggest Microsoft Project, but that may be an overkill)
• Against each task try to assign your best guess of estimate
• Add them up and add a 20-30 percent buffer and give to Management

Over time you will get better.

Update:

Well, it doesn't have to be 2 days. You basically don't give an answer right away. There is no way you can know (except for the most trivial tasks) how much effort it will take right away. If it's a large task (maybe 3-4 weeks) it will take you a couple of days to estimate it, but if its a task which is a 3-4 days, then it will only take you 3-4 hours to break it up and estimate.

Answer 2

This question has a long answer and a slightly less longer answer (but no short answer).

The long answer is 255 pages (and some appendices) long and is called "Software Estimation: Demystifying the Black Art" by Steve McConnell. It comes highly recommended.

The slightly less longer answer is actually just a set of heuristics/tips:

1. Break the task down into small items - preferably 1-3 days each
2. Try to find a similar task - in the same domain or involving similar technologies. It's almost never the case that you're estimating a task that you've done before (because it's already done...). Example: If you're task is "enhance SharePoint in way X",maybe you've done a task that involved integrating with SharePoint and/or another Office product and/or another "Content management system"
3. Give your estimation as a range rather than a single point estimation
4. Understand your margins - if you're estimating something completely new you might need to take a 100% or even 400% factor (i.e. if your estimation is 2 days and the factor is 400%, it could take between .5 a day to 8 days). The more uncertainty, the higher the factor.
5. Understand that as you progress on the task, you need to revisit and refine your estimation
6. Communicate, communicate, communicate! Make sure management is aware of how reliable a specific estimate is (or isn't).
7. Give estimates in a scales and numbers that reflect the accuracy (e.g. estimating in hours implies more accuracy than giving the same estimate in days; Estimating it would take 3.75 days causes people to automatically assume your estimate is more accurate than if an estimate of 4 would have been given)
8. Acknowledge the "Cone of Uncertainty" (see more details below)
9. When giving an estimate, let's say "3 days", think of the following question: "Can it take less than 3 days?". If the answer is No, then this is an optimistic estimate, since it would take 3 days if and only if nothing goes wrong...

Several more notes:

• Inaccuracies can go both ways, either up or down (although software projects are notorious for under-estimates rather than over-estimates)
• Note that the effects of underestimating are usually more detrimental to the project than overestimating (this is also discussed in the first part of the aforementioned book)

---

The Cone of Uncertainty:

The idea of "the cone" is that you start a project with high-level/low-accuracy estimates and your confidence and accuracy increase as you make progress on the project. For example:

• At the time of requirement gathering your estimates are much less accurate than after the high-level design is ready
• Which are far less accurate than after the detailed design was done
• Which is less accurate than halfway through coding.

This is a fact of life, and instead of fighting it, this understanding should be built into your plan. This is mostly relevant when estimating a whole project or a big task (i.e. longer than 2 weeks), since splitting it into fine-grained items like I suggested in item 1 is not really feasible (or cost-effective).

Answer 3

Try evidence-based scheduling.

Over the last year or so at Fog Creek we’ve been developing a system that’s so easy even our grouchiest developers are willing to go along with it. And as far as we can tell, it produces extremely reliable schedules. It’s called Evidence-Based Scheduling, or EBS. You gather evidence, mostly from historical timesheet data, that you feed back into your schedules. What you get is not just one ship date: you get a confidence distribution curve, showing the probability that you will ship on any given date. It looks like this:

The steeper the curve, the more confident you are that the ship date is real.

Here’s how you do it...

Answer 4

You can try reading these two books, they are both great (I read them more than once)

• Software Estimation
• Rapid Development

And they deal also with the human aspect in software construction (which seems may be useful to you)

Answer 5

estimate quickly, re-estimate regularly and most importantly keep management updated.

It's all about managing expectations, so make sure they understand the limitations of your estimate, this is key, especially for non technical project managers.

Answer 6

Notwithstanding his caveat that it is now obsolete, Joel's article on Painless Software Schedules is extremely helpful if you are new to the concept.

So, you have to make a schedule. This is something almost no programmer wants to do. In my experience, the vast majority just try to get away with not making a schedule at all. Of the few that make a schedule, most are only doing it because their boss made them do it, halfheartedly, and nobody actually believes the schedule except for upper management, which simultaneously believes that "no software project is ever on time" and in the existence of UFOs...

Here's a simple, painless way to make schedules that are actually correct...

Answer 7

This may sound glib, but it actually does help. I have had a couple of internal customers over the years who would burst into my office with a new idea. "How long would it take to write a new feature for our big program that does [extremely vague description]?" I found that asking for more details was often futile. "Just give me a rough estimate," he or she would demand.

So I would reach into my desk drawer, and pull out a six-sided die. I would shake it up in my hand and toss it on the desk. "Four weeks," I would pronounce.

"C'mon, be serious," the customer would complain.

At that point, I would explain that without at least a coherent idea of how this new feature would work, I had absolutely no idea, and rolling the die was as good as any other method of estimating. Ultimately, they all got the hint. Most ideas went away to die. With the rest, my customer would come back with something I could work with.

In a nutshell, what I was doing was illustrating the absurdity of trying to come up with an estimate with virtually no information. A picture is worth a thousand words, as they say.

Answer 8

There is some pretty good written material out there on the topic of estimation. Steve McConnell's books (previously mentioned here) are excellent, and I would also recommend Mike Cohn's Agile Estimating and Planning.

I studied this book before I planned and estimated a large project with my team 18 months ago, and it was tremendously helpful. The approaches and techniques in the book are not just for large projects, though - they are useful for any size of task.

Answer 9

Estimating is hard. Working with NO estimates is a pipe-dream. So it's sort of a necessary evil.

• Start by making it clear that initial estimates are way off so if you come to a X Unit estimate, give a range 0.6X - 1.6X.
• Then there is the trick of doubling the estimate (however saying a ~2 year will take 4 years might be a problem)
• (There's one more involving scaling the magniture and bumping to the next higher unit. Need to refer to my book for this one.)

Next comes the part of how do I reach that estimate so that I can double/multiply? No easy answer.

• Historical trends make sense if you're doing the same kind of project with the same level of team skills.
• Next is the 'Let me steer the ship for 2-3 iterations' and then I'll get back with a better estimate.
• Finally comes the "Tell me how much it will cost.. Till then no one works on this." In this case, you have no guardian angel. Sit and break it down and then apply piecemeal guesses till you have a cumulative estimate. Then do the double-n-bump.

In my 5 years of living in the trenches.. The only thing I've seen is 'guessing by the seat of your pants' estimation... No FP.. No process.. helps in real-world constraints. Still things can improve.. for that I'd suggest getting Mike Cohn's Book 'Agile Estimating and Planning' - that's the book that is closest to us sinful programmers and reality. The rest are preachers on their high horses... Caper Jones which planet is your abode?

Nice question.. I keep asking this when every project begins and then once every week i'm doing the GRIND. Still no answer from the voices in my head.

Answer 10

Also, since it sounds like you don't work a purely software company, keep in mind that nowhere near 100% of developer's time is spent on coding. Make sure you put in time for planning, documentation, research, testing. These are usually the steps that managment would like to cut out of the process, but they won't be happy with the results if they do. If you get any backlash you should remind them, that A) You will end up spending time on those items anyway, whether they are budgeted are not B) You will end spending more time on maintaining and fixing bugs without proper planning. If you haven't read "The Mythical Man Month" by Frederick Brooks, I would hightly recommend it. It has a lot of good concepts and you'll probably relate more than a little to some of the scenarios. Some of the items are a little dated, but there's a lot of good info and you should be able to get one on the cheap.

Answer 11

Don't estimate. Instead, use a project management technique that's not reliant on estimate. Scrum, for example, stresses doing work rather than talking about doing work (ie estimates).

If you spend a day or two or 5 or whatever estimating, wouldn't you be better off having coded or designed or tested on those days, instead of spending the time "estimating"?

Answer 12

Remember Hofstadter's Law:

It always takes longer than you expect, even when you take into account Hofstadter's Law.

It's funny and all, but I actually usually make reasonable estimates, then put some (at least) 10% on top of it and then round it all up. In occasions I was not allowed to do it, I regret to say I ended up failing miserably in keeping up to schedule.

Answer 13

Double your initial estimation.

Seriously though, try to break up the task into smaller, estimate-able bits. And then double it.

Answer 14

Mostly it's just a matter of practice.

Here are some major things to remember when estimating:

1. Use some method of keeping track of how long you thought it was going to take vs. how long it actually took.
2. Make sure to break your tasks down into the smallest possible steps before coming up with your estimate.
3. Don't forget to include testing/bug-fixing time in your estimate.
4. Don't let people force you into artificial schedules.

The Joel article that harpo mentions is a great expansion of these ideas.

Answer 15

I don't think there is a universal step-by-step prescription.

Every project is different. If you know your code, and you know your processes, then you know enough to do a serious estimate. Just think through all the steps, all the components, and be realistic. Don't be overly optimistic. Build in contingencies for all the places where there is potential risk. It is far better to be accurate than to be fast.

Like anything else: you get better at estimating by doing it a lot. The more you force yourself (or are forced) to make estimates, the better you'll get at doing it.

Answer 16

To get better at making estimates, you need to compare your previous estimates against the actual time it took and then look for patterns in the difference.

As others have suggested, the first step is to break the project down as far as you can so you're working with manageable discrete tasks. Record your estimates for each task and then, as you do the work, record the actual time it takes (including the time for all the bits you forgot about when making the estimate - they're often where the biggest problems lie).

At the end of the project, compare your two sets of figures to see where you went wrong (because, even after years of experience, we all go wrong somewhere). You can then look to improve in two ways:

• Try to spot where you're consistently underestimating or forgetting tasks and then take that learning forward to your next estimate - ie try to improve the accuracy with which you estimate each task. The more you work on similar projects each time, the easier this is to do.

• Also look for an overall pattern in the difference between the total estimate and actual time taken. If you're consistently taking, say, 50% longer than you estimated for each project, you have a guideline for increasing your future estimates. So even if the accuracy of your individual task estimates doesn't improve, you should have a more realistic overall esimate.

You can benefit from the first approach after one project - the second approach becomes more effective as you have more figures to compare.

I also find the second approach to be more helpful if you need to estimate the time other developers will take. I find it hard to estimate on behalf of another developer in a team, but I know that certain people consistently take more or less time than me on particular types of task. So rather than try to work out how long they'll take, I estimate how long I'd take and then apply an overall multiplier based on previous projects. (Ideally, of course, I'd get them to make their own estimates, but this isn't always possible.)

Answer 17

Ask your boss what decisions s/he is going to make based on the estimate you give.

Estimates are decision support. What most people really want are better decisions.

Maybe we can get that by improving our estimates (and there have been a lot of good suggestions given) but, let's face it, estimates are never going to be as accurate as the average PM wants, nor as low as the average salesman wants.

All estimates are ranges and probabilities (even when we're forced to give a single number), and you can provide a lot better decision support if you know what the estimate is going to be used for.

There's a lot more to be said about this, but I wanted to get the perspective out there as no one else has mentioned it.

Answer 18

George, I've spent most of my career as a one man team, so I know what you are talking about.

Somebody suggested you get MS Project, if you happen to have a lot of experience with Project, that might be ok. But if not, IMHO it will just make things more complicated.

My first real insight into estimates came from Painless Software Schedules by Joel Spolsky. Joel now suggests you use Evidence Based Scheduling instead, which I believe is in thier bug tracking software FogBugz. I haven't used it, but I'm sure it's a hell of a lot better than what I'm about to tell you ... but if you want to go down the DIY road:

What you need is a simple way to manage a list of tasks. Personally, I added 4 custom fields to my bugtracker software (I use BugTracker.NET), a minimum, maximum, probable, and actual estimate times. I add all tasks and features to bugtracker along with my min, max, & probable estimates. When I finish each task, I update the Actual time. As I move through my tasks, I can see from my history that on average, my 'probable' estimates are off by X% (for me, it's 26%), so I can take my total time, and project all remaining tasks with my min, max, and adjusted probable time. Then on future tasks or projects, I can go through my history of similar tasks, and thier actual times, and form my estimates from those. Having the history will also help avoid missing tasks.

If you do this, you may also run into political BS. When I started this, I gave my manager access, showed him how it works, and how he could set my priorities and figure out the projects progress himself. However, because the software is a bug tracking tool, I believe all features, enhancements, and tasks are being reported as bugs! This is one thing I wish I could undo.

And the Steve McConnell book is fantastic.

Answer 19

Something I use is COCOMO. It's a technique that provides probabilistic models for software cost estimating. It's something that your PMs (if formally trained) will likely be familiar with. Instead of being concerned about hours, you simply estimate the SLOC (source lines of code) you expect to write. That right there should set off some alarms, but it's only one piece of the technique. You then plug these estimates into a model and adjust the metrics for fine tuning.

• Number of developers
• Capability of each developer
• Your work calendar (hours per week, per resource)
• Development language(s) & percentages (if using more than one)
• Familiarity with the domain (has anyone written this type of app before)?
• Management style
• Development process (Agile, waterfall, RUP, ...)
• etc.

What this whole process does is let you focus on what you're good at estimating: code. The time estimate is abstracted and encapsulated by the model itself. Each model has been constructed by analyzing 100's of real-world projects. You can even tune them yourself, by plugging in your own actuals.

I've had a lot of resistance to this method, especially when I forget to round. "Oh, right. This will take 29.02 hours." All in all, it's been the most accurate method for me. My colleagues can disagree with the usage of SLOC all they want, but they cannot debate the science that went into creating this technique.

As a sole developer, I have used it, but had problems with my ego at the beginning. I didn't want to acknowledge that my C++ skills were weaker than I thought at the time. It caused me to underestimate. After I plugged in my actual skill levels, everything lined up. Lesson learned.

Another technique I throw on top (some tools do this for you), is the three estimate technique, which is broadly applicable in all places. You take your worst case, best case and expected case then average them together. Weight your expected by a factor of four. So... "(worst + best + expected * 4) / 6" It's a moderate help for risk, but it's actually a psychological trick to get accurate estimates more quickly... in favor of your gut feeling.

Answer 20

Overestimate and break down the project in multiple small tasks wich can be devided in either 15 minutes, 30 minutes, 1 hour, 2 hours, 4 hours or 8 hours; If it doesn't fit in 8 hours split the task up again. This should give a fair idea of how long it should take.

Answer 21

Estimating software projects is a shot in the dark. Don't let anyone tell you otherwise. Project managers, especially non-technical project managers, often don't realise this.

Although there are lots of guidelines and formulas out there I've never found a better way than just taking a previous project, applying some kind of scaling factor and using that as a baseline. For example, my last project was 6 months long and my next one has simpler rules but lots more data migration plus another external interface so I reckon its about 50% harder = 9 months.

Some tips:

Don't get hung up on low-level breakdowns. I frequently find estimating tasks of duration less than 3-4 days is too fine grained and frequently leads to underestimating as it doesn't take into account the overheads.

Take your SDLC and your commercial constructs into account. Estimating a waterfall project is much different than an agile one. Estimating a fixed price project is much different than a time and materials one.

Take into account the impacts of over- or under-estimating. Will you lose the work if you estimate too high? Will you be held personally (or financially) accountable if you estimate too low?

Don't underestimate the complexity of the unknowns.

Look for opportunities to provide incremental estimates. For example you may be able to provide a ballpark estimate at project inception, a more accurate estimate after requirements gathering, another after design.

Don't fall for the trap that doubling the resources halves the duration. It might reduce the duration but frequently it will increase the duration due to training and communication overheads. However, the next project will be faster.

Don't get hung up about estimates. The quicker they're done, the quicker you can get on with the project, the quicker you can get into the nitty gritty and get a better feel for the real size of the project.

Answer 22

There are a lot of good suggestions on how to do the estimate but they all take time. To get you the time to do an estimate correctly I use this tactic.

When I’m asked for an estimate with out being given time to figure it out how long the project will take I always come back and give an estimate I know management will not like. Let’s say you think a project will take about 3 months. I will say something like 6 months to a year. They usually give me a bad look and I say, “Well, if you let me do a real estimate I can probably give you a better estimate.”

Also, NEVER say a project will take 2-3 months! You are thinking the project will take 3 months and your manager will only hear 2 months.

Answer 23

i'm late in the game for this question, but your comment struck a chord: you are often asked for estimates on the spot - no time to research, detail, plan, consult with the team, etc. Just a go-with-your-gut number - and probably a number that will come back to haunt you.

This happens - frequently. The trick is to trust your instincts but also compensate for your errors.

In other words, take your best guess and multiply it times some factor that helps to cancel out the lack of information, time to do a proper analysis, and eternal hubris and optimism.

For example, if your last 3 estimates were:

• 1 day
• 2 weeks
• 3 months

and the actual time was:

• 4 days
• 2 months
• 1 year

then clearly your "coefficient of optimism" is 4.

As you get better at estimating - or avoiding estimating, as the case may be - this coefficient should trend downwards. But it may go back up again for a while if you start working in a different domain or technology or team, etc. So it's important to always track it, at least mentally.

For a brand-new area with little info and unfamiliar technology in extreme circumstances - such as repairing the hyperdrive on a Hutt battle cruiser in the middle of a firefight when you can't read Huttese - it's best to employ the Scotty Rule: times two and add a few ^*

[^* a friend of mine insists that the true Montgomery Scott Rule of Estimation is to take your best estimate, round up, multiply it by 10, and change to the next higher order of units. So if you think a task will take 2.5 hours, you round up to 3 hours, multiply by 10 to get 30 hours, and change the units from hours to days to get the final estimate of 30 days.]

Answer 24

I would recommend to split task as much as possible before performing any evaluation, this exercise only will force you to have a deep understanding of what you have to do, and how you plan to do it. Once this is done, you increase your chances of comparing the subtask to something you have already done and have something closer to reality. If the specification of what you have to do is unclear, you will also find out at this moment, which is a good thing.

Tasks over 40h can hardly be evaluated with precision, if your project manager is good, he'll know that! It's ok to be wrong, experience is the only real thing that will make a real difference in your evaluations (and in the end, will only help you be closer to a correct answer, not perfect).

I also had a very nice experience with poker planning, the more my team uses it, the better their evaluations (plus it's fun!) but it takes more than 1 person which can't help you much.

Answer 25

I used many but maybe somthing like the 3-point estimating tool might be worth a look at.

3 Point Estimating Tool

With this you take the Optimistic (if everything went to plan), Mostly Likely (how long you think the project will take) and the Pessimistic (longest you think it could take)

You can then work out a Mean and add a weighted average to parts as well if needed. The more you break you project down the better the esitimate you might get.

As a quick start, you could have something like this as your main headers, with any numbers of tasks in them, with there Optimistic, Most Likely and Pessimistic ratings

• Research
• Function Spec
• Design Spec
• Coding
• Testing
• Deployment

Of course you might want to expand on some heading, or add you own.

Answer 26

There's usually an asymmetric penalty function here, for the manager and the dev team. That's actually part of the logic of padding.

Late: If people tell their bosses to expect Tsched, and instead it is longer than Tsched, then schedules get bumped and people get faulted for various things and in the worst cases sales gets missed or customers go away or projects are re-organized or trash-canned.

Early: If the true T is less than Tsched, then hey, pat the dev team on the back and get busy on the next project. Or -- heaven forbid -- refactor and improve the code quality (no, don't do that... do the next project. workworkwork).

Answer 27

You should listen to the stackoverflow podcast and hear Jeff Atwood's estimation process for this site you are on right now. He was a little too aggressive and Joel called him out on it.

Answer 28

These are good points to consider, but you'll need to weed out things that don't work or apply. Revisit the suggestions after you do more estimates, and see if anything stands out as more applicable after using these suggestions.

Keeping track of what you estimated and then how long it actually took is a GREAT idea, and one too often not followed up on, because you need to move on to the next "fire". But, you can only get better at estimating if you spend the time to determine how your estimate was off, and figure out what to change for the next time.

You also mentioned you are working with existing code. As you make more changes to this, I would suggest increasing the time needed to test. You need to make sure the change works, and you need to make sure the other functionality of the code still works. If your tests are somehow automated, then you may need to change the to test this change to the software. If your tests are manual then you need to be sure to test what you changed (adding or changing how you tested previously).

Good luck.

Answer 29

You can try to break it down in smaller tasks, and estimate each task. For example, you could break it into something like:

1. Install SharePoint
2. Create proof of concept web part.
3. Deploy custom code into SharePoint.

These are tasks that you should be able to find tutorials for, and by reading them, you may be able to grasp how long it will take (hint: installing SharePoint is long!).

Besides this, I agree that it's hard to estimate tasks that you haven't done before, so try to break it down to tasks that are small and comparable (in scope) to things you've done in the past.

Answer 30

If someone else in the company has done similar stuff yell over the cube wall and see what they think. Compare that to how long it actually takes you, then when you get another task ask the same developer how long it would take them. Multiply that result by the difference you got doing the initial task. There's really no good way to estimate new tasks with out a historical baseline but at least this way will show you made an effort to base your estimate on some kind of fact.