The company where I work is aiming to have a 10% maximum error margin.They expect analysts to not miss the estimate by more or less than 10%.
I don't know what to think about it, since I got nothing to compare it to.
What could be a good baseline to measure if we are estimating too wrong, for more or less? How much (in %) do you think is okay to to miss?
Unless you are estimating something very similar to that which you and your co-workers have done before, +/-10% is ridiculously optimistic. Your management either doesn't have a lot of experience with software, or they're not aware of Large Limits to Software Estimation. That paper has some accompanying supporting material, and a lot of punditry can be found.
Let's examine a far simpler system than a typical software project: Rubik's Cube. You can solve any position in 20 moves, max. But since you're estimating, you can only look at a given cube for a few minutes before giving the solution. Can you give a good estimate? No, sometimes estimating a process takes longer than doing the process.
Another simple system: Pinocchio. A wooden automaton, his nose-piece grows when he utters a lie. What happens when Pinocchio is at rest, and then says "My nose is growing"? Some systems aren't amenable to prediction, they're undecidable.
These two problems are built-in to most software systems. Because of that, you'll never get estimates close to +/-10%.
My advice is to give a heavily padded estimate, work like a slave to get the project done as fast as you can, and then look busy until you're within 10% under or over. At that point, announce a spectacular success.
I'd be very hesitant about any sort of "target error margin" because it really depends on the project. If you are trying to estimate how long it is going to take to install, configure, and customize a new CRM system where no one is quite sure what sorts of customizations are going to be necessary and what sort of business process changes will be needed and the company has no history of similar large projects, your error margin ought to be pretty large (i.e. you might guess that it will take 18 months +/- 50% and quote 9-27 months). As the project goes forward, the specifications get clearer, decisions get made about business processes, your developers get more comfortable, etc. those error margins should get tighter. If you are trying to estimate how long it is going to build the 101st basic ecommerce site when you've got a good history from the first 100, you would expect to be able to give a much more accurate estimate. Most projects, though, are going to fall somewhere in the middle.
Now, if you are quoting a single number rather than a range, the answer is probably to start quoting the range so that the person doing the estimating can specify how accurate they expect their estimate to be.
A good baseline would be one based upon real data you have collected.
The first step to doing this is recording all estimates. The second step is recording the actual results. Be honest, don't be tempted to 'self-adjust' the actual. Gather enough of this information and you have some data to statistical base how good your estimates are. Note, this can/will vary wildly based upon who did the estimation and who the work. Only by doing this can you be reasonably expected to give a 'margin of error' that is anything other pure rubbish.
It doesn't stop there either; analyzing what causes estimates to be off can help improve the accuracy of your future estimates. Note: They still remain estimates, and as such, are only estimates.
Estimation also doesn't finish after the first estimate; this is something that can be adjusted as the project progresses as more knowledge is gained, this reducing the possible margin of error as you go along. The more open you are with communication, the earlier surprises are discussed - leading to people been less surprised and allow more time to make adjustments in either the project or customers expectations.
Second, perhaps a better way of handling error margin is 'confidence internals' rather than merely % margins of error. You than based you estimated delivery date on a confidence interval, rather than a singular date.
PERT is one example of a framework to handle estimation based upon statistical reasoning. For example:
"Based upon what I know now, we have a 90% confidence level we can deliver in 8 months. 95% confidence in 10 months, 99% confidence in 2 years, etc"
Note here: The more confident they want you, the larger the estimate will be. Depending on upon the complexity (aka, how accurate you could possibly be), it could be a small difference between 80% and 90%, or it could be huge!
Lastly - Good luck ;) If you ever solve 'maximum error margin' in software estimation whereby you can specify something like 'all our estimates will be +/- 10%', be sure to commission a box-office movie for the rest of us in the software industry. I'm thinking something like a cross between Office Space and The Matrix :D
It actually depends a lot on the size and complexity of the project.
If your project estimate is say 1 week - 10% is reasonable. It means +/- 1/2 day.
If it is 1 month 10% is shaky - from my experience it is impossible to forecast what you will discover in 1 month.
More than one month - all bets are off :).
These are per developer - so for a 4 developer team 1week == 1 month.
For a 4 developer team, mostly its good to come up with a list of features that can be done in 1 month, and be within 10% for those feature. Then, re-estimate for the next month.
I have made some naive assumptions here
You have to factor those in.. but this is the general idea.
There are lots of variables:
How long is the project?
How will the project be managed? Waterfall? Agile? SCRUM?
I would assume from the question Waterfall. If that is the case you are pretty much expected to fail by some percentage given the request for a margin.
If the answer is some Agile methodology, especially something like SCRUM. It doesn't really matter what the margin percentage is. A 50% margin of error on a 2 week Sprint is 1 week, on a 1 week Sprint it is 2.5 days, both extremely worst case scenarios. This is because the delivery date is re-estimated every Sprint, and it will get to be more and more accurate over time, instead of less and less.
With Waterfall 50% margin of error isn't un-heard of either, but over a 12 month project that is 6 months, and it doesn't really get discovered / admited it is that bad until 10 months into the 12.
Back when I used to lead software teams, roughly around the Cretaceous/Tertiary boundary, we actually came close to achieving +/- 10% on estimation. It was about +/- 15% if my very dodgy memory serves. But this was because we were estimating for things we'd already done: relatively simple real-time communications firmware which took bytes from A and moved them to B using a language that we were familiar with, in a real-time environment designed by us, talking to hardware designed in-house by guys a few offices away. Lots of slight variants on the above theme, for literally years.
Expecting to achieve this kind of error rate in normal software project estimation is, frankly, ludicrous. When you see it apparently achieved, it's because either people over-estimated and padded (doing extra stuff and pet projects to use up the budget), or under-estimated and worked like dogs at evening and weekends to make up time.
You've probably heard the 300% thing right?
I actually use it. Fully based on what I've seen for years. When I hear a day or two, that's a week or more to be really done. And Tested. In All environments. Documentation updated. Usability tested. Performance tested. Load tested. A couple of hours is really more like a day.
I think we're really bad at estimating because of:
So at the highest level, with business folks who need better than 300% estimates, what we end up doing is aiming for reasonably good estimates but at the price of being higher level and more general. "We will have an user edit function" even if version 1 means only for 1 group of users for changing 1 field.
When it comes to "What is the acceptable error margin when estimating a project duration?" I find that this approach, used in many Agile environments helps change the question to what is the minimum functionality to get an alpha or beta version live and then iterate on it.
