Estimating in groups - How to extract useful statistics / estimates?

Question

As many other projects we have to estimate our jira tasks up front. This is obviously not fun and I've started hacking on a simple webapp to "gameify" this tedious process.

My idea is that an issue summary and description is presented to all participants and that when everyone has provided an upper plus a lower estimate the results are displayed to everyone. I prefer estimating in intervals instead of fixed points as it communicates uncertainty.

1. If I ask each participants something in the line of "provide a lower and upper estimate you feel 90% certain will contain the actual number of hours required to complete this task" - is this a 90% confidence interval?

My end goal is to reduce this to a single point estimate that can be fed into Jira + something that communicates the uncertainty related to this estimate.

2. What would be the best / most "correct" way of reducing the suggested intervals into a single fixed number of hours?

3. What would be a good indicator of the uncertainty / spread of the estimates?

Answer 1

In the commonly referenced book Software Estimation: Demystifying the Black Art, McConnell speaks specifically to this technique in Chapter 13 (Expert Judgment in Groups).

The answer to "how to reduce this" is "by consensus."

• Don’t just average your estimates and accept that. You can compute the average, but you need to discuss the differences among individual results. Do not just take the calculated average automatically.
• Arrive at a consensus estimate that the whole group accepts. If you reach an impasse, you can’t vote. You must discuss differences and obtain buy-in from all group members.

The technique itself is a very good and useful one. It is known as the "Wideband Oracle" or "Wideband Delphi" - and searching for that will find other essays and presentations out there on the net.

One approach (summarized from Software Engineering Economics):

• All estimations are given simultaneously / anonymously.
• The coordinator then provides a summary of estimates and presents it (this way people can see how their estimations compare to others).
• The group then meets to discuss the variations in the estimates.
• The group then votes (anonymous) on if they want to accept the average estimates. If any say 'no', repeat.

This approach tends to a single point, though could easily be adapted to high and low single points to give a range.

I have also seen it done in a non-anonymous form where the high and low estimators are then asked to give a short "this is why..." and repeat the process there. This has the possibility of having strong personalities dominating the estimate (or reserved ones going along with the flow rather than raising their concerns).

So yes, this is a very useful technique and can be used to give a convergence. McConnell gives a reduction of estimate error by 40% using multiple iterations compared to the initial average.

Answer 2

An approach I have seen is to ask for 2 numbers. The first number is the "expected" time, which you want to be an unbiased estimator of how long the task will take. The second is the 90% confidence time, you want 90% of your tasks to fall below this estimated line. These numbers are then rephrased in terms of the "expectation" and an "uncertainty" term which is the difference between the two.

The idea behind this is that if I have 10 tasks, each of which will take 1 week "on average" and 2 weeks "90% worst case," then I don't really need to budget for 10-20 weeks. The probability of getting a 90% worst case on every single task is very low. We get to use the central limit theorem to cut down on the final task uncertainty.

The final numbers are found by adding the "expectation" terms and doing a root-sum-squared of the "uncertainty" terms. In my pathological case where I have 10 tasks, each of which is 1-2 weeks, I add the 10 expectations to get 10 weeks. Then I RSS 10 1-week uncertainties together to get 3.16 weeks of uncertainty. Thus my final estimate, instead of a scary looking 10 to 20 weeks, is presented as 10 weeks to 13 weeks 1 day.