Estimating in groups - How to extract useful statistics / estimates?

As many other projects we have to estimate our jira tasks up front. This is obviously not fun and I've started hacking on a simple webapp to "gameify" this tedious process.

My idea is that an issue summary and description is presented to all participants and that when everyone has provided an upper plus a lower estimate the results are displayed to everyone. I prefer estimating in intervals instead of fixed points as it communicates uncertainty.

1. If I ask each participants something in the line of "provide a lower and upper estimate you feel 90% certain will contain the actual number of hours required to complete this task" - is this a 90% confidence interval?

My end goal is to reduce this to a single point estimate that can be fed into Jira + something that communicates the uncertainty related to this estimate.

1. What would be the best / most "correct" way of reducing the suggested intervals into a single fixed number of hours?

2. What would be a good indicator of the uncertainty / spread of the estimates?

• Joel Spolsky has a nice article on Evidence Based Scheduling which may be of interest to you. Sep 23, 2015 at 19:14
• JIRA does have velocity charts you can use to see how things worked out historically (since you are using JIRA). I've also seen that many estimation techniques simply deal with issue magnitude. They then fit the time estimate using a function based on historical velocity. I.e. story points / week or something. Sep 23, 2015 at 20:10
• Minor nitpick, but I would definitely avoid using the term "confidence interval" because that has a precise meaning in statistics. If you want to estimate a ticket that's nearly identical to 30 or more past tickets, you could use their completion times to produce a real confidence interval, but if that situation ever arises you should probably be looking into automating the task rather than automating its estimates. Sep 24, 2015 at 7:25

In the commonly referenced book Software Estimation: Demystifying the Black Art, McConnell speaks specifically to this technique in Chapter 13 (Expert Judgment in Groups).

The answer to "how to reduce this" is "by consensus."

• Don’t just average your estimates and accept that. You can compute the average, but you need to discuss the differences among individual results. Do not just take the calculated average automatically.
• Arrive at a consensus estimate that the whole group accepts. If you reach an impasse, you can’t vote. You must discuss differences and obtain buy-in from all group members.

The technique itself is a very good and useful one. It is known as the "Wideband Oracle" or "Wideband Delphi" - and searching for that will find other essays and presentations out there on the net.

One approach (summarized from Software Engineering Economics):

• All estimations are given simultaneously / anonymously.
• The coordinator then provides a summary of estimates and presents it (this way people can see how their estimations compare to others).
• The group then meets to discuss the variations in the estimates.
• The group then votes (anonymous) on if they want to accept the average estimates. If any say 'no', repeat.

This approach tends to a single point, though could easily be adapted to high and low single points to give a range.

I have also seen it done in a non-anonymous form where the high and low estimators are then asked to give a short "this is why..." and repeat the process there. This has the possibility of having strong personalities dominating the estimate (or reserved ones going along with the flow rather than raising their concerns).

So yes, this is a very useful technique and can be used to give a convergence. McConnell gives a reduction of estimate error by 40% using multiple iterations compared to the initial average.

• Thanks! I assumed that the process we use had a name, but I didn't know it was called "Wideband Delphi". I actually just bought the Kindle edition of that book before I read your answer. Sep 23, 2015 at 20:30
• @Kimble it is a very good book, and now you know which one to look at. You've also given me another one of those "here's another web app to write" (the wide band coordinator application) for my list of personal projects.
– user40980
Sep 23, 2015 at 20:38

An approach I have seen is to ask for 2 numbers. The first number is the "expected" time, which you want to be an unbiased estimator of how long the task will take. The second is the 90% confidence time, you want 90% of your tasks to fall below this estimated line. These numbers are then rephrased in terms of the "expectation" and an "uncertainty" term which is the difference between the two.

The idea behind this is that if I have 10 tasks, each of which will take 1 week "on average" and 2 weeks "90% worst case," then I don't really need to budget for 10-20 weeks. The probability of getting a 90% worst case on every single task is very low. We get to use the central limit theorem to cut down on the final task uncertainty.

The final numbers are found by adding the "expectation" terms and doing a root-sum-squared of the "uncertainty" terms. In my pathological case where I have 10 tasks, each of which is 1-2 weeks, I add the 10 expectations to get 10 weeks. Then I RSS 10 1-week uncertainties together to get 3.16 weeks of uncertainty. Thus my final estimate, instead of a scary looking 10 to 20 weeks, is presented as 10 weeks to 13 weeks 1 day.