How consistent can estimates be expected to be?

Question

I recently did some statistics on the estimates of our two scrum teams.

These graphs plot "original estimate" against "actual time spent" (note that logging time spent is much less objectionable, in a scrum sense, than using actual-time units for planning).

Estimation history (logarithmic scale)

Estimation history (linear scale)

It seems to me that the resulting effort has very little correlation to the original estimates. I had of course expected overlaps, and maybe overlaps between only adjacent estimates, but I certainly don't feel that it makes sense to have 1-2-3 all touching the very low end and the high end.

5│          ══════════════
3│     ═════════
2│  ══════
1│════
 └───────────────────────────
 My expectation: Partially overlapping estimates

Of course, estimates are crude, and occasionally things blow up, but to me this data screams "we suck at estimating" which to some reasonable extent is okay but I feel that the data displayed here is far beyond reason. I want to make an "estimation workshop" with the teams in order to come up with a better culture for estimates, but I want to make sure I have a reasonable frame of reference for it. So...

How consistent can estimates be expected to be? Is my "expected" graph way unrealistic? What, then, should I expect?

Edit: Yes, we have acceptance criteria and a "Definition of Done" (including tests, reviews, etc), but they may be insufficiently policed. Our planning poker usually shows cards of adjacent size (no huge gaps), but even so the gap is between the agreed-upon outcome and the actual size.

Answer 1

Your numbers actually seem to have your expected distribution, though it's hidden by the log-scale. You might get better intuition with a plot that illustrates the density, e.g. a violin-plot. A box-plot can quickly provide a few summary statistics which are not obvious for your data.

Comparing the median might be better than comparing the min/max, as the median tells you “50% of 1-SP stories were completed within X hours”. For task estimation, it's expected that there will be outliers in both directions. Some tasks will be much easier than expected (happens occasionally for bugfixes). Quite a lot of tasks will take much longer than expected.

But yes, the high variance looks weird. But this depends on how SPs are used by those teams. E.g. if the teams are using SPs to rank their tasks according to effort (an ordinal scale) then it's pointless/invalid to try to reinterpret that as a duration (interval scale). (See: Statistical data type.) Your analysis is valid iff the SPs are also an interval scale, i.e. three 1-SP tasks are expected to be as much effort as one 3-SP task.

I'd consider stop doing SPs and start estimating in {½day, 1 day, 2 days, week} increments. Yes, yes, this is non-Scrum and evil. But it allows a feedback cycle to form. In the retrospective, the team can go through the completed stories and compare the estimate with the used time. The team can then discuss which unexpected problems turned up, and how they can be avoided or accounted for in the future. This simple act of comparing expectations to reality should lead to a quick improvement of estimation variance over the course of a few sprints, assuming of course that the team is interested in producing better estimates.

TL;DR:

• yes that variance is a bit extreme
• the team should try to improve their (estimation) process during a retrospective, using real-time estimates makes that easier
• no, an estimation workshop is unlikely to help with that
• if you do statistics, please do them a bit more carefully

Answer 2

You are now only looking at the accuracy of the estimate itself. In my experience, the inaccuracy is more often caused by the user stories, rather than by the estimate it self.

Some things to consider here

• More work is done on the user story than defined by the story itself (some additional low-hanging fruit features).
• The acceptance criteria are not well defined. Acceptance criteria should be clear and testable. In other words, is there a definition of ready?
• There is no Definition of Done. Do you include code reviews, writing tests, refactoring, code clean-up in the DoD? Is that also included in estimation?

Sometimes you find user stories like: "Make a window that lists this and this". Everybody knows what is meant, so why define it further? Well, it's vague, and people will do different things. If during the planningspoker the estimates from team members are far apart, this is a good sign of ill-defined user stories (not always)

What might help is to keep a list of user stories from the past few sprint, where the team thinks the estimate was ok. Show that list at the start of the planningsmeeting, such that there is a baseline, and new user stories can be compared to those.

Answer 3

Complexity (which is what story points are supposed to be) and hours spent on a story aren't always correlated. It's possible to have an extremely simple change that takes a long time to implement or a complex change that takes relatively little time to implement these should balance each other out somewhat over time and that seems to be the case in you example, the couple 5s that were less than 10 hours makes up for a handful of 1s that are longer.

There are some things that can be done to get more consistent estimates on your team. You can take previous stories and use them as a canonical example of what an X point story is. this can be useful because it's generally easier to estimate if story X is closer to Y or Z than just estimating X outright. Another thing that sticks out to me is the time spent on stories of all points seem extremely long, it would probably be worth discussing some of those cases in retrospectives. There appear to be quite a few 3 or less point stories taking more than 40 hours to complete which I think is a good sign something is not right. If you take some time to investigate this you can get a better idea if you have poor requirements, scope creep, people not getting help soon enough, dependencies on external teams, not breaking stories down enough, lots of bugs causing a story to be rejected multiple times, or something else.