# How consistent can estimates be expected to be?

I recently did some statistics on the estimates of our two scrum teams.

These graphs plot "original estimate" against "actual time spent" (note that logging time spent is much less objectionable, in a scrum sense, than using actual-time units for planning).

Estimation history (logarithmic scale)

Estimation history (linear scale)

It seems to me that the resulting effort has very little correlation to the original estimates. I had of course expected overlaps, and maybe overlaps between only adjacent estimates, but I certainly don't feel that it makes sense to have 1-2-3 all touching the very low end and the high end.

``````5│          ══════════════
3│     ═════════
2│  ══════
1│════
└───────────────────────────
My expectation: Partially overlapping estimates
``````

Of course, estimates are crude, and occasionally things blow up, but to me this data screams "we suck at estimating" which to some reasonable extent is okay but I feel that the data displayed here is far beyond reason. I want to make an "estimation workshop" with the teams in order to come up with a better culture for estimates, but I want to make sure I have a reasonable frame of reference for it. So...

How consistent can estimates be expected to be? Is my "expected" graph way unrealistic? What, then, should I expect?

Edit: Yes, we have acceptance criteria and a "Definition of Done" (including tests, reviews, etc), but they may be insufficiently policed. Our planning poker usually shows cards of adjacent size (no huge gaps), but even so the gap is between the agreed-upon outcome and the actual size.

• As much as interesting as this question is, this is highly subjective and heated topic. One group of people argue estimates can work given enough experience. Other group of people says that estimates are useless waste of time. Neither has any objective data, only personal experience. Aug 3, 2018 at 6:33
• No such thing as "official opinion" exist. Aug 3, 2018 at 6:48
• `How consistent can estimates be expected to be?` how consistent is the knowledge of those who do estimations? In which grade of certainty do they estimate? Do they have all the inputs required to estimate wisely? Do they miss relevant information during the estimation? Are they fully aware of the possible risks beneath of each decision? and so on. Even if all the answers are "yes", the estimations will differ from developer to developer.
– Laiv
Aug 3, 2018 at 6:54
• @Laiv Some would claim, that even with all that information, estimation will vary too much to be useful for any kind of forecasting. Aug 3, 2018 at 7:04
• Have you gone back on some of estimates that were way off (e.g. 1 or 2 story points but took 100 hours) and gotten some idea as to why the estimates were off? Say during retrospective? Aug 3, 2018 at 12:15

Your numbers actually seem to have your expected distribution, though it's hidden by the log-scale. You might get better intuition with a plot that illustrates the density, e.g. a violin-plot. A box-plot can quickly provide a few summary statistics which are not obvious for your data.

Comparing the median might be better than comparing the min/max, as the median tells you “50% of 1-SP stories were completed within X hours”. For task estimation, it's expected that there will be outliers in both directions. Some tasks will be much easier than expected (happens occasionally for bugfixes). Quite a lot of tasks will take much longer than expected.

But yes, the high variance looks weird. But this depends on how SPs are used by those teams. E.g. if the teams are using SPs to rank their tasks according to effort (an ordinal scale) then it's pointless/invalid to try to reinterpret that as a duration (interval scale). (See: Statistical data type.) Your analysis is valid iff the SPs are also an interval scale, i.e. three 1-SP tasks are expected to be as much effort as one 3-SP task.

I'd consider stop doing SPs and start estimating in {½day, 1 day, 2 days, week} increments. Yes, yes, this is non-Scrum and evil. But it allows a feedback cycle to form. In the retrospective, the team can go through the completed stories and compare the estimate with the used time. The team can then discuss which unexpected problems turned up, and how they can be avoided or accounted for in the future. This simple act of comparing expectations to reality should lead to a quick improvement of estimation variance over the course of a few sprints, assuming of course that the team is interested in producing better estimates.

TL;DR:

• yes that variance is a bit extreme
• the team should try to improve their (estimation) process during a retrospective, using real-time estimates makes that easier
• no, an estimation workshop is unlikely to help with that
• if you do statistics, please do them a bit more carefully
• Thank you for your answer! Just one point, I don't agree that changing the axis makes a difference, as the extreme points of the ranges are still the same (in the first chart, both 2 and 3 go from almost-nothing to almost-100 hours). Aug 3, 2018 at 8:47
• @KlaymenDK yes but you are comparing the extreme values, which may be outliers. Because the data points overlap it's not easy to see whether they are representative – a plot with density estimation such as a violin plot could help. And instead of min/max, compare robust statistics such as quantiles. You might find that the 2-SP records only go until 50 after outlier removal, not “almost-100”. The log-plot is helping to obscure that. If you are expecting a linear relationship, both axes should use the same scale (both linear, or a log-log-plot).
– amon
Aug 3, 2018 at 9:42
• One approach my team had was to put a bunch of programmers in a room, explain the project, and get their estimates. The most precise calculation we got for the time required was to take the largest estimate by a programmer and double it.
– Neil
Aug 3, 2018 at 12:04
• @KlaymenDK I saw your edit. I think your SP's and actual estimates are correlated, but it's hard to tell. If you want to you can send me the raw data (address see below so that I can delete it later) and I can see whether there are some useful insights after I wave my data science wand over it :)
– amon
Aug 7, 2018 at 15:22
• @amon, oh you're using a sneaky "@" character there, it confused me. :-) Aug 8, 2018 at 11:11

You are now only looking at the accuracy of the estimate itself. In my experience, the inaccuracy is more often caused by the user stories, rather than by the estimate it self.

Some things to consider here

• More work is done on the user story than defined by the story itself (some additional low-hanging fruit features).
• The acceptance criteria are not well defined. Acceptance criteria should be clear and testable. In other words, is there a definition of ready?
• There is no Definition of Done. Do you include code reviews, writing tests, refactoring, code clean-up in the DoD? Is that also included in estimation?

Sometimes you find user stories like: "Make a window that lists this and this". Everybody knows what is meant, so why define it further? Well, it's vague, and people will do different things. If during the planningspoker the estimates from team members are far apart, this is a good sign of ill-defined user stories (not always)

What might help is to keep a list of user stories from the past few sprint, where the team thinks the estimate was ok. Show that list at the start of the planningsmeeting, such that there is a baseline, and new user stories can be compared to those.

Complexity (which is what story points are supposed to be) and hours spent on a story aren't always correlated. It's possible to have an extremely simple change that takes a long time to implement or a complex change that takes relatively little time to implement these should balance each other out somewhat over time and that seems to be the case in you example, the couple 5s that were less than 10 hours makes up for a handful of 1s that are longer.

There are some things that can be done to get more consistent estimates on your team. You can take previous stories and use them as a canonical example of what an X point story is. this can be useful because it's generally easier to estimate if story X is closer to Y or Z than just estimating X outright. Another thing that sticks out to me is the time spent on stories of all points seem extremely long, it would probably be worth discussing some of those cases in retrospectives. There appear to be quite a few 3 or less point stories taking more than 40 hours to complete which I think is a good sign something is not right. If you take some time to investigate this you can get a better idea if you have poor requirements, scope creep, people not getting help soon enough, dependencies on external teams, not breaking stories down enough, lots of bugs causing a story to be rejected multiple times, or something else.

• "There appear to be quite a few 3 or less point stories taking more than 40 hours to complete which I think is a good sign something is not right." - That is precisely my concern. Aug 3, 2018 at 13:08