Are story points really a good measure for velocity?

Question

Story points are used quite a lot nowadays, especially in Scrum to measure how much work the team can complete during a sprint. There are tons of articles out there explaining why estimating in story points is better than estimating in hours. This article by Atlassian seems to capture the essence quite well:

• We want to estimate the complexity of the task, not the time that it takes to complete (because in most cases we don't know the time)
• Hours don’t account for the non-project related work (emails, meetings, and other distractions)
• Story points make it impossible to play politics and claim that one team is working better/faster than the other

Those are all good points, but articles like this one rarely mention how we're supposed to measure team velocity (aka sprint capacity) if we're using story points. To be precise, I have observed the following difficulties when trying to measure sprint capacity in story points:

1. Story points are not additive. If you have two five-pointers in a sprint, completing them usually takes a vary different amount of time that completing five two-pointers. It is impossible to have predictable sprints unless you always have similar number of similarly pointed tasks in each sprint, which is often very hard to achieve.
2. Story points measure complexity, but don't tell you when the task will be completed. So, for example, if your task is to create a simple script and run it against a huge production database, the story point estimate will be low. But if the script takes a week to process all data, the task to "run the script" will not be formally completed for a long time, and is very likely to roll over to the next sprint. Another example is when you have to manually punch in 500 rows of data into the database. The complexity of such task is low, but it requires a lot of time.
3. Stakeholders always want predictability, and they always want it in days (ie 'Will feature XYZ be ready by October?'), not in story points. I've seen many articles on the web suggest that for this reason it's useful to choose your measurement in such a way that a story point can be converted to a certain number of hours/days. The most common approach of all is '1 story point = 1 day'. Or, alternatively, let each task have two separate estimates: one in story points, and the other in hours. But both of these approaches defeat the purpose of story points - to make it impossible to play politics and compare teams' velocities.

For these reasons, I have always found it very difficult to measure sprint capacity in story points. It just makes sprints much less predictable than when you simply measure in hours.

I'm curious to know if there are any approaches where you can use the advantages of story points (no playing politics, etc), but mitigate the above mentioned problems. Or are story points actually pointless (no pun intended)?

Answer 1

I've found story points to be terrible indicators of how long something will take once you get down to the sprint-sized chunk of work that a story should be. Velocity? It's a numbers game too. Track how many story points the team completes each sprint. It's even useful to track how many people are on the team each sprint. This doesn't predict when works gets done, but it can help justify when work doesn't get done — but that's a different question entirely.

I've found that velocity is better at predicting when large chunks of work will be done as long as your minimum unit of time is one sprint. The further out you plan, the fuzzier your prediction gets. If your team averages 24 points per two week sprint and you have a 100 point story, the naïve math says you should be done in 100/24*2 = 8.33 weeks (or 41.67 work days). And if you plan the timeline that tightly, you will be wrong.

At 24 points per sprint, a 100 point story will take longer than 4 sprints, but just barely. So, you tell management it will get done 5 sprints after it starts. When will it be deployed? Not in 41.67 days. At the end of the fifth sprint after you start.

Now the planning part comes in. This 100 point story needs to be deployed by Oct 1. Today is August 9. If, for example, this coming Monday is the first sprint you work on the story, look out 10 weeks. What date is that? Around October 18 (in the year 2024). Standing here today you can bank on being 2 weeks late.

Don't get any more fine-grained than that.

How about that 3-point story? When will that be done? In the same sprint we start it. Don't get any more fine-grained than that. If someone really wants an estimate in hours, then spend some time and estimate that 3-point story in hours. Don't hand-wave a math formula to derive hours from story points. You will be wrong. Do an honest assessment of the work ignoring all that story point mumbo jumbo.

Say you've got a backlog of stories totaling 200 points. Someone says you need total hours for that work. Those of us in the public sector frequently need to deal with that simply because some bureaucrat wrote a policy to prove tax money is being spent wisely. Don't fuss over each story. Do some naïve math, and round up to the nearest sprint: 200 / 24 = 8.33 sprints — nope, scratch that; 9 sprints. That's 18 weeks if doing a two week sprint. How many hours of labor will your team log in 18 weeks? That's your estimate in hours.

The trouble is, some people want exact numbers. They won't be satisfied with the numbers above, because they might want to know the hours for each story. At some point I just started doing naïve math and simply not caring if the numbers were wrong for each story. What is the bigger picture? Are you still on track overall? One story takes too long and now management is having a heart attack. I bet some of those stories took less time, too. Show management that it averages out over many stories. I bet they calm down a little. And sometimes management just has an axe to grind, and you just need to let them. It makes them feel better. Everyone likes to be heard, so hear them, and then move on. No need to change unless you can see the overall trend heading in the wrong direction.

That 2-point SQL script that took a week? It was easy, right? Low complexity. Low story points. This is where complexity is not a well-defined concept, in my opinion. If a simple 2 story point SQL script takes half a sprint to write, test, run, and verify, I would argue the team missed some complexity, and the points should be higher.

Why did that script take so long? That's where you need to lean on the details of the work to come up with the estimate. It isn't just the act of writing SQL that you are estimating. You need to analyze the data and tables. Maybe you need to create a rollback script. Coordinate with DBAs. Depending on the organization, you might have paperwork to submit just to get access to the database to start your analysis. What other applications are impacted? You need to notify those teams. Suddenly you're spending 4 days herding cats so you can spend an afternoon writing and debugging a SQL script. This is more complicated than you thought.

Estimating in complexity is more than the code you write. It involves all activities necessary to analyze, design, develop, test, and deploy that work. Complexity ramps up easily due to those non-coding activities. Don't forget testing, too. Every edge case potentially adds to the permutations required to ensure this "simple" SQL script works and doesn't destroy data. Oh, and that rollback script needs testing, too.

The challenge comes when you get pushback because the 30 line SQL file has a 34 point estimate. That's when you start rattling off all the non-code tasks you need to do and say it increases complexity. If nobody agrees, I'm a fan of just going with a 2 point estimate and then noting your concerns in an email or a comment on the work item. Do the thing, and lo and behold, that 2 point story took half the sprint.

That's why the sprint retrospective is essential. You can bring this story up for discussion. I bet after management made their perfect air-tight plan and watched every word you said come true that they will believe you next time. You can steer the Titanic, but not before it hits the iceberg. So state your concerns, do the work, watch it hit the iceberg, and then watch how people learn to listen to you next time.

I've been doing this for 20 years. Over this time I've noticed some patterns emerge in people's behavior regarding estimates. People demand precise estimates after a series of botched estimates and timelines. They think a more rigorous estimation process leads to more precise estimates, thereby solving the problem of missed timelines and budgets. This has the opposite effect, because so far as I know, humans don't have precognition. So we are wrong. This only serves to reinforce their belief that the estimates need to be more exact.

I've found tracking both estimates to be useful in this situation. Do the elaborate estimation process. Jump on one leg, spin three times and shout "boondoggle!" three times after the first full moon of the month. Jump through all the hoops management wants you to.

And also estimate in points and provide that fuzzy "it will be done during sprint X" estimate. Compare the two estimates with when the thing actually got done. I bet you won't need to shout "boondoggle!" after a full moon too many more times. Management will start to see the fuzzier estimate takes less time and gives them enough granularity to plan for the future.

Remember that all this talk about story points and estimates in hours is about predicting the future. You have far-off predictions and short-term predictions. They require different strategies, but you can use story points either way. Just remember that "complexity" also includes analysis, testing, and coordinating with other teams, too.

Now you can justify that 2-point SQL script being 34 points and it will be done sometime during the second sprint after you start it. And you can predict that because you track sprint velocity, which allows you to do some believable if not naïve math. It's hard to argue with numbers.

Answer 2

Stakeholders always want predictability

Most of the problems in this area arise because non-technical management are determined that alterations of their bespoke information systems and business processes should be a fully predictable activity, when all expert management knows this is not the case.

The vast majority of working time for developers is taken on gaining an understanding of the existing situation, and on searching (by using their intellectual faculties) for a new solution. This might be intermingled with writing code, but it's certainly not the physical act of typing which is the work.

I was tempted to write a more detailed answer about exactly what development involves, but found myself deviating too far from the question.

Suffice to say it is poorly articulated what exactly developers do, and most developers are more apt to do their work than to explain what it is they are doing - because they are generally self-taught practitioners, not educators or philosophers of the practice.

For non-developers, the difference in tacit knowledge with the developer can be unbridgeable by anything the developer is able to say.

But how long does it take someone to learn an arbitrary subject they don't yet know anything about? How long does it take to search for a design you don't yet even know exists?

If non-technical managers were in the 19th century gold mine trade, they'd be asking the surveyors how long it will take to find a gold mine in unfamiliar country, and insisting it must certainly be found in 3 weeks' time when the workforce is booked to arrive - already assuming one is present.

Obviously, it would be very desirable to be sure to find a new gold mine in an arbitrary place by applying a fixed amount of survey labour, but it would also show a breathtaking failure to engage with the reality of the situation.

The analogy here is not that developer activity is a physical process like surveying or mining, which has been amenable to improvements in survey technology and geological study over the centuries.

The analogy is that developers have only slow and speculative methods for acquiring the relevant kinds of knowledge - these methods do not produce outputs that correspond remotely to the certainties being demanded.

We want to estimate the complexity of the task, not the time that it takes to complete

Ironically, if there is one good rule of thumb, it is that more complexity in the picture requires more time. Complexity means more to learn, more to reason about, and more space for uncertainties.

So you are attempting to get at a proxy measure of time. But complexity is just as difficult to measure - indeed, it is complexity, and the risk of finding unexpected interactions or constraints in the design only once a significant amount of work has been done over a significant amount of time, that makes the timeframe and certainty of success so difficult to gauge.

There is nothing wrong with trying to estimate, of course. Experienced developers can rely on experience to calibrate their perceptions somewhat, although it is surprising how often non-technical managers assume a young and inexperienced developer must have the same ability.

Why are developers able to do their work if they don't know how long it takes? Because the work is about learning and reasoning, and a confident and demonstrable ability to learn new things and reason about complicated things in general, doesn't mean you suddenly know how long it takes to learn an arbitrary thing which you don't know anything about yet. That's why it's possible to do work without being able to describe how long it takes - no learner of an unfamiliar subject, is capable of reckoning themselves how long it takes to learn an arbitrary topic or innovate in that field.

Only teachers, who teach the same thing over, know roughly how long it takes - because they're just looking at past learners, not assessing the current ones.

Experienced developers are also less likely to encounter something completely unfamiliar, and therefore can use accumulated knowledge instead of applying time and effort to developing that knowledge. Effectively, sunk work on previous development is applied partly to a new proposal.

Some approach to rationing the development capability is usually required in a business. But this approach is only better than the alternative, of accepting or declining work randomly. It doesn't mean it is occurring as an exact science, or that people can rely on individual estimations for the purposes of deadlines or as guarantees of eventual success.

When you estimate 20 small things, and one takes ten times as long, and one gets cancelled as impossible, people take it in their stride. They don't when the company is being bet on a single, massive task.

Or are story points actually pointless (no pun intended)?

I would say applying some work to estimation, and expected usefulness of the development, is useful for deciding how to apply limited manpower to a choice of development tasks.

But it's not a perfect science and isn't supposed to be, and it can be a serious, harmful overhead when it starts to draw a significant amount of limited manpower in its own right.

Its purpose is to help you and your team allocate your time to the most useful work when there is too much to do overall, balance the total workload between multiple people, and so on. Its purpose is not to solve the estimation problem.

Answer 3

Story points originate with Extreme Programming, and they were a solution to a specific problem that the early XP team was facing.

Ron Jeffries writes the details in his blog post, Story Points Revisited. Initially, the team estimated in time and specifically used Ideal Days. Ideal time is how long it would take to complete the work in the best situations - the team can work uninterrupted without dependencies or coordination from start to finish. However, the ideal time is never the same as the real time, since the team deals with interruptions and context switching. To address these interruptions, the team multiplied the ideal time by a load factor to estimate how much real time would be needed to complete the work. However, when talking to stakeholders, they often dropped the word "ideal" and spoke in hours or days, which led to confusion when it took 2 or 3 days to get 1 day's worth of work done. So, they just started calling the value "points".

So, initially, story points were a unit of time. And, specifically, a time calibrated to a specific team's working environment and ability to focus and progress the work. The notion of relative estimates and effort estimation came later and wasn't a part of the original definition of story points.

A lot of the problems you present - trying to measure complexity, numbers that you can't perform mathematical operations on, a lack of meaning to stakeholders - plus others - sensitivity to context, including process changes and knowledge changes within the team, for example - are common problems with story points, as they are defined and used today as a form of relative estimation.

Although I have seen teams succeed with story points, the problems I've seen and the problems I've read about far outweigh those successes. I'd put myself strongly in the camp of story points being pointless.

If you need to estimate, the original ideas behind story points are the most sound. Estimating ideal time and planning work based on capacity and load factor works well. I've also found that, when working in hours, it becomes easy to account for vacations, holidays, and other planned events, including company overhead work. In my experience, project managers and other more senior business types understand ideal time. Talking about minimizing interruptions and reducing the load factor can be measured and is generally understood to help promote process improvement.

However, there's also the No Estimates movement, originally led by Wood Zuill and Neil Killick, which Ron Jeffries wrote about. Instead of estimating, you can decompose the work into the smallest possible unit of value delivery and then use flow metrics, like throughput and cycle time, to forecast completion rates. Dan Vacanti's When Will It Be Done?: Lean-Agile Forecasting to Answer Your Customers' Most Important Question is a good guide to applying this technique.

Answer 4

• We want to estimate the complexity of the task, not the time that it takes to complete (because in most cases we don't know the time)

The problem with this is that we often don't really know the complexity either. Indeed, one of the main problems I see is that teams are often asked to estimate stories which fall woefully short of any sensible 'Definition Of Ready'

Any quantitative measure must be seen in the context of how confident the team are in even understanding the work and expectations in the first place, as well as all the other context and inter-connected dependencies and risks surrounding the work (As best as we might wish to insulate a team and isolate each story, reality tends not to work that way).

Of course, low-levels of of uncertainty are perfectly normal - the team may not precisely know the area of the code, nor exactly how the system behaves at current, but they should at least be sufficiently confident in many of the basics so that any estimate is within a reasonable margin of error.

While not an exhaustive list, the sources of uncertainty can come from almost any direction, so there are plenty of things for a team to consider when deciding whether they are confident in their estimate:

• Team members sufficiently experienced working within the system and the code, following the team's ways-of-working, familiar with its tools and technologies
• Confidence that they are not missing any inter-dependencies (other teams, 3rd-party systems, external partners, other incomplete work items, etc)
• Understand the context behind the story - The business problem to be solved, the requirements and user/stakeholder expectations, and acceptance criteria.
• They know how to test the story and how to deliver into production.
• The story has already been reviewed and agreed by stakeholders and QA testers with test cases already identified
• The existing test coverage is known, including the quality of existing tests, and the team understand how to assure against regression issues.
• Awareness of the current quality and stability of affected parts of the system.
• Corporate IT policies (e.g. security, data protection, etc) have been considered
• Impact on the wider system architecture have been agreed.
• Any necessary changes to the underlying platform and infrastructure have been identified, and discussed with platform/infrastructure engineers if necessary

Again, this is not intended to be an exhaustive list, but many of these are the kinds of details which teams may not always think about when they're asked to produce a Story Point estimate -- many of which, if not considered ahead of time, will probably only crop up after the work has already started, likely invalidating the original estimate.

For example, if a story which everybody thought was just a code change, worked fine in a developer's local environment, yet broke a bunch of other stuff when running against the "real" infrastructure, then it suggests that the team didn't really have sufficient knowledge of the system when elaborating the task, so perhaps some aspect of the story could have been raised as a time-boxed investigative spike.

Lastly, getting everything above 'right' doesn't necessarily mean you will end up with a useful velocity; but consistently getting these things wrong almost guarantees that whatever measure you use for estimation will be a meaningless predictor of the team's capacity.

Answer 5

Story points are relative to one another and merely a means of estimation. 1 story point versus 5 story means that the 5 pointer is more risky, more complex, and more time consuming than a 1-, 2-, or 3-point story. How much time? Well, you have to measure actuals. A one-point story many take 3 days or 3 hours. A 5-point story may take 5 days or 15 days. One will not know the actual until the story is done and work is tracked, and actuals are plotted.

The problem comes with applying fuzzy math trying to equate story points with actual time. For example, 1 point = 6 hours or 1 point = X days. This will always be a fallacy. It then becomes a numbers game which can be rigged by anyone to show positive outcomes.

Teams should be using retros to determine if any stories were more complex/time consuming or less complex/time consuming and adjusting as necessary. For example, if you had a 1-point story that took longer than a 5-point story. Something was missed in that original story estimate. This will eventually produce a team velocity which can help plan when backlog items should finish.

Note, velocity many take several sprints to stabilize into a useable number. Once that happens, one can start to build in confidence and predictability. Let's say after 6 sprints velocity has stabilized to 25 story points and there is 100 points in the backlog, well that's about 4 sprints worth of effort. The confidence should be pretty high in that estimation. That doesn't mean the work will finish in 4 sprints for those 100 points, but there is high probability that will be based on the current story point velocity. That's where every development team should eventually end up.

Answer 6

Story points measure complexity, but don't tell you when the task will be completed.

I haven't read through all the existing answers (some are really long) so this might have been addressed already, but I think the thing that most people miss about this is that a project manager (scrum-master, whatever you want to call them) is supposed to take these story points and do statistical analysis to determine how many 5-pointers a team can produce in a sprint and how many 2-pointers they can produce, etc.

Over time, they can start to produce timelines from new story point estimates and with details like e.g., we have a 90% chance of completing this work in 5 weeks 97% in 7 weeks.

The idea is that people are really bad at estimating how long things will take but they are pretty good at relative estimates. So instead of trying to have the developers say how long, they say it should take about as long as something that has been done before. Then the PM can then back into the duration from those relative estimates based on historical story point estimates (or, alternately, use T-shirt sizes which avoids the error of summing points) and historical delivery times.

If you are working with PMs who do not have the statistical skills to do this kind of thing or don't understand that's what they should be doing, it does become somewhat pointless. The one benefit you still get from collecting that data is that someone competent might be able to use later. It's also possible this is happening, and you are just not aware of how the information is being used.

Answer 7

In a previous project we modeled story points as if they were complex numbers, where the real part is time (which you can measure) and the imaginary part is technical complexity (you can't measure it). In the end, we decided that even if a script is trivially simple to implement, we can't give it 1 story point if that script needs to run on a 100TB database which would take 3 days. We would fit it into a 3 story point, seeing how it would be a 1 from a complexity perspective but a 2 from time perspective. There is still subjectivity involved, but in the long run it averages out.

In the first few sprints, we were missing the estimations by a lot, but after 4-5 sprints, we had very good examples of tasks for all the story point values between 1 and 13. We even had examples of tasks that would certainly not fit within a 2 week sprint. So by having these examples it was then pretty easy to compare new tasks and give pretty accurate estimations.

This lead to some interesting effects:

1. Most tasks would actually fit in the 5 and 8 story point categories.
2. There was always at least a 13 in the sprint, sometimes even more. This tells that the project had high technical complexity - which it objectively did, so it seemed the math checks out.
3. It would scare new comers which normally are very frightened of values bigger than 3 story points, some would even want to split a 5 in 2+3 but in the end everyone was pretty happy as it gave them a good framework for time management.
4. Because we also took time into consideration, the long term estimations were fairly accurate.
5. There would still be outliers that would get the wrong estimation from the start, these would be added to the examples list for future comparison.

Answer 8

Are story points really a good measure for velocity?

According to a definition available from Scrum Inc.:

Velocity is a measure of the amount of work a Team can tackle during a single Sprint and is the key metric in Scrum. Velocity is calculated at the end of the Sprint by totaling the Points for all fully completed User Stories.

...it seems story points is same with what in physics it is distance for speed that is computed dividing distance by time. It could be called differently than story points if that is convenient though the concept stays the same it's a unit measure of a distance. The strange part is trying to get from story points to time because it is same with trying to transform distance to time, it cannot be done, they are two different concepts. To get to time it should be starting from velocity that is story points per spring length expressed in time's unit of measure and calibrate the result by the coeficient of planed and delivered from team's historical data, when historical data is available otherwise skip calibration, and remember that the result is just an estimate, use it accordingly.

Are story points really a good measure for velocity?

Indeed they are together with sprint dimension. Solely no, they are not since they are intended to measure a component of velocity, the complexity to be delivered, that together with sprint size measures velocity.

Answer 9

It's a bit of a tautological question.

Story points are the units of measurement for sprint velocity, by definition of velocity being the amount of story points that you process in a sprint; in the same sense that metres are a measure of length. That says nothing about how reliable your measurement is, it just means that one is the numerical expression of the other.

As is the case with all measurements, the outcome is only as reliable as the original measurement is. If you are very accurate, precise and consistent about the story points you assign to the work, so that it is an accurate expression of the workload required to achieve the task, then the velocity measured by the story points will be relatively indicative of how much the team achieved in a given sprint.

But that is a very, very big if.

The question is more what you're trying to achieve by using velocity in the first place.

Are you trying to communicate to the stakeholders what the expected delivery date is for a particular features?
If so, sprint goals should be an adequate communication tool here. If any of the goals has not been met at the end of the sprint, you can address that, without having needed a numerical expression of the size of each task.

Are you trying to track employee productivity and find people who are willfully underperforming?

At first blush, story points seem like a very good metric for this, but who is making the estimates in the first place? The developers. Therefore, it's a biased measure because the developers can pad their estimates to hide any wilful drop in productivity. This only really works if you assume that the majority of your developers are giving you honest and accurate estimates, and you're trying to find the outliers.

Secondly, it's going to give you a lot of false positives whenever the work has genuinely turned out to be more complex than was initially understood. Odds are, you're going to accuse people of underperforming, or get them to be defensive before you even bring it up, more so than you're actually going to catch people who are wilfully being unproductive.

Are you trying to detect subconscious work delays and give your developers a sense of progress?
While story points can be used as a measure of sprint progress, a visual medium like your sprint board and swim lanes do the same job without needing you to come up with precise numbers. Within a given sprint, the developer should have a reasonable feeling about which tasks are big and which tasks are small, meaning that they can infer progress based on seeing the board.

Are you comparing the estimates to the actual results, in order to retroactively understand where your technical analysis was under/overcooked?

I actually think that in this last case, story points are a very good measure. Because what they measure is the difference between the initial expectation of how complex the work is, and the actual time worked on the task; which indicates that your developers may be over or underestimating things.

However, the problem is that once you start doing this, some manager will eventually start using this as a metric of productivity. This in turn causes the developers to become defensive, as their estimates are being used against them if ever they end up being wrong; at which point the system falls apart again.

---

For all the above reasons, if there is a healthy working relationship between the developers and the stakeholders; I actually favour a kanban approach, where you keep developers honest not by putting numbers on a piece of paper, but by having them openly communicate with stakeholders.
However, in order for this to work, you need stakeholders who listen to technical concerns that developers face, and you need developers who understand that product is there to deliver business value, not just to build an elegant algorithm.

In my years as a tech/team lead, I have found it to be more efficient to get both sides to understand each other and have a working relationship, rather than trying to boil it down to some numbers on a piece of paper and some blind expectations based on what those numbers express.

In my role, I continually work towards helping both sides understand the other. I listen to my developers' technical concerns, but I also point out when the work they propose is a low business value proposition. Similarly, when the stakeholders push a deadline that I find unrealistic or that gives me pause for technical debt concerns, I point out the long-term ramifications of their short-term deadline, and I try to get them to agree to at least fix the technical debt after the initial delivery. So far, I found that most people are reasonable once you actually convey the core of the issue.

But I've also worked as a consultant developer with stakeholders/management who are wilfully deaf to any technical considerations/pleas, instead micromanaging everything. And when those lines of communication simply aren't possible, yeah then you just put numbers on a piece of paper and work towards those, so that they can leave you alone while you are meeting your numbers.
But the sad reality is that the numbers will be skewed in the developers favour, because the developers are making the estimates in the first place and clearly this workplace is not treating their developers with respect for their technical expertise.

---

Can story points be a valuable measure of something? Yes. But you have to make sure that it's the right measurement for what you're trying to measure, and you have to consider whether there are better options than putting numbers to paper and have people work with those.