Can story points be used to describe size of several projects estimated by several teams?

Question

KLOC has been - I think justly - criticized as too inaccurate to express size when comparing several projects.
In only one source I found a calculation that uses Story Points and thus allow to compare small/large projects with respect to their Number Of Bugs per 100 SP.
Providing that the teams all have a common definition/reference story, is there something I am missing that prevents using story points in this sense? I mean, I understand the productivity of teams will vary, but the estimates of "amount" (in terms of relative effort) should not vary that much, in my opinion. To me it seems much better than KLOC.

Answer 1

Story Points are a "good enough" measurement. That is, you can use them to approximate the size of a block of work compared to other blocks of work. As such, any measurement of X per n Story Points will be inherently imprecise. However, they will probably be "good enough" for most purposes.

Even setting aside the question of comparing across teams, this measurement can show you trends if certain efforts increase or decrease your defect density. If your goal is to reduce defect density and the team will be identifying actions in their retrospectives to attempt to make progress in this area, this metric will be valuable.

So, as far as comparing defect density across teams, there are two things to consider. First, why does comparing it across teams help you? What will you do differently if one team is drastically different from another? or is slightly different from another? The answer to this will tell you to what degree you can rely on the metric.

Let's say you apply this metric and team A has a defect rate of 20 defects per 100 story points. Team B has a rate of 2 per 100 story points. Odds are that the story point scale on those teams can't be different enough to not tell the same story - that team B's process is creating far fewer defects than team A. On the flip side, if Team A has 8 defects per 100SP and Team B has 9 defects per 100SP, they are, again, about the same. Even if their SP scales are different, maybe if you could effectively normalize, the numbers would be 7 and 10, but still the same story - they're about the same.

So let's say you were trying to measure something that required you to make the distinction. This is going to be problematic. The reason is that having a common reference story is nearly impossible. Reference stories work because a team did the work before and they are familiar with what it took to do that work. They then make a guess on other stories in relation to that work. To have a common reference story, you would need something that the members of every team were involved in. Maybe you can get that on two teams with a particularly big story, but it's just impractical. Furthermore, even if you could find one, there is no reason to believe that the team's guesses about future work would be consistent with each other. If you needed the precision and accuracy that would require you to normalize points across teams, story points are not the right measurement.

Answer 2

Assuming a common reference of story point size that seems reasonable. However generally story points are a per team concept and can't be used across teams like that.