1

KLOC has been - I think justly - criticized as too inaccurate to express size when comparing several projects.
In only one source I found a calculation that uses Story Points and thus allow to compare small/large projects with respect to their Number Of Bugs per 100 SP.
Providing that the teams all have a common definition/reference story, is there something I am missing that prevents using story points in this sense? I mean, I understand the productivity of teams will vary, but the estimates of "amount" (in terms of relative effort) should not vary that much, in my opinion. To me it seems much better than KLOC.

  • I'm confused. What is it that you are trying to measure? What is your goal? – Daniel Oct 8 '18 at 15:47
  • 1
    @Daniel Defect density among several projects. Thus I need to normalize the size. – John V Oct 8 '18 at 15:49
3

Story Points are a "good enough" measurement. That is, you can use them to approximate the size of a block of work compared to other blocks of work. As such, any measurement of X per n Story Points will be inherently imprecise. However, they will probably be "good enough" for most purposes.

Even setting aside the question of comparing across teams, this measurement can show you trends if certain efforts increase or decrease your defect density. If your goal is to reduce defect density and the team will be identifying actions in their retrospectives to attempt to make progress in this area, this metric will be valuable.

So, as far as comparing defect density across teams, there are two things to consider. First, why does comparing it across teams help you? What will you do differently if one team is drastically different from another? or is slightly different from another? The answer to this will tell you to what degree you can rely on the metric.

Let's say you apply this metric and team A has a defect rate of 20 defects per 100 story points. Team B has a rate of 2 per 100 story points. Odds are that the story point scale on those teams can't be different enough to not tell the same story - that team B's process is creating far fewer defects than team A. On the flip side, if Team A has 8 defects per 100SP and Team B has 9 defects per 100SP, they are, again, about the same. Even if their SP scales are different, maybe if you could effectively normalize, the numbers would be 7 and 10, but still the same story - they're about the same.

So let's say you were trying to measure something that required you to make the distinction. This is going to be problematic. The reason is that having a common reference story is nearly impossible. Reference stories work because a team did the work before and they are familiar with what it took to do that work. They then make a guess on other stories in relation to that work. To have a common reference story, you would need something that the members of every team were involved in. Maybe you can get that on two teams with a particularly big story, but it's just impractical. Furthermore, even if you could find one, there is no reason to believe that the team's guesses about future work would be consistent with each other. If you needed the precision and accuracy that would require you to normalize points across teams, story points are not the right measurement.

  • Thank you. I certainly do not need a precise measurement rather than an indication about quality. But story points being about effort (I just read several articles explaining that), I am not sure that even in this rough use the results would make sense. – John V Oct 8 '18 at 16:17
  • 1
    I genuinely don't know. That depends, I think, on the nature of your work and what's happening in your teams. It may show you what you want to know by virtue of the fact that effort put in by the team is (hopefully) correlated to product created. However, it might not be. KLOC may also be perfectly useful. The problem with that is if one team rights really dense code and the other team doesn't. Code reviews can tell you that pretty fast though. – Daniel Oct 8 '18 at 16:30
  • some teams use the fibonacci scale, some teams use 8 per day. you could easily see a 10x difference in SP estimates – Ewan Oct 10 '18 at 21:38
6

Assuming a common reference of story point size that seems reasonable. However generally story points are a per team concept and can't be used across teams like that.

  • Unless you have a common reference story so that teams can understand it and estimate relatively to that story. Then, I do not see anything preventing us from using that across the teams. – John V Oct 8 '18 at 15:07
  • 1
    It's somewhat difficult to calibrate story points across teams and it's also difficult to keep them in sync. It can also lead to other negative usages like using story point totals to compare teams. – Sign Oct 8 '18 at 15:36
  • I think comparing totals across teams is different. Our reasons are simply to express the size. I mean, story points are about effort. If two teams agree that a certain story is 1 SP (even though for team A it means 2 days and for team B it means 4 days), the total size should be the same. – John V Oct 8 '18 at 15:45
  • @JohnV - another thing to be wary of is that Story Points are an estimate. Actual story complexity will vary. And it's also fairly common that a single feature or bit of code is the result of a few User Stories over time. If a bug is found, which story gets it? – Telastyn Oct 8 '18 at 16:42

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.