Handling "related" work within a single agile work item

Question

I'm on a project team of 4 devs, myself included. We've been having a long discussion on how to handle extra work that comes up in the course of a single work item.

This extra work is usually things that are slightly related to the task, but not always necessary to accomplish the goal of the item (that may be an opinion). Examples include but are not limited to:

• refactoring of the code changed by the work item
• refactoring code neighboring the code changed by the item
• re-architecting the larger code area around the ticket. For example if an item has you changing a single function, you realize the entire class now could be redone to better accommodate this change.
• improving the UI on a form you just modified

When this extra work is small we don't mind. The problem is when this extra work causes a substantial extension of the item beyond the original feature point estimation. Sometimes a 5 point item will actually take 13 points of time. In one case we had a 13 point item that in retrospect could have been 80 points or more.

There are two options going around in our discussion for how to handle this.

1. We can accept the extra work in the same work item, and write it off as a mis-estimation. Arguments for this have included:

   • We plan for "padding" at the end of the sprint to account for this sort of thing.
   • Always leave the code in better shape than you found it. Don't check in half-assed work.
   • If we leave refactoring for later, it's hard to schedule and may never get done.
   • You are in the best mental "context" to handle this work now, since you're waist deep in the code already. Better to get it out of the way now and be more efficient than to lose that context when you come back later.

2. We draw a line for the current work item, and say that the extra work goes into a separate ticket. Arguments include:

   • Having a separate ticket allows for a new estimation, so we aren't lying to ourselves about how many points things really are, or having to admit that all of our estimations are terrible.
   • The sprint "padding" is meant for unexpected technical challenges that are direct barriers to completing the ticket requirements. It is not intended for side items that are just "nice-to-haves".
   • If you want to schedule refactoring, just put it at the top of the backlog.
   • There is no way for us to properly account for this stuff in an estimation, since it seems somewhat arbitrary when it comes up. A code reviewer might say "those UI controls (which you actually didn't modify in this work item) are a bit confusing, can you fix that too?" which is like an hour, but they might say "Well if this control now inherits from the same base class as the others, why don't you move all of this (hundreds of lines of) code into the base and rewire all this stuff, the cascading changes, etc.?" And that takes a week.
   • It "contaminates the crime scene" by adding unrelated work into the ticket, making our original feature point estimates meaningless.
   • In some cases, the extra work postpones a check-in, causing blocking between devs.

Some of us are now saying that we should decide some cut off, like if the additional stuff is less than 2 FP, it goes in the same ticket, if it's more, make it a new ticket.

Since we're only a few months into using Agile, what's the opinion of all the more seasoned Agile veterans around here on how to handle this?

Answer 1

Agile planning and user stories are focused around providing value and transparency to the project stakeholders and users of the software. This is a good thing, but it doesn't and should not supersede, include or demote the importance of good architectural guidelines, design stewardship, good development practices and maintaining technical debt.

Agile doesn't do the latter well because it was not intended as an answer to these mostly technical problems and issues.

Knowing that I highly disagree that refactoring tasks, technical debt handling, and design work should account for seperate user stories in a given sprint. These are merely tasks that a developer might undertake to help meet the user story for that sprint.

Remember that a Task is any unit of assignable work that helps move a given user story to completion within the architectural guidelines and maintaining the good design and devlopment practices of the software as a whole.

This is why the hours estimation should be on tasks and not on user stories. This is also why some tasks are critical to the completion of multiple user stories.

Answer 2

Red, Green, Refactor. That is the scope of a single work item. Write a failing test suite covering the scope of change, code the minimum required amount to pass your test, then refactor to meet coding standards while still passing the tests. All three of these steps are required; you cannot code the solution until you have defined the problem, if you refactor as you write the line of code you will invariably violate YAGNI, but if you do not come in behind yourself and refactor after passing the tests, by definition you incur technical debt that you will eventually have to repay.

Given this definition, and that it was followed, a 5-pointer that turns out to be a 13-pointer was a mis-estimation. What you would consider refactoring work was probably more like restructuring; you had to reorganize a pretty significant area of the codebase in order for the new functionality to be included in an understandable, maintainable way. That usually indicates a failure of the team to understand the general future path of development, leading to something being implemented very simply in a previous iteration when it would eventually be required to be very SOLID. Better communication between the BAs and PM, who know what's further down in the backlog, and the development team who are generally focused on the current sprint, can mitigate this. Alternately, this story exposed a large amount of technical debt incurred in past development, and it simply caught up with the team. Better code review processes, in addition to better conceptual knowledge of design patterns and of the general future path of the project, can help reduce such occurrences.

One thing to keep in mind is that refactoring is "non-ideal" work. In Agile SCRUM, tasks are estimated in "ideal hours"; that is, the number of hours spent heads-down writing brand-new code that never existed and furthers the feature base of the project. An 8-hour developer-day might realistically only have 5 ideal hours; sometimes you can count on 6, especially in the "stretch" of a project where the team's really humming along. Refactoring, or going back and making changes that do not affect the functionality of the project but that improve the codebase in other ways, is non-ideal work, as is planning, design, communication, review, breaks, or technical downtime. Other than technical downtime, non-ideal work is important, but does not make progress in the eyes of the product owner.

So, provided that refactoring doesn't double the actual hours spent, a certain amount of refactoring work is to be expected when you estimate in ideal hours. Let's say, because I don't know exactly how your team's point scale is calibrated, that a 5-pointer is equivalent to one ideal developer-week, or about 25 ideal hours. That 5, which turned into a 13 (more than two developer-weeks by the same scale), is cause for some retrospection about what caused the complexity to balloon. Perhaps the codebase didn't need as much refactoring as was actually done, perhaps a large amount of technical debt had piled up unbeknownst to the team that had to be resolved to make the new features work, or perhaps this was an honest mis-estimation of the amount of new work required (perhaps the team thought extending an existing part of the domain would be enough, when in reality several new domain classes and related schema changes were necessary).

In an alternate universe, let's imagine that a 5 as estimated in ideal hours became a 7 (~35 hours) based on actual hours, because you needed 10 hours of additional refactoring to put the new code and some previous bits into a properly-patterned design architecture. In that case, the extra is within the "gap" between ideal and total hours during the number of developer-days the story was supposed to take. So, as a project manager, I'd call a 5 that became a 7 a reasonable estimate and move on.

Answer 3

Story points are an estimate of the relative complexity of a given user story. It sounds like you're using story points to say this will take X man days/hours. Instead, strive for two goals

1. Break down stories until they are in a consistent range (3, 5, or 8 points)
2. Assume that the story includes any refactoring necessary

Over time this will give you a baseline for velocity. Every 5 point story won't take the same amount of time as the others but the average velocity per sprint (how many story points the team can complete) will be consistent.

Worrying about how much time a specific story will take is counter productive. The estimates only average out over consistently sized stories in volume (I.E. one 5 pointer might take a little longer due to refactoring but you pick up the benefit of that effort on a related one).

Answer 4

There should be a relative cut-off of how much is within the original work item and how much is something else. User stories are starting points for discussions and thus there may be all kinds of scope elements to nail down during a sprint while working on a story.

There may be times where in a sprint planning a story may get additional criteria added to it in an effort to avoid "scope creep" which can happen where a user wants a new form and then 101 changes to that form which isn't realistic to get done in a 2 week sprint sometimes.

A flip side to keep in mind is how much value is being gained from this extra work. There may be tons of possible refactorings that could be done but how much benefit is there to anyone for all that work? This is where there have to be some guidelines to help keep the team working well but not get lost in trying to make the code perfect.