Should non-priority technical debt tickets be pruned from backlog?

Question

I regularly review the technical debt tickets from my backlog, to prioritize them and remove those which are no longer relevant (fixed by some other development, obsolete...) Among those with high priority, we take 2 or 3 in each sprint, and this way our codebase is healthy for the moment.

The problem is that all those tickets that are still relevant but have not been prioritized, represent a big part of the backlog (50%), and my PO insists that they should be removed, the same way that he deletes regular Story tickets that he knows won't be prioritized in the next semester, in order to have a "lean and healthy backlog".

I acknowledge the fact that with our current "tech debt velocity", we won't be able to take most of them in the next semester, but it frightens me to delete tickets that are pointing to spots in our code that may rot if not fixed, lending further developments more difficult (well, you all know the point of tech debt and why it is important).

So my question is: should I prune the tech debt tickets with lower priority?

Answer 1

I would say that there isn't a single best answer for this problem. You've got a few overall strategies, you would probably be best doing all of them. But the information should never be deleted, but it can be transformed.

Keep them in the issue tracker with the rest of the project

For high priority or fresh debt I would keep this strategy, once tickets start getting older or are deemed very low priority I would move on to the next two strategies

Pros:

1. Keeps the context to one area making it easier to manage the technical debt
2. Gives them high visibility

Cons:

1. Religious management and filtering of these will become important (it may be a large time commitment)
2. If there's a very large number, the small unimportant ones may drown out more important ones

Keep them in the issue tracker separated from the project

Move older tickets or less important tickets into a separate project and pull them into the main project as required for working on them.

Pros:

1. Keeps the backlog clean and sane
2. Makes it easy to determine what is high priority vs low priority by changing their visibility

Cons:

1. This second project becomes a dumping ground and can easily fill up, it will still need to be managed
2. A lot of these tickets will be moved here and never look at again, begging the question if they'll ever get done?

Move the information into code

By using a special commenting style, ie // TECHDEBT: .... you can mark the areas required for clean up directly in your code base instead and these can be fixed up when there is spare time in the project

Pros:

1. The information about the debt is located where the debt roughly is
2. Doesn't pollute the issue tracker
3. Can create a good culture around identifying debt
4. If it's apparent that there is a lot of notes about technical debt in a particular area, it makes it easier for a developer to flag that something needs to be done (the collection is more important than the individual)
5. A developer may fix this when otherwise working on the feature (thanks Heinzi for adding this)
6. When cleanup of the debt has been performed the note should naturally be removed (thanks Steve for adding this)

Cons:

1. Can create noise in the code itself
2. May still never get fixed
3. From a project management perspective, the work may never get prioritised, so it may never get fixed
4. The information is now split across two different places, making it harder to find and more likely to file a duplicate ticket

Answer 2

You are considering deleting the records of genuine problems with the codebase because the product owner wants a shorter backlog?

For me, the only reason to delete (close) an item in the backlog is because you decide it will never be implemented, not because it won't be implemented for a while. Also, in an agile environment, priorities may change quickly and the backlog can be re-ordered. If you have trimmed the list only to what you can do in the near future, you lose the ability to bring lower priority items up the list.

Maybe you should re-assess the tech debt issues if they represent such a large proportion of the backlog; you might be able to close a proportion of them as "won't do".

I am not sure what is concerning the PO to be honest. A healthy backlog contains a mix of items with lower priorities naturally floating down at the bottom. If it's really a problem, just filter the backlog or even create a second list (still logically a single backlog but split into 2 lists for mangeability). Maybe the PO is already doing something like this; I doubt he is actually deleting stories on the basis that they are not part of the current commitment?

Answer 3

I agree with Dan Saunders's answer, but I'm going to go one step further.

I agree that the only reason to close a request for a new feature or a modification to an existing feature is if the change will never be implemented. However, for bugs and technical debt, the only way to close the issue is to "fix" it. There are different ways to fix such issues. One would be resolving it - fixing the bug or refactoring the tech debt away. The other would be removing it - deprecating the feature that the bug or the tech debt exists in from the system. For bugs, reports that don't represent bugs (that is, the actual result of the execution steps is indeed the expected result, even if it's not that way to the user) can also be closed. Still, it may be worth understanding why the user thought it was a bug and correcting it at a root cause.

The most significant factor for keeping all known bugs and technical debt in an issue tracker is visibility and transparency. Users can run across bugs as the system is changed, and changes may increase the likelihood or impact of a bug to the intended operation of the system. Likewise, developers may run across technical debt when working with impacted functionality. If you can track bugs and technical debt into parts of the system, you can make additional work to resolve them known to stakeholders earlier and build it into the plan to develop and release the changes. By keeping all of the known work in a single tool, you can let all of the stakeholders make informed decisions.

Backlog size was a much bigger problem before electronic tooling that comes with the ability to search and filter records. Today, especially in a more remote and highly distributed world, there is little reason for anyone to be using index cards and sticky notes. We should no longer be constrained by physical space on a wall. Electronic tools range from lightweight task tracking to heavyweight project management. It's pretty straight-forward to set up filters that hide these lower priority issues without closing or deleting them in most tools that I've worked with, yet making them visible to the right people at the right time.

Answer 4

Does your bug tracking software allow you to mark a bug according to its priority? If so, then you can ask your boss if it's OK to create a "lean and healthy backlog" by simply viewing the database with filtering based on priority.

Is the bug tracker publicly visible? If so, then maybe the problem is purely marketing. Your competitor may be telling prospective customers that you have thousands of bugs, so your product is obviously bad. Maybe point out that users like you more because you're open about bugs, rather than hiding bugs -- which costs users time and effort because they can't get information.

If you're contemplating deleting a lot of bug reports, then one thing to ask yourself is this. Before I deleted them, would I consider it worthwhile to copy them all somewhere so that the information was preserved? If so, then this is a problem of perception or database design, not a problem that should be solved by a database purge. You don't want to maintain two databases, a big one and a small one. That's just an inefficient way of maintaining a single database with a binary priority field in it.

Answer 5

Yes, you likely should prune your backlog of un-prioritized technical debt, as you won't fix it (similar to YAGNI).

Think of fixing technical debt as an investment opportunity. Something developers can invest time in, and hope that it pays off in the future, and hope that it pays at a higher rate of return than other things you could have invested time in.

Many times we invest days or weeks into "fixing" technical debt, with no demonstrable benefits. Or the rate of return on the investment is low, and we miss out on other opportunities that would benefit the team/code more. Often times, we end up cleaning up regions of code that don't get modified again for years, or refactor code that needs to be re-written weeks later when we have to accommodate a new feature that moves the product in a direction we didn't anticipate.

Furthermore, keeping items in a backlog, has cost a non-zero cost. Either you ignore the items in your backlog, and it keeps growing so large as to not be groom-able. Or, you do a good job and groom it weekly team, in which case you're wasting minutes of everyone's time talking about it each week.

The tough reality is, you need to write good code the first time. As soon as you accumulate technical debt, you're never likely to have time to directly tackle it. You may be able to chip away at it, by making sure every modification you make to the code, cleans the regions it touches (Robert C. Martin's "Clean Code"). Perhaps one of the most important parts of good software design, is to be able to compartmentalize these regions of technical debt within a code base.

That being said, never confuse technical debt with bugs (and possible bugs). Bugs must be prioritized, investigated, and fixed or documented as features (#5 of the Joel Test). The lack of testing is not technical debt, its just not yet discovered bugs.

Answer 6

I expected to see someone reference Joel Spolsky's old article.

Since you say that these tickets haven't been prioritized it sounds like no one's looking at them, and in that case it's ok, leave them, but to quote Joel:

Every minute you spent writing down, designing, thinking about, or discussing features that are never going to get implemented is just time wasted.

If you find yourself spending time reading through these tickets with any frequency, then they're in the way -- get rid of them.

Answer 7

It makes no sense to have a "lean and healthy" backlog for a codebase which isn't equally "lean and healthy". The backlog's job is to reflect the state of what must/could/should/would (not) change in the product. A good backlog isn't pretty, it's representative and up-to-date. It informs all its users on the oughts and ought nots.

Generally, tech debt items in the backlog will be added by developers for the PO to read, thus making information flow the "wrong" way unlike other items. Maybe your PO doesn't want to read them? It doesn't sound like you have an acute tech debt problem though.

The thing which is supposed to be pretty is the list of closed issues.

You should probably investigate why your PO wants the backlog to be pretty. There might be a misunderstanding somewhere. Perhaps the PO is reporting something to a higher level manager using the backlog, which is probably wrong.

Answer 8

You and your PO have conflicting uses for the backlog, which need to be reconciled.

You want a list of every good idea that anyone has ever had. Perhaps above a certain threshold of good, since you're calling it "tech debt" rather than just "potential improvements".

Your PO wants something that you can look through on a regular basis to choose tickets: a realistic and manageable list of what actually needs doing. Your increasing tech debt is not quite the same as "low priority", though, because to the PO a low priority ticket in the backlog means something that's still worth periodically reviewing and perhaps raising its priority. Your PO strongly believes that for these tickets it's not worth doing even that.

You are also concerned that something in the backlog might secretly be really important, since it points to an area that might rot. If you had time to do all of these tickets, then maybe you could deal with every potential point of failure that you've identified. But you do not have time, which is why you prioritize. The things that are persistently not prioritized have a low(er) chance of causing serious failure, and hopefully a genuinely low chance of causing failures that will not be caught in testing in the event that they do happen.

So, by not prioritizing them you have decided not to do them. "We're not going to do this any time soon" means, "we're not going to do this ever, unless it gets worse". It doesn't literally mean that, but be honest with yourself: that's what happens.

So, own that decision. Other than maybe a small sample pulled out for one reason or another (like, it's mid-afternoon, you're going on vacation, and there are no small tickets left in the sprint) you're not going to do these tickets any time soon. Unless you link them to higher-priority symptoms you very likely aren't going to do them ever. You should not be grooming these nuisances every sprint.

If your backlog doesn't satisfy both needs then that is an issue with your issue-tracking system, that needs to be solved in a way your issue-tracking system supports:

• Tag them "slush pile" and exclude the tag from the view used for the majority of your planning. Then include it back in when someone joins the team and you're filtering the entire backlog for "tickets that a novice could tackle before lunch" to get them started.

• Close them with a "won't fix" resolution, so that they're still searchable if they contain any useful insights how to address the issue identified. OK, so it's embarrassing if you eventually re-open or duplicate a "won't fix" ticket and actually fix it, because that means you were "wrong" when you predicted you wouldn't. But should it be more embarrassing than never fixing hordes of tickets you predicted you would fix?

• Move them to another "code improvements" project. You can occasionally look at this to assess whether it's time to focus a sprint on some developer quality-of-life issues that will improve future efficiency, or on some ticking time-bomb issues (like Y2K, or the tiresome inevitability that for every dependency you have, some day upstream will release a critical security fix which is not back-ported to the version you're using).

• Define more priority levels, hide the lowest level from the default backlog view, and make a rule that you do not waste your time looking at those tickets unless you run into them for some reason other than, "they're on the list".

In all of these approaches, the basic idea is that you keep them on your backlog (used for recording everything you know about the state of the product), and remove them from your PO's backlog (used for sprint planning).

Anyway I suspect the root issue here is that you take "backlog" to mean, "every issue we've identified and not fixed", whereas the PO takes "backlog" to mean "todo list that I actually have to think about". Those are both worth having, but one is a filtered view of the other. Feel free to argue with your PO which of those two things deserves the name "backlog" in a well-regulated Scrum methodology. But that might be one for the pub rather than work time.

If you move this junk out of sight and the PO still objects to it, then it's possible that there's some political reason for rigging some "code healthiness" metric. Then maybe you actually need to drill into whether it's right or wrong to include this stuff in that metric. Beware Goodhart's law: "When a measure becomes a target, it ceases to be a good measure" (or, "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes"). Especially beware that knowing the law does not make your own KPIs immune to it. This is why you don't pay bonuses for increased velocity (and if you do, what you get is ticket point inflation). You shouldn't pay bonuses for hiding bugs, either.

Answer 9

A technical debt must be paid or forgiven. The reasons for the latter can be anything from implementation complexity, revised importance of said feature, movement of feature to another version(not even a very strong reason), etc.

Of all the reasons a debt should be forgiven, too many post-it notes shouldn't be one of them. If it means having a backlog of your current backlog, that makes more sense than deleting a report that might eventually find its way back to the backlog when history repeats itself.

You can deescalate, but don't remove until you know it's never going back on that board.

Answer 10

Remove them.

If developer hits the smelly code, he can create a tech debt ticket once again.