46

Disclaimer: I don't expect zero tech debt. In this post, technical debt problem refers to severity that has been causing negative impact, say productivity.

Recently I was thinking to build a tool to automatically generate tech debt report from issue tracker - introduction rate vs cleanup rate over the time. Apart from the total, there'll also be numbers broken down by project team and by manager, so that managers could easily get insight on current tech debt level, without delving into issue tracker and details (such tool might already exists, I need to research to avoid reinventing wheel).

Motivation wise, tech debts have been snowballing for years. Whenever developers increase project estimate to include tech debt clean up, more often they will be asked to remove those numbers from estimate, so refactoring/clean up works usually ends up indefinitely postponed. I hope the periodic report will help to improve tech debt management issue.

However, on second thought, I wonder will increasing visibility of tech debt level really helps to raise priority. Generally, is tech debt issue an org culture issue or just lack of tool/insight? I supposed there's no universal answer, I wonder which is the more common cause. What's your experience?

--- Update 2/28

Clarification: I believe most management are intelligent enough to realise there's impact, especially after teammates reported pain in terms of project productivity. My gut feeling is that, they don't have a concrete picture about how serious problem is. My idea is to help management to gain clearer picture, via two steps:

  1. Have techdebts logged, and have their impact tracked. (there are challenges, but that's beyond scope of this question)
  2. Have a report for introduction rate vs cleanup rate (there could be further breakdown by high/low impact).

My curiosity comes from, will these efforts help or are they just waste of time, generally speaking (not specific within my org) - hence the question what's your experience. If it's org culture issue, then most likely these efforts won't help much.

  • 2
    Instead of analysing the bug tracker, analyse the source code repository. You can see how complex the software is, how often the complex parts are being edited, and how many and who are across the contents of each file. Place that in some pretty graphics and you can highlight why 5 years ago code change was so much easier than now. – Kain0_0 Feb 26 at 5:10
  • 24
    This seems to assume that the technical debt is known and recorded - but its usually like an iceberg, 90% of it is out of sight and only becomes known when it causes a problem. – afaulconbridge Feb 26 at 14:41
  • 12
    "tech debt clean up, more often they will be asked to remove those numbers from estimate" That is a red flag right off the bat. It's a culture issue. It's a trust issue. It's a management and leadership issue. – Brandon Feb 26 at 18:17
  • 3
    Cleanup rates can be gamed. Google has over 50,000 improperly closed bug reports in Android alone. – StackOverthrow Feb 26 at 20:41
  • 8
    My rep is too low for an answer but I feel that flaters answer, good as it is, is too verbose to really bring the crux of the matter home: Management loves cowboys The more you prioritize short term gains over long term viability, the more you will be rewarded. That senior dev that writes spaghetti code and dumps the fallout on your desk? He's gonna get a promotion, cause he "understands the buisness". – Douwe Feb 27 at 10:32
95

Anecdotally

I am a consultant developer. I have been hired on several occasions specifically to "fix the development issues". Some customers are aware of issues in their development process, whereas others only see it as a bunch of bugs that need to be fixed without looking at the cause of the bugs (i.e. bad coding practices).

In my experience, one company that asked for help in fixing the development process was actually interested in taking steps to improve the development process. In other companies, their interest existed up until it required action from their side (e.g. reprimanding a developer who actively rolls back refactoring or improvements, or actually giving me access to the tools needed to set up a CI/CD pipeline).

Based on my experience, bad practice starts off as a developer deficiency. Not a willful one, but rather a matter of either inexperience or general corner-cutting attitude. Whatever the cause, these developers will show quick results due to not taking the time for due diligence such as testing, reviewing or refactoring.

Management will notice those quick results, and will over time come to expect this efficiency. They don't handle the fallout from bad practice (i.e. bugs) directly, but they do benefit from the shorter development times.

At this point, it becomes a feedback loop. Management communicates expected (quick) deadline. Developers are forced to cut corners to achieve it. The codebase degrades. The initial quick release turns into a maintenance cycle of unclear and erratic bugs, regressions, and general lack of readability. In order to cope, while keeping up with the continuing demand for quick results, developers are forced to cut corners in their bugfixes.

The cycle continues, while the quality and performance of the codebase is eroded, and also the good practice skills of developers erode and start being regarded as "needlessly" time consuming. If some developers stick to good practice and others don't, management will judge them based on how quickly they deliver - without observing the bugs or the causes of the bugs.
The good practice developers are deincentivized, the bad practice developers are incentivized. Over time, due to positive/negative feedback from management, the bad practice developers take on a more leading role than the good practice developers, and the bad practice becomes the law of the land.

Speaking from the experience of a company whose main workforce was external consultants, the good practice devs simply leave or become disenfranchised bad practice devs. The (initial) bad practice devs stick around. This perpetuates the imbalance of the bad practice devs having seniority over the good practice devs.

At this point, bad practice has become an endemic company culture. It is reinforced from all sides (including the sales department in case of dev companies), and any good practice suggestion that pops up is often drowned out by the popular support for bad practice, combined with management's intolerance for longer deadlines.

This devolution is something I've observed with at least three different companies. The same events and general work climate pervaded through all three companies.


The monkeys and the ladder

Whenever I talk about detrimental company culture, which often manifests as a "this is how we've always done it" attitude, I am reminded of the parable of the monkeys and the ladder.

enter image description here

Suppose I had turned off the shower around the time of picture 4. The monkeys could have gone up that ladder without any repercussions, but their "company culture" prevented it from happening, based on what is now an outdated idea (since the shower is no longer active).

This parable touches exactly on the erosion of good practice that takes place. Popular but misguided support for bad practice inhibits anyone who tries to make a change for the better by introducing good practice.

The issue isn't with social checks and balances. The same principle is used in other companies to keep up the good practice and squash any bad practice suggestions.

The issue is with the blind acceptance of "things are done this way" without ever being able to re-evaluate. When it reaches that stage, the behavior is a company culture.

Answering your questions

Generally, is tech debt issue an org culture issue or just lack of tool/insight?

It depends what stage of the process you are on. In the beginning, it's a lack of insight and/or tooling. But when combined with management that looks only at results and not ongoing issues and thus wrongly (possibly unknowingly) incentivizes the bad practice, it becomes a feedback loop and over time turns into company culture.

  • 17
    Ironically to the "blind acceptance" moral, the five monkeys experiment is completely made up – BlueRaja - Danny Pflughoeft Feb 26 at 18:38
  • 3
    Well said. The short explanation is that fixing technical debt (and thus preventing future bugs) has no business value while fixing immediate bugs has. So basically improving a software product is destroying business. Also see Iron Gremlin's excellent answer. We should never talk about TD in public, it is just awkward to outsiders and a very hard sale. Labeling an issue as TD in Jira is the most certain way to send it straight to the bottom of the backlog. Split it up and distribute it among popular issues instead and claim the time you need. – Martin Maat Feb 27 at 7:02
  • 7
    There is one insight missing from your answer: Corner-cutting is not always wrong! Sometimes high code quality (== low long term maintenance cost) is important. Sometimes quick delivery (low cost and short time to market) is a higher priority. Knowing which and communicating this to development is the managements responsibility. – Guran Feb 27 at 11:48
  • 3
    @shoan: As with all things, balances indeed need to be struck. However, it is often forgotten (in bad practice environments) that clean coding speeds up the delivery process over the long-term. The usual flawed reasoning pro-bad-practice (or should I say anti-more-good-practice) trades away long term benefits (maintainability and future efficiency) for short term benefits (getting to the next deadline). There are certainly cases of excessive good practice, but anecdotally I see many more cases of reasonable good practice unfairly being painted as if it were excessive. – Flater Feb 28 at 13:04
  • 2
    Just pitching my 2¢ to point out I agree with this answer 100%, and that it isn't specific to software engineering. I have seen this in busy Accident & Emergency departments in hospital contexts, for instance, where there is a pressure for each doctor to "see as many patients as possible". If one has seen 10 patients in their shift, and another 3, the second doctor is criticized, while the first one praised, even if the latter only had time to see 3 because they were dealing with 'technical debt' from the first person cutting corners and putting the patient at risk. – Tasos Papastylianou Feb 28 at 14:01
22

Usually, it is neither. Usually it is mainly a problem of communication.

What many fail to realize is that technical debt is not a big problem for a company, precicely like financial debt is not.

Interest, on the other hand, is. It would be irresponsible to make unneccessary installments on a loan with very low or zero interest.

When you talk about reducing technical debt, what you are actually asking to do is installments och that debt. Since management is unaware of the interest (ie cost) of that debt, this is not viewed as important.

What you should do is to highlight, not the work required to reduce technical debt, but the extra work required to deliver new features (anr/or fix bugs) due to this debt.

So (to take some easy examples):

Do not ask permission to set up a CI/CR pipeline. Do specify that every release takes an extra half hour per environment. Do not ask for automate testing. Do specify that you use two days of manual testing for things that could have been automated. Do not ask for time to refactor. Do specify those extra hours you spent searching through poorly structured code.

This way you raise issues that actually matter to management. And if management is smart they can even decide if it is actually worth fixing the problems or not. (Why fix tech debt in a product that won't be supported next year)

  • 2
    agree. 99% of "tech debt" is just "code not following the latest trend" You have to focus on idenitfing and solving actual problems – Ewan Feb 27 at 12:15
  • @Guran I like the do, do not advice you wrote. I have seen that pattern and observed most of the time, folks are hesitant to step up to the plate. If you have any insight on the nature of that hesitation, I would like to see you add more to your answer. – learning2learn Feb 27 at 16:40
  • 2
    @learning2learn Sorry, I don’t really understand what you’re after. Can you give an example? – Guran Feb 27 at 19:05
  • 2
    @learning2learn In my experience, TD may be a cause of underestimated effort, but rarely the no 1 factor. (As long as we're talking about TD in our own code and not external dependencies) Developers usually have a pretty good grasp about how easy/painful it is to add functionality in their own codebase. As for why people don't speak up... well "we have a TD problem" easily reads as "we wrote crap code" – Guran Feb 28 at 9:36
  • 1
    @shiouming I wish I had. Unfortunately all sane methods of improving code quality requires trust in the first place. Due to this, my first priority in any new environment is to establish trust. (By delivering features and being transparent) If trust cannot be established, everything spirals downwards. The subject of reestablishing trust is definitely a good subject for a separate SE question. (Bet there are several already) – Guran Mar 2 at 15:19
9

We already had such a report available in our issue tracker (Jira). When the introduction vs cleanup graph started getting out of hand, management indeed devoted more resources to fixing it. I think visibility definitely makes a difference in prioritization.

The main problem is, the visibility is all about the time between a defect being recorded and it being fixed. This resulted in more time being devoted to fixing recorded bugs. This resulted in fewer people being available to create new features. Developers feel stretched thin and rushed so they are skipping things that don't show up directly in reports, like refactoring and automated tests.

We're now in this weird situation where we are paying down debt faster, but also generating it faster. That goes to your culture question. Having priority to fix highly visible bugs is very different than having a culture of building quality in to begin with. The latter is much more difficult to cultivate because it is more difficult to measure and feels like more up front effort.

  • Hmm..., by any chance, are trivial/low impact debts given more attention than needed in your case? On the other hand, I believe the outcome your team's experiencing is definitely not unique, the larger the resource gap, the more serious it will be. It seems that we're just dealing with another form of resource issue. – shiouming Feb 26 at 6:16
  • 2
    How do you track technical debt in jira? I couldn't find anything in jira related to technical debt – BЈовић Feb 26 at 12:45
  • 1
    @BЈовић when you see concrete examples, filing a "fix-it" ticket with a known tag is usually enough to let people track stuff that needs fixed – Morgen Feb 26 at 17:42
  • 1
    If your paying it off faster but generating it faster doesn't that mean you're just moving it around? – Bloke Down The Pub Feb 26 at 19:29
  • Basically yes. Moving it from older code to newer code. – Karl Bielefeldt Feb 26 at 22:24
7

The SCRUM canonical answer is that the team owns the tech debt. The team builds the sprint backlog. The team thus controls the rate tech debt is reduced.

In practice, angry managers will appear if you actually do this and escort you out of the building :)

More seriously, it's important to have good communication with the PO about what tech debt is causing issues in production and which tech debt causes current feature implementation to be slow. That will help you drive resolution of critical tech debt without alienating the PO and have one strategy for your deliverables.

  • "which tech debt causes current feature implementation to be slow" That's a good point. – shiouming Feb 27 at 5:39
4

Whenever developers increase project estimate to include tech debt clean up, more often they will be asked to remove those numbers from estimate, so refactoring/clean up works usually ends up indefinitely postponed

Why does this happen?

Ask your management. Make sure they're aware of this phenomenon, and get their justifications for it.

Working on tech debt needs to be 'sold' as a feature. Your management needs to be able to apply a cost/benefit analysis to the effort necessary to resolve the debt/the value of resolving the debt.

That having been said -

The development team owns the code. If you need to refactor before you introduce X feature because the team, as a whole, understands that getting further out on that limb endangers the future of the project, then include that in your estimate, and tell them that's what needs to happen.

They don't get to tell you what needs to happen to implement a feature - They can't. It's not their job or their skillset, and they don't have enough knowledge to make a judgement anyway, and they know it. If you tell them it'll take X story points, and the team has consensus around that figure, they'll accept it on fiat.

Have standards, stick to those standards, and have faith in the significance of your own standards. This is part of your job, because the people feeding you features have no context to set those standards for you.

If you can't get consensus on your team regarding what refactor work needs to get done for a given feature (IE, whether or not it's up to aformentioned standards), that's either a cultural problem that can be addressed internally, or an indicator that this tech debt isn't actually accruing in areas that are as critical as they seem from your perspective - And you'll need to pursue that conversation at length with your team to find out which one it really is, and be willing to understand that it's probably at least a little of both.

  • "get their justifications for it" – shiouming Feb 27 at 5:52
  • "get their justifications for it" Most common reason given, either "Project deadline is fixed, we don't have more manpower to be assign to the project" or "We have many business projects to be deliver. We'll come back to work on tech debts in future whenever we have time" (we rarely have free time, of course). – shiouming Feb 27 at 5:58
  • @shiouming - That means you're failing to sell the importance of resolving technical debt to your management. They need to understand, clearly, what it means to your velocity in adding features to do work involving fixing things. Focus on communicating that doing X work will enable all future work involving Y topic to go faster. If they're still making the call that it's not worth it to do the tech debt, that's their job, and then it's just a question about whether or not you trust their judgement enough to keep working there. – Iron Gremlin Feb 27 at 20:16
  • I believe most management are intelligent enough to realise there's impact, especially after teammates reported pain in terms of project productivity. My guts feeling is that, they don't have a concrete picture about how serious problem is. My idea is to help management to gain clearer picture, via two steps: (1) have techdebts logged, and have their impact tracked (2) have a report for introduction vs cleanup rate. My doubt was that, will these efforts solve the issue or just a waste of time, generally speaking (not specific within my org). – shiouming Feb 28 at 2:16
  • 1
    @shiouming I've seen such an effort succeed (for certain values) multiple times. Standardizing how technical debt is communicated, and making sure it gets added to the backlog as user stories, goes a really long way towards helping management get a handle on proper prioritization. I would not consider the first step on your two step plan to be a "waste of time" on any project, rather, I would consider that to be best practice. The second step may or may not be of value, depending on org and politics. – Iron Gremlin Mar 2 at 19:19
2

XP solved this problem a couple of decades ago in a very simple way.

Customers, or product owners, or whatever you care to call them, cannot ever declare how long a story will take. They can only put stories in the order they'd like them to be done and, if they don't like certain estimates, negotiate with the developers to change stories to be cheaper.

The development team (which cannot change story priority, ever) is responsible for maintaining an appropriate level of technical debt, determining when it can be increased for business purposes and under what schedule it must be paid back, and so on. The time and effort necessary to handle this at any particular point is never shown to the customer, but merely built into the current estimates for stories.

In particular, you never have a story about refactoring or reducing technical debt or anything like that because the customer has full control over the scheduling of the stories, up to and including the ability to say, "I want you never to do that one." If you are telling the customer to make decisions about technical debt, you are abdicating your responsibility as a developer to ensure that the project can move forward at a predictable rate. (Do you seriously think that the customer can be better, or even as good, as you at understanding the future effects of technical debt in the codebase?)

So split your management of projects into techinical management (the "developer team," which cannot set story priorities) and product management (the product owner or "customer" in XP lingo) who cannot make technical decisions, including decisions about how much technical debt one should hold or how much time and effort one should spend at the moment paying back existing technical debt.

If for cultural or other reasons you do need to have someone making decisions about both of these, at least try to have them differentate these two roles and keep them independent, i.e., they're wearing the "developer hat" or "product owner hat," but never both at the same time. (This worked well for me when running my own company.)

1

I tend to be in favor of tracking technical debt in the issue tracking tool and tagging it appropriately. In Jira instances, that often means an issue type. In Github, a label. Other tools may have other methods. However, doing so requires awareness of the technical debt to begin with. I'd be concerned that any reports don't give an accurate picture of the state of technical debt in a product, but rather the state of known technical debt in the product. Encouraging people to evaluate and track technical debt in the issue tracking would be a process, and perhaps a cultural, change for the organization.

It's going to depend on how your organization views technical debt. Before working on a given module or component, the developer can query the issue tracker for known technical debt and this can help guide estimates. Knowing known deficiencies in test coverage or tight coupling or other things that are now seen as poor design decisions can give the necessary insights to help guide estimation and planning activities. However, the product and project managers can always push back on these estimates with deadlines and committed dates. It's a balancing act.

1

It sounds like a culture issue

Whenever developers increase project estimate to include tech debt clean up, more often they will be asked to remove those numbers from estimate, so refactoring/clean up works usually ends up indefinitely postponed.

To me this screams culture issue. It seems like management is aware of the problem (maybe not fully, but at least somewhat), and doesn't want to hear it. That something is more important to them than code quality right now. How is management incentivized? What counts as success for them? Stable, maintainable code? Or splashy new features, time to market, etc.? It sounds like it's the latter. In which case telling them the code stinks isn't going to help anything, because quality isn't how they define success. To them, it doesn't stink. So ultimately you need to first learn what your corporate dev culture is, before you can decide whether and if so how to tackle this problem.

  • 1
    "How is management incentivized? What counts as success for them? Stable, maintainable code? Or splashy new features, time to market, etc.?" Time to market is definitely part of management's KPI. I'm not sure if maintainability is included. – shiouming Feb 27 at 6:02
  • It's also possible for an org that doesn't have the minimum resources required to function properly. – shiouming Feb 28 at 6:34

Not the answer you're looking for?Browse other questions tagged or ask your own question.