3

I'm working with a codebase that is almost 4y old, just a little under 2000 source files and roughly 100 developers have committed to it.

One of our third-party dependencies is not actively maintained anymore and we suspect that it will no longer work as we keep upgrading our stack. We would like to estimate how much effort it would take to refactor all the modules that depend on it, however most of us aren't familiar with that area of the codebase.

Problem is that most people who worked on these modules are either no longer with the company or not in the team anymore. So we're looking at ways to guide us whilst dealing with other imperatives.

So in an attempt to make a relatively well informed decision I thought I would crunch some numbers first. I have identified 32 modules to refactor and for each I want to know:

(The following assumptions are made for the modules we need to refactor and not for the whole codebase)

  1. When was it last updated?
    Assumption: older code needs more attention.

  2. How many revisions?
    Assumption: too many revisions is a sign of unstable code.

  3. How many different developers worked on it?
    Assumption: too many developers leads to inconsistent design.

  4. How many lines of code?
    Assumption: the more code the longer or more difficult the task is.

  5. How many references to it exist?
    Assumption: probability of bugs increases with the number of dependents.

I wrote a shell script and got the following results:

File Last Updated Revs Committers LOC Refs
Module 1 2018-06-13 15 5 56 3
Module 2 2021-02-05 8 7 64 13
Module 3 2020-08-31 27 12 203 2
Module 4 2020-08-31 16 8 103 2
Module 5 2019-07-04 2 1 51 2
Module 6 2020-02-27 6 4 61 2

On a scale of 0 (easy) to 100 (hard) I want to know how much effort each module would take. I start at 50 (medium) and add or subtract points based on the following rules: (please do note that I came up with these rules myself; this not based on any science)

-10 0 +10
Last Updated 6 months ago or less 6 months ago or later last year or later
Revisions < 5 >= 5 >= 10
Committers < 5 >= 5 >= 10
LOC < 50 >= 50 >= 100
Refs < 5 >= 5 >= 10

Which gives me the following results:

File Last Updated Revs Committers LOC Refs Effort
Module 1 10 10 0 0 -10 60
Module 2 -10 0 0 0 10 50
Module 3 0 10 10 10 -10 70
Module 4 0 10 0 10 -10 60
Module 5 10 -10 -10 0 -10 30
Module 6 10 0 -10 0 -10 40

And so on the effort scale:

0 10 20 30 40 50 60 70 80 90 100
M5 M6 M2 M1 M3
M4

This methodology is only an additional tool to try to come up with better estimations. We do have (some) tests and coverage already.

Question: Is there a flaw with this method? Is there another way to approach this?

8
  • 2
    Two books by Adam Tornhill might be interesting to you: Your Code As A Crime Scene, and Software Design X-Rays. They both address this question, and provide strategies for dealing with technical debt, legacy code, etc. I recommend both. (He also has a commercial product using these techniques on your software, codescene.)
    – davidbak
    Apr 8, 2021 at 0:54
  • @davidbak Thanks I'm half-way through Adam Tornhill 's book already. Very nice read. Didn't know about the second one. Will have a look. Apr 8, 2021 at 0:58
  • 2
    It's hard to counter assumptions. How would you answer "I think criminals were hugged much less by their parents when they were a child. Is there a flaw with this assumption?" You can't specifically prove that the opposite of an assumption is inherently more likely. But that doesn't mean that your assumptions are therefore true. Just as a quick mathematical exercise, if all your assumptions are individually 90% correct (which is heavily erring in your favor), then the odds of being correct on 5 assumptions are roughly 59%. At 7 assumptions, you'd be doing worse than a coin flip (48%).
    – Flater
    Apr 8, 2021 at 1:05
  • 3
    Did you notice - there is currently one answer saying "all Git based metrics are irrelevant" and another one saying "look at other metrics - which are all not Git based". Thought experiment: what if for the same code base you would not have a Git history to look at - do you think that would really change anything for the estimation?
    – Doc Brown
    Apr 8, 2021 at 7:04
  • ... I think the important part is how the code looks today, and how it uses the dependency you want to replace in its latest revision - the Git history is almost irrelevant for this. The LOC in each of the modules you listed is also irrelevant - but the LOC which are affected by changing the dependency, the size of the code base which depends on those parts and so on. And one of the biggest flaws: the estimation does not seem to look at the availability of automated tests for the affected modules...
    – Doc Brown
    Apr 8, 2021 at 7:14

5 Answers 5

4

Yes, there is a serious flaw in this method: it has no scientific foundation (such as a proven causal relationships, or statistical analysis of large sets of codebases that could confirm correlation). As a consequence, its result might be unreliable and with no advantage over an educated guess.

Moreover, the formula appears to be based on rational parameters that might mislead about its scientific foundation. This bears the risk that its result are not questioned with sufficient criticism, which might lead to wrong decisions.

Additional thoughts to challenge this method:

  • the number of committers cannot be directly linked to the effort. If a programmer writes a very reliable module, and during the 3 next years, every year one new contributor corrects one small bug, you’ll get 4 committers. But is the effort to maintain it really higher? Can this situation be compared with a complex module written by 4 contributors at the same time?
  • the number of revisions is also a poor predictor for the future: are we speaking of major versions? are we speaking of any versions? are we speaking of number of successive commits? Is there a continuity in the versioning practice?
  • Let’s reason by the extreme. Let’s imagine that some GIT-based estimate estmates the effort to 100 because many modules, many contributors, many revisions... Now let’s download that code, copy it to another directory. Create a new git repository and commit all the code: you’ll find one committer, one revision, and a lot more of the git indicators will be lost. So the estimated effort will be completely different FOR THE SAME CODE !
  • Last but not least: LOC was, is, and will always be a poor predictor of maintenance or refactoring effort. Because lines do not tell anything about structure. The risk is to compare highly engineered, state of the art class design, with poorly conceived copy-pasted redundant code, just because they are the same size.

Additional thoughts following exchange of comments:

  • Using git metrics in effort estimates is an interesting idea. But probably more to obtain objective contextual information about the project history and observable facts that may influence the estimate rather than define it (e.g. if commits affect in average many modules one could fear undesired couplings and some uncontrolled change propagation).
  • You’d probably also need more structural metrics such as Chidamber & Kemerer or others to achieve your intent. These metrics depend on the code itself, and are not affected by code history.
7
  • Thanks for your answer. I'm aware that the numbers I came up with are to be taken with a pinch of salt; I just wanted to find a way to prioritise work. These numbers aren't going to do to the work of course but they may be helpful during planning. I've seen some literature on the subject of using Git to find patterns in codebases so I was inclined to think that this wasn't entirely without merit. Apr 7, 2021 at 22:26
  • @customcommander I edited, adding some more arguments about potential flaws and issues. It’s not that it’s uninteresting: git can give you information about pas history (many small changes, or many bigger ones? are concerns well isolated, i.e. many commits with very few modules, or commits of a large number of changes suggesting risk of uncontrolled change propagation?) that can certainly be used in the estimation process. But more as contextual information than as predictor.
    – Christophe
    Apr 7, 2021 at 22:45
  • I didn't intend this to be a prediction machine but more of a guide during planning. I may have presented this in the wrong way quite possibly. I do agree that Git only gives some contextual information that may or may not be helpful. I don't necessarily disagree with this answer (and the other). Perfectly valid arguments. If you don't mind the feedback: I find the tone of this answer a little bit on the passive aggressive side (but that might just be me on the receiving end ;) Apr 7, 2021 at 22:54
  • @customcommander feedback is always welcome. No intent to be agressive. Sorry if I caused such a feeling.
    – Christophe
    Apr 7, 2021 at 22:59
  • 1
    I like your additional thoughts section — I think I may have mislead (although that wasn't the intention) people to believe I want to estimate only with Git. Really what I want to do is have additional metrics to look at to either 1) orientate discovery work in the areas that are most needed or 2) make "better" estimation in combination with other metrics and methodologies. These areas of the codebase are pretty much unknown to everybody in the team. So we want to approach this in any sensible way whilst dealing with other imperatives. Appreciate your edits. Thanks. Apr 8, 2021 at 8:44
2

Other answers have pointed out flaws in the criteria you're using; Many languages have tools available which can understand the code and therefore provide a range of metrics which are more commonly associated with maintainability.

That's not to say you can rely solely on these metrics either (To rely on any automated approach would still be just as flawed as the git approach) - the only way to arrive at reliable estimates would be for developers to analyse and familiarise themselves with the code, however such tools are a far better indicator on where to focus and what to look out for.

  1. Automated test coverage - Test coverage has a strong correlation with maintainability; so look for a tool which can measure the extent of test coverage. Tests serve a dual purpose as a form of reliable documentation as well as protection against regression issues when you refactor.

  2. Code quality - Static code analysis (for example, Lint) may identify potential bugs and 'code smells'.

  3. Cyclomatic complexity - This is a measure of how many different paths exist through some code; higher complexity values generally indicate code which is harder to reason about and harder to test

  4. Class coupling - If you have a dependency analysis tool available, this may be able to show the extent of tight-coupling between classes/modules and the direction of coupling. It may also be able to show where coupling exists between modules you control as well as to external, 3rd-party components. Tight coupling generally makes code harder to test and therefore harder to maintain and refactor.

You may also be able to find tools which can calculate a Maintainability index or Halstead volume, both of which have their flaws but should be somewhat better indicators for the size and complexity of a codebase than simply looking at lines of code.


For the refactoring effort itself, consider a strategy which involves adding any missing automated tests for the existing code before you refactor it - as per point #1, the presence of tests and high test coverage naturally makes the code much easier to refactor and eliminates a lot of the risk. The process of creating tests for existing code also forces someone to analyse and understand its purpose and behaviour before they refactor.

Writing tests against the existing code involves significant time early in the refactoring effort but should also be something developers can estimate if they've spent sufficient time familiarising themselves with it. The goal of this is not to save time but substantially mitigate the risks involved in refactoring and ensuring the refactored code is more maintainable than the code you started with.

2

Making assumptions

It's hard to counter assumptions. How would you answer "I think criminals were hugged much less by their parents when they were a child. Is there a flaw with this assumption?"

You can't specifically prove that the opposite of an assumption is inherently more likely. But that doesn't mean that your assumptions are therefore true.

Just as a quick mathematical exercise, if all your assumptions are individually 90% correct (which is heavily erring in your favor), then the odds of being correct on 5 assumptions are roughly 59%. At 7 assumptions, you'd be doing worse than a coin flip (48%).


All your assumptions can be countered

When was it last updated?
Assumption: older code needs more attention.

This is at odds with assumptions 2 and 3. The less something has been updated (assumption 1: bad sign), the less likely it is to have had multiple committers (assumption 3: good sign?) or to have had a lot of commits (assumption 2: good sign?).

To put it another way, if observed on a random day, a code file with many revisions is on average "fresher" (i.e. changed more recently) than a code file with fewer revisions. This pits your assumptions against one another.

Furthermore, I can very easily argue the opposite. The code that hasn't been touched in ages clearly didn't need to be touched, so it's arguable better than the code that was touched recently. Clean code leads to readable code, readable code leads to lowered chance of bugs/mistakes and higher chance of being able to implement a fix even if there is a bug. In either case, doing things well means they can be left undisturbed for a longer time.

How many revisions?
Assumption: too many revisions is a sign of unstable code.

Michael paints a picture in 10 minutes. Leo takes 30 days. Whose painting is the best?

Well, you could argue that Michael must be better because Leo is wasting 29 days 23 hours and 50 minutes compared to Michael, so Leo must be a worse painter, right? But then you'd be assuming that Michael and Leo were delivering a painting of equal detail and skill level. It's highly unlikely that this is the case, and even if it were, it defeats the purpose of calling one painting better than the other.

It's more likely that Michael's painting looks slapdash, and Leo's painting looks significantly more detailed and skillful due to the extra time taken.

Or, to drop the analogy, the additional time spent revising code is not more likely to have decreased its quality than to have improved it, as per your assumption. It can be argued either way, and it's much too contextual and volatile to make any blanket assumption about.

How many different developers worked on it?
Assumption: too many developers leads to inconsistent design.

The biggest dragons in legacy code are codebases built by a single person. Single developers work in their own personal style (as opposed to one with team consensus), tend not to document (since knowledge transfer is rarely if ever needed), and tend to learn to live with their own code's quirks (as opposed to people bumping into each other's quirks and fixing them).

I have years worth of experience to argue the opposite of your assumption.

How many lines of code?
Assumption: the more code the longer or more difficult the task is.

Clean code tends to take up more space (by line count, and often file count too), but is orders of magnitude easier to read, understand, refactor or change. That defeats your assumption and argues the exact opposite.

How many references to it exist?
Assumption: probability of bugs increases with the number of dependents.

My aunt lives on a little rural dead end street. Only her and her neighbor use the little road to their properties. It's a bumpy gravel road with potholes, and horrible to drive on.
I live in a city center, on a main road. The road is made with high quality whisper asphalt, and is a joy to drive on.

We both live in the same city. Same local government, same transportation budget, same staff fixing potholes. And yet my road has less potholes than my aunt's road, because more people use it.

A piece of code that has more references to it suggests that it is being used more, and you're much more likely for consumers to stumble on bugs or issues, and thus for more bugs or issues to be noticed and subsequently fixed.


Just to be clear, I am not saying you should assume the opposite of what you have been assuming. I'm trying to point out that all your assumptions are not guarantees, as I can very easily come up with scenarios where your assumptions simply do not hold.


Pudding, meet proof

You've made your assumptions, you've got your answer based on your assumptions. What's missing is confirming the validity of your theory.

Rather than ask some strangers on the internet who have zero contextual knowledge of your environment, monitor if the actual resolution time reasonably approximates the estimated effort calculated by your as of yet unproven theory.

No developer has ever been able to estimate work correctly, other than by trying to estimate, realizing they estimated inaccurately, and then learning to counterbalance their inaccuracy. With each new project/environment/company, that learning curve somewhat resets, as certain environmental factors will have changed.

Your calculation is no different. It's probably very imprecise, but could possibly be tweaked by monitoring and adjusting it to your specific environmental factors.
For example, if Bob revised the same file 4 sprints in a row, that may be indicative of shoddy code. I've known a few Bobs like that.

This is not something a stranger on the internet can tell you, as there are too many contextual factors at play. You simply need to put in the effort to test your assumptions and finetune them based on your findings.

1
  • Thank you for your detailed answer. Contrary to what others may think I never claimed that my methodology had any scientific foundation at all. I like to explore things and look at data (admittedly I'm not very good at that) and see if something useful can be worked out. I'm grateful for all the comments (yours and all the others) and will take them into account when looking at how this pans out in practice. Again I never intended this to be a prediction machine but more of a tool to aid us in prioritising work in areas that nobody in the team is actually familiar with. Apr 8, 2021 at 8:30
1

Questions 1 through 3 are largely irrelevant. Refactoring assumes you are not changing the behavior of the code base. If that is what you are truly doing then the number of references and volume of code will have a bigger impact on overall effort.

Git is a version control system which is ignorant of programming language syntax. A good IDE is what you really need, or just a good honest code analysis session. You won't solve this with Git. It can help a little, perhaps, but this is not what version control was built for.

0

Several good answers here already, and I am not going to repeat things already written by others, but let me stress one additional aspect:

The effort for your planned changes will exclusively depend on two things:

  • how the latest revision of the code looks today (assuming you are not exchanging the dependency for older releases)

  • how well your current team knows the code base and the entanglement with the dependency you are going to replace

My point is: the history on how the code got into the current state is largely irrelevant. If, for example, in one of your modules is just one line of code which calls the 3rd party dependency, it does not matter if the module was changed by 100 devs, has 12345 different revisions and 20K lines of code in total, the replacement will most probably simpler than a for a module which uses the 3rd party dependency in 100 places in various ways, regardless if the latter has only a few revisions and 2000 lines of code.

Hence any Git based metrics is flawed from the ground and will tell you things which won't be of use for your current objective. Of course, the Git history may be useful

  • to find out which of the team members has worked on a certain module in the last months (if for any unknown reason you have trouble to ask them directly about their knowledge).

  • to get additional information when there are usages of the current dependency you don't understand (so you may have luck to find out by looking at the change logs).

So the Git history can be useful for the task, but not for deriving a metrics for effort estimation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.