Can we use Git as an additional tool to better estimate maintenance effort?

Question

I'm working with a codebase that is almost 4y old, just a little under 2000 source files and roughly 100 developers have committed to it.

One of our third-party dependencies is not actively maintained anymore and we suspect that it will no longer work as we keep upgrading our stack. We would like to estimate how much effort it would take to refactor all the modules that depend on it, however most of us aren't familiar with that area of the codebase.

Problem is that most people who worked on these modules are either no longer with the company or not in the team anymore. So we're looking at ways to guide us whilst dealing with other imperatives.

So in an attempt to make a relatively well informed decision I thought I would crunch some numbers first. I have identified 32 modules to refactor and for each I want to know:

(The following assumptions are made for the modules we need to refactor and not for the whole codebase)

1. When was it last updated?
   Assumption: older code needs more attention.

2. How many revisions?
   Assumption: too many revisions is a sign of unstable code.

3. How many different developers worked on it?
   Assumption: too many developers leads to inconsistent design.

4. How many lines of code?
   Assumption: the more code the longer or more difficult the task is.

5. How many references to it exist?
   Assumption: probability of bugs increases with the number of dependents.

I wrote a shell script and got the following results:

File	Last Updated	Revs	Committers	LOC	Refs
Module 1	2018-06-13	15	5	56	3
Module 2	2021-02-05	8	7	64	13
Module 3	2020-08-31	27	12	203	2
Module 4	2020-08-31	16	8	103	2
Module 5	2019-07-04	2	1	51	2
Module 6	2020-02-27	6	4	61	2

On a scale of 0 (easy) to 100 (hard) I want to know how much effort each module would take. I start at 50 (medium) and add or subtract points based on the following rules: (please do note that I came up with these rules myself; this not based on any science)

	-10	0	+10
Last Updated	6 months ago or less	6 months ago or later	last year or later
Revisions	< 5	>= 5	>= 10
Committers	< 5	>= 5	>= 10
LOC	< 50	>= 50	>= 100
Refs	< 5	>= 5	>= 10

Which gives me the following results:

File	Last Updated	Revs	Committers	LOC	Refs	Effort
Module 1	10	10	0	0	-10	60
Module 2	-10	0	0	0	10	50
Module 3	0	10	10	10	-10	70
Module 4	0	10	0	10	-10	60
Module 5	10	-10	-10	0	-10	30
Module 6	10	0	-10	0	-10	40

And so on the effort scale:

0	10	20	30	40	50	60	70	80	90	100
			M5	M6	M2	M1	M3
						M4

This methodology is only an additional tool to try to come up with better estimations. We do have (some) tests and coverage already.

Question: Is there a flaw with this method? Is there another way to approach this?

Answer 1

Yes, there is a serious flaw in this method: it has no scientific foundation (such as a proven causal relationships, or statistical analysis of large sets of codebases that could confirm correlation). As a consequence, its result might be unreliable and with no advantage over an educated guess.

Moreover, the formula appears to be based on rational parameters that might mislead about its scientific foundation. This bears the risk that its result are not questioned with sufficient criticism, which might lead to wrong decisions.

Additional thoughts to challenge this method:

• the number of committers cannot be directly linked to the effort. If a programmer writes a very reliable module, and during the 3 next years, every year one new contributor corrects one small bug, you’ll get 4 committers. But is the effort to maintain it really higher? Can this situation be compared with a complex module written by 4 contributors at the same time?
• the number of revisions is also a poor predictor for the future: are we speaking of major versions? are we speaking of any versions? are we speaking of number of successive commits? Is there a continuity in the versioning practice?
• Let’s reason by the extreme. Let’s imagine that some GIT-based estimate estmates the effort to 100 because many modules, many contributors, many revisions... Now let’s download that code, copy it to another directory. Create a new git repository and commit all the code: you’ll find one committer, one revision, and a lot more of the git indicators will be lost. So the estimated effort will be completely different FOR THE SAME CODE !
• Last but not least: LOC was, is, and will always be a poor predictor of maintenance or refactoring effort. Because lines do not tell anything about structure. The risk is to compare highly engineered, state of the art class design, with poorly conceived copy-pasted redundant code, just because they are the same size.

Additional thoughts following exchange of comments:

• Using git metrics in effort estimates is an interesting idea. But probably more to obtain objective contextual information about the project history and observable facts that may influence the estimate rather than define it (e.g. if commits affect in average many modules one could fear undesired couplings and some uncontrolled change propagation).
• You’d probably also need more structural metrics such as Chidamber & Kemerer or others to achieve your intent. These metrics depend on the code itself, and are not affected by code history.

Answer 2

Other answers have pointed out flaws in the criteria you're using; Many languages have tools available which can understand the code and therefore provide a range of metrics which are more commonly associated with maintainability.

That's not to say you can rely solely on these metrics either (To rely on any automated approach would still be just as flawed as the git approach) - the only way to arrive at reliable estimates would be for developers to analyse and familiarise themselves with the code, however such tools are a far better indicator on where to focus and what to look out for.

1. Automated test coverage - Test coverage has a strong correlation with maintainability; so look for a tool which can measure the extent of test coverage. Tests serve a dual purpose as a form of reliable documentation as well as protection against regression issues when you refactor.

2. Code quality - Static code analysis (for example, Lint) may identify potential bugs and 'code smells'.

3. Cyclomatic complexity - This is a measure of how many different paths exist through some code; higher complexity values generally indicate code which is harder to reason about and harder to test

4. Class coupling - If you have a dependency analysis tool available, this may be able to show the extent of tight-coupling between classes/modules and the direction of coupling. It may also be able to show where coupling exists between modules you control as well as to external, 3rd-party components. Tight coupling generally makes code harder to test and therefore harder to maintain and refactor.

You may also be able to find tools which can calculate a Maintainability index or Halstead volume, both of which have their flaws but should be somewhat better indicators for the size and complexity of a codebase than simply looking at lines of code.

---

For the refactoring effort itself, consider a strategy which involves adding any missing automated tests for the existing code before you refactor it - as per point #1, the presence of tests and high test coverage naturally makes the code much easier to refactor and eliminates a lot of the risk. The process of creating tests for existing code also forces someone to analyse and understand its purpose and behaviour before they refactor.

Writing tests against the existing code involves significant time early in the refactoring effort but should also be something developers can estimate if they've spent sufficient time familiarising themselves with it. The goal of this is not to save time but substantially mitigate the risks involved in refactoring and ensuring the refactored code is more maintainable than the code you started with.

Answer 3

Making assumptions

It's hard to counter assumptions. How would you answer "I think criminals were hugged much less by their parents when they were a child. Is there a flaw with this assumption?"

You can't specifically prove that the opposite of an assumption is inherently more likely. But that doesn't mean that your assumptions are therefore true.

Just as a quick mathematical exercise, if all your assumptions are individually 90% correct (which is heavily erring in your favor), then the odds of being correct on 5 assumptions are roughly 59%. At 7 assumptions, you'd be doing worse than a coin flip (48%).

---

All your assumptions can be countered

When was it last updated?
Assumption: older code needs more attention.

This is at odds with assumptions 2 and 3. The less something has been updated (assumption 1: bad sign), the less likely it is to have had multiple committers (assumption 3: good sign?) or to have had a lot of commits (assumption 2: good sign?).

To put it another way, if observed on a random day, a code file with many revisions is on average "fresher" (i.e. changed more recently) than a code file with fewer revisions. This pits your assumptions against one another.

Furthermore, I can very easily argue the opposite. The code that hasn't been touched in ages clearly didn't need to be touched, so it's arguable better than the code that was touched recently. Clean code leads to readable code, readable code leads to lowered chance of bugs/mistakes and higher chance of being able to implement a fix even if there is a bug. In either case, doing things well means they can be left undisturbed for a longer time.

How many revisions?
Assumption: too many revisions is a sign of unstable code.

Michael paints a picture in 10 minutes. Leo takes 30 days. Whose painting is the best?

Well, you could argue that Michael must be better because Leo is wasting 29 days 23 hours and 50 minutes compared to Michael, so Leo must be a worse painter, right? But then you'd be assuming that Michael and Leo were delivering a painting of equal detail and skill level. It's highly unlikely that this is the case, and even if it were, it defeats the purpose of calling one painting better than the other.

It's more likely that Michael's painting looks slapdash, and Leo's painting looks significantly more detailed and skillful due to the extra time taken.

Or, to drop the analogy, the additional time spent revising code is not more likely to have decreased its quality than to have improved it, as per your assumption. It can be argued either way, and it's much too contextual and volatile to make any blanket assumption about.

How many different developers worked on it?
Assumption: too many developers leads to inconsistent design.

The biggest dragons in legacy code are codebases built by a single person. Single developers work in their own personal style (as opposed to one with team consensus), tend not to document (since knowledge transfer is rarely if ever needed), and tend to learn to live with their own code's quirks (as opposed to people bumping into each other's quirks and fixing them).

I have years worth of experience to argue the opposite of your assumption.

How many lines of code?
Assumption: the more code the longer or more difficult the task is.

Clean code tends to take up more space (by line count, and often file count too), but is orders of magnitude easier to read, understand, refactor or change. That defeats your assumption and argues the exact opposite.

How many references to it exist?
Assumption: probability of bugs increases with the number of dependents.

My aunt lives on a little rural dead end street. Only her and her neighbor use the little road to their properties. It's a bumpy gravel road with potholes, and horrible to drive on.
I live in a city center, on a main road. The road is made with high quality whisper asphalt, and is a joy to drive on.

We both live in the same city. Same local government, same transportation budget, same staff fixing potholes. And yet my road has less potholes than my aunt's road, because more people use it.

A piece of code that has more references to it suggests that it is being used more, and you're much more likely for consumers to stumble on bugs or issues, and thus for more bugs or issues to be noticed and subsequently fixed.

---

Just to be clear, I am not saying you should assume the opposite of what you have been assuming. I'm trying to point out that all your assumptions are not guarantees, as I can very easily come up with scenarios where your assumptions simply do not hold.

---

Pudding, meet proof

You've made your assumptions, you've got your answer based on your assumptions. What's missing is confirming the validity of your theory.

Rather than ask some strangers on the internet who have zero contextual knowledge of your environment, monitor if the actual resolution time reasonably approximates the estimated effort calculated by your as of yet unproven theory.

No developer has ever been able to estimate work correctly, other than by trying to estimate, realizing they estimated inaccurately, and then learning to counterbalance their inaccuracy. With each new project/environment/company, that learning curve somewhat resets, as certain environmental factors will have changed.

Your calculation is no different. It's probably very imprecise, but could possibly be tweaked by monitoring and adjusting it to your specific environmental factors.
For example, if Bob revised the same file 4 sprints in a row, that may be indicative of shoddy code. I've known a few Bobs like that.

This is not something a stranger on the internet can tell you, as there are too many contextual factors at play. You simply need to put in the effort to test your assumptions and finetune them based on your findings.

Answer 4

Questions 1 through 3 are largely irrelevant. Refactoring assumes you are not changing the behavior of the code base. If that is what you are truly doing then the number of references and volume of code will have a bigger impact on overall effort.

Git is a version control system which is ignorant of programming language syntax. A good IDE is what you really need, or just a good honest code analysis session. You won't solve this with Git. It can help a little, perhaps, but this is not what version control was built for.

Answer 5

Several good answers here already, and I am not going to repeat things already written by others, but let me stress one additional aspect:

The effort for your planned changes will exclusively depend on two things:

• how the latest revision of the code looks today (assuming you are not exchanging the dependency for older releases)

• how well your current team knows the code base and the entanglement with the dependency you are going to replace

My point is: the history on how the code got into the current state is largely irrelevant. If, for example, in one of your modules is just one line of code which calls the 3rd party dependency, it does not matter if the module was changed by 100 devs, has 12345 different revisions and 20K lines of code in total, the replacement will most probably simpler than a for a module which uses the 3rd party dependency in 100 places in various ways, regardless if the latter has only a few revisions and 2000 lines of code.

Hence any Git based metrics is flawed from the ground and will tell you things which won't be of use for your current objective. Of course, the Git history may be useful

• to find out which of the team members has worked on a certain module in the last months (if for any unknown reason you have trouble to ask them directly about their knowledge).

• to get additional information when there are usages of the current dependency you don't understand (so you may have luck to find out by looking at the change logs).

So the Git history can be useful for the task, but not for deriving a metrics for effort estimation.