What's the best way to handle refactoring a big file?

Question

I'm currently working on a bigger project which unfortunately has some files where software quality guidelines where not always followed. This includes big files (read 2000-4000 lines) which clearly contain multiple distinct functionalities.

Now I want to refactor these big files into multiple small ones. The issue is, since they are so big, multiple people (me included) on different branches are working on these files. So I can't really branch from develop and refactor, since merging these refactorings with other peoples' changes will become difficult.

We could of course require everyone to merge back to develop, "freeze" the files (i.e. don't allow anyone to edit them anymore), refactor, and then "unfreeze". But this is not really good either, since this would require everyone to basically stop their work on these files until refactoring is done.

So is there a way to refactor, don't require anyone else to stop working (for to long) or merge back their feature branches to develop?

Answer 1

You have correctly understood that this is not so much a technical as a social problem: if you want to avoid excessive merge conflicts, the team needs to collaborate in a way that avoids these conflicts.

This is part of a larger issue with Git, in that branching is very easy but merging can still take a lot of effort. Development teams tend to launch a lot of branches and are then surprised that merging them is difficult, possibly because they are trying to emulate the Git Flow without understanding its context.

The general rule to fast and easy merges is to prevent big differences from accumulating, in particular that feature branches should be very short lived (hours or days, not months). A development team that is able to rapidly integrate their changes will see fewer merge conflicts. If some code isn't yet production ready, it might be possible to integrate it but deactivate it through a feature flag. As soon as the code has been integrated into your master branch, it becomes accessible to the kind of refactoring you are trying to do.

That might be too much for your immediate problem. But it may be feasible to ask colleagues to merge their changes that impact this file until the end of the week so that you can perform the refactoring. If they wait longer, they'll have to deal with the merge conflicts themselves. That's not impossible, it's just avoidable work.

You may also want to prevent breaking large swaths of dependent code and only make API-compatible changes. For example, if you want to extract some functionality into a separate module:

1. Extract the functionality into a separate module.
2. Change the old functions to forward their calls to the new API.
3. Over time, port dependent code to the new API.
4. Finally, you can delete the old functions.
5. (Repeat for the next bunch of functionality)

This multi-step process can avoid many merge conflicts. In particular, there will only be conflicts if someone else is also changing the functionality you extracted. The cost of this approach is that it's much slower than changing everything at once, and that you temporarily have two duplicate APIs. This isn't so bad until something urgent interrupts this refactoring, the duplication is forgotten or deprioritized, and you end up with a bunch of tech debt.

But in the end, any solution will require you to coordinate with your team.

Answer 2

Do the refactoring in smaller steps. Let's say your large file has the name Foo:

1. Add a new empty file, Bar, and commit it to "trunk".

2. Find a small portion of the code in Foo which can be moved over to Bar. Apply the move, update from trunk, build and test the code, and commit to "trunk".

3. Repeat step 2 until Foo and Bar have equal size (or whatever size you prefer)

That way, next time your teammates update their branches from trunk, they get your changes in "small portions" and can merge them one-by-one, which is a lot easier than having to merge a full split in one step. The same holds when in step 2 you get a merge conflict because someone else updated trunk in between.

This won't eliminate merge conflicts or the need for resolving them manually, but it restricts each conflict to a small area of code, which is way more manageable.

And of course - communicate the refactoring in the team. Inform your mates what you are doing, so they know why they have to expect merge conflicts for the particular file.

Answer 3

You are thinking of splitting the file as an atomic operation, but there are intermediate changes you can make. The file gradually became huge over time, it can gradually become small over time.

Pick a part that hasn't had to change in a long time (git blame can help with this), and split that off first. Get that change merged into everyone's branches, then pick the next easiest part to split. Maybe even splitting one part is too big a step and you should just do some rearranging within the large file first.

If people aren't frequently merging back to develop, you should encourage that, then after they merge, take that opportunity to split off the parts they just changed. Or ask them to do the splitting off as part of the pull request review.

The idea is to slowly move toward your goal. It will feel like progress is slow, but then suddenly you'll realize your code is a lot better. It takes a long time to turn an ocean liner.

Answer 4

I'm going to suggest a different than normal solution to this problem.

Use this as a team code event. Have everyone check-in their code who can, then help others who are still working with the file. Once everyone relevant has their code checked in, find a conference room with a projector and work together to start moving things around and into new files.

You may want to set a specific amount of time to this, so that it doesn't end up being a week worth of arguments with no end in sight. Instead, this might even be a weekly 1-2 hour event until you all get things looking how it needs to be. Maybe you only need 1-2 hours to refactor the file. You won't know until you try, likely.

This has the benefit of everyone being on the same page (no pun intended) with the refactoring, but it can also help you avoid mistakes as well as get input from others about possible method groupings to maintain, if necessary.

Doing it this way can be considered to have a built-in code review, if you do that sort of thing. This allows the appropriate amount of devs to sign off on your code as soon as you get it checked in and ready for their review. You might still want them to check the code for anything you missed, but it goes a long ways to making sure the review process is shorter.

This may not work in all situations, teams, or companies, as the work isn't distributed in a way that makes this happen easily. It can also be (incorrectly) construed as a misuse of dev time. This group code needs buy-in from the manager as well as the refactor itself.

To help sell this idea to your manager, mention the code review bit as well as everyone knowing where thing are from the beginning. Preventing devs from losing time searching a host of new files can be worthwhile to avoid. Also, preventing devs from getting POed about where things ended up or "completely missing" is usually a good thing. (The fewer the meltdowns the better, IMO.)

Once you get one file refactored this way, you may be able to more easily get approval for more refactors, if it was successful and useful.

However you decide to do your refactor, good luck!

Answer 5

Fixing this problem requires buy-in from the other teams because you're trying to change a shared resource (the code itself). That being said, I think there's a way to "migrate away" from having huge monolithic files without disrupting people.

I would also recommend not targeting all the huge files at once unless the number of huge files is growing uncontrollably in addition to the sizes of individual files.

Refactoring large files like this frequently causes unexpected problems. The first step is to stop the big files from accumulating additional functionality beyond what's currently in master or in development branches.

I think the best way to do this is with commit hooks that block certain additions to the large files by default, but can be overruled with a magical comment in the commit message, like @bigfileok or something. It's important to be able to overrule the policy in a way that's painless but trackable. Ideally, you should be able to run the commit hook locally and it should tell you how to override this particular error in the error message itself. Also, this is just my preference, but unrecognized magical comments or magical comments suppressing errors that didn't actually fire in the commit message should be a commit-time warning or error so you don't inadvertently train people to suppress the hooks regardless of whether they need to or not.

The commit hook could check for new classes or do other static analysis (ad hoc or not). You can also just pick a line or character count that's 10% larger than the file currently is and say that the large file can't grow beyond the new limit. You can also reject individual commits that grow the large file by too many lines or too many characters or w/e.

Once the large file stops accumulating new functionality, you can refactor things out of it one at a time (and reduce the tresholds enforced by the commit hooks at the same time to prevent it from growing again).

Eventually, the large files will be small enough that the commit hooks can be completely removed.

Answer 6

Wait until hometime. Split the file, commit and merge to master.

Other people will have to pull the changes into their feature branches in the morning like any other change.