How can I refactor a code base while others rapidly commit to it?

Question

I'm on a private project that eventually will become open source. We have a few team members, talented enough with the technologies to build apps, but not dedicated developers who can write clean/beautiful and most importantly long-term maintainable code.

I've set out to refactor the code base, but it's a bit unwieldy as someone in the team out in another country I'm not in regular contact with could be updating this totally separate thing.

I know one solution is to communicate rapidly or adopt better PM practices, but we're just not that big yet. I just want to clean up the code and merge nicely into what he has updated. Would using a branch be a suitable plan? A best-effort-merge? Something else?

Answer 1

One thing people often fail to consider is that a clean architecture doesn't only speed up long term maintenance, it also speeds up development right now. Don't try to insulate your changes from your colleagues until they are "done." Your changes will help them be more productive and less prone to bugs.

The most frequent mistake people make when undertaking a large refactor is to not merge often enough, instead trying to do it in one "big bang." The right way to do it is to make the smallest possible refactor you can, test it, then merge it into your colleague's branch, and teach him about the change so he can incorporate it going forward. Ideally you're doing one merge per day, or one per week at the very least.

Answer 2

You are never "not big enough to communicate". If you fingers can do the typing, your lips can do the talking too. At the end of the day technology improvement is 85% communication and 15% technical. Just because you'd rather sit there coding than having difficult conversations with someone... doesn't mean it's a good idea. Communication is actually the hard piece of what you are trying to, but don't just avoid it.

Answer 3

Yes, a branch is a good solution for this.

I would suggest you start working on this on a branch and make sure it applies cleanly on top of your current HEAD in the meantime (i.e. make test rebases and merges at regular intervals to make sure you can apply your changes easily and your tests still pass -- also look into git rerere for help from git with this). Then once you are done rebase and merge your changes into your HEAD.

The sooner you start working on this the better since changing architecture becomes more and more work the colder the code gets. Also, there might be many instances of handcoded code scattered throughout the code base where e.g. your new and shiner helper function might have made things much simpler.

Answer 4

Have you considered the option "Don't do it yet"?

While doing this work in a seperate branch is probably the best approach, you are setting yourself up for a massive painful merge down the line.

The other guys are presumably adding lots of new functionality, changing existing functionality and possibly removing some functionality.

Once mainstream dev worked has slowed down a bit at some point in the future, then you could be in a much easier position to refactor.

Answer 5

tl;dr - Sounds like it's time to step up to the big leagues. Putting lipstick on a pig doesn't make it any prettier, unless you're into that sort of thing...

The people problem

The first issue is commit synchronization. IF you have multiple people working on the same code at the same time you only need one rule to prevent problems:

Rule 1: Always pull before you merge/rebase

When it comes to DVCS, it's hard to make changes to a remote branch (ie the main repository) and very easy to make changes to the local. Every person is responsible for making their own code additions fit into the greater whole without issues. Unless 2 people commit at the exact same time, you shouldn't experience. Commit access to the origin/remote master should be limited to only a few developers and they should pull changes from the other developers via remote tracking branches.

The code problem

How do you know that the changes you make don't break code?

Simple answer... Write tests to prove they don't. If you ignore the TDD (Test Driven Design) school of thought, the whole point of tests is to add a level of verification that enables you to change code without breaking it.

Rule 2: Don't make assumptions, write proofs (ie tests).

In addition to this, the full gamut of tests should be run before you push to the origin/remote master.

Keep your commits as small and concise as possible. That way, if you need to back out a change that broke something later on, you'll save from having to re-implement the parts that didn't break the code.

You may need some organizational re-structuring first

If the above solutions can't easily be applied, there are probably some issues with the development structure that need to be addressed first.

The owner of the project should be the gatekeeper. If there are commit syncing issues, there are probably too many people with commit access. Even on massive projects like the Linux kernel, only a handful of developers have commit access to the origin/remote master repository. There are actually multiple levels of repositories to manage commits. Instead of a single layer commit model where everybody is pushing their changes to the origin, the hierarchical model has gatekeepers that pull changes and verify their quality before inclusion into the project. The hierarchical commit model can scale a lot bigger and more effective than the single layer model without sacrificing quality.

For the devs that don't get commit access, they should learn to create their own remote tracking branches (git and gitorious are good for this) so the devs who do have commit access can easily pull/integrate branches into the origin. If the changes are small, patches will work just as well.

The ability to pull changes before doing a merge/rebase assumes that you're not developing on your local master branch. The easy way to handle this is to make an initial pull before you start to code, then do all your work on that branch. The hard way is to branch it just before merging and roll back the master.

Define the coding style for the project overall and make the devs follow it. Contributing devs should be writing code that conforms to the project's standards/norms to minimize cleanup. Coding style can be a big ego barrier in an open project. If no standard is set, everybody will code in their own preferred style and the codebase will get very ugly very fast.

The myth of "The Mythical Man Month"

Believe it or not, the larger/more successful open source projects aren't run like a democracy. They're run as a hierarchy. Stating that a project can't effectively grow beyond 8-10 developers is naive. If that were true then mega-projects like the Linux Kernel wouldn't exist. The deeper issue is that giving everybody commit access just makes effecive communication too hard to handle.

The problem of the mythical man month can actually be overcome. You just need to run your project like the military. There are many levels within the hierarchy because it's common knowledge that individual people are really only effective at managing communications with a handful of people. As long as no single individual is responsible for managing the work of more than 5-7 people, the system can scale indefinitely.

It may limit the best/experienced developers to doing more integration and higher-level design/planning but that's not a bad thing. Part of scaling up is making the move to decide that the project needs a long-term plan. The people at the highest levels who have the greatest investment (time is also a resource) in the projects future should be charged with making the big decisions.

It's nice to hear about an open source project going through growing pains. Congrats and good luck.

Answer 6

clean/beautiful and most importantly long-term maintainable code.

In my experience clean/beautiful is the enemy of maintainable. Beautiful code often:

• Has a layer on the framework that introduces a higher level of abstraction
• Optmizes for code reuse, resulting in a lot of dependencies
• Tries to solve the generic problem instead of the specific one

On the other hand, maintainable code:

• Is written directly on the framework, so all developers can read it
• Optimizes for a low number of dependencies, so a change in one area doesn't impact another
• Does not try to solve more problems than it has to