Moving from a CVCS to a DVCS in a big company, what is the right way to do it?

Question

I just started a new job and I'm currently at the process of trying to introduce a DVCS (mercurial, but the actual DVCS is not really a matter of discussion at this point, it could also be git) to the company (big, multinational). So far I've gotten nothing but positive attention, even at managerial and directive level, but I want to make sure I'm not at all disruptive to the organization or anyone's jobs.

The way the developers are set up is in a team room, so there's a lot of communication going on. There is also an integrator, that communicates on a daily basis two or three times with the team leader. Also, there are more development teams on other parts of the world, maybe 20 offices in 6 or 7 countries.

Company culture is open and there would be no problem pilot testing the tool but the real question is (in capital letters):

HOW DO YOU CONVINCE MANAGEMENT AROUND ALL THE OFFICES OVER THE WORLD THAT ITS A SANE THING TO DO AND WILL ACTUALLY BRING HAPPINESS AND PRODUCTIVITY TO DEVELOPERS? (say pilot testing and showing it works is just one part of the answer).

Elaborating on the context...

Does it make sense to make the switch?, yes (note: current SCM is Perforce). I've witnessed several sources of frustration classic to centralized source control management, including:

• People accidentally breaking other's builds or stepping on each others' toes while submitting code revisions.

  This is not often as the responsible must wear a pink hat (hilarious stuff) until the build is fixed, but the hat is still there, and when builds get broken it affects the other offices around the world.

  DVCS to the rescue: Builds are now broken in a single office, people lose the fear to use version control and can actually second-guess themselves after or check with a peer by sharing their changes easier and not necessarily with the rest of the group.

• Unnecessary bureaucracy when it comes to granting access (e.g. looks like everyone is granted access to most of the code).

  There is a dedicated group of admins who take 1 to 3 days to grant access to the code, meanwhile other developers must ask their peers for the code just to look at it... but AFAIK, the often wait instead.

  DVCS to the rescue: Administration does not have to be centralized on a little group of overwhelmed people, new employees get the code from the project's responsible (1st level) or a peer with repository access (2nd level) who could even review their code and submit it along with his/her changes.

• Complaints with regards to slowness due to network latency. Not that directly, but sometimes a) The perforce server is down, b) Internet is down, c) Internet or perforce server is slow. Yes, not that often, but it is annoying enough to kick you off your train of thought sometimes.

  DVCS to the rescue: When using the DVCS you obviously don't need a network connection. Worst case scenario?, share your changes over LAN or sneakernet (with patch queues).

• Trouble when trying to collaborate with external providers that do not necessarily belong to the organization. This happens often enough and security is a concern here AFAIK, and clients suffer from the access bureaucracy.

  DVCS to the rescue: So granting access to a subrepo here would be the ideal solution here... also, clients themselves could use version control among themselves without needing to set up their own server and they would be able to integrate better to the internal workflow as currently Perforce access access to the local network, and thus more bureaucracy which is not necessarily security here, as external collaborators still have access to thumbdrives and email. Actual security would be having externals work in-situ... in a dungeon or something, and grope them à la TSA on their way out to make sure they are not getting the code out.

• Workaround to integrate modules from other repositories. Requires additional maintenance/effort from the developers. **They actually wrote themselves (I think) a plugin to make Perforce fetch files from other repositories in other parts **

  DVCS to the rescue: Their solution looks a lot like the mercurial subrepo feature, which uses a file (.hgsub) to indicate what code goes where, but they have to maintain the other repositories' states in the same file (which is done separately and automatically by mercurial in the .hgsubstate file).

• Working with the source code requires to mark files for edit in advance. This adds an annoying extra step IMHO. If you don't do this the file will be in read only mode and you will not be able to edit it on your IDE, also, this in my opinion makes the local copy prone to low integrity. This trait of Perforce makes me think there was a lazy developer that didn't want to make the client check the files against the repository before versioning the changes. It also makes me think/personify Perforce as an idiot paranoid that says "DON'T CHANGE THE FILES WITHOUT TELLING ME OR THE SQUIRRELS WILL COME AND EAT ME!".

  DVCS to the rescue: Well... I guess any other SCM software knows there are no carnivore squirrels.

• Little project integrity in developers' machines (see previous point; its harder to add changes without previously indicating them to Perforce). You don't need a DVCS for this... but if you make changes locally and for some reason forgot to add them, you are going to have a hard time detecting it. Once you detect it there is a two step command line instruction thing I saw somewhere to detect changes you did without telling Perforce.

Given that context what would you say is the best way to... ?:

1. Know/demonstrate if the migration would actually benefit the employees and the company.
2. Ease the embracement of change, or at least make it look like its worth the effort
3. Not complicate others' jobs.
4. Convince management worldwide.
5. Avoid making the integrator's job a royal pain.

My second question (and focusing on the last point) I've always been a DVCS evangelist mainly because it eases the work and collaboration between developers... but up to this point I've never been faced with having to consider the integrators' perspective (the one in charge of coordinating teams' source code, creating and releasing the builds... from across the globe) who at some point might have to face two SCM sources or at least the migration itself. Then, how can the migration process affect integrators the least?.

Answer 1

The best way is to do a pilot with one existing team and record what the issues are. In my experience, the biggest obstacle to using a DVCS in an enterprise is the lack of structure. A DVCS is more of an SCM framework than a full solution. You have to add a lot of glue to make it work in a large organization, which is why things like kiln exist.

You may also want to do a gradual migration. Perforce has recently added some features to make it more appealing to DVCS fans, namely streams and sandbox. Being familiar with the benefits of those concepts would ease the transition and maybe even obviate it. There are also things like git-p4 that could enable a gradual transition.

Answer 2

People accidentally breaking other's builds or stepping on each others' toes while submitting code revisions

This is an issue with organizational communication and programmer discipline. A DVCS is actually likely to make things worse in this situation, because individual programmers can keep working in their isolated repositories and not discover merge conflicts until they've become all but intractable.

Unnecessary bureaucracy when it comes to granting access (e.g. looks like everyone is granted access to most of the code).

How will a DVCS change this? Unless you have broad-based access rules (eg, everyone in "developer" group has permission to write the repo), you're going to wind up with bureaucracy. After all, someone has to maintain the ACL.

Trouble when trying to collaborate with external providers that do not necessarily belong to the organization.

A big win for DVCS. But it begs the question: how often do you expect to share repositories with these external providers? And how often do you expect to exchange updates?

Workaround to integrate modules from other repositories. Requires additional maintenance/effort from the developers.

How will a DVCS help here? I hope you're not planning to import the other module's source into your repo.

Working with the source code requires to mark files for edit in advance. This adds an annoying extra step IMHO.

I don't know Perforce, but optimistic locking has been available in every SCM that I've used since SCCS.

---

As I see it, your real problem is organizational, with developers happily doing their own thing without talking to one-another. In this situation, just adding a DVCS will make things worse, as I described above. To really make it work, you'd need to create the role of integration czar: a person who is responsible for pulling finished work from the various developers and making sure that it fits together. That's the way that Linus works, and it's the way that GitHub makes you work, but I'm not sure that it will go over well in most companies.

That said, there are a lot of benefits that a DVCS can bring to a team that communicates well. And Mercurial's server mode allows for informal sharing between team members, which I think is a Good Thing. But I'll second Karl's answer: start with a pilot project. And roll out slowly.

Answer 3

Just some type of personal subjective opinions and experience regarding mentioned problems and questions

Know/demonstrate if the migration would actually benefit the employees and the company.

Will it (benefits) will appears really? Which troubles in current workflow you want eliminate fist? Are they really troubles and showstoppers or just some annoyances, significance of which is exaggerated for bosses?

• "People accidentally breaking other's builds..." is problem of planning, workflow and management (missing QA, work-patterns, not used automated *-test), not tools
• "Unnecessary bureaucracy when it comes to granting access..." some type of ACL and restrictions is a must, from my POV. This (almost) missing feature in Mercurial can be a bigger headache
• "slowness due to network latency" - DVCS just delay (can delay) a time of sync with remotes, not eliminate it totally
• "Trouble when trying to collaborate with external providers..." Mercurial will change nothing technically and organizationally, may do process only harder, because it can get relation many-2-one instead old one-one (single repo to single repo)
• "Workaround to integrate modules from other repositories..." Mercurial's subrepos aren't easiest and fully automatic part of... Handwork and accuracy is a must
• Last two points are closed by Mercurial more or less, but sufficiently justified to consider the transition

Ease the embracement of change, or at least make it look like its worth the effort

It's heavy depend from local situation and current workflow (which I can't know) - some devs consider just movement from "commit" to "commit + push" as life drama. But all changes in common can be more deep and hard.

No opinions. But if you'll simulate CVCS (to some degree) with Mercurial, it can may facilitate the initial stages of migration

Not complicate others' jobs.

Pure development can be as autonomous, as the developer wants, thus - doesn't interfere with other devs. But, besides code-monkey, changes will appear for everybody (testers, PMs, releasers, doc-writers)

how can the migration process affect integrators the least?

Sorry, if I don't know how integration process works now, I can say nothing how don't broke it and how to do minimal impact... Also think about possible wanted and requested changes in old style, which can bring new tool from new class