34

Say there is a team of ten agile developers. Every day they each pick a task from the board, commits several changes against it, until (by the end of the day) they have completed the task. All developers check in directly against trunk (Google-style, every commit is a release candidate, using feature toggles etc).

If they were using a centralized CVS like SVN, every time one of them commits, the build server will integrate and test their changes against the other nine developers' work. The build server will be pretty much running continuously all day.

But if they were using a DCVS like git, the developer may wait until they complete the task before pushing all their local commits together up to the central repository. Their changes will not be integrated until the end of the day.

In this scenario, the SVN team is continuously-integrating more frequently, and discovering integration problems much faster than the git team.

Does this mean DVCSs are less suitable for continuous teams than older centralized tools? How do you guys get around this deferred-push issue?

  • 15
    Will people commit at least once before completing the task when using SVN? And will people only push once a day when using a DVCS? Your reasoning assumes neither is true, but my impression indicates otherwise. – user7043 May 16 '12 at 15:03
  • 3
    Very good question. – Michael Brown May 16 '12 at 15:05
  • 1
    @delnan: assume both teams commit several times per day, but the git guys only push those commits together when the task is completed. – Richard Dingwall May 16 '12 at 15:15
  • 2
    I think you are looking at the wrong end of the pipe, you get issues, not if you don't push till complete, but if you don't pull regularly while developing – jk. May 17 '12 at 7:38
  • 2
    I've seen the opposite: Developers using a centralized source-control system like TFS commit rarely, because their code affects everyone when they do. They end up temporarily saving their work in monster shelvesets, and when they're finished, it all goes in one monster commit. – Kyralessa Jan 16 '14 at 20:22

12 Answers 12

26

Disclaimer: I work for Atlassian

DVCS does not discourage Continuous Integration as long as the developer pushes remotely on a regular basis to their own branch and the CI server is setup so that it builds the known active branches.

Traditionally there are two problems with DVCS and CI:

  • Uncertainty of integration state - unless the developer has been merging regularly from master and running the build, you don't know what the state of the combined changes are. If the developer has to do this manually, chances are it won't be done often enough to pick up problems early enough.
  • Duplication and drift of build configuration - if the build configuration has to be copied from a 'master' build to create a branch build, the configuration for the branch can quickly become out of sync with the build it was copied from.

In Bamboo, we introduced the ability for the build server to detect new branches as they are created by developers and automatically setup builds for the branch based off the build configuration for master (so if you change masters build config, it also changes the branches config to reflect the change).

We also have a feature called Merge Strategies that can be used to either update the branch with changes from master before the branch build runs or automatically push the changes in a successful build branch to master, ensuring changes between branches are tested together as soon as possible.

Anyhow, if your interested in learning more, see my blog post "Making Feature Branches effective with Continuous Integration"

14

My small team switched to a DVCS a year or two ago, and the rest of my company followed suit a couple of months ago. In my experience:

  • People using a centralized VCS still tend to hold off on commits when they are doing a large project. This isn't a problem unique to DVCSes. They'll have change sets that wait several days before doing a commit. The big difference is that if they make a mistake at some point during this time, or if the computer crashes, it requires vastly more effort to fix it.
  • We use a commit workflow where each developer works on their own named branch, and only the person who reviewed their code is allowed to merge their changes into the head. This reduces the likelihood of a commit causing problems, so people really pay attention when the build server produces an error message. It also means that other developers can continue working on their own branches until the head gets fixed.
  • On a DVCS, people do tend to spend more time programming before merging their code in with the head. So it does tend to introduce a little lag into the continuity of the build. But the difference is not significant enough to counter the advantages of the DVCS.
  • The build server builds all named branches so each committer has his own build server job? – user1249 May 17 '12 at 7:13
  • Doesn't the code reviewer become a serious bottleneck in this scenario? – Andres F. May 17 '12 at 12:45
  • @ThorbjørnRavnAndersen: No, the build server only builds the "head" or "default" branch, and the release branches. So each user can commit to his own named branch without fear of breaking the build. We could conceivably set up a build server to build everybody's branches, but in some cases I want to commit some work that I've done, knowing full well that it puts my own branch in an unusable state. I'll make sure my branch is stable before I do a code review and merge. I only care that the main branches that everyone else use are stable. – StriplingWarrior May 17 '12 at 15:15
  • @AndresF.: No, it hasn't become a serious bottleneck for us. For one thing, we have multiple people who can do code reviews, so each developer can usually find at least one reviewer who is available for a review at any given time. Also, part of the beauty of a DVCS is that even if you can't merge right away, you can start working on something else, and other developers can merge your changes into their branches if they depend on your changes for their work. Once your code is reviewed, there is a specific changeset node that the reviewer can merge in. – StriplingWarrior May 17 '12 at 15:24
13

I recently observed on about 19 projects that used Mercurial over SubVersion (I was a subversion geek): developers started to become really individualists by working on their own branch and integrating only after serveral days or weeks. This caused serious integration troubles and concerns.

Another problem we faced is with the continuous integration server. We were notified of problems (failing test for instance), only when the sync of commits were made to the server.

It seems that Martin Fowler wrote about it on his site.

That said, some of the project I mention did a sync at least once a day reducing the problems. So to answer your question, I do think that DVCS may discourage continuous integration and increase individualism. However, DVCS is not the direct cause.

Developer is still in charge regardless the VCS they use.

  • Did said projects emphasize on a common goal or did the developers have to work on specific, disconnected targets? – user1249 May 17 '12 at 7:16
  • We can't generalize on 19 projects. But when we faced integration problems, that's also because some principles like separation of concerns were not respected. What I say is that, yes, DVCS seems to encourage individualism and reduce the benefits of continuous integration, but, if the developers are well trained, it's possible to reduce or eliminate the problem. – user2567 May 17 '12 at 7:22
  • In that case I would suggest that you also do continuos delivery, or at least frequent customer deliveries, so the deadline for when the merge MUST happen is much shorter. Did you do that in these projects? – user1249 May 17 '12 at 7:44
  • Of course, we use Scrum – user2567 May 17 '12 at 7:53
  • 1
    I was looking for your definition of continuous delivery (still can't find something decent, I will appreciate if you could give me some references), and found this: continuousdelivery.com/2011/07/… – user2567 May 17 '12 at 8:56
10

Idea you base your reasoning on is very shaky, softly speaking. It is the matter of team / management / process that developer may wait until they complete the task.

Doing it one way or another, "wait" or "hurry", shared trunk or isolated branch, is known as branching strategy, and if you study information available online, you'll find out that choosing particular strategy has basically nothing to do with VCS being centralized or distributed.

Say, for distributed VCS like Mercurial, you can easily find strong recommendation for frequent merges:

First, merge often! This makes merging easier for everyone and you find out about conflicts (which are often rooted in incompatible design decisions) earlier...

Studying recommendations like above, one can easily find out that these appeal to considerations having nothing to do with Mercurial being distributed.

Now, let's look at the situation at side of centralized VSC, Subversion. Studying online information, one can find among top popular strategies so called stable trunk and unstable trunk - each having opposite impact on frequency of merges. You see, people choose one or other way of doing things without even paying attention to VCS being centralized.

  • I've seen severely delayed merges happening (even encouraged by lame management) with centralized VCS, as well as frequent merges done with DVCS when team/management just thought that it's the right way. I've seen that nobody cares if VCS is distributed or centralized at deciding one way or another.

Given above, it looks like the right answer to Do DVCSes discourage continuous integration? would be Mu.

VCS being distributed or not does not have a substantial impact on that.

  • 1
    +1 I agree with you that management is the key to solve the problem. However, we must admit that there is something in DVCS that discourages continuous integration. In fact, one of the key feature of DCVS encourage that behavior. – user2567 May 17 '12 at 13:19
  • 1
    @Pierre303 maybe - I somehow feel like that too, but that's pretty much a theory. As I wrote, I've seen team integrating like crazy with DVCS and on the other hand, the most "isolationist" project I ever worked in (and that was a nightmare) was with centralized VCS. So much for the feelings, so much for the theory... – gnat May 17 '12 at 13:58
  • I admit that is only empirical observation, but on large number of projects, and there are probably a huge "skill" bias involved. – user2567 May 17 '12 at 14:10
10

My experience is the exact opposite, teams using svn would not push for days, because the code they were working on would cause the trunk to not compile for everyone else without wasting time on manual merging. Then near the end of the sprint, everyone would commit, merging madness would take place, things would get over-written and lost and have to be recovered. The CI system would go RED and finger pointing would ensue.

Never had this problem with Git/Gitorious.

Git lets you pull and merge other peoples changes at your convenience, not because someone else checked something and you want to check in but you have 20 mins of manual merging to do.

Git also lets you pull everyone else's commits, merge your code in and then push a working version to everyone else so they don't have to guess at what they should have to merge based on what you changed.

Having something like Gitorious as a mediator for code reviews via merge requests makes managing many branches and many contributors very painless.

Setting up Jenkins/Hudson to track all the active branches in a Git repository is very easy as well. We got more traction with CI and more frequent feedback about the state of the repositories when we moved to Git from SVN.

  • why would they commit directly to trunk? I think this was your problem. – gbjbaanb May 17 '12 at 14:16
  • 1
    @gbjbaanb because that is the traditional idiomatic CVS way of working, because this is the traditional centralized repo idiom. SVN users are usually former CVS users, and branching and merging in SVN is only marginally better than in CVS; which was beyond painful/next to impossible to get correct. This is the 99% workflow case in 99% of all the SVN shops because of the tools and group think. – user7519 May 17 '12 at 14:27
  • @JarrodRoberson : nonsense. My old SVN users were refugees from VSS :) Merging in SVN isn't nearly as bad as you think. In this case, he complains his users would break the build by checking in broken code directly to trunk - and then having to merge, frankly, having to merge your code with your colleague's is not an optional thing if you're all working directly on the same branch. – gbjbaanb May 18 '12 at 18:22
4

Build servers are cheap. Just have your CI server pick up all the branches you know about.

Jenkins has support to check multiple git repositories and get the 'latest' from any of those in a single job. I'm sure there are similar solutions with other tools.

  • And what happens if you want to commit something that breaks head but helps a colleague or is required so a colleague can help you? You could create a diff and email that to your colleague, but somehow that doesn't feel right. – Arjan May 17 '12 at 21:22
  • 1
    Team/feature branch? Or direct pull from your colleague repository? If more than one person is working on something that would break head but still requires a timeline/multi stage commit, it deserves its feature/work branch anyway. Merge to head when it's ready. – ptyx May 17 '12 at 21:41
  • A team branch of feature branch won't work if your CI tool picks up all branches you know of. And if your CI tool also processes multiple repositories you still don't want to include developer repos, just because they may not have been fully tested. – Arjan May 17 '12 at 22:24
  • 1
    The CI server won't automatically know of a private branch until it's told about it. Up to individuals to choose wether they want their branches on CI or not. (There is no miracle solution) – ptyx May 17 '12 at 23:40
  • So CI should not pick up all branches you know about, but only those you want in in CI. To me that is a difference. Still, I do think I understand what you're trying to say, so +1 – Arjan May 18 '12 at 19:24
4

This old question just got marked as a duplicate of a new one, and since a lot of the answers reference some outdated ideas, I thought I'd post an updated one.

One thing that apparently wasn't very common five years ago was running CI tests on pull request branches before merging them into master. I think this reflects a changing attitude that although merging frequently is desirable, sharing every change with everyone, as soon as you make it, isn't optimal.

DVCS has spawned a more hierarchical mode of integrating your commits. For example, I often work on tasks very closely with the developer who sits next to me. We will pull from each other's branches several times per day. Today, we collaborated with another developer via changes pushed to a pull request every few hours.

We were making extensive changes to the build scripts. Jenkins locally merges every PR branch with master and runs tests, so we got automated feedback that way, without disturbing any other developer who needed a clean build. It will probably be another day or so before that PR is ready to merge to master and share outside our group of three developers.

However, if someone can't wait for our changes to merge to master, because their change is dependent on ours, they can merge our dev branch locally, and continue with their work. This is what a lot of people who are accustomed to CVCS miss. With CVCS, the only way to share your changes is to merge them into the central repo, and that's why merging often is more critical. With DVCS, you have other options.

3

I'd say the DVCS is more conducive to continuous integration. Merges are not as irritating with them. It does require more discipline however. You should follow a local commit with a pull from the base to merge and then push when your task is complete (before going to the next).

2

When my team switched to Git, we explicitly laid out our process such that a push was to be treated exactly like a commit in the older VCS, and local commits could be done as frequently/infrequently as each individual chose. With that, there is no difference to the CI system whether we are using a DVCS or a centralized VCS.

1

The answer is both yes, and no.

The difference here is between committing directly to the central CI-viewed repo, and pushing your changes to the central CI-viewed repo. The 'problem' you might find is that DVCS users may not actually perform that push regularly.

I'd say this is an inherent design feature of a DVCS, it's not designed to push your changes to the central server all the time - if it were, you might as well use a CVCS instead. So the answer is to enforce a better workflow amongst your developers. Tell them to push changes every night. Simples!

(and if your SVN users are not committing every night, tell them to - its the exact same problem).

0

Git isn’t preventing continuous integration. Your trunk-based workflow is.

That may sound counterintuitive, but: if the developers work on feature branches, they can be encouraged to integrate frequently on their own machines (and required to do so before submitting their feature for merge). By contrast, a trunk-based workflow favors larger commits and therefore less frequent integration.

I maintain that a Google-style trunk-based workflow is counterproductive with a VCS such as Git where merging is easy. Here’s what I’d advise instead:

  • Break features down small enough that none will take more than a day or two to develop.
  • Each feature is developed on a private branch.
  • Developer integrates frequently on private branch (git fetch origin; git merge master). I typically do this many times a day when working this way.
  • When developer submits branch for merge and review, CI server does an automated build. Merge only happens when this passes.

So there you have it: small commits, frequent integration, and a traceable history of what commits belonged to which feature. Branches, used properly, are the key to all that is worthwhile about Git, so avoiding them is a big mistake.

-1

There are awesome technical solutions like @jdunay mentioned, but for us it's a people issue - in the same way that fostering an environment where people commit to svn often is a people issue.

What's worked for us is: (replace 'master' with the currently active dev branch)

  1. Frequent merges/rebases from master
  2. Frequent-enough pushes to master
  3. Awareness of things that cause merge hell, such as certain refactorings, and mitigating this by communicating. For example:

    • Make sure everyone pushes before lunch
    • Perform and push the refactoring during lunch
    • Make sure everyone pulls after lunch

Not the answer you're looking for? Browse other questions tagged or ask your own question.