So I'm a developing a workflow with vagrant+git…does this make sense?

Question

Relevant Background Details

1. We've got two types of VMs (Utility Boxes & Web Servers) that developers need.
2. We are going to be using git for version control.
3. We have developers who have different preferences for their working environment (e.g. Some Linux, Some Windows).
4. I'm in the Linux camp and I believe Git and Windows doesn't mix as well as Linux & Git. However, this could be personal bias.
5. We using Linux in production.

Building the Vagrant VMs for distribution

1. Build a base box with the relevant OS for vagrant.
2. Using a configuration manager (e.g. Chef) to build out the Utility & Web images, convert them to new base boxes. It will clone service configurations from a centralized git repository.
3. Distribute the base boxes (really just virtual machine images) for the users to develop locally with vagrant. The distributed box will automatically pull in source code from certain git repos (e.g. libraries).
4. If changes are planned for the production environment, all developers will need to pull down new base boxes for vagrant as they come prepackaged. I think this is the simplest way for a new developer to deal with it. Staging is updated to match the new development VMs in preparation.

Developer Workflow

1. Get assigned an issue from the issue tracker.
2. Use the vagrant VM to clone the current dev repository into the folder it shares with the host OS (so the Developer can use their favorite IDE).
3. Developer commits changes and tests locally.
4. When satisfied, Developer merges his changes to the dev repository. If conflicts, work with the Developer commited the conflicting code to resolve the issue.
5. When Dev is in a stable state, Dev is merged with the current Staging repository for QA of the new features. Nothing is pushed from Dev to Staging until Step #6 is completed. Hooks generate new copy of documentation for Staging.
6. Staging is cloned into Production once the QA is completed. Hooks generate new copy of documentation for Production.

Is there any obvious flaws/pitfalls in the above or steps that are generally considered 'best practices' that should be added?

Answer 1

Your plan sounds great! I think you are off to a really good start.

My only advice is in regard to you Developer Workflow. I think you're dev branch may become problematic, because developers will be merging code willy-nilly then they think its ready.

If both branchA and branchB, C, and D are merged into dev, and it fails, you can't be certain which parts are necessarily failing. Also, if someone pushes up something with a conflict, you dont know who it was. Madness and insanity are bound to ensue.

If a feature turns out not to be ready, and code needs to be backed out of dev, other developers will have already pulled down their additions. Those developers then will merge back into dev, unwittingly re-introducing the broken code. Madness and insanity are bound to ensue.

You're going to need a couple steps of separation to keep untested code away from tested code.

This all depends on the skill sets of your team, and how you actually work together. What follows is my solution to problems of a quickly expanding team with differing levels of git knowledge and different levels of developers code dependability.

I always try to tell people to not simply think about developer workflow, but testing procedure, and release process. They all need to be planned as part of a singular process.

1. Lead Dev or Release Mngr (whoever) creates a new release branch based off master. This branch will be a container for anything going on it in the next release.
2. Lead/ReleaseMngr creates integration branch based off release.
3. Developer creates new feature branch (or topic branch, whatever you want to call it), based off the current release branch.
4. Developer tests locally, is happy.
5. The developers feature branch deployed somewhere and tested by QA independently of any untested code.
6. QA signs off on feature - it gets merged into an integration branch. (ideally, IMHO, only after the feature branch has been rebased off release, to force the conflict resolutions )
7. QA tests the integration branch which is just the release branch + this one feature.
8. QA signs off - integration is merged into release (if not signed off, integration is blown away and recreated based on release). This is the reason for integration. No one pulls form this branch, and it can be blown away as needed.
9. Now the feature is in release, and can be shared with other developers making features based off release branch.
10. Release is good, merge to master. Deploy, make a new release branch based off master.

I know it sounds like a lot, but it really will save you from the headaches and, in my experience, are inevitable with large projects with logs of people having differing levels of knowledge.

If you have a small team with a simple release process, or a team that is very experienced - all this may not be necessary - but do be aware of the inherent problem with testing multiple people's code at the same time in your dev branch.

All that said, its my understanding the GitHub team just lets everyone merge into master directly (after a brief code review) and auto deploys ~30 times a day.

Answer 2

The general outline sounds good. The only thing I would change is that the base boxes themselves should be self-updating. Presumably your provisioning scripts also live in git so you could automatically update the boxes by a properly configured /etc/rc.local script. The base box should just have the bare minimum necessary to start and then download all the necessary pieces for configuration. This way you decouple the distribution of the boxes from the actual update process and don't need to ship an entire new base box every time you make changes, excepting OS upgrades of course.