Source control branch design for deployed environments

Question

I currently work with a young team of developers using ASP.NET, TeamCity and Octopus Deploy. One of our developers thinks that our system is flawed and introduces the inability to test production hotfixes. What is the best solution to this problem?

We have three environments: development, staging and production. Basically, our workflow only goes forward.

Git push to master -> TeamCity Build -> Octopus Deploy (Dev) -> Promote to Staging -> Promote to Prod

His solution is to have three branches: master (development), staging (staging) and prod (production). We do all active development in our master branch, do any QA in staging and merge from staging to prod when we need to perform a release. When a bug occurs in production, we make the change in staging, and then remerge into production, leaving our development branch alone. We eventually merge all development changes into the production branch once we are ready to release new features.

I don't think this is necessarily right however. My solution is to have the latest production version in the master branch. All development occurs in feature branches. When we've tested a feature branch, we merge back to production. Simple.

So what is the best solution to this problem? Also, some follow-up questions:

• Should we improve our QA process? Should we be doing QA on feature branches or on full releases?
• How should we set up our continuous integration? Should we have servers deployed for every version?

I'd love to hear some examples or suggestions you guys may have regarding this scenario. Thanks!

Answer 1

It is good practice to building your code only once and not per environment or branch.

Your GIT branches should not be directly connected to your environment stages.

The pipeline should be as follows:

develop > merge to baseline > build > upload to repo > deploy on dev > test > deploy to Staging > test > approve > deploy to 1 Prod > test > to n Prod.

Always using the same Artifact between promotions. All environments should be mirrors, with environment depend configurations being injected and minimised.

"promotion" is done by the deployment tool and not via GIT branch merging.

I also apply the QA to the full product and not to feature branches.

Testing branches in a vacuum will lead to integration or merging errors not being found and a false sense of security about the quality of the code. It can also create confusion with the Product Owner "I thought it was done and tested ? Why do we get errors now ?"

If you are not applying continuous integration, branching makes sense to split your hotfix from your already developed newer code. As such, simply feed the hotfix into the start pipeline and deploy it over the newest version.

Problems may arise if there are environment or data changes that are not backward compatible between the newest version and the hotfix.

In this case, one normally also includes all the environment and data configurations as a part of the "version" thus rebuilding the full environment every deployment.

This can be easily achieved if the servers are virtual or if they are provisioned via a configuration management tool.

Pratical example: vagrant file for each version. When you rollback from version 1.4.5 to version 0.5.6 you also rollback the vagrant configuration and thus the environment configuration.

While not my initial reference, here is an MSDN Team foundation Server best pratices and patterns: https://msdn.microsoft.com/en-us/library/dn449951.aspx I also advice the reading of the Addison-Wesley signature series books, Continuous Integration and Continuous Delivery

Answer 2

This is the solution I've come up with.

All projects which need to be kept in different deployed server environments will have two branches: master and development. The master branch will be the branch with the latest production version. The development branch will contain all active development.

Our CI will be hooked up to both the development and master branch. When the development branch is updated, a build will be created and then deployed to the development server. QA will be performed on the development server. Once the QA is done, the development branch will be merged to the master branch.

When the master branch is updated, a build will be created and then deployed to the staging server. From the staging server someone will need to promote the build to production. This extra step is done to ensure that nothing goes in production that shouldn't be there.

When hot fixes need to happen, a branch of master will be created, developed on, and once that hotfix has been tested locally it will be merged back into the master branch. Here someone can test the hotfix in the staging environment and from there promote it to production.

Hope this helps anyone.

Answer 3

As well as the above answers which sound reasonable, I also am a fan of ENV Branching with Git which has been useful in my own experience so sharing this here maybe could be helpful for some. This approach more or less supports "His solution is to have three branches: master (development), staging (staging) and prod (production)" from the question.

Basically, to have different branches according to the environments your code will deploy to. In other words, every branch should exactly reflect what has been deployed to the correspondent environment. There are of course a few of minimum rules that should follow to make this approach more effective. In short, they are:

• Never commit directly to an environment
• Never merge directly into an env branch
• Each feature branch should be autonomously deployable

To find out more, please have a look at the following link.

ENV Branching with Git