6

I currently work with a young team of developers using ASP.NET, TeamCity and Octopus Deploy. One of our developers thinks that our system is flawed and introduces the inability to test production hotfixes. What is the best solution to this problem?

We have three environments: development, staging and production. Basically, our workflow only goes forward.

Git push to master -> TeamCity Build -> Octopus Deploy (Dev) -> Promote to Staging -> Promote to Prod

His solution is to have three branches: master (development), staging (staging) and prod (production). We do all active development in our master branch, do any QA in staging and merge from staging to prod when we need to perform a release. When a bug occurs in production, we make the change in staging, and then remerge into production, leaving our development branch alone. We eventually merge all development changes into the production branch once we are ready to release new features.

I don't think this is necessarily right however. My solution is to have the latest production version in the master branch. All development occurs in feature branches. When we've tested a feature branch, we merge back to production. Simple.

So what is the best solution to this problem? Also, some follow-up questions:

  • Should we improve our QA process? Should we be doing QA on feature branches or on full releases?
  • How should we set up our continuous integration? Should we have servers deployed for every version?

I'd love to hear some examples or suggestions you guys may have regarding this scenario. Thanks!

  • 3
    Here's an example of the model of proposing; I just don't know how we'd set up our continuous integration. nvie.com/posts/a-successful-git-branching-model – dimiguel Jun 19 '15 at 19:47
  • It doesnt matter too much because git will let you branch from any commit. So if you need to do a hot fix you can create a branch from the last commit in master which was in the live build – Ewan Jun 20 '15 at 7:24
  • @dimgl using that model (which is good) you CI on the develop or master branches, whichever you merge most to. You should also then create CI builds using the build server (or continually update a single release) for each production build. – gbjbaanb Jun 25 '15 at 9:59
9

It is good practice to building your code only once and not per environment or branch.

Your GIT branches should not be directly connected to your environment stages.

The pipeline should be as follows:

develop > merge to baseline > build > upload to repo > deploy on dev > test > deploy to Staging > test > approve > deploy to 1 Prod > test > to n Prod.

Always using the same Artifact between promotions. All environments should be mirrors, with environment depend configurations being injected and minimised.

"promotion" is done by the deployment tool and not via GIT branch merging.

I also apply the QA to the full product and not to feature branches.

Testing branches in a vacuum will lead to integration or merging errors not being found and a false sense of security about the quality of the code. It can also create confusion with the Product Owner "I thought it was done and tested ? Why do we get errors now ?"

If you are not applying continuous integration, branching makes sense to split your hotfix from your already developed newer code. As such, simply feed the hotfix into the start pipeline and deploy it over the newest version.

Problems may arise if there are environment or data changes that are not backward compatible between the newest version and the hotfix.

In this case, one normally also includes all the environment and data configurations as a part of the "version" thus rebuilding the full environment every deployment.

This can be easily achieved if the servers are virtual or if they are provisioned via a configuration management tool.

Pratical example: vagrant file for each version. When you rollback from version 1.4.5 to version 0.5.6 you also rollback the vagrant configuration and thus the environment configuration.

While not my initial reference, here is an MSDN Team foundation Server best pratices and patterns: https://msdn.microsoft.com/en-us/library/dn449951.aspx I also advice the reading of the Addison-Wesley signature series books, Continuous Integration and Continuous Delivery

4

This is the solution I've come up with.

All projects which need to be kept in different deployed server environments will have two branches: master and development. The master branch will be the branch with the latest production version. The development branch will contain all active development.

Our CI will be hooked up to both the development and master branch. When the development branch is updated, a build will be created and then deployed to the development server. QA will be performed on the development server. Once the QA is done, the development branch will be merged to the master branch.

When the master branch is updated, a build will be created and then deployed to the staging server. From the staging server someone will need to promote the build to production. This extra step is done to ensure that nothing goes in production that shouldn't be there.

When hot fixes need to happen, a branch of master will be created, developed on, and once that hotfix has been tested locally it will be merged back into the master branch. Here someone can test the hotfix in the staging environment and from there promote it to production.

Hope this helps anyone.

  • 1
    Seems reasonable to me. You're basically using feature branches. Only thing is I would merge your dev branch to main, the branch that and deploy. The reason is you can mess something up performing the merge. My preferred alternative is to do rev in trunk and when you get to code freeze/RC you branch that, and then do fixes in the branch and merge to trunk. – Andy Jun 23 '15 at 23:01
  • I wouldn't recommend ever merging a development branch to anything tbh. Dev branches are usually unstable -- if that is your remote testing environment. Here, everyone merges to dev from their feature branches and that gets deployed to the dev environment so that you aren't blowing away each other's changes. The PRs go to a release branch and that's what gets deployed to qa/uat/staging. When release is ready, it's merged to master and deployed to prod. – Sinaesthetic Sep 6 '17 at 21:48
0

As well as the above answers which sound reasonable, I also am a fan of ENV Branching with Git which has been useful in my own experience so sharing this here maybe could be helpful for some. This approach more or less supports "His solution is to have three branches: master (development), staging (staging) and prod (production)" from the question.

Basically, to have different branches according to the environments your code will deploy to. In other words, every branch should exactly reflect what has been deployed to the correspondent environment. There are of course a few of minimum rules that should follow to make this approach more effective. In short, they are:

  • Never commit directly to an environment
  • Never merge directly into an env branch
  • Each feature branch should be autonomously deployable

To find out more, please have a look at the following link.

ENV Branching with Git

  • 1
    Could you please give a short summary or quote the key points of the article? From How to Answer: "Always quote the most relevant part of an important link, in case the target site is unreachable or goes permanently offline." – doubleYou Jun 2 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.