I want a quick way to roll back our production build in the event of any mishaps - a one-click solution, if you will. My gut is telling me that this is important - and I believe that my gut is correct, but I want implementation advice.

My current thought is to start a local repository on our production server, and simply commit the entire production build to the repo - binaries, web assets, everything. When we make an update to the production site, we can commit the changes to the repo. This way, if anything breaks for any reason and we need a quick rollback - it's simply a case of checking out the previous commit.

What better methods are available for this type of rollback? I know that there are multiple backup solutions, but a VCS is familiar to work with and provides similar benefits. Am I stepping into a trap?

  • 3
    I can think of 2 things that your strategy doesn't cover off the top of my head, database updates and registry settings, and I suspect there are others. – Sign Apr 23 '14 at 20:16
  • Database rollbacks would have to be handled separately, yes. Aside from that, I don't think we're doing anything with the registry (it's an IIS site; we're updating the binaries in the www directory). – lunchmeat317 Apr 23 '14 at 20:18
  • Looks OK for reverting the software, but does it undo all changes made for the new version and user interaction? So look at stored data (remove new users?), data in queues, changes to data structure, server configurations, infrastructure, scheduled task execution, etc. – Kwebble Apr 23 '14 at 22:07
  • @Kwebble Stored data and data structure would be handled in the DB rollback - server configuration won't change, and scheduled task execution is handled by a separate program. However, this is the kind of stuff that I was wondering about; great comment! – lunchmeat317 Apr 23 '14 at 23:11

If you use continuous deployment, then the easiest solution may be to rollback the last commit and let the system do the deployment stuff.

Otherwise, it depends on both your current workflow and the acceptable duration of the rollback. For example, if you don't rely on in-process sessions and you need the fastest possible rollback, you:

  • Create virtual machines during the original deployment,

  • Deploy the new version of the product on them,

  • Transfer your users from the old virtual machines to the new ones,

  • Keep the old virtual machines active, up and running.

Then, if you detect something wrong happening, all you have to do is to transfer the users back to the old virtual machines, which can happen nearly instantaneously.

In all cases, you'll rely either on:

  • swapping of servers (virtual machines or instances of IIS sites),
  • or a backup,
  • or snapshots (I'm not sure if this is doable in Windows for ordinary files; it is with LVM in Linux). You have snapshots feature in SQL Server, on the other hand.
  • I don't think that we can do a virtual machine deal like that.....however, spinning up a new IIS site instance and switching the bindings actually seems like a really good idea! – lunchmeat317 Apr 23 '14 at 23:20

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.