Blue/Green Deployment - How do I deal with persistent data added during cutover

Question

I'm currently looking at blue/green deployments and I'm not quite sure if my use case is even applicable.

The following simplified scenario of a user management tool:

I have 3 services - GUI, BL, Persistence. I have blue and green environment setup. Green is currently idle and users access blue. Now I want to deploy to Green.

I update my 3 services, and update the database with the schema changes. Tests run fine, so I'll make the switch.

How do I handle data that has been added/modified during the deployment of green?

For example: For some reason the deployment to green takes an hour - a username was changed during the deployment - so the changes will only affect the database of blue. If I just do the cut to green, this change would be lost?

Answer 1

This is handwaved a lot but is actually quite important. e.g. this guide says only

If your app uses a relational database, blue-green deployment can lead to discrepancies between your Green and Blue databases during an update. To maximize data integrity, configure a single database for backward and forward compatibility.

Martin Fowler says

There's still the issue of dealing with missed transactions while the green environment was live, but depending on your design you may be able to feed transactions to both environments in such a way as to keep the blue environment as a backup when the green is live

That is, run the secondary database as a replica of the primary database.

This glosses over a lot of potential work in the background. All your schema changes need to be forwards-only, for example; you can't delete a column or a record. If you want to do that you have to do two upgrades, one of which stops accessing the record/column and only later once that's fully switched over can you actually delete it.

In your example you'd have to sync up all the changes from blue to green as part of preparing for the cutover, and even then you need a way of handling transactions that happen curing the few seconds that the cutover actually takes. And a way of feeding changes back in case you need to cut back.

The CloudFoundry guide suggests using a single database for both versions of the frontend, but that carries higher risk (what if the new frontend corrupts data?) and forces you to do schema upgrades on the live database as well as designing your application to be tolerant of differences in the schema.

Answer 2

I've never seen Blue/Green work routinely with a (non-distributed) database.

The answer of how to do it is fairly boring though. Import/Export scripts. Just have two databases Green and Blue and have a script that can export v1 data and import it as v2 and vice-versa.

Now you can update Green to v2, copy all the v1 data across and start running it.

If you have a problem you can switch back to Blue and use the scripts to bring the extra v2 data back into v1

If the deployment takes an hour, run the script at the end to bring the data up to date

I'm sure lots of people will be familiar with this kind of thing for big migrations of data from old to new systems.

However, It's a pain to write and test those scripts, does it really reduce risk when you add these extra moving parts. Plus your overall system has to be written with a distributed approach in mind. It mustn't go wrong while you wait for the data to be synced up.

Green/Blue works great for Apis and apps, but its problematic with databases. Moving to a distributed or sharded architecture can make it work.